Re: make check fails if no en_US.iso88591 locale

[Neil Jerram]

Mike Gran <address@hidden> writes:

> My bad.  Actually, I should have enclosed the 'with-locale' in the
> context of a 'pass-if', which would have caught the exception.

Yes, but at the cost of not running the tests...

>> I can allow make check to complete by changing that line to
>> 
>>   (false-if-exception (with-locale "en_US.iso88591"
>> 
>> but I doubt that's the best fix.  Is the "en_US.iso88591" locale
>> actually important for the enclosed tests?
>
> It is important.  This is one of the problems with the whole Unicode
> effort.  There is no Unicode-capable regex library.  The regexp.test
> tries matching all bytes from 0 to 255, and it uses scm_to_locale_string
> to prep the string for dispatch to the libc regex calls and
> scm_from_locale_string to send them back.  
>
> If the current locale is C or ASCII, bytes above 127 will cause errors.
> If the current locale is UTF-8, bytes above 127 will be converted into
> multibyte sequences that won't be matched by the regular expression
> being tested.  To pass the test in regexp.test, we need to use the 
> encoding that matches all of the codepoints 0 to 255 to single byte
> characters, which is ISO-8859-1.
>
> So until a better regex comes along, wrapping regex in an
> 8-bit-clean-friendly locale like Latin-1 is necessary to avoid encoding
> errors when encoding arbitrary 8-bit data like the test does.
>
> The reason why this problem is cropping up now and didn't occur before
> is because the old scm_to_locale_string was just a stub that passed
> 8-bit data through unmodified.

Thanks for explaining; I think I understand now.  So then Ludovic's
suggestion of with-latin1-locale should work, shouldn't it?

> This regex library actually can be used with arbitrary Unicode data
> but it takes extra care.  UTF-8 can be used as the locale, and, then
> regular expression must be written keeping in mind that each non-ASCII
> character is really a multibyte string.

Can you give an example of what that ("keeping in mind...") means?  Is
it being careful with repetition counts (as in "[a-z]{3}"), for
example?

Thanks,
        Neil