Re: Bug#760841: pspp: [kfreebsd] random testsuite failures

[Steven Chamberlain]

Hi again,

The failing test ("test 940", but that number may change) is this:

> AT_SETUP([read UTF-16 as Auto])
> AT_KEYWORDS([u8_istream])
> AT_CHECK([i18n-test supports_encodings UTF-16 UTF-16BE UTF-16LE])

This check seems fine:
> AT_CHECK([printf '\0e\0n\0t\0r\0\351\0e\0\n' | u8-istream-test read - Auto],
>   [0], [entrée
> ])

This is the check that is usually seen to fail (u8-istream.at:146) :
> AT_CHECK([printf 'e\0n\0t\0r\0\351\0e\0\n\0' | u8-istream-test read - Auto],
>   [0], [entrée
> ])

It is supposed to decode the UTF-16LE string to UTF-8.  The following
method reproduced it 1.9% of the time on my kfreebsd-amd64 system.  When
it fails, it outputs the raw input string not converted at all:

> ~/pspp-0.8.3$ for i in $(seq 1 1000) ; do printf 'e\0n\0t\0r\0\351\0e\0\n\0' 
> | tests/libpspp/u8-istream-test read - Auto | xxd ; done | sort | uniq -c | 
> sort -bn
>      19 0000000: 6500 6e00 7400 7200 c3a9 0065 000a 00    e.n.t.r....e...
>     981 0000000: 656e 7472 c3a9 650a                      entr..e.

This check seems fine:
> AT_CHECK([printf '\376\377\0e\0n\0t\0r\0\351\0e\0\n' | u8-istream-test read - 
> Auto],
>   [0], [entrée
> ])

This check seems to have issues too (UTF-16LE with Byte-Order Marker):
> AT_CHECK([printf '\377\376e\0n\0t\0r\0\351\0e\0\n\0' | u8-istream-test read - 
> Auto],
>   [0], [entrée
> ])

> ~/pspp-0.8.3$ for i in $(seq 1 1000) ; do printf 
> '\377\376e\0n\0t\0r\0\351\0e\0\n\0' | tests/libpspp/u8-istream-test read - 
> Auto | xxd ; done | sort | uniq -c | sort -bn

Raw input string output about 1.4% of the time:
>      14 0000000: c3bf c3be 6500 6e00 7400 7200 c3a9 0065  ....e.n.t.r....e
>      14 0000010: 000a 00                                  ...

Correct output:
>     986 0000000: 656e 7472 c3a9 650a                      entr..e.

Regards,
-- 
Steven Chamberlain
address@hidden