bug#26948: ‘write-file’ output should not be locale-dependent

[Ludovic Courtès]

Maxim Cournoyer <address@hidden> skribis:

> address@hidden (Ludovic Courtès) writes:

[...]

>> Strangely that file name has question marks instead of the non-ASCII
>> characters on my GuixSD system:
>>
>> $ ls -l /etc/ssl/certs/*Certi*mara*
>> lrwxrwxrwx 8 root root 162 Jan  1  1970 
>> '/etc/ssl/certs/AC_Ra?z_Certic?mara_S.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem'
>>  -> 
>> '/gnu/store/3ql0vilc0zv6ra42ghi04787vrg6bb71-nss-certs-3.30.2/etc/ssl/certs/AC_Ra?z_Certic?mara_S.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem'
>
> Hmm. That is strange. It seems like you also have a locale problem, but
> that it is handled in a way that doesn't break nss-certs?

AFAICS the file is really called that way, with question marks:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> (stat 
"/gnu/store/3ql0vilc0zv6ra42ghi04787vrg6bb71-nss-certs-3.30.2/etc/ssl/certs/AC_Ra?z_Certic?mara_S.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem")
$2 = #(64768 4719936 33060 8 0 0 0 2444 1496043280 1 1492867575 4096 8 regular 
292 130744281 0 1492867575)
--8<---------------cut here---------------end--------------->8---

And:

--8<---------------cut here---------------start------------->8---
$ wget -O - 
https://mirror.hydra.gnu.org/guix/nar/gzip/3ql0vilc0zv6ra42ghi04787vrg6bb71-nss-certs-3.30.2
 |gunzip -c | guix archive -x t
--2017-05-29 10:55:36--  
https://mirror.hydra.gnu.org/guix/nar/gzip/3ql0vilc0zv6ra42ghi04787vrg6bb71-nss-certs-3.30.2
Ni solvigas mirror.hydra.gnu.org (mirror.hydra.gnu.org)... 131.159.14.26, 
2001:4ca0:2001:10:225:90ff:fedb:c720
Konektado al mirror.hydra.gnu.org (mirror.hydra.gnu.org)|131.159.14.26|:443... 
konektita.
HTTP peto sendita, ni atendas respondon... 200 OK
Grando: 171969 (168K) [application/octet-stream]
Ni konservas al: 'STDOUT'

-                            
100%[==============================================>] 167.94K  --.-KB/s    en 
0.08s   

2017-05-29 10:55:37 (2.02 MB/s) - skribita al ĉefeligujo [171969/171969]

$ find t -name AC_Ra\*
t/etc/ssl/certs/AC_Ra?z_Certic?mara_S.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem
$ locale
LANG=en_US.utf8
LC_CTYPE="en_US.utf8"
LC_NUMERIC="en_US.utf8"
LC_TIME="en_US.utf8"
LC_COLLATE="en_US.utf8"
LC_MONETARY="en_US.utf8"
LC_MESSAGES="en_US.utf8"
LC_PAPER=fr_FR.utf8
LC_NAME="en_US.utf8"
LC_ADDRESS="en_US.utf8"
LC_TELEPHONE="en_US.utf8"
LC_MEASUREMENT="en_US.utf8"
LC_IDENTIFICATION="en_US.utf8"
LC_ALL=
--8<---------------cut here---------------end--------------->8---

But wait!  “guix build nss-certs --check -K” fails, and the diff is:

--8<---------------cut here---------------start------------->8---
$ LANGUAGE= diff -ur 
/gnu/store/3ql0vilc0zv6ra42ghi04787vrg6bb71-nss-certs-3.30.2{,-check}
Only in 
/gnu/store/3ql0vilc0zv6ra42ghi04787vrg6bb71-nss-certs-3.30.2-check/etc/ssl/certs:
 
AC_Raíz_Certicámara_S.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem
Only in 
/gnu/store/3ql0vilc0zv6ra42ghi04787vrg6bb71-nss-certs-3.30.2/etc/ssl/certs: 
AC_Ra?z_Certic?mara_S.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem
diff -ur 
/gnu/store/3ql0vilc0zv6ra42ghi04787vrg6bb71-nss-certs-3.30.2/etc/ssl/certs/ae8153b9.0
 
/gnu/store/3ql0vilc0zv6ra42ghi04787vrg6bb71-nss-certs-3.30.2-check/etc/ssl/certs/ae8153b9.0
--- 
/gnu/store/3ql0vilc0zv6ra42ghi04787vrg6bb71-nss-certs-3.30.2/etc/ssl/certs/ae8153b9.0
       1970-01-01 01:00:01.000000000 +0100
+++ 
/gnu/store/3ql0vilc0zv6ra42ghi04787vrg6bb71-nss-certs-3.30.2-check/etc/ssl/certs/ae8153b9.0
 1970-01-01 01:00:01.000000000 +0100
@@ -3,10 +3,10 @@
 # distrust=
 # openssl-trust=codeSigning emailProtection serverAuth
 -----BEGIN CERTIFICATE-----
-MIIHyTCCBbGgAwIBAgIBATANBgkqhkiG9w0BAQUFADB9MQswCQYDVQQGEwJJTDEW
+MIIHhzCCBW+gAwIBAgIBLTANBgkqhkiG9w0BAQsFADB9MQswCQYDVQQGEwJJTDEW
 MBQGA1UEChMNU3RhcnRDb20gTHRkLjErMCkGA1UECxMiU2VjdXJlIERpZ2l0YWwg
 Q2VydGlmaWNhdGUgU2lnbmluZzEpMCcGA1UEAxMgU3RhcnRDb20gQ2VydGlmaWNh
-dGlvbiBBdXRob3JpdHkwHhcNMDYwOTE3MTk0NjM2WhcNMzYwOTE3MTk0NjM2WjB9
+dGlvbiBBdXRob3JpdHkwHhcNMDYwOTE3MTk0NjM3WhcNMzYwOTE3MTk0NjM2WjB9
 MQswCQYDVQQGEwJJTDEWMBQGA1UEChMNU3RhcnRDb20gTHRkLjErMCkGA1UECxMi

[...]

+O3NJo2pXh5Tl1njFmUNj403gdy3hZZlyaQQaRwnmDwFWJPsfvw55qVguucQJAX6V
+um0ABj6y6koQOdjQK/W/7HW/lwLFCRsI3FU34oH7N4RDYiDK51ZLZer+bMEkkySh
+NOsF/5oirpt9P/FlUQqmMGqz9IgcgA38corog14=
 -----END CERTIFICATE-----
Only in 
/gnu/store/3ql0vilc0zv6ra42ghi04787vrg6bb71-nss-certs-3.30.2-check/etc/ssl/certs:
 Certinomis_-_Autorité_Racine:2.1.1.pem
Only in 
/gnu/store/3ql0vilc0zv6ra42ghi04787vrg6bb71-nss-certs-3.30.2/etc/ssl/certs: 
Certinomis_-_Autorit?_Racine:2.1.1.pem
Only in 
/gnu/store/3ql0vilc0zv6ra42ghi04787vrg6bb71-nss-certs-3.30.2-check/etc/ssl/certs:
 NetLock_Arany_=Class_Gold=_Főtanúsítvány:2.6.73.65.44.228.0.16.pem
Only in 
/gnu/store/3ql0vilc0zv6ra42ghi04787vrg6bb71-nss-certs-3.30.2/etc/ssl/certs: 
NetLock_Arany_=Class_Gold=_F?tan?s?tv?ny:2.6.73.65.44.228.0.16.pem
Only in 
/gnu/store/3ql0vilc0zv6ra42ghi04787vrg6bb71-nss-certs-3.30.2/etc/ssl/certs: 
T?B?TAK_UEKAE_K?k_Sertifika_Hizmet_Sa?lay?c?s?_-_S?r?m_3:2.1.17.pem
Only in 
/gnu/store/3ql0vilc0zv6ra42ghi04787vrg6bb71-nss-certs-3.30.2/etc/ssl/certs: 
T?RKTRUST_Elektronik_Sertifika_Hizmet_Sa?lay?c?s?_H5:2.7.0.142.23.254.36.32.129.pem
Only in 
/gnu/store/3ql0vilc0zv6ra42ghi04787vrg6bb71-nss-certs-3.30.2-check/etc/ssl/certs:
 TÜBİTAK_UEKAE_Kök_Sertifika_Hizmet_Sağlayıcısı_-_Sürüm_3:2.1.17.pem
Only in 
/gnu/store/3ql0vilc0zv6ra42ghi04787vrg6bb71-nss-certs-3.30.2-check/etc/ssl/certs:
 
TÜRKTRUST_Elektronik_Sertifika_Hizmet_Sağlayıcısı_H5:2.7.0.142.23.254.36.32.129.pem
--8<---------------cut here---------------end--------------->8---

See?  (The difference in the first certificate is weird too…)

There are two ways to create nars.  One is via the ‘export-paths’ RPC
(implemented in the daemon in C++), which does not interpret file names
and thus leaves them untouched.  The other one is via ‘write-file’ from
(guix serialization), which is written in Scheme and thus converts file
names from locale encoding (specifically, ‘scandir’ does that.)

‘guix publish’ uses the latter, so ‘guix publish’ is sensitive to locale
settings, which is pretty bad.

Guile currently does not allow us to specify whether/how file names
should be decoded, but possible solutions have been discussed for 2.2.

In the meantime, solutions are:

  1. To run ‘guix publish’ in a UTF-8 locale, which apparently was not
     the case.

  2. Add to (guix build syscalls) a separate locale-independent
     ‘scandir’ implementation and use that.

Thoughts?

Ludo’.