Re: [WIP Patch] Adding an FHS container to guix shell

[John Kehayias]

Hi Ludo’,

------- Original Message -------
On Monday, July 18th, 2022 at 7:40 AM, Ludovic Courtès wrote:

> Hello!
>
> John Kehayias skribis:
>
> > 2. Typically binaries will expect the ld loader to use
> > /etc/ld.so.cache for finding libraries.
>
> Not necessarily. /etc/ld.so.cache is an optimization, but it’s entirely
> possible to opt out: if that file is missing, things will work fine, but
> libc will ‘stat’ more to find files. That would make glibc-for-fhs
> unnecessary.
>

I was thinking mostly of binaries, though I've also encountered code that 
wanted to read the ldcache (e.g. parsing ldconfig -p directly), for some 
reason. Is there any difficulty with having already some /etc/ld.so.cache from 
some package in the container profile? Perhaps that leads some programs to rely 
on the cache and then not continue searching for libraries? I'm not sure how 
that works, but only know that I've found the ldcache necessary (at least 
without modifying source) for things I would need the FHS container for.

My overall goal with this FHS option is to mimic both the standard and the 
typical expectations of distros that follow it (global ld cache, certain 
directories in PATH, etc). In my testing I didn't get very far without the 
modified glibc, though perhaps there are other ways around it. (There's 
patchelf, though hard to scale that I think.)

For example, lots of language build sytems/package managers will bring in some 
binaries, whether it is just to install (rustup does this) or because something 
like a web automation framework downloads browser binaries to use. My goal here 
was to make this all "just work," but I can understand wanting to keep it 
minimal.

I'll proceed with the glibc-for-fhs and ldcache for now, but open to discussing 
and hearing what everyone thinks. Overall, I find it necessary to enable the 
use-cases I'm thinking of.

> > + ;; Set up an FHS container.
> > + (when fhs-container?
> > + ;; Set up the expected bin and library directories as symlinks to
> > + ;; the profile lib directory. Note that this is assuming a 64bit
> > + ;; architecture.
> > + (let ((lib-dir (string-append profile "/lib")))
> > + (symlink lib-dir "/lib64")
> > + (symlink lib-dir "/lib")
> > + (mkdir-p "/usr")
> > + (symlink lib-dir "/usr/lib"))
>
> Instead of adding code here, maybe you could do in a more declarative
> fashion, like:
>
> (define fhs-mappings
> (list (file-system-mapping
> (source (string-append profile "/bin")) (target "/bin"))
> …))
>
> and append that to the ‘mappings’ variable there. It’s not necessarily
> more compact, but maybe marginally easier to read?
>

Thanks, that's all a good idea for tidying up! I did so, further breaking down 
the FHS directories to ones we can map directly like this, others that need 
directory contents symlinked (because the container already has e.g. /bin not 
empty), and then creating the optional FHS directories to link to these. For 
example, /lib is mapped to profile/lib and /usr/lib is linked to /lib. Again, 
not strictly necessary, but I was trying to follow the structure of other 
distros here. And this way there is still "one" place where everything from the 
profile ends up, /lib, /bin, etc.

> > + ;; Define an entry script to start the container: generate
> > + ;; ld.so.cache, supplement $PATH, and include command.
>
> I’d leave ld.so.cache generation out.
>

As discussed above, I'm finding this rather necessary and makes for a smooth 
FHS-container experience. I'm open to other options, maybe making this an 
additional option or hiding it away, though I think the design here fits the 
"pretend to be like another distro and just work" that I was going for.

> > + (call-with-output-file "/tmp/fhs.sh"
> > + (lambda (port)
> > + (display "ldconfig -X -f /tmp/ld.so.conf" port)
> > + (newline port)
> > + (display "export PATH=/bin:/usr/bin:/sbin:/usr/sbin:$PATH" port)
>
> I think the default value of PATH in our libc is the FHS one:
>
> --8<---------------cut here---------------start------------->8---
>
> $ env -i $(type -P strace) -e execve -f $(type -P guile) -c '(system* 
> "whatever")'
> execve("/gnu/store/lpcjxka7hx3ypv4nz47g08k4m2syqwlj-profile/bin/guile", 
> ["/gnu/store/lpcjxka7hx3ypv4nz47g0"..., "-c", "(system* \"whatever\")"], 
> 0x7ffede27ad38 /* 0 vars /) = 0
> /home/ludo/.guix-home/profile/bin/strace: Process 9727 attached
> /home/ludo/.guix-home/profile/bin/strace: Process 9728 attached
> /home/ludo/.guix-home/profile/bin/strace: Process 9729 attached
> /home/ludo/.guix-home/profile/bin/strace: Process 9730 attached
> /home/ludo/.guix-home/profile/bin/strace: Process 9731 attached
> /home/ludo/.guix-home/profile/bin/strace: Process 9732 attached
> [pid 9732] execve("/bin/whatever", ["whatever"], 0x7ffed6d967c8 / 0 vars /) = 
> -1 ENOENT (No such file or directory)
> [pid 9732] execve("/usr/bin/whatever", ["whatever"], 0x7ffed6d967c8 / 0 vars 
> */) = -1 ENOENT (No such file or directory)
> In execvp of whatever: No such file or directory
> --8<---------------cut here---------------end--------------->8---
>
>
> So you could leave it undefined, but ‘load-profile’ in
> ‘launch-environment’ will define it.
>

I'm not sure what you were showing here/I'm a bit confused. If I do just 'guix 
shell -C' for example, I can't just do 'sh' or 'exec sh' in this environment 
although /bin/sh exists (as a link to /gnu/store/...bash-.../bin/sh). While 
your example does indeed find sh (if I include the needed inputs to run the 
example), I'm not sure how that relates to running things directly. But maybe 
you just mean in terms of libc directly. Sorry, I don't know quite enough in 
this area.

In the container PATH only has profile/bin. Additionally, we frequently see 
hardcoded path assumptions like /bin or /usr/bin, or expectations of PATH being 
set with these. Adding, e.g. coreutils and which to the shell, which will 
happily find 'ls' but not 'sh', as which just searches PATH of course.

This could be worked around by the user, but again, I was going for "just 
works" as much as possible. Or at least as other distros "just work". (Side 
note: not that I think this is better! All this FHS stuff gives me even better 
appreciation for the design of Guix, despite the initial complications one sees 
as a new user.)

> Instead of the wrapper script, maybe you could extend
> ‘launch-environment’ so the caller can make it override certain
> variables? I would find it a bit clearer.
>

Would this still be better if we do keep the ldcache being generated? I didn't 
see a quick way to just add in some setup commands to the container, though 
that could be a nice extension. Admittedly, my script here was the simplest 
hack I saw that worked, so I'm not married to it by any means. Again, the user 
could also generate the ldcache, but was trying to make it all automatic.

> Thanks,
> Ludo’.

Thanks for the constructive comments! It definitely helped me clean up the 
implementation, which I'll be sending shortly to the patches list. I'll 
followup on this thread as well to point people to a patch they can try.

For my testing I've been able to do work with web automations (uses binaries of 
browsers), tested that AppImages can work, built a Rust project, etc. A useful 
tool, even if it is one I hope I don't have to reach for too often.

John