[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Guix on clusters and in HPC

From: Ludovic Courtès
Subject: Guix on clusters and in HPC
Date: Tue, 18 Oct 2016 16:20:43 +0200
User-agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1 (gnu/linux)


I’m trying to gather a “wish list” of things to be done to facilitate
the use of Guix on clusters and for high-performance computing (HPC).

Ricardo and I wrote about the advantages, shortcomings, and perspectives

I know that Pjotr, Roel, Ben, Eric and maybe others also have experience
and ideas on what should be done (and maybe even code? :-)).

So I’ve come up with an initial list of work items going from the
immediate needs to crazy ideas (batch scheduler integration!) that
hopefully make sense to cluster/HPC people.  I’d be happy to get
feedback, suggestions, etc. from whoever is interested!

(The reason I’m asking is that I’m considering submitting a proposal at
Inria to work on some of these things.)

TIA!  :-)


  - non-root usage
    + file system virtualization needed
      * map ~/.local/gnu/store to /gnu/store
      * user name spaces?
      * [[][PRoot]]? but performance problems?
      * common interface, like “guix enter” spawns a shell where
        /gnu/store is available
    + daemon functionality as a library
      * client no longer connects to the daemon, does everything
        locally, including direct store accesses
      * can use substitutes
    + or plain ’guix-daemon --disable-root’?
    + see 
with Ben Woodcroft and Roel]]
  - central daemon usage (like at MDC, but improved)
    + describe/define appropriate setup, like:
      * daemon runs on front-end node
      * clients can connect to daemon from compute nodes, and perform
        any operation
      * use of distributed file systems: anything to pay attention to?
      * how should the front-end offload to compute nodes?
    + technical issues
      * daemon needs to be able to listen for connections elsewhere
      * client needs to be able to 
[[][connect remotely]] 
instead of using
        [[][‘socat’ hack]]
      * how do we share localstatedir?  how do we share /gnu/store?
      * how do we share the profile directory?
    + admin/social issues
      * daemon runs as root
      * daemon needs Internet access
      * Ricardo mentions lack of nscd and problems caused by the use of
        NSS plugins like [[][SSSD]] 
in this context
    + batch scheduler integration?
      * allow users to offload right from their machine to the cluster?
  - package variants, experimentation
    + for experiments, as in Section 4.2 of 
[[][the RepPar paper]]
      * in the meantime we added 
 et al.]]; need more?
    + for 
    + somehow support -mtune=native (and even profile-guided
    + simplify the API to switch compilers, libcs, etc.
  - workflow, reproducible science
    + implement [[][channels]]
    + provide a way to see which Guix commit is used, like “guix channel
    + simple ways to 
[[][test the 
dependents of a package]] (see also
      discussion between E. Agullo & A. Enge)
      * new transformation options: --with-graft, --with-source
    + support 
 and pipelines]]?
    + add [[][Guix support 
in Galaxy]]?

reply via email to

[Prev in Thread] Current Thread [Next in Thread]