Re: Progress on 'guix deploy'

[Ludovic Courtès]

Hello Jakob,

address@hidden (Jakob L. Kreuze) skribis:

> As a quick refresher, my GSoC project this summer is 'guix deploy', a
> deployment automation tool for GuixSD that's been discussed more
> thoroughly in [1] and [2]. Development has taken place on my personal
> branch of Guix, specifically the 'wip-deploy' branch [3], and is
> represented by three new Scheme source files:
>
> - 'gnu/machine.scm', which provides an abstraction for /something/ that
>   can be deployed to in a heterogeneous deployment. Currently, the only
>   concrete implementation of this is the simple case of in-place updates
>   to machines running SSH whose names and IP addresses we know.
> - 'gnu/tests/machine.scm', which implements some tests for
>   'gnu/machine.scm'. This is where I'm most interested in receiving
>   feedback. More on that later.
> - 'guix/scripts/deploy.scm', which implements the rudimentary
>   command-line interface for 'guix deploy'.

Nice!

> The command-line interface hasn't really been fleshed out yet, but if
> you'd like to play around with it, it takes a "deployment" file as a
> parameter, which is a Scheme file looking something like the following:
>
> #+BEGIN_SRC scheme
> (use-modules ...)
>
> (define %system
>   (operating-system
>    (host-name "gnu-deployed")
>    (timezone "Etc/UTC")
>    (bootloader (bootloader-configuration
>                 (bootloader grub-bootloader)
>                 (target "/dev/sda")
>                 (terminal-outputs '(console))))
>    (file-systems (cons (file-system
>                         (mount-point "/")
>                         (device "/dev/vda1")
>                         (type "ext4"))
>                        %base-file-systems))
>    (services
>     (append (list (service dhcp-client-service-type)
>                   (service openssh-service-type
>                            (openssh-configuration
>                             (permit-root-login #t)
>                             (allow-empty-passwords? #t))))
>             %base-services))))
>
> (list (make <sshable-machine>
>         #:host-name "localhost"
>         #:ssh-port 5556
>         #:system %system))
> #+END_SRC
>
> Basically, you attach an 'operating-system' declaration to a 'machine'.
> In this case, 'sshable-machine' is the specific type of 'machine' that
> we're deploying to (one that's running an SSH daemon and has a known IP
> + port + hostname).

Looks great to me.

> I've found that the GuixSD QEMU images work well for playing around with
> this, provided that you add the SSH service to the system configuration
> and start it. In the case of this deployment file, I had a GuixSD guest
> with port 22 forwarded to my host's port 5556. You'll also need to set
> up some sort of public key auth in your SSH config. The current code
> isn't capable of handling other forms of SSH authentication.

That’s reasonable.

> In terms of implementation, GOOPS feels like a bit of an unusual choice
> when juxtaposed with the rest of the Guix codebase, but I've come to
> really enjoy it. I'll roll with it for now, since I think it will make
> it easier to flesh out the vocabulary for specifying deployments.

It’s more than unusual.  :-)  Perhaps it’s the right choice here, we’ll
have to discuss it at some point!

> The implementation of '<sshable-machine>' is doing what
> 'switch-to-system' and 'install-bootloader' in 'guix/scripts/system.scm'
> do, but in terms of data that can be sent with 'remote-eval'. I imagine
> the code will make more sense if you read both simultaneously.

OK, sounds good.

Some time ago, I proposed to have ‘remove-eval’ where you could do:

  (remote-eval #~(begin …) #:session …)

which would take care of building and copying everything the gexp refers
to (see
<https://lists.gnu.org/archive/html/guix-devel/2019-03/msg00127.html>.)
That would generalize a bit what you describe above.  But anyway, that’s
something that can always come later.

> Okay, on to the test suite.
>
> My understanding of the system test suite (tests run with 'check-system'
> as opposed to those run with 'check') is that the meat of the test code
> exists in a G-Expression and should _not_ be interacting with the store
> of the machine running the test suite (i.e. that's the reason we're
> using marionettes in the first place). 'gnu/tests/install.scm' seems to
> be somewhat of an exception, and because the code in '(gnu machine)'
> depends heavily on having access to a store, I've tried to extend what's
> done in 'gnu/tests/install.scm' so that my tests have access to store
> while instrumenting the marionettes.

When you say “access the store”, you mean interact with the build
daemon, right?

The (gnu tests install) tests are indeed the only ones that talk to the
build daemon.

> To be specific, the chicken and egg scenario I'm working around is that
> the SSH daemon on the marionette needs to be running in order for
> 'deploy-os' to work, but I can't call 'deploy-os' from the test
> G-Expression because the store wouldn't be accessible then.
>
> My gut is telling me that this is absolutely the wrong way to go about
> it, though. 'call-with-marionette' is one of a couple red flags making
> me feel that way -- I don't think marionettes were meant to be started
> outside the context of a derivation...

Marionettes are just a way to instrument the system under test, which is
running in a VM.

I think what you need is to ensure the target VM (the one you’re
deploying to) has a writable store.  To do that, you could create the
target VM with the equivalent of ‘guix system vm-image’, which is the
‘system-qemu-image’ procedure.  (That will require a bit of disk space,
because that image needs to be large enough to hold the store of the
machine.)

How does that sound?

> I'm going to continue to spend time on the "internals" of 'guix deploy'
> this coming week, incorporating any feedback I receive and ensuring that
> I have a framework to build upon when I extend this to more complicated
> scenarios like deploying to a VPS provider. After that, I'll tackle
> fleshing out the way deployments are specified (the file part. I'm
> holding off on the command-line tool part for right now).

Sounds good!

Consider hanging out on #guix and sending email here—it’s usually easier
to get feedback on specific topics.

Anyway, glad to see so much progress already!  Thanks for the update!

Ludo’.