shepherd-action debug help needed

[Attila Lendvai]

dear guixers,

i have a custom shepherd action that calls tar with (system ...):

https://github.com/attila-lendvai/guix-crypto/blob/staging/src/guix-crypto/services/swarm.scm#L311

it works fine in a `guix system vm` environment, where i'm usually testing it. 
but when i pull it to my server, it fails with exit code 127.

it has logging:

2023-10-30T14:24:12 Will run backup cmd: "install --mode=600 
<(/gnu/store/iiyf5ns10j8sgzfpmcgbfyb0byk5sbjl-tar-1.34/bin/tar --verbose 
--create --directory /var/lib/swarm/mainnet/bee-0 keys/ statestore/ | 
/gnu/store/gjsxzcc0gqpz4lpbsrbidlnn5ij1lfm1-gzip-1.12/bin/gzip) 
/tmp/2023-10-30-serlap-bee-mainnet-0.tgz", PATH is 
/gnu/store/d4rqw481nwvrzs09nd8ad647nczgm9k1-coreutils-9.1/bin:/gnu/store/gjsxzcc0gqpz4lpbsrbidlnn5ij1lfm1-gzip-1.12/bin,
 uid is 0
2023-10-30T14:24:12 Cmd returned exit code 127

if i copy-paste the above command into the same root prompt from where i'm 
invoking `herd backup-identity bee-0 /tmp/`, then it works as expected. even if 
i prefix it with `PATH="" ...`.

my question is:

what could be the difference between running a `guix system vm` from my git 
checkout, and guix pull'ing the same to a proper guix system?

could it be file or other permissions? according to the log, (getuid) is 0 when 
the action is executed.

alternatively, how could i debug this?

for completeness, this is how i start the system vm test:

$(./pre-inst-env guix system --no-graphic vm 
--share=$HOME/workspace/guix/var-lib-of-guest-vm=/var/lib 
~/workspace/guix/guix-crypto/tests/swarm-tests.scm) -m 2048

and this is the file:

https://github.com/attila-lendvai/guix-crypto/blob/staging/tests/swarm-tests.scm

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“Tyranny is defined as that which is legal for the government but illegal for 
the citizenry.”
        — Thomas Jefferson (1743–1826)