bug#55441: [cuirass] hang in "In progress..."; runs out of pgsql connections

[Maxim Cournoyer]

Hello Guix,

Since a couple days, Craps exhibits a new problem, where the
evaluations on master appear stalled, showing as "In progress..."
forever.

Eventually, it seems the maximum number of PostgreSQL connections
allowed (300, if what Ricardo wrote on #guix is accurate :-)) is
reached, and we get the following error:

--8<---------------cut here---------------start------------->8---
Backtrace:
In ice-9/boot-9.scm:
  1752:10 11 (with-exception-handler _ _ #:unwind? _ # _)
In unknown file:
          10 (apply-smob/0 #<thunk 7f4455f8b340>)
In ice-9/boot-9.scm:
    724:2  9 (call-with-prompt _ _ #<procedure default-prompt-handle?>)
In ice-9/eval.scm:
    619:8  8 (_ #(#(#<directory (guile-user) 7f4455f8ec80>)))
In cuirass/ui.scm:
   104:10  7 (run-cuirass-command _ . _)
In cuirass/scripts/evaluate.scm:
    101:7  6 (cuirass-evaluate _)
In cuirass/utils.scm:
    123:6  5 (make-worker-thread-channel _ #:parallelism _)
In srfi/srfi-1.scm:
    634:9  4 (for-each #<procedure 7f4451dd9e40 at cuirass/utils.sc?> ?)
In cuirass/utils.scm:
   125:21  3 (_ . _)
In cuirass/scripts/evaluate.scm:
    101:7  2 (_)
In cuirass/database.scm:
   375:13  1 (db-open #:database _)
In squee.scm:
   257:10  0 (connect-to-postgres-paramstring _)

squee.scm:257:10: In procedure connect-to-postgres-paramstring:
Throw to key `psql-connect-error' with args `(connection-bad "FATAL:  remaining 
connection slots are reserved for non-replication superuser connections\n")'.
--8<---------------cut here---------------end--------------->8---

(see: https://ci.guix.gnu.org/eval/323648/log/raw)

Restarting postgresql workarounds the above error, but it doesn't fix
the hang, and the problem connection problem recurs.

I've logged at the Cuirass and messages logs, but I haven't spot
anything abnormal.

Thanks,

Maxim