Re: iotest 040, 041, intermittent failure in netbsd VM

[Peter Maydell]

On Mon, 17 Jan 2022 at 20:35, John Snow <jsnow@redhat.com> wrote:

> Can you please try applying this temporary patch and running `./check
> -qcow2 040 041` until you see a breakage and show me the output from
> that?

With this temporary patch the VM doesn't launch at all:

peter.maydell@hackbox2.linaro.org:~/qemu-netbsd$ make -C build/
vm-build-netbsd J=8 V=1 2>&1 | tee netbsd.log
make: Entering directory '/home/peter.maydell/qemu-netbsd/build'
/usr/bin/python3 -B /home/peter.maydell/qemu-netbsd/meson/meson.py
introspect --targets --tests --benchmarks | /usr/bin/python3 -B
scripts/mtest2m
ake.py > Makefile.mtest
{ \
  echo 'ninja-targets = \'; \
  /usr/bin/ninja -t targets all | sed 's/:.*//; $!s/$/ \\/'; \
  echo 'build-files = \'; \
  /usr/bin/ninja -t query build.ninja | sed -n '1,/^  input:/d; /^
outputs:/q; s/$/ \\/p'; \
} > Makefile.ninja.tmp && mv Makefile.ninja.tmp Makefile.ninja
(GIT="git" "/home/peter.maydell/qemu-netbsd/scripts/git-submodule.sh"
update ui/keycodemapdb meson tests/fp/berkeley-testfloat-3
tests/fp/berkeley-softfloat-3 dtc capstone slirp)
(GIT="git" "/home/peter.maydell/qemu-netbsd/scripts/git-submodule.sh"
update ui/keycodemapdb meson tests/fp/berkeley-testfloat-3
tests/fp/berkeley-softfloat-3 dtc capstone slirp)
/usr/bin/python3 -B /home/peter.maydell/qemu-netbsd/tests/vm/netbsd
--debug  --jobs 8 --verbose    --image
"/home/peter.maydell/.cache/qemu-vm/images/netbsd.img"  --snapshot
--build-qemu /home/peter.maydell/qemu-netbsd --
DEBUG:root:Creating archive
/home/peter.maydell/qemu-netbsd/build/vm-test-72ra6_8s.tmp/data-f706c.tar
for src_dir dir: /home/peter.maydell/qemu-netbsd
DEBUG:root:QEMU args: -nodefaults -m 4G -cpu max -netdev
user,id=vnet,hostfwd=:127.0.0.1:0-:22,ipv6=no -device
virtio-net-pci,netdev=vnet -vnc 127.0.0.1:0,to=20 -smp 8 -enable-kvm
-drive 
file=/home/peter.maydell/.cache/qemu-vm/images/netbsd.img,snapshot=on,if=none,id=drive0,cache=writeback
-device virtio-blk,drive=drive0,bootindex=0 -drive
file=/home/peter.maydell/qemu-netbsd/build/vm-test-72ra6_8s.tmp/data-f706c.tar,if=none,id=data-f706c,cache=writeback,format=raw
-device virtio-blk,drive=data-f706c,serial=data-f706c,bootindex=1
DEBUG:asyncio:Using selector: EpollSelector
DEBUG:qemu.aqmp.qmp_client.qemu-2335-7f3b78d7f128:Registering
<qemu.aqmp.events.EventListener object at 0x7f3b76bfc9b0>.
DEBUG:qemu.machine.machine:VM launch command: 'qemu-system-x86_64
-display none -vga none -chardev
socket,id=mon,path=/var/tmp/qemu-machine-0m15ou19/qemu-2335-7f3b78d7f128-monitor.sock
-mon chardev=mon,mode=control -machine pc -chardev
socket,id=console,path=/var/tmp/qemu-machine-0m15ou19/qemu-2335-7f3b78d7f128-console.sock,server=on,wait=off
-serial chardev:console -nodefaults -m 4G -cpu max -netdev
user,id=vnet,hostfwd=:127.0.0.1:0-:22,ipv6=no -device
virtio-net-pci,netdev=vnet -vnc 127.0.0.1:0,to=20 -smp 8 -enable-kvm
-drive 
file=/home/peter.maydell/.cache/qemu-vm/images/netbsd.img,snapshot=on,if=none,id=drive0,cache=writeback
-device virtio-blk,drive=drive0,bootindex=0 -drive
file=/home/peter.maydell/qemu-netbsd/build/vm-test-72ra6_8s.tmp/data-f706c.tar,if=none,id=data-f706c,cache=writeback,format=raw
-device virtio-blk,drive=data-f706c,serial=data-f706c,bootindex=1'
DEBUG:qemu.aqmp.qmp_client.qemu-2335-7f3b78d7f128:Transitioning from
'Runstate.IDLE' to 'Runstate.CONNECTING'.
DEBUG:qemu.aqmp.qmp_client.qemu-2335-7f3b78d7f128:Awaiting connection
on /var/tmp/qemu-machine-0m15ou19/qemu-2335-7f3b78d7f128-monitor.sock
...
DEBUG:qemu.aqmp.qmp_client.qemu-2335-7f3b78d7f128:Connection accepted.
DEBUG:qemu.aqmp.qmp_client.qemu-2335-7f3b78d7f128:Awaiting greeting ...
DEBUG:qemu.aqmp.qmp_client.qemu-2335-7f3b78d7f128:<-- {
  "QMP": {
    "version": {
      "qemu": {
        "micro": 1,
        "minor": 11,
        "major": 2
      },
      "package": "(Debian 1:2.11+dfsg-1ubuntu7.38)"
    },
    "capabilities": []
  }
}
DEBUG:qemu.aqmp.qmp_client.qemu-2335-7f3b78d7f128:Negotiating capabilities ...
DEBUG:qemu.aqmp.qmp_client.qemu-2335-7f3b78d7f128:--> {
  "execute": "qmp_capabilities",
  "arguments": {
    "enable": []
  }
}
DEBUG:qemu.aqmp.qmp_client.qemu-2335-7f3b78d7f128:<-- {
  "error": {
    "class": "GenericError",
    "desc": "Parameter 'enable' is unexpected"
  }
}
ERROR:qemu.aqmp.qmp_client.qemu-2335-7f3b78d7f128:Negotiation failed:
AssertionError
DEBUG:qemu.aqmp.qmp_client.qemu-2335-7f3b78d7f128:Negotiation failed:
  | Traceback (most recent call last):
  |   File 
"/home/peter.maydell/qemu-netbsd/tests/vm/../../python/qemu/aqmp/qmp_client.py",
line 306, in _negotiate
  |     assert 'return' in reply
  | AssertionError

ERROR:qemu.aqmp.qmp_client.qemu-2335-7f3b78d7f128:Failed to establish
session: qemu.aqmp.qmp_client.NegotiationError: Negotiation failed:
DEBUG:qemu.aqmp.qmp_client.qemu-2335-7f3b78d7f128:Failed to establish session:
  | Traceback (most recent call last):
  |   File 
"/home/peter.maydell/qemu-netbsd/tests/vm/../../python/qemu/aqmp/qmp_client.py",
line 306, in _negotiate
  |     assert 'return' in reply
  | AssertionError
  |
  | The above exception was the direct cause of the following exception:
  |
  | Traceback (most recent call last):
  |   File 
"/home/peter.maydell/qemu-netbsd/tests/vm/../../python/qemu/aqmp/protocol.py",
line 371, in _new_session
  |     await self._establish_session()
  |   File 
"/home/peter.maydell/qemu-netbsd/tests/vm/../../python/qemu/aqmp/qmp_client.py",
line 253, in _establish_session
  |     await self._negotiate()
  |   File 
"/home/peter.maydell/qemu-netbsd/tests/vm/../../python/qemu/aqmp/qmp_client.py",
line 312, in _negotiate
  |     raise NegotiationError(emsg, err) from err
  | qemu.aqmp.qmp_client.NegotiationError: Negotiation failed:

DEBUG:qemu.aqmp.qmp_client.qemu-2335-7f3b78d7f128:disconnect() called.
DEBUG:qemu.aqmp.qmp_client.qemu-2335-7f3b78d7f128:Transitioning from
'Runstate.CONNECTING' to 'Runstate.DISCONNECTING'.
DEBUG:qemu.aqmp.qmp_client.qemu-2335-7f3b78d7f128:Scheduling disconnect.
DEBUG:qemu.aqmp.qmp_client.qemu-2335-7f3b78d7f128:Closing StreamWriter.
DEBUG:qemu.aqmp.qmp_client.qemu-2335-7f3b78d7f128:Waiting for
StreamWriter to close ...
DEBUG:qemu.aqmp.qmp_client.qemu-2335-7f3b78d7f128:StreamWriter closed.
DEBUG:qemu.aqmp.qmp_client.qemu-2335-7f3b78d7f128:Disconnected.
DEBUG:qemu.aqmp.qmp_client.qemu-2335-7f3b78d7f128:QMP Disconnected.
DEBUG:qemu.aqmp.qmp_client.qemu-2335-7f3b78d7f128:Transitioning from
'Runstate.DISCONNECTING' to 'Runstate.IDLE'.
ERROR:qemu.machine.machine:Error launching VM
ERROR:qemu.machine.machine:Process was forked, waiting on it
DEBUG:qemu.aqmp.qmp_client.qemu-2335-7f3b78d7f128:disconnect() called.
DEBUG:qemu.aqmp.qmp_client.qemu-2335-7f3b78d7f128:Transitioning from
'Runstate.IDLE' to 'Runstate.DISCONNECTING'.
DEBUG:qemu.aqmp.qmp_client.qemu-2335-7f3b78d7f128:Scheduling disconnect.
DEBUG:qemu.aqmp.qmp_client.qemu-2335-7f3b78d7f128:Disconnected.
DEBUG:qemu.aqmp.qmp_client.qemu-2335-7f3b78d7f128:QMP Disconnected.
DEBUG:qemu.aqmp.qmp_client.qemu-2335-7f3b78d7f128:Transitioning from
'Runstate.DISCONNECTING' to 'Runstate.IDLE'.
WARNING:qemu.machine.machine:qemu received signal 9; command:
"qemu-system-x86_64 -display none -vga none -chardev
socket,id=mon,path=/var/tmp/qemu-machine-0m15ou19/qemu-2335-7f3b78d7f128-monitor.sock
-mon chardev=mon,mode=control -machine pc -chardev
socket,id=console,path=/var/tmp/qemu-machine-0m15ou19/qemu-2335-7f3b78d7f128-console.sock,server=on,wait=off
-serial chardev:console -nodefaults -m 4G -cpu max -netdev
user,id=vnet,hostfwd=:127.0.0.1:0-:22,ipv6=no -device
virtio-net-pci,netdev=vnet -vnc 127.0.0.1:0,to=20 -smp 8 -enable-kvm
-drive 
file=/home/peter.maydell/.cache/qemu-vm/images/netbsd.img,snapshot=on,if=none,id=drive0,cache=writeback
-device virtio-blk,drive=drive0,bootindex=0 -drive
file=/home/peter.maydell/qemu-netbsd/build/vm-test-72ra6_8s.tmp/data-f706c.tar,if=none,id=data-f706c,cache=writeback,format=raw
-device virtio-blk,drive=data-f706c,serial=data-f706c,bootindex=1"
ERROR:root:Failed to launch QEMU, command line:
ERROR:root:qemu-system-x86_64 -nodefaults -m 4G -cpu max -netdev
user,id=vnet,hostfwd=:127.0.0.1:0-:22,ipv6=no -device
virtio-net-pci,netdev=vnet -vnc 127.0.0.1:0,to=20 -smp 8 -enable-kvm
-drive 
file=/home/peter.maydell/.cache/qemu-vm/images/netbsd.img,snapshot=on,if=none,id=drive0,cache=writeback
-device virtio-blk,drive=drive0,bootindex=0 -drive
file=/home/peter.maydell/qemu-netbsd/build/vm-test-72ra6_8s.tmp/data-f706c.tar,if=none,id=data-f706c,cache=writeback,format=raw
-device virtio-blk,drive=data-f706c,serial=data-f706c,bootindex=1
ERROR:root:Log:
ERROR:root:VNC server running on 127.0.0.1:5900

Failed to prepare guest environment
Traceback (most recent call last):
  File 
"/home/peter.maydell/qemu-netbsd/tests/vm/../../python/qemu/aqmp/qmp_client.py",
line 306, in _negotiate
    assert 'return' in reply
AssertionError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File 
"/home/peter.maydell/qemu-netbsd/tests/vm/../../python/qemu/aqmp/protocol.py",
line 371, in _new_session
    await self._establish_session()
  File 
"/home/peter.maydell/qemu-netbsd/tests/vm/../../python/qemu/aqmp/qmp_client.py",
line 253, in _establish_session
    await self._negotiate()
  File 
"/home/peter.maydell/qemu-netbsd/tests/vm/../../python/qemu/aqmp/qmp_client.py",
line 312, in _negotiate
    raise NegotiationError(emsg, err) from err
qemu.aqmp.qmp_client.NegotiationError: Negotiation failed:

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File 
"/home/peter.maydell/qemu-netbsd/tests/vm/../../python/qemu/machine/machine.py",
line 399, in launch
    self._launch()
  File 
"/home/peter.maydell/qemu-netbsd/tests/vm/../../python/qemu/machine/machine.py",
line 437, in _launch
    self._post_launch()
  File 
"/home/peter.maydell/qemu-netbsd/tests/vm/../../python/qemu/machine/machine.py",
line 340, in _post_launch
    self._qmp.accept(self._qmp_timer)
  File 
"/home/peter.maydell/qemu-netbsd/tests/vm/../../python/qemu/aqmp/legacy.py",
line 69, in accept
    timeout
  File 
"/home/peter.maydell/qemu-netbsd/tests/vm/../../python/qemu/aqmp/legacy.py",
line 42, in _sync
    asyncio.wait_for(future, timeout=timeout)
  File "/usr/lib/python3.6/asyncio/base_events.py", line 484, in
run_until_complete
    return future.result()
  File "/usr/lib/python3.6/asyncio/tasks.py", line 339, in wait_for
    return (yield from fut)
  File 
"/home/peter.maydell/qemu-netbsd/tests/vm/../../python/qemu/aqmp/protocol.py",
line 274, in accept
    await self._new_session(address, ssl, accept=True)
  File 
"/home/peter.maydell/qemu-netbsd/tests/vm/../../python/qemu/aqmp/protocol.py",
line 390, in _new_session
    raise ConnectError(emsg, err) from err
qemu.aqmp.protocol.ConnectError: Failed to establish session:
Negotiation failed:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File 
"/home/peter.maydell/qemu-netbsd/tests/vm/../../python/qemu/machine/machine.py",
line 529, in _do_shutdown
    self._soft_shutdown(timeout)
  File 
"/home/peter.maydell/qemu-netbsd/tests/vm/../../python/qemu/machine/machine.py",
line 514, in _soft_shutdown
    self._subp.wait(timeout=timeout)
  File "/usr/lib/python3.6/subprocess.py", line 1469, in wait
    raise TimeoutExpired(self.args, timeout)
subprocess.TimeoutExpired: Command '('qemu-system-x86_64', '-display',
'none', '-vga', 'none', '-chardev',
'socket,id=mon,path=/var/tmp/qemu-machine-0m15ou19/qemu-2335-7f3b78d7f128-monitor.sock',
'-mon', 'chardev=mon,mode=control', '-machine', 'pc', '-chardev',
'socket,id=console,path=/var/tmp/qemu-machine-0m15ou19/qemu-2335-7f3b78d7f128-console.sock,server=on,wait=off',
'-serial', 'chardev:console', '-nodefaults', '-m', '4G', '-cpu',
'max', '-netdev', 'user,id=vnet,hostfwd=:127.0.0.1:0-:22,ipv6=no',
'-device', 'virtio-net-pci,netdev=vnet', '-vnc', '127.0.0.1:0,to=20',
'-smp', '8', '-enable-kvm', '-drive',
'file=/home/peter.maydell/.cache/qemu-vm/images/netbsd.img,snapshot=on,if=none,id=drive0,cache=writeback',
'-device', 'virtio-blk,drive=drive0,bootindex=0', '-drive',
'file=/home/peter.maydell/qemu-netbsd/build/vm-test-72ra6_8s.tmp/data-f706c.tar,if=none,id=data-f706c,cache=writeback,format=raw',
'-device', 'virtio-blk,drive=data-f706c,serial=data-f706c,bootindex=1')'
timed out after 30 seconds

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/peter.maydell/qemu-netbsd/tests/vm/basevm.py", line 647, in main
    vm.boot(img)
  File "/home/peter.maydell/qemu-netbsd/tests/vm/basevm.py", line 296, in boot
    guest.launch()
  File 
"/home/peter.maydell/qemu-netbsd/tests/vm/../../python/qemu/machine/machine.py",
line 409, in launch
    self.wait()
  File 
"/home/peter.maydell/qemu-netbsd/tests/vm/../../python/qemu/machine/machine.py",
line 576, in wait
    self.shutdown(timeout=timeout)
  File 
"/home/peter.maydell/qemu-netbsd/tests/vm/../../python/qemu/machine/machine.py",
line 558, in shutdown
    self._do_shutdown(timeout)
  File 
"/home/peter.maydell/qemu-netbsd/tests/vm/../../python/qemu/machine/machine.py",
line 533, in _do_shutdown
    from exc
qemu.machine.machine.AbnormalShutdown: Could not perform graceful shutdown
/home/peter.maydell/qemu-netbsd/tests/vm/Makefile.include:105: recipe
for target 'vm-build-netbsd' failed
make: *** [vm-build-netbsd] Error 2
make: Leaving directory '/home/peter.maydell/qemu-netbsd/build'

-- PMM