Hello community,
I am working on a gnuradio application that
synchronously retrieves data from a usrp_source, does some
DSP on each of the streams and selects the signal to send
on a usrp_sink based on some metric computation. The
sources and sinks represent 3 USRP X310 devices, each with
2 UBX-160 daughterboards, and are synchronized using an
Octoclock-G. Thus, creating a 6x1 MISO system. The USRP
synchronization is done using the Python API, as well as
the flowgraph. The DSP blocks are coded in C++. The
flowgraph would be something like:
self.connect((usrp_source, 0), my_dsp_0,
(my_switch, 0))
self.connect((usrp_source, 1), my_dsp_1,
(my_switch, 1))
self.connect((usrp_source, 2), my_dsp_2,
(my_switch, 2))
self.connect((usrp_source, 3), my_dsp_3,
(my_switch, 3))
self.connect(my_switch, another_dsp_0, (usrp_sink,
0))
The application runs for a while until UHD outputs a
message saying that the poke32 has timed out on one of the
USRP. After that, a misalignment error on the receiver
streams is shown and the green lights on the USRP go off.
We are under the impression that the DSP blocks working
independently may be requesting samples from the source at
different times, causing the misalignment. Could this be
the case? Is there an issue with my network connection?
I have tried to get rid of the misalignment error by
increasing the threshold in the file
{uhd_prefix}/host/lib/transport/super_recv_packet_handler.hpp,
through the function set_alignment_failure_threshold.
That got rid of the misalignment error, but the failure
coming from the poke function is still there.
Interestingly, the exact same behavior arises when I
just connect file_sinks to the outputs from the
usrp_source and a simple analog signal to the usrp_sink.
Below the flowgraph:
self.connect((usrp_source, 0), file_sink_0)
self.connect((usrp_source, 1), file_sink_1)
self.connect((usrp_source, 2), file_sink_2)
self.connect((usrp_source, 3), file_sink_3)
self.connect(analog_sig_c, (usrp_sink, 0))
I'd appreciate any help on this since I'm running out
of ideas here. Please, find below more details.
The error:
[ERROR] [X300] 192.168.100.2: x300 fw
communication failure #1
EnvironmentError: IOError: x300 fw poke32 - reply timed
out
[ERROR] [X300] 192.168.100.2: x300 fw
communication failure #1
EnvironmentError: IOError: x300 fw poke32 - reply timed
out
[ERROR] [X300] 192.168.100.2: x300 fw
communication failure #3
EnvironmentError: IOError: x300 fw poke32 - reply timed
out
[ERROR] [UHD] An unexpected exception was caught in a
task loop.The task loop will now exit, things may not
work.EnvironmentError: IOError:
192.168.100.2: x300 fw
communication failure #3
[ERROR] [STREAMER] The receive packet handler
failed to time-align packets.
1002 received packets were processed by the handler.
However, a timestamp match could not be determined.
Here are some details of my network configuration:
- UHD built from source, tag v3.14.1.1.
- Gnuradio built from source, tag 3.7.13.5.
- X310 USRP, each equipped with two UBX-160
daughterboards.
- The communication is over Ethernet using the 1Gb port
on the X310.
- Each USRP is connected to a dedicated NIC.
- The MTU is set to 1500 for the all interfaces.
- The USRPs are synchronized using an Octoclock-G and
the parameters tuned in the Python script. The procedure
is similar to the one followed in benchmark_rate (uhd),
for the usrp_source and usrp_sink.
- The NIC ring buffers are set to the maximum for both
TX and RX, to 4096, using ethtool.
- The OS ring buffers are set as suggested bu Ettus, using
the following commands:
>> sudo sysctl -w net.core.rmem_max=33554432
>> sudo sysctl -w net.core.wmem_max=33554432
>> sudo sysctl -w net.core.rmem_default=33554432
>> sudo sysctl -w net.core.wmem_default=33554432