Re: Reducing rare underflows to zero occurrences

[Chad Spooner]

Daniel:

Thanks much for the tips.

I tried to monitor /proc/net/udp while the channel flowgraph was running over a couple hours. Every second I called

ss -ulm

and recorded the results.

I never saw Recv-Q or Send-Q deviate from 0, and all occurrences of the dropped-packed indicator were also 0:

State  Recv-Q Send-Q Local Address:Port   Peer Address:PortProcess                        
                        
UNCONN 0      0      127.0.0.53%lo:domain      0.0.0.0:*   
     skmem:(r0,rb212992,t0,tb212992,f4096,w0,o0,bl0,d0) 
UNCONN 0      0            0.0.0.0:49206       0.0.0.0:*   
     skmem:(r0,rb212992,t0,tb212992,f0,w0,o0,bl0,d0)    
UNCONN 0      0            0.0.0.0:631         0.0.0.0:*   
     skmem:(r0,rb212992,t0,tb212992,f0,w0,o0,bl0,d0)    
UNCONN 0      0            0.0.0.0:mdns        0.0.0.0:*   
     skmem:(r0,rb212992,t0,tb212992,f4096,w0,o144,bl0,d0)

During this multi-hour experiment, there were several reported underflows, and I noticed that many of them are separated by 128 seconds, 512 seconds, 256 seconds, and combinations like 384 = 128 + 256 seconds:

Uusrp_sink :error: In the last 8307 ms, 1 underflows occurred.
usrp_sink :error: In the last 340937 ms, 1 underflows occurred.
Uusrp_sink :error: In the last 171004 ms, 1 underflows occurred.
Uusrp_sink :error: In the last 127998 ms, 1 underflows occurred.
UUusrp_sink :error: In the last 640023 ms, 1 underflows occurred.
usrp_sink :error: In the last 511982 ms, 1 underflows occurred.
UUusrp_sink :error: In the last 256048 ms, 1 underflows occurred.
usrp_sink :error: In the last 127946 ms, 1 underflows occurred.
Uusrp_sink :error: In the last 1963006 ms, 1 underflows occurred.
Uusrp_sink :error: In the last 128000 ms, 1 underflows occurred.
UUusrp_sink :error: In the last 384001 ms, 1 underflows occurred.
usrp_sink :error: In the last 171006 ms, 1 underflows occurred.
UUusrp_sink :error: In the last 85083 ms, 1 underflows occurred.
usrp_sink :error: In the last 42916 ms, 1 underflows occurred.
UUusrp_sink :error: In the last 85082 ms, 1 underflows occurred.
Uusrp_sink :error: In the last 127925 ms, 1 underflows occurred.
Uusrp_sink :error: In the last 43042 ms, 1 underflows occurred.
usrp_sink :error: In the last 127950 ms, 1 underflows occurred.
Uusrp_sink :error: In the last 128000 ms, 1 underflows occurred.
Uusrp_sink :error: In the last 127998 ms, 1 underflows occurred.
Uusrp_sink :error: In the last 127999 ms, 1 underflows occurred.
Uusrp_sink :error: In the last 128000 ms, 1 underflows occurred.
Uusrp_sink :error: In the last 127999 ms, 1 underflows occurred.
Uusrp_sink :error: In the last 384001 ms, 1 underflows occurred.
UUusrp_sink :error: In the last 127999 ms, 1 underflows occurred.
usrp_sink :error: In the last 128000 ms, 1 underflows occurred.
UUusrp_sink :error: In the last 256082 ms, 1 underflows occurred.
Uusrp_sink :error: In the last 127940 ms, 1 underflows occurred.
Uusrp_sink :error: In the last 427075 ms, 1 underflows occurred.
Uusrp_sink :error: In the last 127941 ms, 1 underflows occurred.
Uusrp_sink :error: In the last 384023 ms, 1 underflows occurred.
usrp_sink :error: In the last 127942 ms, 1 underflows occurred.
Uusrp_sink :error: In the last 128000 ms, 1 underflows occurred.
UUusrp_sink :error: In the last 384023 ms, 1 underflows occurred.
usrp_sink :error: In the last 255977 ms, 1 underflows occurred.
Uusrp_sink :error: In the last 511999 ms, 1 underflows occurred.
UUusrp_sink :error: In the last 384024 ms, 1 underflows occurred.

which is weird and probably significant?

I hope my use of ss was correct.

Thanks again,

Chad

On 1/16/25 1:49 PM, Daniel Estévez wrote:

> Hi
>       Chad, 
>       
>       Something that you can do to monitor the problem is to look in
>       /proc/net/udp. This will show you the fill level of kernel UDP
>       receive buffers. If you've set the buffer size to 32 MiB as
>       recommended by the performance tuning tips, usually these sporadic
>       drops are the result of the flowgraph consuming data at a rate
>       slightly lower than the nominal sample rate for multiple seconds,
>       rather than a brief but larger performance reduction. If this is
>       the case, you can see how the buffer fill level slowly increases
>       over the course of several seconds until the buffer fills up and
>       packets are dropped. Looking at this won't solve your problem, but
>       at least it will give you a way to monitor if you do some changes
>       to try to improve the performance. 
>       
>       Best, 
>       Daniel. 
>       
>       On 16/01/2025 19:38, Chad Spooner wrote: 
> > All: 
> >         
> >         I'm developing a comm system involving three SDRs: transmit,
> >         channel effects, and receiver. 
> >         
> >         The channel SDR is controlled by a flowgraph that has various
> >         options for the channel: AWGN (h(\tau) = \delta(\tau)), LTI, and
> >         a couple time- variant frequency-selective channel models. 
> >         
> >         I need to run the full three-SDR system over long periods of
> >         time to get stable average values for system-performance
> >         parameters. 
> >         
> >         My problem is that the channel flowgraph reports underflows on
> >         the order of every hundred seconds or so (sometimes much less).
> >         I've tracked down these rare underflows in the channel to
> >         transient problems in the receiver flowgraph. 
> >         
> >         So I put together a minimum flowgraph that shows the problem,
> >         which is attached. When I run it, I get output like this: 
> >         
> >             [INFO] [UHD] linux; GNU C++ version 11.2.0; Boost_107400;
> >         UHD_4.1.0.5-3 
> >             [INFO] [X300] X300 initialization sequence... 
> >             [INFO] [X300] Maximum frame size: 8000 bytes. 
> >             [INFO] [X300] Radio 1x clock: 200 MHz 
> >             usrp_source :info: set_min_output_buffer on block 1 to
> >         2097152 
> >             usrp_sink :info: set_min_output_buffer on block 2 to 2097152
> >         
> >             mmse_resampler_cc :info: set_min_output_buffer on block 4 to
> >         2097152 
> >             fir_filter_blk<IN_T,OUT_T,TAP_T> :info:
> >         set_min_output_buffer on 
> >             block 5 to 2097152 
> >             multiply_cc :info: set_min_output_buffer on block 9 to
> >         2097152 
> >             sig_source :info: set_min_output_buffer on block 8 to
> >         2097152 
> >             add_cc :info: set_min_output_buffer on block 6 to 2097152 
> >             fastnoise_source :info: set_min_output_buffer on block 7 to
> >         2097152 
> >             *Uusrp_sink :error: In the last 579300 ms, 1 underflows
> >         occurred. 
> >             Uusrp_sink :error: In the last 256008 ms, 1 underflows
> >         occurred. 
> >             usrp_sink :error: In the last 84992 ms, 1 underflows
> >         occurred. 
> >             UUusrp_sink :error: In the last 128008 ms, 1 underflows
> >         occurred.* 
> >         
> >         The PSD of the filtered input signal appears correctly on a
> >         spectrum analyzer. 
> >         
> >         System Information: 
> >         
> >             24-core Intel i9 system 
> >             Ubuntu 22.04.5 LTS, kernel 5.19.0-46-generic
> >         #47~22.04.1-Ubuntu 
> >             gnuradio 3.10.1.1 
> >             UHD 4.1.0.5-3 
> >         
> >         I've taken some pains to optimize the OS settings for
> >         gnuradio/UHD along the lines suggested by the online tips at 
> >         
> >             https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks
> >         
> >         
> >         I've also tinkered with the minoutbuf parameter in the various
> >         blocks. 
> >         
> >         Any advice? 
> >         
> >         Thanks, 
> >         
> >         Chad 
> >         cyclostationary.blog 
> >         cmspooner@nwra.com 
> >         
> >         
> >         
> >         
> >         
> >         -- 
> >         Chad M. Spooner, PhD 
> >         NorthWest Research Associates 
> >         149 Bonifacio Place, Suite 200 
> >         Monterey, CA 93940 
> >         cmspooner@nwra.com 
> >         831 521 6743 
> >         NWRA Scientist/Engineer, VP, BoD 
> >         cyclostationary.blog 
> >         
> >       
>       
>     
-- 
Chad M. Spooner, PhD
NorthWest Research Associates
149 Bonifacio Place, Suite 200
Monterey, CA 93940
cmspooner@nwra.com
831 521 6743
NWRA Scientist/Engineer, VP, BoD
cyclostationary.blog

smime.p7s
Description: S/MIME Cryptographic Signature