Block timeout during flush

[nzqo]

Issue Description

Occasionally, and non-deterministically, I see the following warning:

[WARNING] [RFNOC::GRAPH] One or more blocks timed out during flush!

The weird part is that it happens during uhd_usrp_probe, i.e. nowhere in any code written by me. I see that this warning pops up for other people as well, but I couldn't find any resource properly explaining why or how one could fix it.

Setup Details

Usrp x310 (2954R), Ubuntu 20.04 (kernel 5.15.0-113-generic), XG FPGA image flavor, X520-DA2 10G Ethernet Card, Uhd version: 4.7.0.0-0ubuntu1~focal1. All performance tips were followed (CPU governor, thread priority, MTU size, network buffer sizes, ring buffer sizes, and disabling KPTI protections)

Expected Behavior

No timeouts.

Actual Behaviour

Sometimes blocks timing out (see logs at the end)

Steps to reproduce the problem

Run uhd_usrp_probe a few times.

Additional Information

Full output of uhd_usrp_probe:

[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100; UHD_4.7.0.0-0ubuntu1~focal1
[INFO] [X300] X300 initialization sequence...
[INFO] [X300] Maximum frame size: 8000 bytes.
[INFO] [GPS] Found an internal GPSDO: LC_XO, Firmware Rev 0.929a
[INFO] [X300] Radio 1x clock: 200 MHz
[WARNING] [RFNOC::GRAPH] One or more blocks timed out during flush!
  _____________________________________________________
 /
|       Device: X-Series Device
|     _____________________________________________________
|    /
|   |       Mboard: X310
|   |   revision: 8
|   |   revision_compat: 7
|   |   product: 30810
|   |   mac-addr0: 00:80:2f:27:0d:ce
|   |   mac-addr1: 00:80:2f:27:0d:cf
|   |   gateway: 192.168.10.1
|   |   ip-addr0: 192.168.10.2
|   |   subnet0: 255.255.255.0
|   |   ip-addr1: 192.168.20.2
|   |   subnet1: 255.255.255.0
|   |   ip-addr2: 192.168.30.2
|   |   subnet2: 255.255.255.0
|   |   ip-addr3: 192.168.40.2
|   |   subnet3: 255.255.255.0
|   |   serial: 30F1626
|   |   FW Version: 6.1
|   |   FPGA Version: 39.3
|   |   FPGA git hash: c37b318
|   |   Device DNA: 0064148262F10854
|   |   RFNoC capable: Yes
|   |   
|   |   Time sources:  internal, external, gpsdo
|   |   Clock sources: internal, external, gpsdo
|   |   Sensors: gps_servo, gps_gprmc, gps_time, gps_gpgga, gps_locked, ref_locked
|     _____________________________________________________
|    /
|   |       RFNoC blocks on this device:
|   |   
|   |   * 0/DDC#0
|   |   * 0/DDC#1
|   |   * 0/DUC#0
|   |   * 0/DUC#1
|   |   * 0/Radio#0
|   |   * 0/Radio#1
|   |   * 0/Replay#0
|     _____________________________________________________
|    /
|   |       Static connections on this device:
|   |   
|   |   * 0/SEP#0:0==>0/DUC#0:0
|   |   * 0/DUC#0:0==>0/Radio#0:0
|   |   * 0/Radio#0:0==>0/DDC#0:0
|   |   * 0/DDC#0:0==>0/SEP#0:0
|   |   * 0/Radio#0:1==>0/DDC#0:1
|   |   * 0/DDC#0:1==>0/SEP#1:0
|   |   * 0/SEP#2:0==>0/DUC#1:0
|   |   * 0/DUC#1:0==>0/Radio#1:0
|   |   * 0/Radio#1:0==>0/DDC#1:0
|   |   * 0/DDC#1:0==>0/SEP#2:0
|   |   * 0/Radio#1:1==>0/DDC#1:1
|   |   * 0/DDC#1:1==>0/SEP#3:0
|   |   * 0/SEP#4:0==>0/Replay#0:0
|   |   * 0/Replay#0:0==>0/SEP#4:0
|   |   * 0/SEP#5:0==>0/Replay#0:1
|   |   * 0/Replay#0:1==>0/SEP#5:0
|     _____________________________________________________
|    /
|   |       TX Dboard: 0/Radio#0
|   |   ID: UBX-160 v1 (0x0079)
|   |   Serial: 30F00A6
|   |   Revision: 3
|   |     _____________________________________________________
|   |    /
|   |   |       TX Frontend: 0
|   |   |   Name: UBX TX
|   |   |   Antennas: TX/RX, CAL
|   |   |   Sensors: lo_locked
|   |   |   Freq range: 10.000 to 6000.000 MHz
|   |   |   Gain range PGA0: 0.0 to 31.5 step 0.5 dB
|   |   |   Bandwidth range: 160000000.0 to 160000000.0 step 0.0 Hz
|   |   |   Connection Type: QI
|   |   |   Uses LO offset: No
|     _____________________________________________________
|    /
|   |       RX Dboard: 0/Radio#0
|   |   ID: UBX-160 v1 (0x007a)
|   |   Serial: 30F00A6
|   |   Revision: 3
|   |     _____________________________________________________
|   |    /
|   |   |       RX Frontend: 0
|   |   |   Name: UBX RX
|   |   |   Antennas: TX/RX, RX2, CAL
|   |   |   Sensors: lo_locked
|   |   |   Freq range: 10.000 to 6000.000 MHz
|   |   |   Gain range PGA0: 0.0 to 31.5 step 0.5 dB
|   |   |   Bandwidth range: 160000000.0 to 160000000.0 step 0.0 Hz
|   |   |   Connection Type: IQ
|   |   |   Uses LO offset: No
|     _____________________________________________________
|    /
|   |       TX Dboard: 0/Radio#1
|   |   ID: UBX-160 v1 (0x0079)
|   |   Serial: 30F00AA
|   |   Revision: 3
|   |     _____________________________________________________
|   |    /
|   |   |       TX Frontend: 0
|   |   |   Name: UBX TX
|   |   |   Antennas: TX/RX, CAL
|   |   |   Sensors: lo_locked
|   |   |   Freq range: 10.000 to 6000.000 MHz
|   |   |   Gain range PGA0: 0.0 to 31.5 step 0.5 dB
|   |   |   Bandwidth range: 160000000.0 to 160000000.0 step 0.0 Hz
|   |   |   Connection Type: QI
|   |   |   Uses LO offset: No
|     _____________________________________________________
|    /
|   |       RX Dboard: 0/Radio#1
|   |   ID: UBX-160 v1 (0x007a)
|   |   Serial: 30F00AA
|   |   Revision: 3
|   |     _____________________________________________________
|   |    /
|   |   |       RX Frontend: 0
|   |   |   Name: UBX RX
|   |   |   Antennas: TX/RX, RX2, CAL
|   |   |   Sensors: lo_locked
|   |   |   Freq range: 10.000 to 6000.000 MHz
|   |   |   Gain range PGA0: 0.0 to 31.5 step 0.5 dB
|   |   |   Bandwidth range: 160000000.0 to 160000000.0 step 0.0 Hz
|   |   |   Connection Type: IQ
|   |   |   Uses LO offset: No

[wordimont]

Thanks for reporting. As far as I can tell, this warning can normally be safely ignored. But we need to look into this more to see if the warning can be removed or if the flush mechanism needs to be improved.