ZMQ sampling rate doubled

[RafalArciszewski]

Hello,
I've just started my adventure with srsRAN and I must say its great!
I am trying to connect srsRAN over ZMQ with my own written ZMQ device.
The gNB is configured with 5MHz bandwith and sampling rate 7.68MHz. I am expecting 7680 samples every 1ms but I receive two times more.
For sure I am missing something here. I assume that default IQ format for ZMQ is Float32? First 4 bytes for Q and next 4bytes for I giving in total 8 bytes per sample?

here is the eNB config:

amf:
  addr: 172.45.0.12                                               # The address or hostname of the AMF.
  bind_addr: 10.35.1.40                                            # A local IP that the gNB binds to for traffic from the AMF.

cell_cfg:
  channel_bandwidth_MHz: 5                                       # Bandwith in MHz. Number of PRBs will be automatically derived.
  plmn: "26024"                                                   # PLMN broadcasted by the gNB.
  tac: 100                                                        # Tracking area code (needs to match the core configuration).
  common_scs: 15                                                  # Subcarrier spacing in kHz used for data.

ru_sdr:
  srate:  7.68                                                 # RF sample rate might need to be adjusted according to selected bandwidth.
  device_driver: zmq
  device_args:  verbose,tx_port=tcp://127.0.0.2:5005,rx_port=tcp://127.0.0.3:6006
  otw_format: default                                             # Float32
  

log:
  filename: /home/ubuntu/srsRAN_Project/build/apps/gnb/gnb.log    # Path of the log file.
  all_level: debug                                                 # Logging level applied to all layers.

Log from gNB:

~/srsRAN_Project/build/apps/gnb$ cat gnb.log | grep zmq:tx | grep 'Requested to transmit' | tail
2024-02-13T13:36:40.845236 [zmq:tx:0:0] [D] Requested to transmit 3840 samples.
2024-02-13T13:36:40.845366 [zmq:tx:0:0] [D] Requested to transmit 3840 samples.
2024-02-13T13:36:40.845695 [zmq:tx:0:0] [D] Requested to transmit 3840 samples.
2024-02-13T13:36:40.845813 [zmq:tx:0:0] [D] Requested to transmit 3840 samples.
2024-02-13T13:36:40.846096 [zmq:tx:0:0] [D] Requested to transmit 3840 samples.
2024-02-13T13:36:40.846264 [zmq:tx:0:0] [D] Requested to transmit 3840 samples.
2024-02-13T13:36:40.846559 [zmq:tx:0:0] [D] Requested to transmit 3840 samples.
2024-02-13T13:36:40.846668 [zmq:tx:0:0] [D] Requested to transmit 3840 samples.


~/srsRAN_Project/build/apps/gnb$ cat gnb.log | grep zmq:rx | grep 'Requested to receive' | tail
2024-02-13T13:36:40.844976 [zmq:rx:0:0] [D] Requested to receive 3840 samples.
2024-02-13T13:36:40.845258 [zmq:rx:0:0] [D] Requested to receive 3840 samples.
2024-02-13T13:36:40.845393 [zmq:rx:0:0] [D] Requested to receive 3840 samples.
2024-02-13T13:36:40.845717 [zmq:rx:0:0] [D] Requested to receive 3840 samples.
2024-02-13T13:36:40.845859 [zmq:rx:0:0] [D] Requested to receive 3840 samples.
2024-02-13T13:36:40.846120 [zmq:rx:0:0] [D] Requested to receive 3840 samples.
2024-02-13T13:36:40.846289 [zmq:rx:0:0] [D] Requested to receive 3840 samples.
2024-02-13T13:36:40.846580 [zmq:rx:0:0] [D] Requested to receive 3840 samples.

Log from my ZMQdevice:
MyZMQreceiver.log

Timedelta: 0.4ms Frame 3941: bytes received: 61440, sample received: 7680.0 sample count 30485960.0
Timedelta: 0.4ms Frame 3942: bytes received: 61440, sample received: 7680.0 sample count 30493640.0
Timedelta: 0.4ms Frame 3943: bytes received: 61440, sample received: 7680.0 sample count 30501320.0
Timedelta: 0.4ms Frame 3944: bytes received: 61440, sample received: 7680.0 sample count 30509000.0
Timedelta: 0.4ms Frame 3945: bytes received: 61440, sample received: 7680.0 sample count 30516680.0
Timedelta: 0.4ms Frame 3946: bytes received: 61440, sample received: 7680.0 sample count 30524360.0
Timedelta: 0.4ms Frame 3947: bytes received: 61440, sample received: 7680.0 sample count 30532040.0
Timedelta: 0.4ms Frame 3948: bytes received: 61440, sample received: 7680.0 sample count 30539720.0
Timedelta: 0.4ms Frame 3949: bytes received: 61440, sample received: 7680.0 sample count 30547400.0
Timedelta: 0.4ms Frame 3950: bytes received: 61440, sample received: 7680.0 sample count 30555080.0
Timedelta: 0.5ms Frame 3951: bytes received: 61440, sample received: 7680.0 sample count 30562760.0

The throuput on the wire I have ~1Gbps when I expect ~491Mbps.
Regards,
Rafał

[pgawlowicz]

Yes, we use cf_t type, that is defined as using cf_t = std::complex<float>;, so two floats.

You might use the GNU-radio flow-graph provided in our Multi-UE tutorial for testing. Just modify it to have 1 gNB and 1 UE.

[RafalArciszewski]

Thanks, I will look at this GNU-radio flow.

Any ideas why gNB is sending 15360 (4x3840) samples in one millisecond, instead of 7680? It looks like in the enb.log.

[pgawlowicz]

@xavierarteaga could you help?

[xavierarteaga]

Hi Rafal,
the ZMQ operation is designed for running at the maximum speed the slowest device in the network can handle. The receiver entities throttle the transmitters through the request/reply pattern.

When you have low bandwidth and use sockets or IPC in the same machine it runs faster than the ideal sampling rate. You could potentially use the following throttling option to limit the sampling rate:

$ gnb -c yourconfig.yml ru_sdr expert_cfg --low_phy_dl_throttling=0.99

However, I suggest your application throttles the transmitter by stopping the ZMQ requests.

[RafalArciszewski]

Hi Xavier,
Thank you for your answer. I did some test changing zmqRes periodicity, however when I slow down ZMQ requests the problem still exists.
When I send zmqReq every 0.5ms I receive on avarage 61440bytes zmqResponse (7680samples)
When I send zmqReq every 1ms I receive on average 122880bytes zmqResponse (15360samples)
When I send zmqReq every 2ms I receive on average 245760bytes zmqResponse (30720samples).
So it is two times more samples in every scenario.

----------------------
Timedelta: 0.7ms Frame 3695: bytes received: 92160, samples received: 11520.0
Timedelta: 0.5ms Frame 3696: bytes received: 92160, samples received: 11520.0
Timedelta: 0.5ms Frame 3697: bytes received: 61440, samples received: 7680.0
Timedelta: 0.6ms Frame 3698: bytes received: 61440, samples received: 7680.0
Timedelta: 0.6ms Frame 3699: bytes received: 61440, samples received: 7680.0
Timedelta: 0.6ms Frame 3700: bytes received: 92160, samples received: 11520.0
Timedelta: 0.6ms Frame 3701: bytes received: 92160, samples received: 11520.0
Timedelta: 0.5ms Frame 3702: bytes received: 61440, samples received: 7680.0
Timedelta: 0.6ms Frame 3703: bytes received: 61440, samples received: 7680.0
Timedelta: 0.6ms Frame 3704: bytes received: 122880, samples received: 15360.0
Timedelta: 0.5ms Frame 3705: bytes received: 61440, samples received: 7680.0
Timedelta: 0.5ms Frame 3706: bytes received: 61440, samples received: 7680.0
-------------------------
Timedelta: 1.0ms Frame 1814: bytes received: 122880, samples received: 15360.0
Timedelta: 1.1ms Frame 1815: bytes received: 122880, samples received: 15360.0
Timedelta: 1.1ms Frame 1816: bytes received: 153600, samples received: 19200.0
Timedelta: 1.1ms Frame 1817: bytes received: 153600, samples received: 19200.0
Timedelta: 1.0ms Frame 1818: bytes received: 122880, samples received: 15360.0
Timedelta: 1.1ms Frame 1819: bytes received: 122880, samples received: 15360.0
Timedelta: 1.1ms Frame 1820: bytes received: 153600, samples received: 19200.0
Timedelta: 1.1ms Frame 1821: bytes received: 153600, samples received: 19200.0
Timedelta: 1.0ms Frame 1822: bytes received: 122880, samples received: 15360.0
Timedelta: 1.1ms Frame 1823: bytes received: 122880, samples received: 15360.0
Timedelta: 1.0ms Frame 1824: bytes received: 122880, samples received: 15360.0
---------------------------
Timedelta: 2.0ms Frame 1231: bytes received: 245760, samples received: 30720.0
Timedelta: 2.1ms Frame 1232: bytes received: 215040, samples received: 26880.0
Timedelta: 2.2ms Frame 1233: bytes received: 215040, samples received: 26880.0
Timedelta: 2.2ms Frame 1234: bytes received: 245760, samples received: 30720.0
Timedelta: 2.1ms Frame 1235: bytes received: 245760, samples received: 30720.0
Timedelta: 2.2ms Frame 1236: bytes received: 276480, samples received: 34560.0
Timedelta: 2.1ms Frame 1237: bytes received: 215040, samples received: 26880.0
Timedelta: 2.1ms Frame 1238: bytes received: 245760, samples received: 30720.0
Timedelta: 2.1ms Frame 1239: bytes received: 215040, samples received: 26880.0
Timedelta: 2.0ms Frame 1240: bytes received: 245760, samples received: 30720.0

Below is the gnb.log when I have send zmqRes every 2ms.

2024-02-14T10:55:47.900199 [zmq:tx:0:0] [D] Requested to transmit 3840 samples.
2024-02-14T10:55:47.900391 [zmq:tx:0:0] [D] Requested to transmit 3840 samples.
2024-02-14T10:55:47.900728 [zmq:tx:0:0] [D] Requested to transmit 3840 samples.
2024-02-14T10:55:47.900927 [zmq:tx:0:0] [D] Requested to transmit 3840 samples.
2024-02-14T10:55:47.901256 [zmq:tx:0:0] [D] Requested to transmit 3840 samples.
2024-02-14T10:55:47.901514 [zmq:tx:0:0] [D] Requested to transmit 3840 samples.
2024-02-14T10:55:47.901871 [zmq:tx:0:0] [D] Requested to transmit 3840 samples.
2024-02-14T10:55:47.902069 [zmq:tx:0:0] [D] Requested to transmit 3840 samples.
2024-02-14T10:55:47.902401 [zmq:tx:0:0] [D] Requested to transmit 3840 samples.
2024-02-14T10:55:47.902736 [zmq:tx:0:0] [D] Requested to transmit 3840 samples.
2024-02-14T10:55:47.902939 [zmq:tx:0:0] [D] Requested to transmit 3840 samples.

Correct me if I am wrong but this log indicates how much samples lower phy is writing into "zmq driver" circular buffer, right? If yes, then is suggests that lower phy is sending two times more samples to "zmq driver" than it should at 7.68Mhz sampling rate.

In my opinion, whatever happens on the transmission layer, the lower phy should write and read the samples to/from the circular buffers according to the sampling rate speed .

Regards,
Rafał

[RafalArciszewski]

Ok. I think I figured it out.
Previously I tried to slow down zmqREQests from my device (zmq:tx stream in gNB), but I did not changed the zmqRESponses from my device (zmq:rx stream in gNB).
When I adjusted zmqRESponses from my device to throttle for given sampling rate, suddenly also the zmq:tx stream slow down.

Probably gNB is not so asynchrounues between RX and TX stream, I will look more in the code.

Thanks for the answers.

PS. The --low_phy_dl_throttling=0.99 also did the trick and lower phy did slow down.

[xavierarteaga]

Great, many thanks for letting us know.

You are welcome.

Xavier

[RafalArciszewski]

One more comment,
I noticed also in gnb.log that depending on ZMQ speed the 'FAPI Slot.indication' also slow down or speed up. I have seen Slots triggered on FAPI for example per 10ms. There is no "internal clock supervisor".
Now I am looking at ru_ofh O-RAN split7.2x and 'FAPI Slot.indications' are correctly logged with high precision at every 1ms, regardless what O-RU is sending.

Nevertheless thanks again.