Bluetooth: controller: tx buffer overflow error

[hardik-proxy]

Describe the bug
I have multiple central devices connecting to a single peripheral asynchronously. The central device writes to one characteristic (around 40 bytes) and reads from another characteristic (around 170 bytes). After few hours of run, the peripheral device runs into the error. The error occurs when the central device tries to read the characteristic.

bt_ctlr_hci: Tx Buffer Overflow
bt_hci_core: Data buffer overflow (link type 0x01)
bt_conn: Unable to send to driver (err -55)

To Reproduce
I've tried developing standalone peripheral and central app to reproduce the issue. But I am unable to do so. I still run into errors on my production code.

Expected behavior
the central device should be able to read the characteristic without tx buffer overflow error.

Impact
impact on upcoming delivery of the project

Logs and console output
cc @vrahane
we've tried to add values of free buffer count into an array and print the array on the error.
something like this tx_mem_free_arr_acq[i_acq].cnt = mem_free_count_get(mem_conn_tx.free); in functions void *ll_tx_mem_acquire(void) and void ll_tx_mem_release(void *tx)

the result is as below:
arr_acp is count in mem_acquire and arr_free is count in mem_release

Environment (please complete the following information):

• OS: MacOS
• Toolchain: arm-none-eabi-gcc (GNU Tools for Arm Embedded Processors 7-2018-q2-update) 7.3.1 20180622 (release) [ARM/embedded-7-branch revision 261907]
• Version: Zephyr 2.3.0, b8c78e2

Additional context

the above screenshot is with these config options related to tx buffer.

CONFIG_BT_CTLR_TX_BUFFERS=6
CONFIG_BT_CTLR_TX_BUFFER_SIZE=251
CONFIG_BT_L2CAP_TX_BUF_COUNT=6

[cvinayak]

@hardik-proxy Please attached the diff file as to the changes you have done with reference to the printing

[hardik-proxy]

Here it is:

git_diff.txt

[cvinayak]

From the arr_acp values, the value 0 returned by mem_free_count_get should cause the ll_tx_mem_acquire to return NULL, and subsequently lead to bt_ctlr_hci: Tx Buffer Overflow, correct me that is the case?
Why do I see another 0 in arr_acp in your logging, if there was already a tx buffer overflow?
Also, CONFIG_BT_CTLR_TX_BUFFERS=6 hence your arr_acp should show numbers from 6 to 1.

Could you confirm that during the long duration, the centrals connection and disconnect many times? and if so, do they disconnect due to supervision timeouts?

[hardik-proxy]

during testing, what I've observed is,
--> tx buffer overflow occurred
--> again buffer were available (not able to understand why it is the case)
--> again application worked as expected
--> again tx buffer overflow occurred
--> the cycle continues few times and then system hangs. Soft reset is the only option to get out of it.

secondly, I even tried to set CONFIG_BT_CTLR_TX_BUFFERS=12 and the result was almost similar with values in the range of 3 to 0

for the last question, the central disconnects as soon as it reads the characteristic value. And as long as central devices and peripherals are in the range, flow is something like this:
the central device --> connect to peripheral --> write to a characteristic --> read a characteristic --> disconnect.

[cvinayak]

I even tried to set CONFIG_BT_CTLR_TX_BUFFERS=12 and the result was almost similar with values in the range of 3 to 0

This indicates loss of tx buffer count very early in your application. Please check if your application has sufficient thread call stack allocated, use the THREAD_ANALYZER feature to get the runtime usage details. Try to keep a free bytes margin of >128 bytes when subjected to worst case scenarios of possible nested interrupts of priority levels.

I've tried developing standalone peripheral and central app to reproduce the issue. But I am unable to do so. I still run into errors on my production code.

You will need to make a small reproducible sample, so that issue, if any, can be analysed.

Experiments that can be done:

• Use nRF sniffer to compare on air traffic between your application and the simple test application to compare if the usecase behavior is similar, in terms of procedures, values/length, and timings
• Experiment with single device, connection and reconnection
• Experiment with multiple devices one after the other, connection and reconnection
• Experiment with multiple devices simultaneously, connection and reconnection