IPStackWalkthru

Zephyr IP stack has, expectedly, different send and receive paths.

The general processing structure of the stack is that there're separate receive (RX) and transmit (TX) threads. (1 thread each in the default case, or separate RX and TX threads for different traffic classes (TC) with different priorities, if enabled. Also, after TC support was introduced, explicit RX/TX threads were replaced with work queues, each serviced by its own thread anyway.)

However, not all processing happens in the dedicated RX/TX threads (and this fact is actually a alleged source of problems in the stack):

1. On the receive path, majority of processing happens in the dedicated RX thread. However, user callbacks are called from it either. Thus, unlike the main flow of the application, which happens in its own thread (usually, the main thread), receive callbacks happen in the context of IP stack RX thread.
2. On the send path, the majority of initial data processing happens in the context of the calling, user thread. Only when the send data is properly "formatted" as packets and support data structures, the packets are handed over to the TX thread.
3. Some (delayed) processing is also "offloaded" to the system workqueue (which is backed by its own thread).

Thus, at least following threads interact in the IP stack: user thread, RX thread, TX thread, system workqueue thread. This may raise concerns if they all are properly synchronized to avoid concurrency issues.

Before analyzing this question, a short reminder of Zephyr threads and priorities:

1. There're 2 types of threads in Zephyr: cooperative and preemptive. A cooperative thread, once being run, can't be preempted by any other thread (even by higher-priority thread). It must yield control to the scheduler explicitly. On the other hand, a preemptive thread, as expected, can be preempted at any time by a higher-priority thread. It must take special measures to avoid that (use irq spinlock as the low-level measure for any preemption, or higher-level synchronization primitives which it shares with other threads, to synchronize with specific thread(s)).
2. Cooperative threads have priorities <0, preemptive >=0. Thus, any cooperative thread has higher priority than preemptive thread, and will preempt it at once.
3. Zephyr also has concept of "relative cooperative priorities", which are non-negative, and are converted to an absolute priority using K_PRIO_COOP(prio) macro. The order of priorities still hold - the lower number, the higher priority. For example, K_PRIO_COOP(0) is the highest cooperative priority, and K_PRIO_COOP(5) is higher priority than K_PRIO_COOP(10).
4. By default (Kconfig-configurable), system workqueue thread has a priority of -1, i.e. the lowest cooperative priority.
5. Default RX and TX threads (num TC=1) have priority of K_PRIO_COOP(7). For a default case of 16 total levels of cooperative priorities, that's equivalent to absolute priority of -9.
6. Default (Kconfig-configurable) main thread priority is 0.

Thus, we have following situation:

1. The user thread can be preempted by system workqueue or RX/TX threads at any time. Note that as mentioned above, large share of send path processing happens in the user thread.
2. However, system workqueue thread vs RX vs TX threads can't preempt each other, being cooperative. Thus, they're free from concurrency artifacts.
3. However, the fact that some delayed processing happens in system workqueue, which has lower priority than RX/TX threads, may be not ideal. For example, if delayed processing includes clean up for resources, with high RX/TX threads activity, this clean up may be postponed (because lower-priority thread isn't scheduled, if higher-level thread(s) can run).
4. User receive callback happens in the context of high-priority cooperative thread.