Add tests to confirm known deadlock issues

[cijothomas]

Adding test to repro #1745

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 45 lines in your changes missing coverage. Please review.

Project coverage is 79.1%. Comparing base (b6a108e) to head (98e1be2).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
opentelemetry-appender-tracing/src/layer.rs	0.0%	45 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #2199     +/-   ##
=======================================
- Coverage   79.2%   79.1%   -0.2%     
=======================================
  Files        121     121             
  Lines      20964   21009     +45     
=======================================
  Hits       16620   16620             
- Misses      4344    4389     +45     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[lalitb]

wondering if we should bring the message queue back for simple log processor ?

[cijothomas]

that might mitigate the issue temporarily, but we need to solve it via context based suppression I think.

[lalitb]

In context-based suppression, there may still be cases where we want to loop back the internal events at least once. This could potentially result in a deadlock situation.

[cijothomas]

Not sure I follow that. Maybe add a unit test to cover the scenario, so its easy to track.

[lalitb]

I meant - even with the suppression logic in place, we may want the internal errors to loop-back to the pipeline once, hoping that the error is transient and doesn't cause infinite loop. In case of non-transient errors, the subsequent error is suppressed. But this can be something discussed when we look into suppression logic.

Not related to this, but moving to message queue will also fix hang issue - #2071 (comment)

[cijothomas]

we may want the internal errors to loop-back to the pipeline once, hoping that the error is transient and doesn't cause infinite loop

Can you share a scenario where this is needed? I am not sure how often it is feasible for a component to determine if an error is transient or not.

Not related to this, but moving to message queue will also fix hang issue - #2071 (comment)

Yes, likely. But this needs a full redesign. I don't think adding a queue in the middle helps all cases.

[lalitb]

Thanks for the tests. nit comments, with the lint fix - should be good to merge.