Enable parallel MediaMTX playback

[mjh1]

We want to try direct playback from MediaMTX to see how much startup time is saved from avoiding studio stream startup.

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 66 lines in your changes missing coverage. Please review.

Project coverage is 32.16198%. Comparing base (33e1bf8) to head (7982356).
Report is 4 commits behind head on master.

Files with missing lines	Patch %	Lines
server/ai_live_video.go	0.00000%	62 Missing ⚠️
server/ai_mediaserver.go	0.00000%	4 Missing ⚠️

Additional details and impacted files

@@                 Coverage Diff                 @@
##              master       #3396         +/-   ##
===================================================
- Coverage   32.19210%   32.16198%   -0.03012%     
===================================================
  Files            147         147                 
  Lines          40687       40722         +35     
===================================================
- Hits           13098       13097          -1     
- Misses         26816       26852         +36     
  Partials         773         773                 

Files with missing lines	Coverage Δ
server/ai_process.go	0.59222% <ø> (ø)
server/auth.go	63.72549% <ø> (ø)
server/ai_mediaserver.go	7.28814% <0.00000%> (-0.03724%)	⬇️
server/ai_live_video.go	0.00000% <0.00000%> (ø)

... and 1 file with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 33e1bf8...7982356. Read the comment docs.

Files with missing lines	Coverage Δ
server/ai_process.go	0.59222% <ø> (ø)
server/auth.go	63.72549% <ø> (ø)
server/ai_mediaserver.go	7.28814% <0.00000%> (-0.03724%)	⬇️
server/ai_live_video.go	0.00000% <0.00000%> (ø)

... and 1 file with indirect coverage changes

[leszko]

I think it looks good. Two things I'd check before merging and deploying to prod:

1. How much CPU/MEM usage of mediamtx pod increases with each viewer?

We have the following mediamtx kubernetes resources set, maybe we'll need to increase it.

resources:
      limits:
        memory: 4Gi
      requests:
        memory: 4Gi

2. If one ffmpeg RTMP command fails, does it still work? E.g. if you can't push it into studio, will the push to MediaMTX work? I think it should, but I'd double check because these multiwriter and all these io.writer in golang behaves weirdly sometimes.

Thanks Max 🙏

[emranemran]

Could we add a graph to our AI dashboard in grafana to track this?

[mjh1]

I'll raise a ticket for that 👍

[emranemran]

If the thread that pushes segments into the pipe:0 crashes, does this ffmpeg process die? what does recovery look like then?

[mjh1]

This was from the previous logic, and yep that's right, if the thread pushing segments stops then stopPipeline would have been called here and so everything shuts down then the input stream can reconnect and try again.

[emranemran]

What is this 5s sleep for?

[mjh1]

@leszko Very good point about the multiwriter, it does indeed fail if there are issues with either ffmpeg output, I've changed it now to a custom implementation which only returns an error if all writes fail.

I'll check out the cpu usage in staging 👍

[j0sh]

This breaks MediaMTX playback during local testing with the default setup unless a separate rtmp output URL is specified, because there are two output streams of the same name going into MediaMTX ... @mjh1 can you fix this?

[j0sh]

Also I believe we should be careful with the multi writer here because one blocked write can still stall the others; ideally they would be asynchronous (which also implies doing some buffer management). Not sure if this PR is meant to be a temp fix during the latency investigations though (in which case we can probably live with this blocking risk as long as things are reverted later on)