PSVAMB-69845: [Dreamforce 2024][ESTT] Buffering and Errors on Live Streams During Extended Streaming-Time Test#215
Open
PSVAMB-69845: [Dreamforce 2024][ESTT] Buffering and Errors on Live Streams During Extended Streaming-Time Test#215
Conversation
in case media_info is not 'bit exact': - flush media pipeline - ack last received frame id so that downstream not attempt to resend the previous media. - send eos to packager - exit
- workaround downstream sending multiple media info after transcoder reconnection
ghost
requested review from
kalturaguy
reviewed
Oct 7, 2024
| @@ -1238,6 +1293,8 @@ ngx_kmp_out_track_write_media_info(ngx_kmp_out_track_t *track) | |||
| return NGX_ERROR; | |||
| } | |||
|
|
|||
Contributor
There was a problem hiding this comment.
Don't we want to log before the other calls in the function in case one fails?
|
|
||
| nelts = cmcf->events[NGX_RTMP_MSG_AUDIO].nelts; | ||
| for ( ; nelts > 1; nelts--, h--) { | ||
| *h = h[-1]; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
issue:
A transcoder as a micro-service does not handle codec changes.
therefore, transcoding fails once codec change is detected (KMP_MEDIA_INFO_PACKET is received).
This cause stream to stuck since downstream KMP node would not get last ack until upstream KMP node sends one, and segmenter won't send one until specific segment is ready to be persisted.
In the meanwhile, transcoder gets re-created and old media info segment is re-played over and over again, causing waste of system resources, keeping transcoder fail over and over again.
solution
The quick and dirty fix is for transcoder recognize the need to skip over last media info segment and send an ack downstream without waiting for upstream to send an ack - which will resolve the issue immediately.
Once the last segment is confirmed, transcoder starts with new segment after a single restart.
downsides
While this solves the issue we have control no more over remaining segment since we give up on acking the rest of segment in favor of new one (segment means here collection of frames using same media info).
special issues
I had to modify init logic in transcoder because after a restart, both old & new media types are pushed to transcoder, making it impossible to do a codec change. the modification is to wait until first media frame, so that media info be the most updated one. I could not easily solve the issue since media info does not bears an id, so it cannot be ack'ed like frames do. perhaps adding acks to media info would make passing around media info more robust. :(
misc
an rtmp file dump, video only ,used with automated tools, having codec change (480p and 720p) in a loop