Upload speed and success rate for larger video files (YMMV)

[denven]

Although the uploader works normally on most occasions when re-transmission is added to tackle large file uploading failures. however, after a whole night's monitoring of the upload logs, I found sometimes it still experiences a high rate of re-transmission of chunks, a few cases of abortion with too many transmission retries

• too many re-transmission happened

upload chunk: 0, length: 3276800, bytes: 0-3276799 
upload chunk: 0, length: 3276800, bytes: 0-3276799 
upload chunk: 0, length: 3276800, bytes: 0-3276799 
upload chunk: 1, length: 3276800, bytes: 3276800-6553599 
upload chunk: 1, length: 3276800, bytes: 3276800-6553599 
upload chunk: 1, length: 3276800, bytes: 3276800-6553599 
upload chunk: 1, length: 3276800, bytes: 3276800-6553599 
upload chunk: 2, length: 3276800, bytes: 6553600-9830399 
upload chunk: 2, length: 3276800, bytes: 6553600-9830399 
upload chunk: 2, length: 3276800, bytes: 6553600-9830399 
upload chunk: 2, length: 3276800, bytes: 6553600-9830399 
upload chunk: 3, length: 614500, bytes: 9830400-10444899 

• failed with max retry times ( I set it 5 as max continuous failures)

upload chunk: 0, length: 3276800, bytes: 0-3276799 
upload chunk: 1, length: 3276800, bytes: 3276800-6553599 
upload chunk: 2, length: 3276800, bytes: 6553600-9830399 
upload chunk: 2, length: 3276800, bytes: 6553600-9830399 
upload chunk: 2, length: 3276800, bytes: 6553600-9830399 
upload chunk: 2, length: 3276800, bytes: 6553600-9830399 
upload chunk: 3, length: 523886, bytes: 9830400-10354285 
upload chunk: 3, length: 523886, bytes: 9830400-10354285 
upload chunk: 3, length: 523886, bytes: 9830400-10354285 
upload chunk: 3, length: 523886, bytes: 9830400-10354285 
upload chunk: 3, length: 523886, bytes: 9830400-10354285 

• slow speed of successful uploads (one file takes more than 1 minute)

Sun Jan 15 10:21:33 PST 2023: /tmp/sd/record/2023Y01M15D07H/34M00S60.mp4
Sun Jan 15 10:28:09 PST 2023: /tmp/sd/record/2023Y01M15D07H/35M00S60.mp4
Sun Jan 15 10:29:53 PST 2023: /tmp/sd/record/2023Y01M15D07H/36M00S60.mp4
Sun Jan 15 10:32:06 PST 2023: /tmp/sd/record/2023Y01M15D07H/37M00S60.mp4
Sun Jan 15 10:36:39 PST 2023: /tmp/sd/record/2023Y01M15D07H/38M00S60.mp4
Sun Jan 15 10:39:30 PST 2023: /tmp/sd/record/2023Y01M15D07H/39M00S60.mp4
Sun Jan 15 11:12:52 PST 2023: /tmp/sd/record/2023Y01M15D07H/40M00S60.mp4
Sun Jan 15 11:13:36 PST 2023: /tmp/sd/record/2023Y01M15D07H/41M00S60.mp4
Sun Jan 15 11:18:15 PST 2023: /tmp/sd/record/2023Y01M15D07H/42M00S60.mp4
Sun Jan 15 11:20:20 PST 2023: /tmp/sd/record/2023Y01M15D07H/43M00S60.mp4

If this situation is intensive and continues for some time, the upload speed will decline dramatically, and if the camera is recording a new video every minute, the video files will never be uploaded in time, and more and more video files will be delayed uploading on camera's sd card.

[denven]

After some investigation, there are many transmission timeouts that happen when uploading file chunks, so it consumes more time to upload a complete file, or fails with max retried times. The following screenshot is an example that has 6 times (60%) re-transmission due to connection timeout(the server does not respond), and it takes about 8 or more minutes to get a 9.93MB file uploaded.

That's not tolerable. I have to say the resumable API is not reliable at all for the time being, at least from where my camera accesses it.

However, sometimes,** the status code 000 timeout returned from the API is not real, it doesn't indicate a failed transmission**. For the below case, the retransmission is not necessary, since it throws 416 fragmentOverlapped error for the retransmission, which means the server has already got the first time's data. But we never know, we have to do the re-transmission in case it's truly failed.

[denven]

After tuning the call of curl request to PUT file data to the OneDrive resumable API endpoint by using:

• set curl timeout headers option --max-time to 15 seconds
• resumable API:
  • set chunk size to 3932160 bytes (3.92MB)
  • do a max of 10 times re-transmission when data chunk sending times out
  • ignore the 416 status code error

the upload speed and success rate have been enhanced significantly over the past 19 hours:

• file upload success rate is 94.95% (15 out of 297 files)
• most files can be uploaded in less than 40 seconds

However, there are lots of re-transmissions which is the main issue of using the resumable API.

• re-transmission rate of chunks is 36.41%, and the average chunks count per file is 2.48

[denven]

I did continuously upload efficiency and success rate improvement:

• add re-transmission for the whole file instead of file chunks, but only retry once
• upload the whole file once more can fix some upload failures caused by token expiry during the first time upload

Test result:

• upload success rate: 💯 %
• upload delay: 1-2 minutes (video files are uploaded to OneDrive within 2 minutes they were created)

I think the test result is awesome!

[denven]

A recent test, the file upload success rate is 99.26%:

• in about 42 hours
• the uploader tried to upload 2175 video files and got 16 failed records(the recorded failures are based on twice uploading the files and eventually having a non-successful API status code.)

And I found another interesting thing: Even sometimes the OneDrive API reports an error code of 416 (fragmentOutOfOrder), but the actual thing is the file is uploaded successfully(the video file can be played back as well). I checked the failed records, and half of the recorded files are already processed by OneDrive successfully. Then, this means the following things:

• false negative API errors result in unnecessary re-transmission and decline the upload speed and efficiency
• few false records of upload failures (we cannot tell before checking the video files on OneDrive manually)
• we cannot avoid this before the resumable API is more reliable, and most of the re-transmissions are required to fix a true API failure from the OneDrive response

[denven]

Some elaborations from the latest upload results for the past 16 days:

1. Failures are few, random, and not often. some days didn't have any failures, some days had multiple. I have to say it really depends on the reliability of the OneDrive API service;
2. An upload success rate of 99.74%(or above 99%) is very good or overall acceptable for non-business application scenario;
3. No abnormal or exceptional camera reboot or overheating, which proves that it will not impact the camera performance cost in the long run;
4. Worry-free of OneDrive space exhaustion with auto-clean enabled. No warnings or errors are reported from my drive each time I checked. The auto-clean job will delete all the record files of the earliest date from the drive each time when it detects space utilization rate surpass the set threshold.

[denven]

Recently I found more upload failures due to resumable API access errors, the failures are much more compared to several months ago since I could see the video files recorded without too much delay, however, there is a delay of 4+ hours before the recent new video is uploaded.

Checked the upload logs, and there are many more re-transmissions, many single video files took more than 1 minute to complete the upload process.

Currently, there seems no good solution to avoid the delay due to these API errors. There is no rule to check and follow, since some days the API access is great, there are rarely failures, while some other days there are many.

For example, on Jan 22, 2024, there are many failures, and there is only 1 failure on Jan 23 and 24, no failure is found on Jan 25.