Add `--load-info-json` when video is no longer on youtube.

[barkoder]

A video I'm trying to upload using tubeup, is no longer available on youtube.
Using tubeup on said video, gives a [DEBUG] tubeup.TubeUp - ERROR - ERROR: [youtube] TEST_VIDEOID: This video has been removed for violating YouTube's policy ....

But, I have the video correctly muxed. I have the corresponding thumbnail from --write-thumbnail and VIDEOID.info.json file as well.
Currently, there doesn't appear to be a way for tubeup to work without connecting to YT servers, i.e, to use --load-info-json VIDEOID.info.json to read the already available info.json and use that to generate ia compatible metadata template and begin uploading.

Please add an option to have tubeup to read the info.json file without connecting to YT servers, and begin uploading.

Thanks!

[vxbinaca]

We don't support uploading dead videos you held onto.

[vxbinaca]

Further reflection:

• How do we know the video is complete?
• Why did it fail to upload?
• Is your item live on IA?

[barkoder]

How do we know the video is complete?

The .info.json file has all the relevant yt-dlp readable details including the yt-dlp --load-info-json-able duration(&dur=) and yt-dlp --load-info-json-able sizes(&clen=) and literally every other detail to validate the completion of the video.
tubeup may, if need be(as an additional measure), then use ffprobe/ffmpeg to check the VIDEOID's corresponding format_id stubs(.f299.mp4 , .f303.webm etc.). And yes, I store the stubs as is, without merging them into a single file.
So tubeup could, after validating the file, trigger yt-dlp(using --load-info-json) to trigger ffmpeg to merge the stubs into a single file, and ready it for upload to IA.

Why did it fail to upload?

tubeup failed to upload to IA because tubeup is triggering yt-dlp to parse a URL on YT that no longer exists on YT. When tubeup should be triggering yt-dlp to read my local info.json file(--load-info-json) , and use that instead.
Currently, I can't use tubeup on videos that are no longer available on YT.
I didn't start using tubeup until recently. Before I started using tubeup, I've been archiving YT videos, in a tubeup-merge-friendly format(video stubs), checksumming them and storing them, along with the info.json and subtitles(also checksummed).
I wrote a script that checks if the videos I have archived on my computer, are no longer available on YT. I found a bunch of unavailable videos. After doing a public safety check, I decided I would like to upload these now unavailable YT videos to IA. Preferably using tubeup.

Is your item live on IA?

No? Obviously not . At least, not yet. Because I need tubeup to read my local VIDEOID.info.json to generate the super nice ia item metadata template. I could also archive the videos manually of course, but I've come to really appreciate tubeup's ia compatible item metadata generation.

[vxbinaca]

This feature would need to leave a piece of metadata in the IA item to let people know it was called otherwise yeah, someone can code this up and put it in. Leaving open.

[brandongalbraith]

@vxbinaca thanks for leaving this open, I like this idea, but to your point we're going to need some way to indicate that the video was dead at the time of upload (provenance, integrity, authenticity concerns).

[vxbinaca]

@vxbinaca thanks for leaving this open, I like this idea, but to your point we're going to need some way to indicate that the video was dead at the time of upload (provenance, integrity, authenticity concerns).

Yeah this is the dilemma that I raise with the 3 loud people who want 'store by default, check if dead link and only upload if dead link'

Okay so silent drive corruption, intentional editing, false triggers for if YouTube is down and it uploads everything but then the user complains to us is it's slow or IA bans them.

I can come up with a ton of reasons why upload from drive later is a bad idea.

I'll leave open but in the meantime @barkoder just manually recreate that one item based off Metadata on disk. Do not hide videoIDs in the future. Ever.

[vxbinaca]

So tubeup could, after validating the file, trigger yt-dlp(using --load-info-json) to trigger ffmpeg to merge the stubs into a single file, and ready it for upload to IA.

Is your FFMPEG up to date? Why is it failing to mux?

[barkoder]

Is your FFMPEG up to date? Why is it failing to mux?

It's not an ffmpeg issue. I can mux it manually just fine.

The problem is that tubeup tries to use yt-dlp to try to download new info.json from YT servers(which no longer exists), and yt-dlp doesn't try to invoke ffmpeg to merge the already existing stub files, because it doesn't even acknowledge their existence unless --load-info-json VIDEO.info.json is specified.

Do not hide videoIDs in the future. Ever.

I understand that you and other devs need the videoID for testing but I'm sorry, I can't do that. I want my github nym decoupled from my ia nym.
If I post here which videoid it is in public, then all one has to do is to go to archive dot org slash details slash youtube-<videoid_that_I_mention_here_on_github> and figure out my ia nym.
I don't want that.

But, here's the rest of the log, if that helps.

[DEBUG] 2025-05-26 19:23:35,672 - tubeup.TubeUp - DEBUG - [debug] yt-dlp version nightly@2025.05.22.232956 from yt-dlp/yt-dlp-nightly-builds [e0d6c0822] (pip) API
[DEBUG] 2025-05-26 19:23:35,672 - tubeup.TubeUp - DEBUG - [debug] params: {'outtmpl': '~\\REDACTED-%(id)s.%(ext)s', 'restrictfilenames': True, 'quiet': False, 'verbose': True, 'progress_with_newline': True, 'forcetitle': True, 'continuedl': True, 'retries': 9001, 'fragment_retries': 9001, 'forcejson': False, 'writeinfojson': True, 'writedescription': True, 'writethumbnail': True, 'writeannotations': True, 'writesubtitles': True, 'allsubtitles': True, 'ignoreerrors': True, 'fixup': 'detect_or_warn', 'nooverwrites': True, 'consoletitle': True, 'prefer_ffmpeg': True, 'call_home': False, 'logger': <Logger tubeup.TubeUp (DEBUG)>, 'progress_hooks': [<function TubeUp.get_resource_basenames.<locals>.ydl_progress_hook at 0x0000000004B1B380>], 'compat_opts': set(), 'http_headers': {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.15 Safari/537.36', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Accept-Language': 'en-us,en;q=0.5', 'Sec-Fetch-Mode': 'navigate'}}
[DEBUG] 2025-05-26 19:23:36,202 - tubeup.TubeUp - DEBUG - [debug] exe versions: ffmpeg 7.0-full_build-www.gyan.dev (setts), ffprobe 7.0-full_build-www.gyan.dev, phantomjs 2.1.1
[DEBUG] 2025-05-26 19:23:36,202 - tubeup.TubeUp - DEBUG - [debug] Optional libraries: Cryptodome-3.23.0, brotli-1.1.0, certifi-2025.04.26, curl_cffi-0.11.1, mutagen-1.47.0, requests-2.32.3, secretstorage-3.3.3, sqlite3-3.49.1, urllib3-2.4.0, websockets-15.0.1
[DEBUG] 2025-05-26 19:23:36,202 - tubeup.TubeUp - DEBUG - [debug] Request Handlers: urllib, requests, websockets, curl_cffi
[DEBUG] 2025-05-26 19:23:36,202 - tubeup.TubeUp - DEBUG - [debug] Plugin directories: none
[DEBUG] 2025-05-26 19:23:36,327 - tubeup.TubeUp - DEBUG - [debug] Loaded 1859 extractors
[DEBUG] 2025-05-26 19:23:37,107 - tubeup.TubeUp - DEBUG - [debug] [youtube] [pot] PO Token Providers: none
[DEBUG] 2025-05-26 19:23:37,107 - tubeup.TubeUp - DEBUG - [debug] [youtube] [pot] PO Token Cache Providers: memory
[DEBUG] 2025-05-26 19:23:37,107 - tubeup.TubeUp - DEBUG - [debug] [youtube] [pot] PO Token Cache Spec Providers: webpo
[DEBUG] 2025-05-26 19:23:37,107 - tubeup.TubeUp - DEBUG - [youtube] Extracting URL: VIDEOID
[DEBUG] 2025-05-26 19:23:37,107 - tubeup.TubeUp - DEBUG - [youtube] VIDEOID: Downloading webpage
[DEBUG] 2025-05-26 19:23:40,539 - tubeup.TubeUp - DEBUG - [youtube] VIDEOID: Downloading tv client config
[DEBUG] 2025-05-26 19:23:41,943 - tubeup.TubeUp - DEBUG - [youtube] VIDEOID: Downloading tv player API JSON
[DEBUG] 2025-05-26 19:23:42,474 - tubeup.TubeUp - DEBUG - [youtube] VIDEOID: Downloading ios player API JSON
[DEBUG] 2025-05-26 19:23:42,973 - tubeup.TubeUp - DEBUG - [youtube] VIDEOID: Downloading web player API JSON
[DEBUG] 2025-05-26 19:23:43,456 - tubeup.TubeUp - ERROR - ERROR: [youtube] VIDEOID: This video has been removed for violating YouTube's policy on nudity or sexual content
[DEBUG] 2025-05-26 19:23:43,456 - tubeup.TubeUp - ERROR -   File "Python313\Lib\site-packages\yt_dlp\extractor\common.py", line 748, in extract
    ie_result = self._real_extract(url)
  File "Python313\Lib\site-packages\yt_dlp\extractor\youtube\_video.py", line 3821, in _real_extract
    self.raise_no_formats(reason, expected=True)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "Python313\Lib\site-packages\yt_dlp\extractor\common.py", line 1269, in raise_no_formats
    raise ExtractorError(msg, expected=expected, video_id=video_id)

Traceback (most recent call last):
  File "Python313\Lib\site-packages\tubeup\__main__.py", line 97, in main
    for identifier, meta in tu.archive_urls(URLs, metadata,
                            ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
                                            cookie_file, proxy_url,
                                            ^^^^^^^^^^^^^^^^^^^^^^^
                                            username, password,
                                            ^^^^^^^^^^^^^^^^^^^
                                            use_download_archive,
                                            ^^^^^^^^^^^^^^^^^^^^^
                                            ignore_existing_item):
                                            ^^^^^^^^^^^^^^^^^^^^^
  File "Python313\Lib\site-packages\tubeup\TubeUp.py", line 412, in archive_urls
    downloaded_file_basenames = self.get_resource_basenames(
        urls, cookie_file, proxy, ydl_username, ydl_password, use_download_archive,
        ignore_existing_item)
  File "Python313\Lib\site-packages\tubeup\TubeUp.py", line 183, in get_resource_basenames
    if info_dict.get('_type', 'video') == 'playlist':
       ^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'get'

An exception just occured, if you found this exception isn't related with any of your connection problem, please report this issue to https://github.com/bibanon/tubeup/issues

[barkoder]

I don't have a Google/YT account anymore, but I think this should help to reproduce this issue on your end.

1. Upload a short 1080p video to YT with videoID being test_videoID

2. Download it using yt-dlp --no-config test_videoID -k -o "TEST-%(id)s.%(ext)s" <+ the CLI equivalents of tubeup's yt-dlp API options. Currently it's the following.>

   'restrictfilenames': True,
'quiet': not self.verbose,
'verbose': self.verbose,
'progress_with_newline': True,
'forcetitle': True,
'continuedl': True,
'retries': 9001,
'fragment_retries': 9001,
'forcejson': False,
'writeinfojson': True,
'writedescription': True,
'writethumbnail': True,
'writeannotations': True,
'writesubtitles': True,
'allsubtitles': True,
'ignoreerrors': True,
'fixup': 'detect_or_warn',
'nooverwrites': True,
'consoletitle': True,
'prefer_ffmpeg': True,
'call_home': False,
'logger': self.logger,

3. Delete the video on your YT account.

4. Check that your video is deleted using yt-dlp --no-config test_videoID . You'll get an "ERROR: Video has been removed by the uploader".

5. Make sure TO DELETE any muxed copy of the video that was created in Step 2, from your tubeup download directory.

6. Make sure to NOT DELETE the stubs - .f299, .f140, .f248, .f251 etc.

7. Rename ia.ini config to ia DOT ini to stop tubeup from automatically uploading this test video to the internet archive. You may rename it later after testing.

8. tubeup -d -o "downloaddirectory\\TEST-%(id)s.%(ext)s" test_videoID

9. You should get the debug log I've posted above. You'll notice that the stubs remain as they are - unmuxed.

[vxbinaca]

We have safeties that don't upload the separated streams, but why are the streams not muxing to where you're having this problem?

[barkoder]

We have safeties that don't upload the separated streams,

I am aware – Video download incomplete, please re-run or delete video stubs in downloads folder, exiting...
The reason for renaming ia.ini(Step 7 in my checklist to reproduce the issue) is so that when this issue is fixed, the stubs would merge into a single file. And after they merge, tubeup would automatically trigger ia to upload to IA.
Renaming ia.ini is to thwart the muxed test video from uploading to IA.

but why are the streams not muxing to where you're having this problem?

yt-dlp only invokes ffmpeg to merge the stubs, after it first validates the completion of the video.
And yt-dlp can only validate the completion of the video if yt-dlp can get the content-length (clen values) for the stubs from the info.json
And yt-dlp can only get the clen values for the stubs from the info.json, if there is an info.json to begin with(either in memory as infodict, or a locally fed info.json file).
If the video is alive on YT, then a info.json will be acquired by yt-dlp from YT servers.
If the video is no longer on YT(as is the case with my video), then an info.json may only be supplied by --load-info-json file.info.json
This is why the merge fails – yt-dlp will not invoke ffmpeg to merge a video, whose completion yt-dlp can't validate.

---

Did you try to reproduce the issue following my checklist above? Do the stubs merge for you?

[vxbinaca]

So your problem is that ffmpeg / yt-dlp has a issue. Were you watching tubeups output?

[barkoder]

Were you watching tubeups output ?

I've posted the tubeup log above.

your problem is that ffmpeg / yt-dlp has a issue

No, it's not an ffmpeg/yt-dlp issue. ffmpeg works fine. yt-dlp works fine. Even tubeup works fine(but only for videos that are currently live on youtube.)

yt-dlp does merge the stub files into a single file using ffmpeg if --load-info-json VIDEOID.json is specified.

But when tubeup invokes yt-dlp, currently, it always tries to retrieve a fresh copy of the info.json from YT servers. This is fine, but only works for videos that are still not deleted on YT.

tubeup needs to be reworked a little with an additional option that allows specifying a local info.json(say --use-local-info-json <path to VIDEOID.info.json>) that let's a user pass a local info.json that makes tubeup invoke yt-dlp to read said info.json and trigger ffmpeg to merge the stubs. Or, if the stubs are already merged(according to yt-dlp), query it for upload to IA.

The problem is tubeup. Not yt-dlp or ffmpeg. Hopefully I've defined the problem clearly now.