Tubeup "Upload Only Mode" Capability and Challenges

[barkoder]

A video I'm trying to upload using tubeup, is no longer available on youtube.
Using tubeup on said video, gives a [DEBUG] tubeup.TubeUp - ERROR - ERROR: [youtube] TEST_VIDEOID: This video has been removed for violating YouTube's policy ....

But, I have the video correctly muxed. I have the corresponding thumbnail from --write-thumbnail and VIDEOID.info.json file as well.
Currently, there doesn't appear to be a way for tubeup to work without connecting to YT servers, i.e, to use --load-info-json VIDEOID.info.json to read the already available info.json and use that to generate ia compatible metadata template and begin uploading.

Please add an option to have tubeup to read the info.json file without connecting to YT servers, and begin uploading.

Thanks!

[vxbinaca]

We don't support uploading dead videos you held onto.

[vxbinaca]

Further reflection:

• How do we know the video is complete?
• Why did it fail to upload?
• Is your item live on IA?

[barkoder]

How do we know the video is complete?

The .info.json file has all the relevant yt-dlp readable details including the yt-dlp --load-info-json-able duration(&dur=) and yt-dlp --load-info-json-able sizes(&clen=) and literally every other detail to validate the completion of the video.
tubeup may, if need be(as an additional measure), then use ffprobe/ffmpeg to check the VIDEOID's corresponding format_id stubs(.f299.mp4 , .f303.webm etc.). And yes, I store the stubs as is, without merging them into a single file.
So tubeup could, after validating the file, trigger yt-dlp(using --load-info-json) to trigger ffmpeg to merge the stubs into a single file, and ready it for upload to IA.

Why did it fail to upload?

tubeup failed to upload to IA because tubeup is triggering yt-dlp to parse a URL on YT that no longer exists on YT. When tubeup should be triggering yt-dlp to read my local info.json file(--load-info-json) , and use that instead.
Currently, I can't use tubeup on videos that are no longer available on YT.
I didn't start using tubeup until recently. Before I started using tubeup, I've been archiving YT videos, in a tubeup-merge-friendly format(video stubs), checksumming them and storing them, along with the info.json and subtitles(also checksummed).
I wrote a script that checks if the videos I have archived on my computer, are no longer available on YT. I found a bunch of unavailable videos. After doing a public safety check, I decided I would like to upload these now unavailable YT videos to IA. Preferably using tubeup.

Is your item live on IA?

No? Obviously not . At least, not yet. Because I need tubeup to read my local VIDEOID.info.json to generate the super nice ia item metadata template. I could also archive the videos manually of course, but I've come to really appreciate tubeup's ia compatible item metadata generation.

[vxbinaca]

This feature would need to leave a piece of metadata in the IA item to let people know it was called otherwise yeah, someone can code this up and put it in. Leaving open.

[brandongalbraith]

@vxbinaca thanks for leaving this open, I like this idea, but to your point we're going to need some way to indicate that the video was dead at the time of upload (provenance, integrity, authenticity concerns).

[vxbinaca]

@vxbinaca thanks for leaving this open, I like this idea, but to your point we're going to need some way to indicate that the video was dead at the time of upload (provenance, integrity, authenticity concerns).

Yeah this is the dilemma that I raise with the 3 loud people who want 'store by default, check if dead link and only upload if dead link'

Okay so silent drive corruption, intentional editing, false triggers for if YouTube is down and it uploads everything but then the user complains to us is it's slow or IA bans them.

I can come up with a ton of reasons why upload from drive later is a bad idea.

I'll leave open but in the meantime @barkoder just manually recreate that one item based off Metadata on disk. Do not hide videoIDs in the future. Ever.

[vxbinaca]

So tubeup could, after validating the file, trigger yt-dlp(using --load-info-json) to trigger ffmpeg to merge the stubs into a single file, and ready it for upload to IA.

Is your FFMPEG up to date? Why is it failing to mux?

[barkoder]

Is your FFMPEG up to date? Why is it failing to mux?

It's not an ffmpeg issue. I can mux it manually just fine.

The problem is that tubeup tries to use yt-dlp to try to download new info.json from YT servers(which no longer exists), and yt-dlp doesn't try to invoke ffmpeg to merge the already existing stub files, because it doesn't even acknowledge their existence unless --load-info-json VIDEO.info.json is specified.

Do not hide videoIDs in the future. Ever.

I understand that you and other devs need the videoID for testing but I'm sorry, I can't do that. I want my github nym decoupled from my ia nym.
If I post here which videoid it is in public, then all one has to do is to go to archive dot org slash details slash youtube-<videoid_that_I_mention_here_on_github> and figure out my ia nym.
I don't want that.

But, here's the rest of the log, if that helps.

[DEBUG] 2025-05-26 19:23:35,672 - tubeup.TubeUp - DEBUG - [debug] yt-dlp version nightly@2025.05.22.232956 from yt-dlp/yt-dlp-nightly-builds [e0d6c0822] (pip) API
[DEBUG] 2025-05-26 19:23:35,672 - tubeup.TubeUp - DEBUG - [debug] params: {'outtmpl': '~\\REDACTED-%(id)s.%(ext)s', 'restrictfilenames': True, 'quiet': False, 'verbose': True, 'progress_with_newline': True, 'forcetitle': True, 'continuedl': True, 'retries': 9001, 'fragment_retries': 9001, 'forcejson': False, 'writeinfojson': True, 'writedescription': True, 'writethumbnail': True, 'writeannotations': True, 'writesubtitles': True, 'allsubtitles': True, 'ignoreerrors': True, 'fixup': 'detect_or_warn', 'nooverwrites': True, 'consoletitle': True, 'prefer_ffmpeg': True, 'call_home': False, 'logger': <Logger tubeup.TubeUp (DEBUG)>, 'progress_hooks': [<function TubeUp.get_resource_basenames.<locals>.ydl_progress_hook at 0x0000000004B1B380>], 'compat_opts': set(), 'http_headers': {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.15 Safari/537.36', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Accept-Language': 'en-us,en;q=0.5', 'Sec-Fetch-Mode': 'navigate'}}
[DEBUG] 2025-05-26 19:23:36,202 - tubeup.TubeUp - DEBUG - [debug] exe versions: ffmpeg 7.0-full_build-www.gyan.dev (setts), ffprobe 7.0-full_build-www.gyan.dev, phantomjs 2.1.1
[DEBUG] 2025-05-26 19:23:36,202 - tubeup.TubeUp - DEBUG - [debug] Optional libraries: Cryptodome-3.23.0, brotli-1.1.0, certifi-2025.04.26, curl_cffi-0.11.1, mutagen-1.47.0, requests-2.32.3, secretstorage-3.3.3, sqlite3-3.49.1, urllib3-2.4.0, websockets-15.0.1
[DEBUG] 2025-05-26 19:23:36,202 - tubeup.TubeUp - DEBUG - [debug] Request Handlers: urllib, requests, websockets, curl_cffi
[DEBUG] 2025-05-26 19:23:36,202 - tubeup.TubeUp - DEBUG - [debug] Plugin directories: none
[DEBUG] 2025-05-26 19:23:36,327 - tubeup.TubeUp - DEBUG - [debug] Loaded 1859 extractors
[DEBUG] 2025-05-26 19:23:37,107 - tubeup.TubeUp - DEBUG - [debug] [youtube] [pot] PO Token Providers: none
[DEBUG] 2025-05-26 19:23:37,107 - tubeup.TubeUp - DEBUG - [debug] [youtube] [pot] PO Token Cache Providers: memory
[DEBUG] 2025-05-26 19:23:37,107 - tubeup.TubeUp - DEBUG - [debug] [youtube] [pot] PO Token Cache Spec Providers: webpo
[DEBUG] 2025-05-26 19:23:37,107 - tubeup.TubeUp - DEBUG - [youtube] Extracting URL: VIDEOID
[DEBUG] 2025-05-26 19:23:37,107 - tubeup.TubeUp - DEBUG - [youtube] VIDEOID: Downloading webpage
[DEBUG] 2025-05-26 19:23:40,539 - tubeup.TubeUp - DEBUG - [youtube] VIDEOID: Downloading tv client config
[DEBUG] 2025-05-26 19:23:41,943 - tubeup.TubeUp - DEBUG - [youtube] VIDEOID: Downloading tv player API JSON
[DEBUG] 2025-05-26 19:23:42,474 - tubeup.TubeUp - DEBUG - [youtube] VIDEOID: Downloading ios player API JSON
[DEBUG] 2025-05-26 19:23:42,973 - tubeup.TubeUp - DEBUG - [youtube] VIDEOID: Downloading web player API JSON
[DEBUG] 2025-05-26 19:23:43,456 - tubeup.TubeUp - ERROR - ERROR: [youtube] VIDEOID: This video has been removed for violating YouTube's policy on nudity or sexual content
[DEBUG] 2025-05-26 19:23:43,456 - tubeup.TubeUp - ERROR -   File "Python313\Lib\site-packages\yt_dlp\extractor\common.py", line 748, in extract
    ie_result = self._real_extract(url)
  File "Python313\Lib\site-packages\yt_dlp\extractor\youtube\_video.py", line 3821, in _real_extract
    self.raise_no_formats(reason, expected=True)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "Python313\Lib\site-packages\yt_dlp\extractor\common.py", line 1269, in raise_no_formats
    raise ExtractorError(msg, expected=expected, video_id=video_id)

Traceback (most recent call last):
  File "Python313\Lib\site-packages\tubeup\__main__.py", line 97, in main
    for identifier, meta in tu.archive_urls(URLs, metadata,
                            ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
                                            cookie_file, proxy_url,
                                            ^^^^^^^^^^^^^^^^^^^^^^^
                                            username, password,
                                            ^^^^^^^^^^^^^^^^^^^
                                            use_download_archive,
                                            ^^^^^^^^^^^^^^^^^^^^^
                                            ignore_existing_item):
                                            ^^^^^^^^^^^^^^^^^^^^^
  File "Python313\Lib\site-packages\tubeup\TubeUp.py", line 412, in archive_urls
    downloaded_file_basenames = self.get_resource_basenames(
        urls, cookie_file, proxy, ydl_username, ydl_password, use_download_archive,
        ignore_existing_item)
  File "Python313\Lib\site-packages\tubeup\TubeUp.py", line 183, in get_resource_basenames
    if info_dict.get('_type', 'video') == 'playlist':
       ^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'get'

An exception just occured, if you found this exception isn't related with any of your connection problem, please report this issue to https://github.com/bibanon/tubeup/issues

[barkoder]

I don't have a Google/YT account anymore, but I think this should help to reproduce this issue on your end.

1. Upload a short 1080p video to YT with videoID being test_videoID

2. Download it using yt-dlp --no-config test_videoID -k -o "TEST-%(id)s.%(ext)s" <+ the CLI equivalents of tubeup's yt-dlp API options. Currently it's the following.>

   'restrictfilenames': True,
'quiet': not self.verbose,
'verbose': self.verbose,
'progress_with_newline': True,
'forcetitle': True,
'continuedl': True,
'retries': 9001,
'fragment_retries': 9001,
'forcejson': False,
'writeinfojson': True,
'writedescription': True,
'writethumbnail': True,
'writeannotations': True,
'writesubtitles': True,
'allsubtitles': True,
'ignoreerrors': True,
'fixup': 'detect_or_warn',
'nooverwrites': True,
'consoletitle': True,
'prefer_ffmpeg': True,
'call_home': False,
'logger': self.logger,

3. Delete the video on your YT account.

4. Check that your video is deleted using yt-dlp --no-config test_videoID . You'll get an "ERROR: Video has been removed by the uploader".

5. Make sure TO DELETE any muxed copy of the video that was created in Step 2, from your tubeup download directory.

6. Make sure to NOT DELETE the stubs - .f299, .f140, .f248, .f251 etc.

7. Rename ia.ini config to ia DOT ini to stop tubeup from automatically uploading this test video to the internet archive. You may rename it later after testing.

8. tubeup -d -o "downloaddirectory\\TEST-%(id)s.%(ext)s" test_videoID

9. You should get the debug log I've posted above. You'll notice that the stubs remain as they are - unmuxed.

[vxbinaca]

We have safeties that don't upload the separated streams, but why are the streams not muxing to where you're having this problem?

[barkoder]

We have safeties that don't upload the separated streams,

I am aware – Video download incomplete, please re-run or delete video stubs in downloads folder, exiting...
The reason for renaming ia.ini(Step 7 in my checklist to reproduce the issue) is so that when this issue is fixed, the stubs would merge into a single file. And after they merge, tubeup would automatically trigger ia to upload to IA.
Renaming ia.ini is to thwart the muxed test video from uploading to IA.

but why are the streams not muxing to where you're having this problem?

yt-dlp only invokes ffmpeg to merge the stubs, after it first validates the completion of the video.
And yt-dlp can only validate the completion of the video if yt-dlp can get the content-length (clen values) for the stubs from the info.json
And yt-dlp can only get the clen values for the stubs from the info.json, if there is an info.json to begin with(either in memory as infodict, or a locally fed info.json file).
If the video is alive on YT, then a info.json will be acquired by yt-dlp from YT servers.
If the video is no longer on YT(as is the case with my video), then an info.json may only be supplied by --load-info-json file.info.json
This is why the merge fails – yt-dlp will not invoke ffmpeg to merge a video, whose completion yt-dlp can't validate.

---

Did you try to reproduce the issue following my checklist above? Do the stubs merge for you?

[vxbinaca]

So your problem is that ffmpeg / yt-dlp has a issue. Were you watching tubeups output?

[barkoder]

Were you watching tubeups output ?

I've posted the tubeup log above.

your problem is that ffmpeg / yt-dlp has a issue

No, it's not an ffmpeg/yt-dlp issue. ffmpeg works fine. yt-dlp works fine. Even tubeup works fine(but only for videos that are currently live on youtube.)

yt-dlp does merge the stub files into a single file using ffmpeg if --load-info-json VIDEOID.json is specified.

But when tubeup invokes yt-dlp, currently, it always tries to retrieve a fresh copy of the info.json from YT servers. This is fine, but only works for videos that are still not deleted on YT.

tubeup needs to be reworked a little with an additional option that allows specifying a local info.json(say --use-local-info-json <path to VIDEOID.info.json>) that let's a user pass a local info.json that makes tubeup invoke yt-dlp to read said info.json and trigger ffmpeg to merge the stubs. Or, if the stubs are already merged(according to yt-dlp), query it for upload to IA.

The problem is tubeup. Not yt-dlp or ffmpeg. Hopefully I've defined the problem clearly now.

[vxbinaca]

Right okay the video was removed I get that, but was it removed when you were first ripping it or was this later when you discovered it and tried to upload it?

This still isn't a problem with Tubeup by the way. I'm leaning more towards yt-dlp. I'd be hesitant in the extreme to merge incomplete data streams and just force something to work.

[barkoder]

No. It wasn't removed when I was ripping it. I discovered it was removed later, as I mentioned earlier.

Before I started using tubeup, I've been archiving YT videos, in a tubeup-merge-friendly format(video stubs), checksumming them and storing them, along with the info.json and subtitles(also checksummed).
I wrote a script that checks if the videos I have archived on my computer, are no longer available on YT. I found a bunch of unavailable videos.

I always use yt-dlp with -k to keep the stubs, so I have the full stubs.
My data stream is complete.
I always download the stubs to ramdisk and checksum it before I mirror it to my btrfs drives.

Here's what I can and can't do with tubeup, currently:
I can use tubeup to merge the stubs of videos(that I've been collecting), that haven't been deleted from YT, and upload to IA. I tested this for a couple of videos, and succeeded. But this is not what I want to do.
I can't use tubeup to merge the stubs of videos(that I've been collecting), that have been deleted from YT, to merge and upload to IA. I tested this and I failed, as logged above. But this is what I want to do.

Even when I do merge it manually, - ffmpeg -i VIDEOID.f137.mp4 -i VIDEOID.f140.mp4 -movflags "+faststart" -c copy VIDEOID.mp4 , I still get the same error as the log I've posted above.

Could you please try to reproduce the issue following my checklist above? Please confirm whether or not the stubs merge and upload to IA for you. Thanks.

[vxbinaca]

Are you trying to do live video?

[barkoder]

No, the video is not a live video.

And whenever I do save the stubs of a live YT video(this one is not), I always wait until the yt-dlp -F has an ID with an actual size(no approximate ~ sizes) in it, [protocol=https]

[vxbinaca]

Internet Archive doesn't want people wholesale dumping things onto the site. I've effectively abandoned uploading. The feature you want will hammer the sites resources more and invites altering the historical record easier. Given they're fighting various court cases over copyright, my suggestion is eat the loss the loss isn't great.

For numerous technical and archival reasons we won't be merging PRs for adding features that enable stored uploads. These reasons are numerous and have been covered in other issues.

[brandongalbraith]

To @vxbinaca's point, the Internet Archive does not want folks dumping tons of video content that may not be relevant to their cultural preservation mission into their storage system. With that said, if you believe the collection of videos you have that is no longer available is relevant, my guidance would be to reach out to the Internet Archive via their public email address, discuss the content library you have with them, and ask them if they want it. If they do, you can then work with them to facilitate a transfer. This could be physical media, your Internet Archive account if sanctioned and previously approved, or if you are of means and are approved, a Vault account where you pay per GB for the storage.

If the content is of cultural value, I would encourage reaching out to them. The conversation is free.

[vxbinaca]

I haven't stopped maintaining this tool I just don't want to use it at scale currently due to YouTube aggressively obstructing scraping and IA throttling use of this tool to very slow speeds.

It's been a good run guys but some y'all need to chill with uploads.

My uploads slowly went down since 2017 just due to less targets alive to collect. But we're half way through this year and I've uploaded only 1,500 items in 6 months, down from tens of thousands mostly YouTube items which weigh nothing in comparison. I don't intend to upload much else this year or after.

Still I will ensure the tool functions in case IA changes things again but not only am I uninterested in adding functionality I'd like to slim the code base if possible.

Increasingly the requested new functionality in the tool is to 100 percent sure hammer IA and make sure what people scrape gets up. I don't want the site to be abused.

[barkoder]

Internet Archive doesn't want people wholesale dumping things onto the site.

Internet Archive does not want folks dumping tons of video content.

I am aware. I've read Jason Scott's caution

Weirdly, that post is also exactly why I haven't setup tubeup to auto upload new vids of the channels I follow to IA.
This is why I hoard the videos stubs for myself instead.
Then I curate and review it and do a public safety check.
And only then I decide to mux and upload to IA.

I'm acutely aware of the limitations of this approach. But I really don't want to tax IA any more than I absolutely have to.

I'm not an expert in assessing "cultural significance" , so I don't know if what I have might be "of interest" , but I'll look into it. Thanks Brandon.

[brandongalbraith]

@barkoder I admit "of cultural significance" is subjective and in the eye of the beholder. I also do not want to dissuade your archiving operations. Tubeup, based on your needs for this specific use case, is probably not the best tool for the job and a call for tooling that rips the content, bundles it into an archive (tar?) that is easily seekable both locally and remotely using HTTP primitives, and maintains a local inventory for future distribution (either into the Internet Archive or perhaps a torrent per item). Like I said, ask the Internet Archive. They estimate it costs them $2/GB to store content in perpetuity, so there is a discussion to be had of "is this content worth $2/GB?" And if so, to whom?

[brandongalbraith]

Converted to discussion for improved forum for additional thoughts, ideas, comments on the topic.