Downloading from ViX.com issue

[GM43299]

Your question

Hello,

I am having problems downloading videos from the ViX.com (Spanish language programs) web site. Essentially, the downloading starts properly but stops with the following error as soon as it hits the first commercial break:

[https @ 000001e11b342d80] Cannot reuse HTTP connection for different host: m110601-fcdn.mp.lura.live:-1 != dcs110-mcdn.mp.lura.live:-1
[https @ 000001e11b340580] Cannot reuse HTTP connection for different host: m110601-fcdn.mp.lura.live:-1 != dcs110-mcdn.mp.lura.live:-1
[aac_adtstoasc @ 000001e11b7bc9c0] Error parsing ADTS frame header!
[mp4 @ 000001e11b808d40] Error applying bitstream filters to an output packet for stream #1: Invalid data found when processing input
[aost#0:1/copy @ 000001e11b7f54c0] Error submitting a packet to the muxer: Invalid data found when processing input
[out#0/mp4 @ 000001e11b7f5e80] Error muxing a packet

The end result is a perfect recording, but only for the first segment of the video, until the first commercial break (as opposed to all of it). I just tested it with the current beta version (v9.1), but the latest stable version did the same.
You can use the following URL as an example to try: https://vix.com/es-es/video/video-4204862 (US- or Mexico-based VPN necessary for access).
I tried may different settings, but nothing seems to work. Can anybody suggest a way to resolve this, or can this be addressed in a future update?
Thank you

Addon & CoApp version and details

version: 9.1.0.39
target: mozilla
channel: beta
lang: fr
coapp: {"found":true,"path":"C:\Program Files\DownloadHelper CoApp\vdhcoapp.exe","version":"2.0.19","new_version":false}
license: {"accepted":true,"email":"","key":"ff269289ee0643ad"}
platform: x86-64 win
UA: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:128.0) Gecko/20100101 Firefox/128.0
{
"valid_license_message_has_been_displayed": true,
"http_media_download_strategy": "inbrowser",
"download_directory": "C:\Téléchargements\dwhelper",
"view_options": {
"all_tabs": false,
"low_quality": true,
"sort_by_status": true,
"sort_reverse": false,
"show_button_clean": true,
"show_button_clean_all": false,
"show_button_convert_local": false,
"hide_downloaded": false
},
"use_legacy_ui": false,
"use_sidebar": false,
"open_count_store": 2,
"successfull_dl": 29
}

[GrampaWildWilly]

I do not work for VDH. I'm just another user.

I visited the URL you gave & for me, VDH just gave an error trying to download that item. It never even began to download it. You must have a magic touch.

So I went to Plan B. For this exercise, I decided to undock the Network Monitor to its own window. I didn't find any of the usual objects for either HLS or DASH master manifests. But I did find this rather suspicious looking json:

You would expect an object of type x-mpegurl to have a name that includes master.m3u8. So this is a bit non-standard. Whatever. This is the hand we're dealt here. Play it.

I double clicked that entry in the Network Monitor. Firefox did open a new tab for the json. It looked like this:

Now that does look like the URL of an HLS master manifest. These things are never guaranteed. The web is full of oddball things. But I can only hope this really is the URL for an HLS master manifest. That means I have to run that URL through ffprobe. I'm not going to post that here because the ffprobe results are duplicated in the 2 ffmpeg logs I'm about to post. I discovered a 1280x720 video in there of duration 31:26. It happened to have Spanish subtitles associated with it. I don't speak Spanish but I decided to see if I could get them. So I ran 2 ffmpeg runs, one to get the video & the other to get the captions. The captions came in no problem, apparently for the full duration of the video. Here are the log for downloading the subtitles & the captions themselves. I had to rename the file from .vtt to .txt to get it to post here.

VIX Test Video vtt Log.txt
VIX Test Video.txt

The video itself was a different matter.

VIX Test Video mp4 Log.txt

Somewhat in keeping with your observation that things went wrong at the first commercial, I think ffmpeg hit a snag there. What I got did play fine, with subtitles.

It seems to be about a guy & his girlfriend going to a dance club. She loves it all. He thinks the music sucks & it's giving him a headache. I can probably identify with the guy.

I am going to speculate about what happened here. Speculate. Wild-ass guess. Not reliable. I am probably completely wrong. But here's what I think is going on. The master manifest points to a first stream manifest. That stream manifest covers about 5 minutes of the clip. When the first stream manifest ends, it chains to a second stream manifest. But somehow, there is an ad inserted into the stream. When the ad finishes, the video continues with a second stream manifest. But it's treated more or less like a whole new video with time stamps starting over again from 0. I think that's what ffmpeg can't handle. Of course, starting the second segment of the video back at time stamp 0 is totally ridiculous. But that's what I think is going on. Anybody who actually knows something would probably have a proper explanation. But until such a person shows up here to post, this is the best I can do.

Here's another wild-ass guess. Maybe it's the ad itself that starts with a time stamp of 0 & that's confusing ffmpeg.

I tried to play the show on the web site with the subtitles activated. I managed to turn them on for the first segment of the show. But they refused to remain on after the first ad & I couldn't make them turn back on. They exist. I downloaded them. You can see that in the file I posted above. But the web site refuses to use them. It's very strange. Maybe because the different segments revert to time index 0 while the subtitles continue with what ought to be the proper time indexes, that's screwing up the display of the captions. Maybe. Remember, I'm guessing here.

This doesn't get us anywhere. Neither VDH nor ffmpeg can download this. I have no solution to work around it. Maybe there's parameters for ffmpeg that tell it to not validate the time stamps & just take what streams in. I don't know what they are. Maybe somebody else can figure that out.

[GrampaWildWilly]

I couldn't leave this alone. I did a web search for how I might ignore the time stamp information in an input stream. I found a discussion that led me to try an idea. But I needed to download the video again (for reasons that are too complicated to explain). This time, ffmpeg made it all the way to a little over 26 minutes of the 31+ minutes of this thing. But the error messages led me to believe that it was encountering something wrong in the audio track. So I downloaded the video & the audio separately. Sure enough, the video downloaded to completion without error. One odd thing is that the ffprobe at the start of the download says the duration is 30:56 but it downloaded only 30:41. However, when I played the thing, which was of course just a silent movie, everything was there, all the closing credits. So something is fishy about the way this is on the web site. On the other hand, the audio did cut short at about 26 minutes with errors complaining about time stamps. According to the web site, the video is, in fact, 30:41. So I don't know what ffprobe was detecting. In addition, I turned on the subtitles during playback on the web site & they stayed on all the way to the end...one time. When I tried to do it again, I couldn't get them to appear after the first ad. This seems very temperamental.

No. I had a brain wave. I disabled uBlock Origin on the page. Then I was able to get the captions to appear all the way to the end. But they're very inconsistent. When I skipped from one point to another in the video, the captions went off, even though the little indicator in the bottom left corner said they were still on. I had to disable then re-enable the subtitles to get them to start appearing again after any skip I did. Also, any ad seems to turn the captions off, & you have to switch them off & back on to restore them after the ad.

The idea I found via web search didn't let ffmpeg download the whole thing properly. The context of the discussion I found was not downloading streams off the internet. When I search for ffmpeg discussions, I practically never find discussions of downloading streams. It's always something about broadcasting streams (the opposite of downloading) or doing some sort of fancy editing on existing videos. Nobody seems to want to just download stuff. So the thing I found was probably irrelevant, especially since it didn't work.

So here's the conclusions I'm drawing. A few hours ago I got one result. Just now I got a different result. Maybe tomorrow I'll get yet another result. This tells me the web site is not reliable. Its contents change from hour to hour, day to day. In addition, there really does seem to be some sort of corruption in the audio track. Maybe corruption isn't the right word. The audio plays in the player on the site after the 26 minute mark. So maybe it's just coded in some non-standard way that ffmpeg doesn't know how to deal with. Plus the way the site handles captions in its player doesn't exactly inspire my confidence.

[Mattmjs]

I tried this video after using a web proxy because of the regional blocking, it was somewhere around 80% before failing.
Not sure if the web proxy caused it to stop, but I needed to try again. This is going to be a familiar tactic.
I also looked in Network Monitor and copied the same m3u8 link. This time without using the web proxy I pasted in a new tab.
The extension detected the media and downloaded to completion. No failure and on the stable extension.

[GrampaWildWilly]

Encouraged by your post, I have tried again a couple of times, both with VDH & directly with ffmpeg. It always fails. VDH does start to download something but it fails fairly quickly. I wish we were getting to see whatever it is that gets sent to @paulrouget Paul when you click the Report button. With ffmpeg, I can reliably download the full video track & I can reliably download the full subtitles track. But the audio track never completes. Sometimes it quits after 5 minutes, sometimes after 26 minutes, sometimes at some other point. But it always fails with that error about time stamps being out of sequence. It looks like the input suddenly starts giving data with negative time indexes. You can also see that the names of the chunks go along with numbers incrementing by 1.ts until the error occurs. Then suddenly the chunk name reverts to 1.ts. You can see this in the ffmpeg log I posted above. It always complains about stream #1 which is the audio stream. Of course, when I download the individual tracks separately, it's stream #0 for the audio as well as the video & subtitles. But the audio track is the only one that is presenting any kind of problem. It's looking to me like you were extremely lucky to manage to get the whole thing, video AND audio, in one go. I think it's pretty clear that the site acts differently from one time to the next. Very frustrating.

[GM43299]

Thank you GrampaWildWilly and Mattmjs for your comments. Using these, in combination with a couple of web searches (to make up for my limited knowledge of streaming), I finally figured out a way to get around VDH's issue downloading from vix.com. Here is how it goes:

1. I navigate to the webpage for the video I want and use VDH to get the link to the media manifest file that is generated (by klicking on the arrow next to the video resolution I want, and then on the Details tab). As far as I can tell, the m3u8 files are generated dynamically (on demand) and are always different even for the same video, because they contain a variable number of ads each time). They also expire after 6 hours, so there is no point in keeping them when done.
2. I downdload the media manifest (prog.m3u8) file using the link provided in the Details.
3. I edit the file (in a text editor), removing all references to the ad segments. Essentially, I seach all occurrences of the "type=ad" labels, and remove them from the beginning of that line until the end of the line just preceding the sequence
   #EXT-X-DISCONTINUITY
   #EXT-X-KEY:METHOD=NONE
   #ANVATO-SEGMENT-INFO: type=master
4. After saving the edited file, I use ffmpeg to download the video with the following command line:
   ffmpeg -protocol_whitelist "file,http,https,tcp,tls" -i prog.m3u8 -c copy video.mp4
   The end result is a clean video without any errors, no commercials, and including the subtitles (which is what I wanted)...
   That is not as simple as clicking on the wanted video with VDH, but at least it works. Maybe the VDH developers could use that info to make VDH work with vix.com directly?

[GrampaWildWilly]

Well whaddaya know? It works, although I got the information directly from the Network Monitor without reference to VDH. But I have a couple of details I'd like to comment on.

First off, nothing useful appeared in the Network Monitor for me until I let the show play for a couple of seconds on the web page. Then I paused it & the necessary manifests were there in the Network Monitor.

The URLs of the manifests differ because we are visiting the web site as different users. The URLs are generated by the web site to relate the URLs to each user's individual visit to the web site. Part of the URL is a session identifier that works only for me. It has nothing to do with varying numbers of ads. The ads are the same for everybody. It's the fact that we are different users that makes the URLs different. That plus, as you pointed out, the URLs expire after a time, so the one that worked for me just now will probably not work for me tomorrow. Or whenever. Whatever the expiration time is.

If you interpret the master manifest, which does appear there as master.m3u8 (the web site was being very cooperative on this point), you can get the URLs for the different variants of different resolutions. Then you can pick the prog.m3u8 that corresponds to the resolution you want to download.

The master manifest also contains a descriptor for the subtitles. As it happens, the subtitles are in a subs.m3u8, in other words, the subtitles have their own stream manifest. In my Network Monitor, I had several entries for subs.m3u8, but they all turned out to have the same URL. I don't know why I would see that duplication but . . . whatever. I downloaded the subtitles in a second invocation of ffmpeg. Then I let VLC find the subtitles automatically because I download the video & the subtitles into the same directory, calling them "Spanish TV Show.mp4" & "Spanish TV Show.vtt".

Editing the stream manifest to remove the segments that refer to the ads is probably very unique to this web site. You probably couldn't do something like that on another web site. However, the fact that we can remove those sections of advertising & suddenly everything downloads just fine points out that there is something fishy about the information for the ads. I don't know what that might be. I would hope @paulrouget Paul could figure that out.

In addition, we are very lucky with this stream manifest. The URLs in the manifest are complete URLs. This is not the case on all web sites. Lots of manifests I've looked at contain only partial URLs. In order to copy a manifest off the web onto your own hard drive & then run ffmpeg against the manifest from there works with this case because the URLs inside the manifest are complete. It is just as likely that the URLs in a stream manifest are only partial, in which case this approach would not work . . . at least, not without some tweaking. You have to figure out what part of each URL is not included in the stream manifest & then you'd have to "repair" every single URL in the manifest. I have seen where you had to both prefix AND suffix something to each URL in the manifest before you could just let ffmpeg download the video using the manifest on your system. It's usually a much more tedious task than the case of this Spanish language TV show.

In the end, I agree with you that VDH ought to be able to handle this. If we can "manually" make ffmpeg download this stuff, VDH ought to be able to do it as well. After all, you can think of VDH as simply a GUI front-end for ffmpeg. The download engine inside VDH is the ffmpeg that comes packaged with the CoApp. So if we can do this manually, VDH ought to be able to do it under program control. In fact, for my test, I used the ffmpeg that comes with the CoApp. So if I can do this, VDH should be able to do the same thing.

[GM43299]

Ok, now here's another one giving me trouble, also from vix.com...
Looking at the following video, https://vix.com/es-es/video/video-4549816 (also limited to access in the US and Mexico, so a VPN is required), VDH now detects a large series of fragments apparently corresponding to video, audio, and (presumably) subtitle streams. As before, I would like to be able to download the full video (including the audio and subtitles). Is there any (reasonably simple) way to do this?

[GrampaWildWilly]

This one is different from the earlier one. When I visited the page, after showing a still image of something while the page was downloading, it went completely black. So I clicked in the middle of the page & the show started playing.

But . . .

You must give up dreaming of downloading this video.

I captured this screenshot in the usual way. I clicked the Print Screen key on my keyboard. What you see is what got pasted as a result of that. The reason the web browser window is black is DRM. When content is protected by DRM, even screenshots don't work. This is a problem we've encountered before.

However, if you don't use the Print Screen key but use the popup context menu of Firefox itself, there is a screenshot option on that menu. Using that, I was able to capture this image:

I do so very much hate DRM. Decades ago, there was a class action lawsuit that decided that people were entitled to record anything that appeared on their TV. This gave rise to the videocassette recorder industry. To my way of thinking, DRM is in contravention of that legal decision. But it's going to take another class action lawsuit to fix this problem.