WMD is throwing errors instead of downloading the way it used to

[greg-raven]

I'm using macOS with an up-to-date version of Ruby installed via Homebrew.

I'm asking WMD to grab files from an old website:

wayback_machine_downloader http://www.racquetresearch.com -l -t 20021122105251

It seems to start fine, with the message:

Getting snapshot pages....

but after a few seconds throws nothing but errors:

/opt/homebrew/Cellar/ruby/3.4.2/lib/ruby/3.4.0/open-uri.rb:393:in 'OpenURI.open_http': 400 BAD REQUEST (OpenURI::HTTPError)
from /opt/homebrew/Cellar/ruby/3.4.2/lib/ruby/3.4.0/open-uri.rb:825:in 'URI::HTTP#buffer_open'
from /opt/homebrew/Cellar/ruby/3.4.2/lib/ruby/3.4.0/open-uri.rb:233:in 'block in OpenURI.open_loop'
from /opt/homebrew/Cellar/ruby/3.4.2/lib/ruby/3.4.0/open-uri.rb:231:in 'Kernel#catch'
from /opt/homebrew/Cellar/ruby/3.4.2/lib/ruby/3.4.0/open-uri.rb:231:in 'OpenURI.open_loop'
from /opt/homebrew/Cellar/ruby/3.4.2/lib/ruby/3.4.0/open-uri.rb:163:in 'OpenURI.open_uri'
from /opt/homebrew/Cellar/ruby/3.4.2/lib/ruby/3.4.0/open-uri.rb:805:in 'OpenURI::OpenRead#open'
from /opt/homebrew/lib/ruby/gems/3.4.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader/archive_api.rb:13:in 'ArchiveAPI#get_raw_list_from_api'
from /opt/homebrew/lib/ruby/gems/3.4.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader.rb:92:in 'block in WaybackMachineDownloader#get_all_snapshots_to_consider'
from internal:numeric:257:in 'Integer#times'
from /opt/homebrew/lib/ruby/gems/3.4.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader.rb:91:in 'WaybackMachineDownloader#get_all_snapshots_to_consider'
from /opt/homebrew/lib/ruby/gems/3.4.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader.rb:105:in 'WaybackMachineDownloader#get_file_list_curated'
from /opt/homebrew/lib/ruby/gems/3.4.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader.rb:164:in 'WaybackMachineDownloader#get_file_list_by_timestamp'
from /opt/homebrew/lib/ruby/gems/3.4.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader.rb:177:in 'WaybackMachineDownloader#list_files'
from /opt/homebrew/lib/ruby/gems/3.4.0/gems/wayback_machine_downloader-2.3.1/bin/wayback_machine_downloader:70:in '<top (required)>'
from /opt/homebrew/lib/ruby/gems/3.4.0/bin/wayback_machine_downloader:25:in 'Kernel#load'
from /opt/homebrew/lib/ruby/gems/3.4.0/bin/wayback_machine_downloader:25:in '

'

What am I doing incorrectly?

[rma92]

Edit: Manually going to an archive.org CDX url made a timeout error. I suspect there's something wrong on their end.
http://web.archive.org/cdx/search/cdx?url=example.com*&output=txt&from=2010&to=2018. I attempted to follow the manual instructions from this page: https://exposureninja.com/blog/extract-urls-archive-org/

That looks like a reasonable command line.

I have also been observing this today. Tried two different machines on different internet connections. I also had gotten a 504 Gateway Time-out before. I suspect either archive.org changed something or is experiencing poor performance.

wayback_machine_downloader http://storage.stardock.com/files/ -o .exe
Downloading http://storage.stardock.com/files/ to websites/storage.stardock.com/ from Wayback Machine archives.

Getting snapshot pagesC:/Ruby33-x64/lib/ruby/3.3.0/open-uri.rb:376:in `open_http': 504 Gateway Time-out (OpenURI::HTTPError)
        from C:/Ruby33-x64/lib/ruby/3.3.0/open-uri.rb:770:in `buffer_open'
        from C:/Ruby33-x64/lib/ruby/3.3.0/open-uri.rb:220:in `block in open_loop'
        from C:/Ruby33-x64/lib/ruby/3.3.0/open-uri.rb:218:in `catch'
        from C:/Ruby33-x64/lib/ruby/3.3.0/open-uri.rb:218:in `open_loop'
        from C:/Ruby33-x64/lib/ruby/3.3.0/open-uri.rb:158:in `open_uri'
        from C:/Ruby33-x64/lib/ruby/3.3.0/open-uri.rb:750:in `open'
        from D:/Users/user/.local/share/gem/ruby/3.3.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader/archive_api.rb:13:in `get_raw_list_from_api'
        from D:/Users/user/.local/share/gem/ruby/3.3.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader.rb:88:in `get_all_snapshots_to_consider'
        from D:/Users/user/.local/share/gem/ruby/3.3.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader.rb:105:in `get_file_list_curated'
        from D:/Users/user/.local/share/gem/ruby/3.3.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader.rb:164:in `get_file_list_by_timestamp'
        from D:/Users/user/.local/share/gem/ruby/3.3.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader.rb:309:in `file_list_by_timestamp'
        from D:/Users/user/.local/share/gem/ruby/3.3.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader.rb:192:in `download_files'
        from D:/Users/user/.local/share/gem/ruby/3.3.0/gems/wayback_machine_downloader-2.3.1/bin/wayback_machine_downloader:72:in `<top (required)>'
        from D:/Users/user/AppData/Local/Microsoft/WindowsApps/wayback_machine_downloader:32:in `load'
        from D:/Users/user/AppData/Local/Microsoft/WindowsApps/wayback_machine_downloader:32:in `<main>'

wayback_machine_downloader http://storage.stardock.com/files/ -o .exe
Downloading http://storage.stardock.com/files/ to websites/storage.stardock.com/ from Wayback Machine archives.

Getting snapshot pages..C:/Ruby33-x64/lib/ruby/3.3.0/open-uri.rb:376:in `open_http': 400 BAD REQUEST (OpenURI::HTTPError)
        from C:/Ruby33-x64/lib/ruby/3.3.0/open-uri.rb:770:in `buffer_open'
        from C:/Ruby33-x64/lib/ruby/3.3.0/open-uri.rb:220:in `block in open_loop'
        from C:/Ruby33-x64/lib/ruby/3.3.0/open-uri.rb:218:in `catch'
        from C:/Ruby33-x64/lib/ruby/3.3.0/open-uri.rb:218:in `open_loop'
        from C:/Ruby33-x64/lib/ruby/3.3.0/open-uri.rb:158:in `open_uri'
        from C:/Ruby33-x64/lib/ruby/3.3.0/open-uri.rb:750:in `open'
        from D:/Users/user/.local/share/gem/ruby/3.3.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader/archive_api.rb:13:in `get_raw_list_from_api'
        from D:/Users/user/.local/share/gem/ruby/3.3.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader.rb:92:in `block in get_all_snapshots_to_consider'
        from <internal:numeric>:237:in `times'
        from D:/Users/user/.local/share/gem/ruby/3.3.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader.rb:91:in `get_all_snapshots_to_consider'
        from D:/Users/user/.local/share/gem/ruby/3.3.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader.rb:105:in `get_file_list_curated'
        from D:/Users/user/.local/share/gem/ruby/3.3.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader.rb:164:in `get_file_list_by_timestamp'
        from D:/Users/user/.local/share/gem/ruby/3.3.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader.rb:309:in `file_list_by_timestamp'
        from D:/Users/user/.local/share/gem/ruby/3.3.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader.rb:192:in `download_files'
        from D:/Users/user/.local/share/gem/ruby/3.3.0/gems/wayback_machine_downloader-2.3.1/bin/wayback_machine_downloader:72:in `<top (required)>'
        from D:/Users/user/AppData/Local/Microsoft/WindowsApps/wayback_machine_downloader:32:in `load'
        from D:/Users/user/AppData/Local/Microsoft/WindowsApps/wayback_machine_downloader:32:in `<main>'

[rma92]

The CDX API seems to be working again, but WMD is still making bad request errors. To work around this quickly, I generated a script that filters through CDX API JSON results and returned curl command lines to do the download. It's hacky and far less nice than WMD. @greg-raven in your case, you should probably curl one of these URLs and parse it. Removing the digest from the request significantly speeds up the request. API for CDX Server is here: https://github.com/internetarchive/wayback/tree/master/wayback-cdx-server

• http://web.archive.org/cdx/search/cdx?url=http://www.racquetresearch.com/*&output=json&fl=original,timestamp&collapse=urlkey
• http://web.archive.org/cdx/search/cdx?url=http://www.racquetresearch.com/*&output=json&fl=original,timestamp

PHP CLI script if it's useful (note that it's set up to expect the parameters in a specific order, looks only for .exe files, and tosses any URLs that have been shortened (contain '...') as I couldn't find a way to expand them:

<?php
//curl "http://web.archive.org/cdx/search/cdx?url=http://www.stardock.com/files*&output=json&fl=original,timestamp&collapse=urlkey" > stardock2.json
// Load the JSON data from the file
$jsonData = file_get_contents('stardock2.json');
$data = json_decode($jsonData, true);

// Dictionary to store the latest .exe files
$latestExeFiles = [];

// Iterate over each entry in the JSON data
foreach ($data as $index => $entry) {
    if ($index === 0) continue; // Skip the header row

    $originalUrl = $entry[0];
    $timestamp = $entry[1];

    // Check if the original URL ends with ".exe" (case insensitive)
    if (strtolower(substr($originalUrl, -4)) === '.exe' && strpos($originalUrl, '...') === false ) {
        $urlKey = $entry[0];

        // If the URL key is not in the dictionary or the current timestamp is later than the stored one
        if (!isset($latestExeFiles[$urlKey]) || $timestamp > $latestExeFiles[$urlKey]['timestamp']) {
            $latestExeFiles[$urlKey] = [
                'timestamp' => $timestamp,
                'original_url' => $originalUrl
            ];
        }
    }
}

// Output curl command lines for each .exe file
foreach ($latestExeFiles as $urlKey => $info) {
    $timestamp = $info['timestamp'];
    $originalUrl = $info['original_url'];
    $curlCommand = "curl -o " . basename($originalUrl) . " https://web.archive.org/web/$timestamp/$originalUrl";
    echo $curlCommand . PHP_EOL;
}
?>

[sub205]

Problem seems to be the upcounting of the pages. At the last page, archive.org gives you a 400:
Also the request baseurlseems to use an older api call (xd instead of cdx)
Additional output from my debug code.
Unfortunately, i don't know ruby and i am not able to skip the 400 error.

Getting snapshot pages
[[output=json&url=http%3A%2F%2Fwww.brackeen.com%2Fvga%2Findex.html&fl=timestamp%2Coriginal&collapse=digest&gzip=false&filter=statuscode%3A200]]
.
[[output=json&url=http%3A%2F%2Fwww.brackeen.com%2Fvga%2Findex.html%2F*&fl=timestamp%2Coriginal&collapse=digest&gzip=false&filter=statuscode%3A200&page=0]]
.
[[output=json&url=http%3A%2F%2Fwww.brackeen.com%2Fvga%2Findex.html%2F*&fl=timestamp%2Coriginal&collapse=digest&gzip=false&filter=statuscode%3A200&page=1]]

/usr/lib/ruby/3.1.0/open-uri.rb:364:in `open_http': 400 BAD REQUEST (OpenURI::HTTPError)
        from /usr/lib/ruby/3.1.0/open-uri.rb:741:in `buffer_open'
        from /usr/lib/ruby/3.1.0/open-uri.rb:212:in `block in open_loop'
        from /usr/lib/ruby/3.1.0/open-uri.rb:210:in `catch'
        from /usr/lib/ruby/3.1.0/open-uri.rb:210:in `open_loop'
        from /usr/lib/ruby/3.1.0/open-uri.rb:151:in `open_uri'
        from /usr/lib/ruby/3.1.0/open-uri.rb:721:in `open'
        from /var/lib/gems/3.1.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader/archive_api.rb:17:in `get_raw_list_from_api'
        from /var/lib/gems/3.1.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader.rb:92:in `block in get_all_snapshots_to_consider'
        from /var/lib/gems/3.1.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader.rb:91:in `times'
        from /var/lib/gems/3.1.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader.rb:91:in `get_all_snapshots_to_consider'
        from /var/lib/gems/3.1.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader.rb:105:in `get_file_list_curated'
        from /var/lib/gems/3.1.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader.rb:164:in `get_file_list_by_timestamp'
        from /var/lib/gems/3.1.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader.rb:309:in `file_list_by_timestamp'
        from /var/lib/gems/3.1.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader.rb:192:in `download_files'
        from /var/lib/gems/3.1.0/gems/wayback_machine_downloader-2.3.1/bin/wayback_machine_downloader:72:in `<top (required)>'
        from /usr/local/bin/wayback_machine_downloader:25:in `load'
        from /usr/local/bin/wayback_machine_downloader:25:in `<main>'

[sub205]

Ok, i finally got it working (kinda) by rescuing from the HTTPError.
Unfortunately, archive.org seems to block the downloader after some files for several minutes.

module ArchiveAPI

  def get_raw_list_from_api url, page_index
    request_url = URI("https://web.archive.org/cdx/search/cdx")
    params = [["output", "json"], ["url", url]]
    params += parameters_for_api page_index
    request_url.query = URI.encode_www_form(params)

    begin
      json = JSON.parse(URI(request_url).open.read)
      if (json[0] <=> ["timestamp","original"]) == 0
        json.shift
      end
      json
    rescue OpenURI::HTTPError => e
      []
    rescue JSON::ParserError
      []
    end
  end

[gingerbeardman]

See the other issues and PRs for proper solution

[cirosantilli]

related: #307

[cirosantilli]

This fork worked: https://github.com/StrawberryMaster/wayback-machine-downloader