How to archive public documentation of trackers?

[zner0L]

As we discussed in #3, we need some kind of archiving solution that can be trusted and that is good enough for archiving modern, JS infested websites with potentially hidden content. We decided to use ArchiveBox, but while researching on how to set it up, I stumbled upon https://webrecorder.net/tools, who have offer very good archiving tools, which they also host at https://conifer.rhizome.org/ and they also have a self hosted service: https://github.com/webrecorder/browsertrix-cloud

Though I think we could also do easily by just having a folder of WACZ files generated by https://archiveweb.page/ in a Cloud somewhere. What do you think?

[baltpeter]

Interesting idea, I hadn't considered Webrecorder. But when I tested them a while back, it did produce very high-fidelity results.

I'm definitely happy if we don't have to host our own stuff. But I do like the simplicity of having a screenshot/PDF. Pretty much anyone can understand/open that.

[baltpeter]

I don't think the hosted version is an option for us, unfortunately. First like I tried didn't work at all. :/

With "Use Current Browser", it only showed me a Cloudflare page and produced a broken snapshot: https://conifer.rhizome.org/einbenni/tweasel-tracker-research/20230831125614/https://support.singular.net/hc/en-us/articles/4411780525979-Types-of-Device-IDs?__cf_chl_rt_tk=Y_8BpAkKD6JEL0fuG9C25M1j69rQvw_50xxYK.bQz0g-1693486574-0-gaNycGzNCyU

Using one of their hosted browsers wasn't much better: https://conifer.rhizome.org/einbenni/tweasel-tracker-research/20230831130617$br:chrome:76/https://support.singular.net/hc/en-us/articles/4411780525979-Types-of-Device-IDs?__cf_chl_rt_tk=Ca3lMAxwRDNfMOPVQCw62S_Fkwq.nATI.xX8jT89XOY-1693487178-0-gaNycGzNCbs

[baltpeter]

Also… :|

[zner0L]

The conifer service isn‘t much good though, or so it seems. My tests were all pretty bad. Webrecoder is really nice, still.

I do agree, that screenshots and PDF are very accessible also. But Webrecorder might be more accurate and they also produce good results for hidden content, it seems.

[zner0L]

How would we add links to public archives in our current schema for the reasoning? We can only add one link, and it would sense to have this be the archived link, but I would like to have the original link there as well…

[baltpeter]

I had two ideas:

• An object ({ link: string, archived: string[] }), possibly with the script automatically replacing a URL string with that.
• But then you have to do code mods in the script, which… ugh… So maybe just a CSV/SQLite DB that records date, adapter, URL, archived URLs?

[zner0L]

Yes, an object seems like the obvious way to go, but this would also break the API for the reasoning field. I guess we could also have some kind of linktree kind of thing that takes the URL and provides a link to the original and the archived version? Like https://sources.tweasel.org/https%3A%2F%2Fsupport.singular.net%2Fhc%2Fen-us%2Farticles%2F360037581952--UPDATED-Android-SDK-Integration-Guide%2325_Sending_the_User_ID_to_Singular_Optional or something similar?

[baltpeter]

I guess we could also have some kind of linktree kind of thing that takes the URL and provides a link to the original and the archived version?

Yeah, I guess. That would work well with option 2.

[baltpeter]

The conifer service isn‘t much good though, or so it seems. My tests were all pretty bad. Webrecoder is really nice, still.

I do agree, that screenshots and PDF are very accessible also. But Webrecorder might be more accurate and they also produce good results for hidden content, it seems.

Hm, if we can't use the hosted service and have to manually make snapshots using a Chrome extension/desktop app, I don't really see the value for us. I mean, high-fidelity archiving is nice and all, but we're not trying to archive Flash games here. If we already have to do manual work anyway, wouldn't a screenshot and/or PDF not be much better for our use case?

[baltpeter]

I've been using a local ArchiveBox instance to archive all interesting tracker documentation pages I've encountered. While doing that, I've discovered yet another annoying quirk. I've been using the CLI to add new pages to archive (reasoning).

I'm currently at 27 archived pages. A lot of them have extractors that failed (the archive.org extractor almost always fails with Failed to find "content-location" URL header in Archive.org response.). When I want to archive new URLs, it first goes through all previously failed snapshots and tries them again. Especially the archive.org extractor takes quite a long time. If you archive a PDF, it will also try all the extractors that will never work (screenshot, PDF, etc.) and each one will fail with Extractor timed out after 600s. every. single. time. And there is no parallelism, so this takes forever.

I've added a few new URLs almost 45 minutes ago and it hasn't even started archiving those, it's still stuck (unsuccessfully) retrying old snapshots. And of course, this is only going to get (a lot) worse the more pages we archive.

At the same time, snapshots do also sometimes fail temporarily, so it is important that they are retried. I'm just really not a fan of the architecture design here. There should be a background job that does that automatically in the background without interfering with new snapshots (and with an exponential backoff, of course).

[zner0L]

Yeah, I guess. That would work well with option 2.

If that works for you, I’d start building something like this and maybe also build a plugin for ArchiveBox to automate sending the links there? I am currently thinking of a very simple server doing just a complete text match lookup in the database and serving the links found. Links could be added using a REST API/a simple js cli.

[baltpeter]

also build a plugin for ArchiveBox

I'm really not convinced that AB is the way to go (see my comment above). It just has so many fundamental problems.

I am currently thinking of a very simple server doing just a complete text match lookup in the database and serving the links found.

Are you sure we want a server for that? I'd definitely want the archived links to be part of the TrackHAR repo, so there's no need for a PUT API. And if we're only GETing, you could just 'compile' the list to _redirects.

[zner0L]

I'd definitely want the archived links to be part of the TrackHAR repo

And how would you save those then? As a big JSON file?