New WARC header to provide hint where to find original records for revisit records.

[ikreymer]

The revisit record generally includes WARC-Refers-To-Target-URI and WARC-Refers-To-Target-Date to indicate the URL / timestamp
of the original record that the revisit refers to.

It would also be useful to provide an additional hint as to where this data may be found, through some sort of user-defined identifier,
which, if provided, could be used to help find the original records. The header can be a URI.

Some name ideas for this header:

• WARC-Revisit-Original-Source
• WARC-Revisit-Original-Location
• WARC-Revisit-Original-Identifier

For our use case, we would this to store the name of a WACZ file that contains the original, but not necessarily the location, as that may change, eg:

WARC-Revisit-Original-Source: file://somefile.wacz
WARC-Revisit-Original-Source: urn:wacz:somefile.wacz

Other identifiers:

WARC-Revisit-Original-Source: https://example.com/path/to/known.warc.gz
WARC-Revisit-Original-Source: urn:sha-256:<...>
WARC-Revisit-Original-Source: urn:customurn:...

Idea it make this generic so that it could be used for variety of identifiers, though could also go with something more specific (like WARC-Revisit-Original-WACZ-Filename instead).

Would anyone else use such a header if it existed? Thoughts on the name, eg. source vs location?

[ato]

What is the motivating problem that this is intended to solve? Is it something to do with export from de-duplicated collections? My first thought was wondering if this was to allow resolving the payload during replay without an index lookup but as it doesn't include the record offset that doesn't seem like the intent.

The examples are WARC and WACZ files, are there concrete use cases in which it would be something other than a container file? It also seems worth considering if there should be a relationship (or maybe intentionally not) to the WARC-Filename field.

Potentially related work

Extension to conversion records

#51 proposes extending WARC-Refers-To-Target-URI and WARC-Refers-To-Target-Date to conversion records. Perhaps the same reasoning applies to this header too? In which case we might not want to call it Revisit. I'd lean towards staying with the WARC-Refers-To- prefix if possible, maybe something along the lines of WARC-Refers-To-File or WARC-Refers-To-Container, but I want to better understand the use cases first.

Locating external payloads

In bag-nabit @jcushman defined a revisit profile so that payloads can be stored in separate files while keeping headers and other metadata in the WARC file:

WARC-Type: revisit
WARC-Profile: file-content; filename="files/data.html"

@ibnesayeed suggested a similar concept on the IIPC Slack with a WARC-Payload-Location header.

[ikreymer]

@ato thanks for mentioning all related ideas in one place!
This is slightly different as we're still following the revisit semantics as is, just adding a hint as to where to find the referred to records.

What is the motivating problem that this is intended to solve? Is it something to do with export from de-duplicated collections? My first thought was wondering if this was to allow resolving the payload during replay without an index lookup but as it doesn't include the record offset that doesn't seem like the intent.

Yes, it is to allow linkage between revisit records and content that it was deduplicated from in an unambiguous way,
to allow efficient access during replay time.
Since WACZ files have a built-in index, just referencing the file is sufficient, the reader would know how to access the index to lookup the record.
For WARC (and I suppose as backwards compatibility for WACZ), could also support optional <offset>:<length>, perhaps

WARC-Refers-To-Container: somefile.wacz
WARC-Refers-To-Container: somefile.warc:200:1000
WARC-Refers-To-Container: {"filename": "somefile.warc", "offset": 200, "length": 1000}

Was considering refers-to-* but thought refers-to-location may be confusing as it doesn't refer to this, but contains the object references in the other refers-to headers, but container might work!

As for the format, using JSON would be most flexible if wanted to support different ways to specify the container. Could also add multiple headers like WARC-Refers-To-Container-Offset but that seems a bit excessive IMO.