Add search_v2() method

[Mr0grog]

🚧 Work in Progress! 🚧

This adds support for the Internet Archive's new, beta CDX search endpoint at /web/timemap/cdx. It deals with pagination much better and is eventually slated to replace the search currently at /cdx/search/cdx, but is a little slower and still being tested. Fixes #8.

There are still a bunch of things to be done before merging:

• Get clarity from Internet Archive folks on fields, order (see if they get output=json working), etc.
• Determine whether all the other parameters still work and work the same (filter, collapse, resolveRevisits, etc) and whether there are new ones we can/should use. (Update: resolveRevisits is badly broken, but the rest are the same as original search and work fine. Checking w/ Wayback folks for more detail.)
• Implement rate limiting
• Determine if search_v2 is the right name, or if it should be something else (search_beta()? search_next()?)
• Add tests

[Mr0grog]

This might not be necessary! I just discovered that sending too high a page value gets a 400 error with the header x-archive-wayback-runtime-error: page must be smaller than numpages, so we can in theory check for that and stop.

That said, that’s a very human-readable message and feels unstable. We should check with folks at the Internet Archive about what approach they’d prefer people use.

[Mr0grog]

Update: the way you control output format in the new search is not with ?output=<format> but instead with the path: /web/timemap/<format>. So the right way to get JSON is:

GET /web/timemap/json?url=<url>&other=params&go=here

[Mr0grog]

Some updates:

• Most of the other parameters continue to work just as they did. But resolveRevisits seems to break searches. I’ve ignored it in the code for now and asked Wayback folks about whether it planned or no longer intended to work, and will figure out the right way to handle it after I hear back.

• I added copies of most of the search() tests. I think I should probably re-organize these so they are interleaved with the original search tests (so it’s easier to make sure a new test covers both) or even parametrize some of these tests so one test covers both implementations. (Not feasible in cases where we are exercising behavior or arguments that differ between the two.)

• There is a weird issue where the search sometimes returns some very recent results (like just the last week-ish?) even if they are after the to timestamp. Our tests currently fail because of this. I’ve asked Wayback folks about it (seems like a bug), and am debating whether the best behavior here is to skip or implement the test differently (and expose users to the bug), or to strip the bad results client-side.

At this point, there’s a little more I can do (rate limiting, cleanup), but we the main blocker by lack of clarity on bugs/intended behavior from Wayback, which we’ll have to wait to hear back on.