Update documentation/structure/etc in mass with fixed support

[nelsonjchen]

Much thanks for @fcfort for implementing and fixing so much in #13

New things noticed right away to fix:

• Stronger emphasis on using a private proxy. We're handling long lived cookies right now. Much more radioactive.
• Speeds are slower. We're no longer transloading straight from GCS. As a result, speeds on a 50GB file seem to now hover around 350MB/s, down from GB/s.
• Higher concurrency though.
• Update the documentation in various parts with the new cookie approach. We aren't using URLs anymore.

Other refactors and stuff aren't in scope

[fcfort]

Speeds are slower. We're no longer transloading straight from GCS. As a result, speeds on a 50GB file seem to now hover around 350MB/s, down from GB/s.

This part is surprising to me that it changed. As far as I know, I didn't change the request flow, only the data that gets passed along in the requests, see https://gist.github.com/fcfort/71ef9739c2cbc982ef65663740e69170. As far as I can tell, the code before my also used the Cloudflare proxy as an intermediary between Azure Storage and Google Takeout.

[nelsonjchen]

I know, but ah, oh well. It's still pretty darn fast.

[nelsonjchen]

@fcfort Out of curiosity, what speeds are you getting?

[fcfort]

Hmm, I'm actually running into a strange problem that I hadn't seen before. It's strange because this didn't happen before during my testing.

When the Chrome Extension does a HEAD against one of my Takeout download URLs (e.g. https://takeout-download.usercontent.google.com/download/takeout-$takeouttime-12-001.tgz?j=$uuid&i=0&user=$userid&authuser=0), that URL will not return a content-length header. An ordinary GET against that URL will return a content-length header as expected. And of course, the extension expects a content-length header so that it can properly set up the "job plan" of downloads.

Maybe there is some way to use GET instead to the content-length, then interrupt that download after the response headers are received and return the content-length value from there.

[fcfort]

I've turned the issue above into nelsonjchen/gtr-proxy#114. I'll send a PR to fix it.

[fcfort]

@fcfort Out of curiosity, what speeds are you getting?

I'm seeing anywhere from 150 MB/s to 450 MB/s in the few that I've done so far.