Warcprox?

[TheTechRobo]

I need two flags for integration with warcprox; are they implemented and if so where? (if not are they planned to be added?)

• Proxy option
• Skip SSL verification (might not be necessary depending on how twarc works with the server)

Thanks!!

[edsu]

This is a good idea, and ought to be easy to implement given that we are using requests for http and it supports proxies?

Does your warcprox support ssl?

[TheTechRobo]

Yes, but it passes an invalid cert (not signed by trusted CAs). I know that with youtube-dl you need to use something like --no-check-certificate.

Here's the official doc for warcprox explaining this:

https://github.com/internetarchive/warcprox#man-in-the-middle

[TheTechRobo]

Looks like requests can do that (albeit without env-vars): verify=False

[edsu]

So you can't set HTTP_PROXY by itself and have it work?

[TheTechRobo]

Sort of - the SSL certs will not be signed by any trusted authority, so requests will throw an SSLError ... I think.

I'll try sometime soon.