Cookie Sync in Burst

[elgatito]

Description

Currently there are websites that are protected by captchas or Cloudflare to block robots.
While there are ways to bypass it using browser emulators (like Jakett, etc), it can be not available for all the platforms.

As an alternative approach we now have a code in Burst that can use cookies, synced from other devices, that run Chrome.
Burst will get cookies into HTTP client and use them for doing requests.

In case providers.json is having configuration, that describes session cookie for specific provider - Burst will try to skip authentication and consider ourselves as logged in (so no login/password required in this case to be filled in Burst settings).

How it works

We use open-source Chrome addon - https://github.com/Andiedie/sync-my-cookie .
It allows setting what websites you want to sync, and puts a file with cookies into your Github profile as a Gist file.
Optionally, if you set an encryption password, it will encrypt contents, so it would be safe to share the gist.

The minimal required setup is:

1. Create Github account (if not having)
2. Create personal Github access token that has access to read/write Gists
3. Install Chrome addon
4. Fill in Github access token in addon settings and optionally password (to encrypt uploaded cookies).
5. Open website you want to enable syncing and enable cookie sync.
6. Fill in Github access token in Burst and optionally password (if you used encryption).

That is it. Burst will read cookies on each search, apply to http client session.

Things to take into mind

Create sessions in the same network as Burst is running. It is a common practice to use IP-dedicated sessions, so IP address of Burst should be the same as the one used by Chrome to log in/authenticate.

It can be that there are multiple domains involved into authentication process on specific provider website. Chrome addon is syncing only the website you ask to, it does not store every cookie everywhere.

Providers configuration

To configure provider as the one that can re-use cookies to bypass authentication, you need to specify cookie name, that we try to find:

        "languages": "en",
        "login_cookie": "sid_session",
        "login_failed": "...",
        "login_object": "...",
        "login_path": "...",

login_cookie in this example is saying which cookie do we need (non expired one) to consider we have required cookie to skip authentication. Cookie is checked for the domain, provided in base_url parameter.

[elgatito]

@antonsoroko I've tested this solution with few trackers that were not working on my test host, both are working now. I guess we will need to verify the ones protected by Cloudflare. And also check with Python2 Kodi, I was not able yet to make a proper testing.

I was not yet releasing new version, as I think there might be issues and it needs a proper testing first.

[elgatito]

This functionality was released with Burst 0.0.87 and can be used.
Few issues were catched and fixed and I've tested.

Cookie sync would not work with Kodi 17 and earlier, as the Cryprography module, required for decrypting cookies is not available there.

Also, as mentioned in #240 , Cookie sync addon does not store browser UserAgent, that was used while creating cookies. Also it does not store TLS fingerprints of the device that we run.

I don't think current addon maintainer would do those changes, and without them we need to make sure Burst is using same UserAgent as the browser.

[antonsoroko]

@elgatito what about public trackers like 1337 and torrentgalaxy? it there a way to use this to bypass cloudflare for public trackers?

i mean, it is possible to login to some public trackers but some trackers might be purely anonymous. and anyway, i guess in general people usually do not have logins for public trackers.

[elgatito]

@antonsoroko

@elgatito what about public trackers like 1337 and torrentgalaxy? it there a way to use this to bypass cloudflare for public trackers?

In Burst we have (for long time) commong cookie jar (not separate per-provider, as it was before).
When you configure Cookie sync - then before starting any searches we fetch all cookies from Github and add them to cookie jar, and then proceed.

So if you sync cookies for a public tracker - they will be synced as well (cf_clearance cookies as well).
I was testing on one of the Cloudflare protected providers and it was bypassing cloudflare with cookie sync, but there you need to make sure that UserAgent in Burst is the same as the one that is used in the browser (otherwise Cloudflare ignore your cf_clearance cookies).

[elgatito]

@antonsoroko You can install addon to Chrome and run it without password to have cookies unencrypted, to see how contents of that Gist looks like.

[antonsoroko]

@elgatito

1. I think it makes sense to add a help message for this "cookie sync" settings.
   Like "Install SyncMyCookie Chrome extension,[CR] see <link_to_this_issue> for details".

otherwise people will not understand what to do and how to use this feature.

2. maybe we should ask for "store user agent" feature here https://github.com/Andiedie/sync-my-cookie/issues ? alternatively, we can add "user agent" as setting in burst, but it is a very long string, so it would be hard to enter it with tv remote and anyway majority of users do not know about "user agent". and changing it every time browser was updated - tedious even of advanced users.

3. were you able to do that?

run it without password

it does not allow me to save settings w/o password. looks like it is intended: https://github.com/Andiedie/sync-my-cookie/blob/6357c0d3a8a09e999c31f00c6bfc3c54d2d0cb43/src/components/setting/setting.tsx#L84

[antonsoroko]

did some testing:

torrentgalaxy (own protection) - previously it was giving me captcha page in burst debug log, with "cookie sync" - ok

cloudflare:
extranet (ext.to and extranet.torrentbay.st) - fail, still gives "Just a moment" page
idope - fail, still gives "Just a moment" page
yggtorrent - fail, still gives "Just a moment" page. but here it looks like extension does not add cf_clearance cookie to gist. if i manually add my cf_clearance to .kodi/temp/burst/common_cookies.jar - site works.

(tried even with/without my own "user agent" - same result)

other providers do not give me "Just a moment" page, so can't say anything about them.

[antonsoroko]

Easy way for users to see if they hit cloudflare protection:
enable debug log in Kodi and enable "Log debug messages for parsing" it Burst.
then open kodi.log and try to find Just a moment string.
if you see it - then you hit cloudflare protection. thus you can't use login.
if you do not see it - find Failed login content: line and see what it says.
in the end - disable debug in Kodi and Burst.

also i noticed that almost always this extesion does not push cf_clearance cookie. but looks like very soon this extension will not function anyway (because of deprecation, https://developer.chrome.com/docs/extensions/develop/migrate/mv2-deprecation-timeline)