CloudProxy gets stuck on jsch challenge

[lululombard]

So I've been implementing CloudProxy on one of my projects, basically I'm trying to download plugins from SpigotMC.org and everything works fine until I want to download a plugin.

When browsing normally, the first time I download a plugin, I'll get the following screen:

And the download starts.

But with CloudProxy, this happens:

2020-08-07T01:06:09.818Z DEBUG REQ-99 Navegating to... https://www.spigotmc.org/resources/cmi-270-commands-insane-kits-portals-essentials-economy-mysql-sqlite-much-more.3742/download?version=349138
2020-08-07T01:06:09.820Z DEBUG REQ-99 {
  headers: {
    'upgrade-insecure-requests': '1',
    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.0 Safari/537.36',
    'accept-language': 'en-US,en;q=0.9'
  }
}
2020-08-07T01:06:09.919Z INFO REQ-99 Cloudflare detected
2020-08-07T01:06:09.923Z DEBUG REQ-99 Waiting for Cloudflare challenge...
2020-08-07T01:06:14.127Z DEBUG REQ-99 {
  headers: {
    'upgrade-insecure-requests': '1',
    origin: 'https://www.spigotmc.org',
    'content-type': 'application/x-www-form-urlencoded',
    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.0 Safari/537.36',
    'accept-language': 'en-US,en;q=0.9',
    referer: 'https://www.spigotmc.org/resources/cmi-270-commands-insane-kits-portals-essentials-economy-mysql-sqlite-much-more.3742/download?version=349138'
  }
}
2020-08-07T01:06:15.926Z DEBUG REQ-99 Found challenge element again...
2020-08-07T01:06:15.928Z DEBUG REQ-99 {
  headers: {
    'upgrade-insecure-requests': '1',
    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.0 Safari/537.36',
    'accept-language': 'en-US,en;q=0.9'
  }
}
TimeoutError: Navigation timeout of 30000 ms exceeded
    at /home/lululombard/Workspace/RedCraft/CloudProxy/node_modules/puppeteer/lib/LifecycleWatcher.js:100:111
    at FrameManager.waitForFrameNavigation (/home/lululombard/Workspace/RedCraft/CloudProxy/node_modules/puppeteer/lib/FrameManager.js:107:23)
    at Frame.waitForNavigation (/home/lululombard/Workspace/RedCraft/CloudProxy/node_modules/puppeteer/lib/FrameManager.js:298:16)
    at Page.waitForNavigation (/home/lululombard/Workspace/RedCraft/CloudProxy/node_modules/puppeteer/lib/Page.js:492:16)
    at async Promise.all (index 0)
    at Page.reload (/home/lululombard/Workspace/RedCraft/CloudProxy/node_modules/puppeteer/lib/Page.js:488:24)
    at resolveChallenge (/home/lululombard/Workspace/RedCraft/CloudProxy/src/routes.ts:112:24)
    at request.get (/home/lululombard/Workspace/RedCraft/CloudProxy/src/routes.ts:330:18)
    at Object.Router [as default] (/home/lululombard/Workspace/RedCraft/CloudProxy/src/routes.ts:349:23)
  -- ASYNC --
    at Frame.<anonymous> (/home/lululombard/Workspace/RedCraft/CloudProxy/node_modules/puppeteer/lib/helper.js:94:19)
    at Page.waitForNavigation (/home/lululombard/Workspace/RedCraft/CloudProxy/node_modules/puppeteer/lib/Page.js:492:53)
    at Page.<anonymous> (/home/lululombard/Workspace/RedCraft/CloudProxy/node_modules/puppeteer/lib/helper.js:95:27)
    at Page.reload (/home/lululombard/Workspace/RedCraft/CloudProxy/node_modules/puppeteer/lib/Page.js:488:48)
    at Page.<anonymous> (/home/lululombard/Workspace/RedCraft/CloudProxy/node_modules/puppeteer/lib/helper.js:95:27)
    at resolveChallenge (/home/lululombard/Workspace/RedCraft/CloudProxy/src/routes.ts:112:35)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at request.get (/home/lululombard/Workspace/RedCraft/CloudProxy/src/routes.ts:330:18)
    at Object.Router [as default] (/home/lululombard/Workspace/RedCraft/CloudProxy/src/routes.ts:349:23)
2020-08-07T01:06:45.956Z ERROR REQ-99 Navigation timeout of 30000 ms exceeded

I think it's probably related to the // TODO: find out why these pages hang sometimes but no mater how many times I try, it will fail on that link. You can add me on Discord (lululombard#1337) if you require more information, I can even give you the credentials to my account so you can try to run it for yourself to debug :)

[NoahCardoza]

I’ve been a little backlogged lately at work but I’ll get back to reviewing issues and PRs shortly. Thanks for being thing this to my attention 🙏

[jairoxyz]

@lululombard I don't think this is a bug or issue of the proxy. Most sites will need the proxy to try several times before the challenge is solved. Don't ask me why but this been my experience so far. So for your site, I ran below curl 2 times and got a successful response then. Note that it is important to create a custom session, so that the correct headers and cookies are sent.

Request:
curl -L -X POST http://localhost:8191/v1 -H "Content-Type: application/json" --data-raw "{\"cmd\":\"request.get\",\"session\":\"mysession\",\"url\":\"https://www.spigotmc.org/resources/cmi-270-commands-insane-kits-portals-essentials-economy-mysql-sqlite-much-more.3742/download?version=349138\",\"userAgent\":\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36\", \"maxTimeout\": 10000}"

Response:

2020-08-07T03:14:52.886Z DEBUG REQ-2 Found challenge element again...
2020-08-07T03:14:53.016Z DEBUG REQ-2 Reloaded page...
2020-08-07T03:14:53.036Z ERROR REQ-2 Maximum timeout reached. maxTimeout=10000 (ms)
2020-08-07T03:15:15.356Z INFO REQ-3 Incoming request: POST /v1  <<---- second call of curl request
2020-08-07T03:15:15.357Z INFO REQ-3 Params: {"cmd":"request.get","session":"mysession","url":"https://www.spigotmc.org/resources/cmi-270-commands-insane-kits-portals-essentials-economy-mysql-sqlite-much-more.3742/download?version=349138","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36","maxTimeout":10000}
2020-08-07T03:15:15.480Z DEBUG REQ-3 Using custom UA: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36
2020-08-07T03:15:15.481Z DEBUG REQ-3 Adding custom headers: {}
2020-08-07T03:15:15.482Z DEBUG REQ-3 Navegating to... https://www.spigotmc.org/resources/cmi-270-commands-insane-kits-portals-essentials-economy-mysql-sqlite-much-more.3742/download?version=349138
2020-08-07T03:15:16.351Z INFO REQ-3 Cloudflare detected
2020-08-07T03:15:16.449Z DEBUG REQ-3 No '.ray_id' challenge element detected.
2020-08-07T03:15:16.483Z DEBUG REQ-3 No '.attack-box' challenge element detected.
ok
2020-08-07T03:15:16.800Z INFO REQ-3 Successful response in 1.444 s

[lululombard]

@jairoxyz thanks for testing dude!
Indeed I wasn't using sessions, I added them, but I still have the same issue :(

When I run the following command, it will get stuck:

curl -L -X POST http://localhost:8191/v1 -H "Content-Type: application/json" --data-raw '
{
    "cmd": "request.get",
    "url": "https://www.spigotmc.org/resources/essentialsx.9089/download?version=343125",
    "method": "GET",
    "download": true,
    "maxTimeout": 120000
}'

I get the following reply:

{
    "status": "error",
    "message": "Navigation timeout of 30000 ms exceeded",
    "startTimestamp": 1596805537584,
    "endTimestamp": 1596805586296,
    "version": "1.0.0"
}

And here are the logs:

2020-08-07T13:05:37.584Z INFO REQ-33 Incoming request: POST /v1
2020-08-07T13:05:37.584Z INFO REQ-33 Params: {"cmd":"request.get","url":"https://www.spigotmc.org/resources/essentialsx.9089/download?version=343125","method":"GET","download":true,"maxTimeout":120000}
2020-08-07T13:05:37.584Z DEBUG REQ-33 Launching headless browser...
2020-08-07T13:05:37.732Z DEBUG REQ-33 Adding custom headers: {}
2020-08-07T13:05:37.732Z DEBUG REQ-33 { headers: [Function (anonymous)] }
2020-08-07T13:05:37.733Z DEBUG REQ-33 Navegating to... https://www.spigotmc.org/resources/essentialsx.9089/download?version=343125
2020-08-07T13:05:37.753Z DEBUG REQ-33 {
  headers: {
    'upgrade-insecure-requests': '1',
    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.0 Safari/537.36',
    'accept-language': 'en-US,en;q=0.9'
  }
}
2020-08-07T13:05:37.851Z INFO REQ-33 Cloudflare detected
2020-08-07T13:05:37.855Z DEBUG REQ-33 Waiting for Cloudflare challenge...
2020-08-07T13:05:41.938Z DEBUG REQ-33 {
  headers: {
    'upgrade-insecure-requests': '1',
    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.0 Safari/537.36',
    'accept-language': 'en-US,en;q=0.9',
    referer: 'https://www.spigotmc.org/resources/essentialsx.9089/download?version=343125'
  }
}
2020-08-07T13:05:41.975Z DEBUG REQ-33 Found challenge element again...
2020-08-07T13:05:41.977Z DEBUG REQ-33 {
  headers: {
    'upgrade-insecure-requests': '1',
    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.0 Safari/537.36',
    'accept-language': 'en-US,en;q=0.9',
    referer: 'https://www.spigotmc.org/resources/essentialsx.9089/download?version=343125'
  }
}
2020-08-07T13:05:42.013Z DEBUG REQ-33 Reloaded page...
2020-08-07T13:05:46.057Z DEBUG REQ-33 {
  headers: {
    'upgrade-insecure-requests': '1',
    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.0 Safari/537.36',
    'accept-language': 'en-US,en;q=0.9',
    referer: 'https://www.spigotmc.org/resources/essentialsx.9089/download?version=343125'
  }
}
2020-08-07T13:05:46.098Z DEBUG REQ-33 Found challenge element again...
2020-08-07T13:05:46.099Z DEBUG REQ-33 {
  headers: {
    'upgrade-insecure-requests': '1',
    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.0 Safari/537.36',
    'accept-language': 'en-US,en;q=0.9',
    referer: 'https://www.spigotmc.org/resources/essentialsx.9089/download?version=343125'
  }
}
2020-08-07T13:05:46.134Z DEBUG REQ-33 Reloaded page...
2020-08-07T13:05:50.182Z DEBUG REQ-33 {
  headers: {
    'upgrade-insecure-requests': '1',
    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.0 Safari/537.36',
    'accept-language': 'en-US,en;q=0.9',
    referer: 'https://www.spigotmc.org/resources/essentialsx.9089/download?version=343125'
  }
}
2020-08-07T13:05:50.223Z DEBUG REQ-33 Found challenge element again...
2020-08-07T13:05:50.224Z DEBUG REQ-33 {
  headers: {
    'upgrade-insecure-requests': '1',
    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.0 Safari/537.36',
    'accept-language': 'en-US,en;q=0.9',
    referer: 'https://www.spigotmc.org/resources/essentialsx.9089/download?version=343125'
  }
}
2020-08-07T13:05:50.260Z DEBUG REQ-33 Reloaded page...
2020-08-07T13:05:54.304Z DEBUG REQ-33 {
  headers: {
    'upgrade-insecure-requests': '1',
    origin: 'https://www.spigotmc.org',
    'content-type': 'application/x-www-form-urlencoded',
    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.0 Safari/537.36',
    'accept-language': 'en-US,en;q=0.9',
    referer: 'https://www.spigotmc.org/resources/essentialsx.9089/download?version=343125'
  }
}
2020-08-07T13:05:56.265Z DEBUG REQ-33 Found challenge element again...
2020-08-07T13:05:56.266Z DEBUG REQ-33 {
  headers: {
    'upgrade-insecure-requests': '1',
    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.0 Safari/537.36',
    'accept-language': 'en-US,en;q=0.9',
    referer: 'https://www.spigotmc.org/resources/essentialsx.9089/download?version=343125'
  }
}
TimeoutError: Navigation timeout of 30000 ms exceeded
    at /home/lululombard/Workspace/RedCraft/CloudProxy/node_modules/puppeteer/lib/LifecycleWatcher.js:100:111
    at FrameManager.waitForFrameNavigation (/home/lululombard/Workspace/RedCraft/CloudProxy/node_modules/puppeteer/lib/FrameManager.js:107:23)
    at Frame.waitForNavigation (/home/lululombard/Workspace/RedCraft/CloudProxy/node_modules/puppeteer/lib/FrameManager.js:298:16)
    at Page.waitForNavigation (/home/lululombard/Workspace/RedCraft/CloudProxy/node_modules/puppeteer/lib/Page.js:492:16)
    at async Promise.all (index 0)
    at Page.reload (/home/lululombard/Workspace/RedCraft/CloudProxy/node_modules/puppeteer/lib/Page.js:488:24)
    at resolveChallenge (/home/lululombard/Workspace/RedCraft/CloudProxy/src/routes.ts:112:24)
    at request.get (/home/lululombard/Workspace/RedCraft/CloudProxy/src/routes.ts:330:18)
    at Object.Router [as default] (/home/lululombard/Workspace/RedCraft/CloudProxy/src/routes.ts:349:23)
  -- ASYNC --
    at Frame.<anonymous> (/home/lululombard/Workspace/RedCraft/CloudProxy/node_modules/puppeteer/lib/helper.js:94:19)
    at Page.waitForNavigation (/home/lululombard/Workspace/RedCraft/CloudProxy/node_modules/puppeteer/lib/Page.js:492:53)
    at Page.<anonymous> (/home/lululombard/Workspace/RedCraft/CloudProxy/node_modules/puppeteer/lib/helper.js:95:27)
    at Page.reload (/home/lululombard/Workspace/RedCraft/CloudProxy/node_modules/puppeteer/lib/Page.js:488:48)
    at Page.<anonymous> (/home/lululombard/Workspace/RedCraft/CloudProxy/node_modules/puppeteer/lib/helper.js:95:27)
    at resolveChallenge (/home/lululombard/Workspace/RedCraft/CloudProxy/src/routes.ts:112:35)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at request.get (/home/lululombard/Workspace/RedCraft/CloudProxy/src/routes.ts:330:18)
    at Object.Router [as default] (/home/lululombard/Workspace/RedCraft/CloudProxy/src/routes.ts:349:23)

Even with sessions, I can login, browse resources, but not download them:

[jairoxyz]

Ah I see you are using the download parameter in the GET request for downloading the file itself:

https://www.spigotmc.org/resources/essentialsx.9089/download?version=343125","method":"GET","download":true

I haven't tried that myself yet. Will have a look at the code and see if something occurs to me ;)

PS:

OK so I added download:true to the request and I get the timeout on first try but on second try I get:
"net::ERR_ABORTED at https://www.spigotmc.org/resources/essentialsx.9089/download?version=343125"

I checked in Fiddler and the response is actually successful but I think there is an issue with MIME type or so because response is not processed correctly:

HTTP/1.1 200 OK
Date: Fri, 07 Aug 2020 14:20:21 GMT
Content-Type: application/octet-stream
Content-Length: 3251905
Connection: keep-alive
Cache-control: private
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1
Last-Modified: Fri, 07 Aug 2020 14:20:21 GMT
Content-Disposition: attachment; filename="EssentialsX-2.18.0.0.zip"
Etag: "1594324507"
X-Content-Type-Options: nosniff
CF-Cache-Status: DYNAMIC
cf-request-id: 046ae45f790000af2d01ab9200000001
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
CF-RAY: 5bf1a34588baaf2d-KIX
alt-svc: h3-27=":443"; ma=86400, h3-28=":443"; ma=86400, h3-29=":443"; ma=86400

PK���   �    P  � /�  X� �   EssentialsX-2.18.0.0.jart UL$ � qwww      pw/Z eqw�   p�w9 pw8 �*i  I'  f ɗy dF]	�� ���� # L� ?����

*** FIDDLER: RawDisplay truncated at 128 characters. Right-click to disable truncation. ***

Maybe sending a header like this would help?

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed

Or it might even work in your py code because you can handle the bin data there. I think you're giving up after 1st timeout. Run a small loop to rerun the POST with timeout 10secs until response is 200.

[lululombard]

So I've been trying to do some retries, I also got the net:ERR_ABORTED, so I tried to debug CloudProxy by running Chrome in a non headless mode, and it actually asks to download the file. I'll play arround and maybe add support for files that get downloaded after a redirect, because I think this is where the issue is

[lululombard]

Update!

By adding this to routes.ts I am able to download files:

Chrome says it failed and will keep the temporary download name in my downloads folder, but it's the right md5, I can rename the file with the correct extension and it will work, so that's good progress. I'll continue working on this and hopefully have a PR ready in a few hours

[jairoxyz]

Yes, I saw that Page.setDownloadBehavior when googling for Puppeteer file download yesterday. Good job :)

[NoahCardoza]

@lululombard Wait so will this download the file to the downloads directory?

That's probably better than the base64 encoded buffer solution I was using but I wonder how this would work with the Docker? Tbh I barely even use the Docker file so I might add a pole to remove it if no one else is making use of it.

The only reason I've been using ENV variables to pass things to the script is because of Docker. I'd much rather use CLI flags or a config file or something.

[lululombard]

@NoahCardoza I'm using the tmp package to set a download directory, it works with Windows, macOS and Linux from my testing. I'll try with Docker once I'm sure that I don't need to make any changes to the code :)

[NoahCardoza]

Ah, so you can set the directory! Good. We might be able to work it like this:

Save the file to the tmp dir, and then return a url rather than the file in the json. Then they can use that url to download the file from the tmp dir and after serving the file it will be deleted?

[lululombard]

Right now I'm returning the base64 from the file like before, and deleting the temporary file and folder when CloudProxy replies, but yes we can do anything we want :)

[NoahCardoza]

I think the idea I just purposed would be the best general solution, since it wouldn't be a good idea to encode larger files into JSON.

Like if someone wanted to download a video or even a large photo.

[lululombard]

Indeed, for now I'll still work like this, I'm downloading files of 3-4 MB and it works, but sure we'll probably hit a limit. I'll continue working on this though :)

[NoahCardoza]

Great!

[lululombard]

Sorry I kind of got off track. I have something working but not in a final state, and I'll probably not update it because dumping cookies and use them directly in my Python program works. That said, it's available here: https://github.com/lululombard/CloudProxy/tree/feature/download-redirect-support