Excessive input buffering on HTTP Server - has anybody noticed this?

[8chanAnon]

Version

Issue found on all versions since 10

Platform

Windows 10

Subsystem

HTTP Server

What steps will reproduce the bug?

I built a local proxy server designed for the purpose of bypassing cross-origin restrictions in web browsers. The server works fine and this is not an issue which affects the functionality of the server. This issue affects the ability to monitor upload progress when a data transfer is passed through the proxy server. This is an example of how the URL might look like in a Javascript program running in the browser: http://localhost:8080/https://somewebsite.com/upload_my_file

This works fine. The upload gets to the destination as it should. The problem is with monitoring the upload progress using XMLHttpRequest. Any file smaller than 5 megabytes or so gets sucked into a black hole so the upload progress monitor will immediately show 100% complete. Files larger than 5 megabytes will bump up the progress monitor in 3 to 5 megabyte increments. Sometimes it is worse than this. I have seen the black hole suck up as much as 15 megs. No idea why this would vary.

The black hole exists both in front of the HTTP server and after it. My server passes the data through a second server which is just a Socks5 proxy. There is another black hole at the output of the HTTP server.

This drove me crazy for a long time. I thought XMLHttpRequest was buggered up but I've seen it work correctly elsewhere. By monitoring the socket inputs and outputs in my server, I have shown that the issue exists outside of my code, in the HTTP server. The Socks5 server, which my web browser is configured to use as a proxy for DNS management, does not exhibit this behaviour so the issue is not with the net server implementation which the HTTP server is based on.

In any case, I have worked around the issue by adding a mechanism for monitoring upload progress via polling the server every few seconds to get a progress report (taken from the output socket in the Socks5 server). The advantage is that I can now use fetch for uploading instead of XMLHttpRequest so it's not like this issue really needs fixing but I think it is problematic for possibly other reasons.

I've searched the Internet to find any mention of this issue and I can't find anything. I know that I'm not imagining this. Other people must have noticed this issue but haven't mentioned it because they can't pinpoint the source. I've looked at the Nodejs source code but can't find what is causing this excessive buffering. Nor can I see any way to turn it off.

If anybody needs to see my server to run tests then you can take a look at my GitHub project. I'm readying a new version so it would be nice to get a fix. Maybe it's just a setting buried in some far corner somewhere?

https://github.com/8chanAnon/Kraker-Local-Proxy-Server

How often does it reproduce? Is there a required condition?

Always. No condition.

What is the expected behavior?

Input should be consumed in smaller chunks, preferably less than 100KB.

What do you see instead?

Input is consumed in chunks of 3 to 5 megabytes.

Additional information

No response

[bnoordhuis]

Do you have a more minimal test case that demonstrates the issue? The fewer moving parts and dependencies, the better.

If not, no problem, but then I'll convert this to a discussion because it isn't really actionable as a bug report.

[8chanAnon]

Do you have a more minimal test case that demonstrates the issue? The fewer moving parts and dependencies, the better.

I can supply that. Download these two files:

https://8chananon.github.io/dl/kraker-demo.js
https://8chananon.github.io/dl/upload_test.htm

Run "kraker-demo.js" in Nodejs and then run "upload_test.htm" in your web browser (do not run from the website if your browser blocks HTTP).

An upload url has been provided but you can supply your own if you have one. Note that the default upload site will return status 400 or 403. That's fine. You will see that the upload progresses in chunks of 3 to 5 megabytes.

If not, no problem, but then I'll convert this to a discussion because it isn't really actionable as a bug report.

There should be an option for discussion but the only options were bug report or feature request. Although I'm leaning in the direction of calling this a bug and not a feature.

[bnoordhuis]

I'm not able to reproduce locally (on Linux, not Windows) but if I'd have to venture a guess, I'd say it's something your browser does, not node. Try uploading a file with curl and see if it makes a difference.

I'll convert this to a discussion because so far there's no indication it's a node bug.

[8chanAnon]

@bnoordhuis

Well, it is possibly a Windows-only issue. Not browser related since this happens on both Brave and Firefox. I'm not terribly familiar with curl. Does it offer an upload progress monitor? The upload, as I stated, works fine. It's just that I can't realistically measure the rate of progress because of the data getting sucked in. This is a local connection to a local server which then connects to a remote server. There is no speed limit on the rate of data transfer to the local server. I hope you're understanding my issue correctly.

I'll convert this to a discussion because so far there's no indication it's a node bug.

It could be a feature. :)

[bnoordhuis]

Yes, curl shows a counter, or a progress bar if you start it with -#.