wget works but lightpanda has error.ConnectionResetByPeer

[marcuskbr]

This works:
wget -O ttt https://www.volksbankinostwestfalen.de/

This fails:

./lightpanda-2025-03-14 fetch --dump https://www.volksbankinostwestfalen.de/
error(cli): 'https://www.volksbankinostwestfalen.de/' fetching error (error.ConnectionResetByPeer)

error: ConnectionResetByPeer

[krichprollsch]

$ zig build run -- fetch  https://www.volksbankinostwestfalen.de/                                                                                                                                                                             15:15:35 [1/9204]
debug(cli): Fetch mode: url https://www.volksbankinostwestfalen.de/, dump false
debug(browser): start js env
debug(browser): setup global env
debug(polyfill): load polyfill-fetch: undefined
debug(browser): inspector context created
debug: Inspector contextCreated called
debug(browser): starting GET https://www.volksbankinostwestfalen.de/
error(cli): 'https://www.volksbankinostwestfalen.de/' fetching error (error.ConnectionResetByPeer)

error: ConnectionResetByPeer
/usr/local/zig-0.13.0/lib/std/posix.zig:843:27: 0x1b96b51 in read (lightpanda)
            .CONNRESET => return error.ConnectionResetByPeer,
                          ^
/usr/local/zig-0.13.0/lib/std/net.zig:1821:9: 0x1bdc490 in read (lightpanda)
        return posix.read(self.handle, buffer);
        ^
/home/pierre/wrk/browser/vendor/tls.zig/src/record.zig:70:31: 0x1bdc0ab in next (lightpanda)
                    const n = try r.inner_reader.read(r.buffer[r.end..]);
                              ^
/home/pierre/wrk/browser/vendor/tls.zig/src/record.zig:78:26: 0x1c44a6a in nextDecrypt (lightpanda)
            const rec = (try r.next()) orelse return null;
                         ^
/home/pierre/wrk/browser/vendor/tls.zig/src/connection.zig:71:55: 0x1deea48 in nextRecord (lightpanda)
                const content_type, const cleartext = try c.rec_rdr.nextDecrypt(&c.cipher) orelse return null;
                                                      ^
/home/pierre/wrk/browser/vendor/tls.zig/src/connection.zig:63:46: 0x1d0c8bc in next (lightpanda)
            const content_type, const data = try c.nextRecord() orelse return null;
                                             ^
/home/pierre/wrk/browser/vendor/tls.zig/src/connection.zig:212:34: 0x1d0c6a4 in readv (lightpanda)
                    c.read_buf = try c.next() orelse break;
                                 ^
/home/pierre/wrk/browser/src/http/Client.zig:233:66: 0x1c48df7 in readvDirectTls (lightpanda)
                error.ConnectionResetByPeer, error.BrokenPipe => return error.ConnectionResetByPeer,
                                                                 ^
/home/pierre/wrk/browser/src/http/Client.zig:243:13: 0x1b88675 in readvDirect (lightpanda)
            return conn.readvDirectTls(buffers);
            ^
/home/pierre/wrk/browser/src/http/Client.zig:260:23: 0x1b2d22e in fill (lightpanda)
        const nread = try conn.readvDirect(&iovecs);
                      ^
/home/pierre/wrk/browser/src/http/Client.zig:990:17: 0x1ae6bce in wait (lightpanda)
                try connection.fill();
                ^
/home/pierre/wrk/browser/src/browser/loader.zig:76:9: 0x1ae23ea in get (lightpanda)
        try resp.req.wait();
        ^
/home/pierre/wrk/browser/src/browser/browser.zig:374:20: 0x1af3e75 in navigate (lightpanda)
        var resp = try self.session.loader.get(arena, self.uri);
                   ^
/home/pierre/wrk/browser/src/main.zig:110:21: 0x1afa2ae in main (lightpanda)
                    return err;
                    ^
run
└─ run lightpanda failure
error: the following command exited with error code 1:
/home/pierre/wrk/browser/.zig-cache/o/5b9deb890d188671421348ab172dfd4c/lightpanda fetch https://www.volksbankinostwestfalen.de/
Build Summary: 3/5 steps succeeded; 1 failed (disable with --summary none)
run transitive failure
└─ run lightpanda failure
error: the following build command failed with exit code 1:
/home/pierre/wrk/browser/.zig-cache/o/d8f113b6a18b1614868b9efb1a78495d/build /usr/local/zig-0.13.0/zig /home/pierre/wrk/browser /home/pierre/wrk/browser/.zig-cache /home/pierre/.cache/zig --seed 0x68a958ae -Zbea3f0ca19c626db run -- fetch https://www.volks
bankinostwestfalen.de/

[karlseguin]

It's responding to the User-Agent. Here, the server immediately closes the connection without sending a response:

curl https://www.volksbankinostwestfalen.de/ \
  -H "Host: www.volksbankinostwestfalen.de" \
  -H "User-Agent: Lightpanda/1.0" -v

Whereas, without the User-Agent, the server sends the response (a redirect):

curl https://www.volksbankinostwestfalen.de/ \
  -H "Host: www.volksbankinostwestfalen.de" -v

[marcuskbr]

Thanks karlseguin!
I have not seen any option for lightpanda to set the UA.
It seems that the fetch command needs an option --user-agent <string>.
There is a commercial scraper that has options similar to --phone and --desktop which set the UA and the screen orientation and screen size. That would be even better.

[krichprollsch]

I don't have any error anymore:

$ zig build run -- fetch --dump  https://www.volksbankinostwestfalen.de/
[...]
!DOCTYPE html>
<html lang="de-DE"><head><script type="text/javascript" src="/ruxitagentjs_ICA7NVfqrux_10317250630095842.js" data-dtconfig="rid=RID_2081936797|rpid=465407|domain=volksbankinostwestfalen.de|re
portUrl=/rb_18f35b8a-28e6-455f-9304-0c5d6c4ac3ba|uam=1|app=35703a38e62069a9|coo=1|owasp=1|featureHash=ICA7NVfqrux|rdnt=1|uxrgce=1|cuc=wjovp48q|mel=100000|expw=1|dpvc=1|lastModification=175465
2854991|tp=500,50,0|agentUri=/ruxitagentjs_ICA7NVfqrux_10317250630095842.js"></script><meta charset="utf-8"
[...]