Distfile caching on failed fetch stage

[tomalexander]

I think the current behavior for poudriere is that it does not save any dist files when the fetch stage fails, which is leaving me stuck unable to build graphics/librsvg2-rust.

For a while now I've had graphics/librsvg2-rust stuck as a failed build in my poudriere instance. I've attached 3 log files of my failed build of that port. Looking at the logs, it is downloading a bunch of crates, alternating between two distinct behaviors:

First, 3-4 crates successfully download from crates.io:

=> librsvg-2.58.0.tar.xz doesn't seem to exist in /portdistfiles/.
=> Attempting to fetch https://download.gnome.org/sources/librsvg/2.58/librsvg-2.58.0.tar.xz
librsvg-2.58.0.tar.xz                                 5592 kB   16 MBps    01s
=> rust/crates/anes-0.1.6.crate doesn't seem to exist in /portdistfiles/.
=> Attempting to fetch https://crates.io/api/v1/crates/anes/0.1.6/download?dummy=/rust/crates/anes-0.1.6.crate
rust/crates/anes-0.1.6.crate                            23 kB   13 MBps    00s
=> rust/crates/approx-0.5.1.crate doesn't seem to exist in /portdistfiles/.
=> Attempting to fetch https://crates.io/api/v1/crates/approx/0.5.1/download?dummy=/rust/crates/approx-0.5.1.crate
rust/crates/approx-0.5.1.crate                          14 kB  136 MBps    00s
=> rust/crates/assert_cmd-2.0.13.crate doesn't seem to exist in /portdistfiles/.
=> Attempting to fetch https://crates.io/api/v1/crates/assert_cmd/2.0.13/download?dummy=/rust/crates/assert_cmd-2.0.13.crate
rust/crates/assert_cmd-2.0.13.crate                     23 kB  108 kBps    00s

Then, there is a block of crates that fail to download from crates.io with Connection reset by peer, so they pull from distcache.FreeBSD.org:

=> rust/crates/bit-set-0.5.3.crate doesn't seem to exist in /portdistfiles/.
=> Attempting to fetch https://crates.io/api/v1/crates/bit-set/0.5.3/download?dummy=/rust/crates/bit-set-0.5.3.crate
fetch: https://crates.io/api/v1/crates/bit-set/0.5.3/download?dummy=/rust/crates/bit-set-0.5.3.crate: Connection reset by peer
=> Attempting to fetch http://distcache.FreeBSD.org/ports-distfiles/rust/crates/bit-set-0.5.3.crate
rust/crates/bit-set-0.5.3.crate                         14 kB 1671 kBps    00s
=> rust/crates/bit-vec-0.6.3.crate doesn't seem to exist in /portdistfiles/.
=> Attempting to fetch https://crates.io/api/v1/crates/bit-vec/0.6.3/download?dummy=/rust/crates/bit-vec-0.6.3.crate
fetch: https://crates.io/api/v1/crates/bit-vec/0.6.3/download?dummy=/rust/crates/bit-vec-0.6.3.crate: Connection reset by peer
=> Attempting to fetch http://distcache.FreeBSD.org/ports-distfiles/rust/crates/bit-vec-0.6.3.crate
rust/crates/bit-vec-0.6.3.crate                         19 kB 2250 kBps    00s
=> rust/crates/bit_field-0.10.2.crate doesn't seem to exist in /portdistfiles/.
=> Attempting to fetch https://crates.io/api/v1/crates/bit_field/0.10.2/download?dummy=/rust/crates/bit_field-0.10.2.crate
fetch: https://crates.io/api/v1/crates/bit_field/0.10.2/download?dummy=/rust/crates/bit_field-0.10.2.crate: Connection reset by peer
=> Attempting to fetch http://distcache.FreeBSD.org/ports-distfiles/rust/crates/bit_field-0.10.2.crate
rust/crates/bit_field-0.10.2.crate                      10 kB  193 MBps    00s

Then the downloads from crates.io start working again briefly.

So looking at this pattern, I think I'm getting rate-limited by crates.io. This doesn't cause problems when the crates also exist in distcache.FreeBSD.org but not all the crates exist there.

There are two interesting bits from this:

1. The crate that the build eventually fails on is different between runs. This is down to random chance, failing the first time I am both rate-limited and the crate doesn't exist on distcache.FreeBSD.org. So if I'm lucky, my non-rate-limited requests happen to be the crates missing from distcache.FreeBSD.org.
2. The crates do not end up in /usr/ports/distfiles/rust/crates/. For example, in 3.log (attached), the build fails on color_quant-1.1.0.crate but in both my earlier builds of 1.log and 2.log that crate successfully downloaded.

Running find / -type f -name '*color_quant*crate*' turns up no files, nor does find /usr/ports/distfiles/rust/crates/ | grep -i quant. But ls -l /usr/ports/distfiles/rust/crates | wc -l shows I have 541 crates cached.

So what I suspect is happening is poudriere only moves the downloaded distfiles into /usr/ports/distfiles if it succeeds in fetching ALL of the crates. Is this statement correct?

And assuming that is correct, this presents a problem because if each individual successfully downloaded crate ended up cached in /usr/ports/distfiles then over a series of builds, the rate-limiting RNG gods would slowly grant me a couple new crates each build which would eventually let me build graphics/librsvg2-rust.

So I've posted this as a discussion instead of filing a bug report because I don't know if my assumptions on poudriere's behavior are correct. I'm hoping someone here can confirm?

Logs:
1.log
2.log
3.log

[garrettboone]

I don't know why this failure to download the patches happens, but you could download the files manually into /usr/local/poudriere/distfiles and then run the build again. It was happening to me with gitlab-ce.

[jhgit]

This is not the same issue as the one describe with a reset when trying to fetch from crates.io. But I'll describe the reset issue I see (with distcache.freebsd.org). This problem is likely different, however, so just be aware this is probably not the same problem.

fetch(1) spews 'Connection reset by peer' for me when I fetch using http://distcache.freebsd.org (when requesting ANY distribution file) on a corporate network. I suspect that the corporate network overlords have monitoring / security measures in place that interfere, but I have not yet been able to diagnose exactly what causes the reset (I don't get the reset when connected to a "regular" ISP).

When I use wget(1), unless I use '-t 1', it retries and fails (with 'Connection reset by peer'). When I use retry counts of '-t 2' or '-t 3' then it gets some of the file, but hangs waiting for the rest. If I use '-t 4' (or more), it succeeds. Hints welcome on diagnosing the cause of the reset (note: I do not have access to packet dumps on freebsd.org or the corporate routes - only on the FreeBSD workstation endpoint).

When I change to https://distcache.freebsd.org (instead of http), there is a certificate error because 'distcache' is not a valid alternate name according to the public cert (the FreeBSD project needs to update the cert to add 'distcache' as an alternative name to fix that). Changing 'distcache' to 'pkg' in the hostname gets past that issue.

So that (use https://pkg.freebsd.org/...) seems to be a workaround at this time for the 'http://distcache.freebsd.org' issue.