Failed to download host image in CI: helios / build TUF repo

[andrewjstone]

This test failed on a CI run on #8517:

https://github.com/oxidecomputer/omicron/pull/8517/checks?check_run_id=45323384328

Log showing the specific test failure:

https://buildomat.eng.oxide.computer/wg/0/details/01JZ8ZHJ4XVXCS5GAWKKXC11Q9/tJkl0XS27a36jT8mKQiplMj26YHk8lrIySvMrYusjb4QhsMM/01JZ8ZJY7PC0QABNFQBDRPWBR5#S4145

Excerpt from the log showing the failure:

host-image] Jul 03 21:21:54.669 INFO O\| Jul 03 21:21:54.669 INFO O\| Download: 17108/22884 items  148.4/207.3MB  71% complete
--
4145 | 2025-07-03T21:21:56.525Z | [      host-image] Jul 03 21:21:54.687 INFO O\| Jul 03 21:21:54.687 INFO E\| Errors were encountered while attempting to retrieve package or file data for
4146 | 2025-07-03T21:21:56.525Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.687 INFO E\| the requested operation.
4147 | 2025-07-03T21:21:56.530Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| Details follow:
4148 | 2025-07-03T21:21:56.530Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| pkg://helios-dev/system/library@0.5.11,5.11-2.0.23466:20250703T143438Z
4149 | 2025-07-03T21:21:56.530Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 1: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4150 | 2025-07-03T21:21:56.530Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/45561808630cf63e5fabc664024a883a40cf549e' (happened 3 times)
4151 | 2025-07-03T21:21:56.530Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 2: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4152 | 2025-07-03T21:21:56.530Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/588380daca3b7f47e15b7b75981482307f691cb8' (happened 3 times)
4153 | 2025-07-03T21:21:56.530Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 3: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4154 | 2025-07-03T21:21:56.530Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/558854ab115fe87591da9d803243c2c41749596d' (happened 3 times)
4155 | 2025-07-03T21:21:56.530Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 4: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4156 | 2025-07-03T21:21:56.530Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/2e3012caf0bf6e40504e9d2076ba796548713c47' (happened 3 times)
4157 | 2025-07-03T21:21:56.530Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 5: Framework stall:
4158 | 2025-07-03T21:21:56.530Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/1c7057316b39e5e8dca7bee64b47838e1edd28a7'
4159 | 2025-07-03T21:21:56.530Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 6: Framework stall:
4160 | 2025-07-03T21:21:56.530Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/6e1babf526ebc0712f3bf37df931791d24df1ba1'
4161 | 2025-07-03T21:21:56.530Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 7: Framework stall:
4162 | 2025-07-03T21:21:56.530Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/239b8357c815fe07d408432a66e2ddd7acb6fb1f'
4163 | 2025-07-03T21:21:56.530Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 8: Framework stall:
4164 | 2025-07-03T21:21:56.530Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/b3cefca89be497b97fe1a8c397d9df3a81232469'
4165 | 2025-07-03T21:21:56.530Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 9: Framework stall:
4166 | 2025-07-03T21:21:56.530Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/87a5017d1d75e354f6f797338f1e39dccfd6a6e1'
4167 | 2025-07-03T21:21:56.530Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 10: Framework stall:
4168 | 2025-07-03T21:21:56.530Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/1980e52d40c1348e19db6914625032d58ec0f4d2'
4169 | 2025-07-03T21:21:56.530Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 11: Framework stall:
4170 | 2025-07-03T21:21:56.530Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/4ec8422d11086f58a810326f57f01b070da9b4f8'
4171 | 2025-07-03T21:21:56.530Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 12: Framework stall:
4172 | 2025-07-03T21:21:56.530Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/ca8ce9ea77a45d1a2bbadd179a70b5fadf2c0d6f'
4173 | 2025-07-03T21:21:56.530Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 13: Framework stall:
4174 | 2025-07-03T21:21:56.530Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/3883f8d860d0179c4ae0cbbff85aed28c722b107'
4175 | 2025-07-03T21:21:56.530Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 14: Framework stall:
4176 | 2025-07-03T21:21:56.531Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/02eaa693b5b4b79bd048c40e7ca2882a937b0cee'
4177 | 2025-07-03T21:21:56.531Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 15: Framework stall:
4178 | 2025-07-03T21:21:56.531Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/420afd281ce3e04e0d4e81234a6d714dc8c84f97'
4179 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 16: Framework stall:
4180 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/efca3b5f1d84a26f6c810558c3186d574db47625'
4181 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 17: Framework stall:
4182 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/b30e1f5d1f3894cf829d3112bdaed11869eb213d'
4183 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 18: Framework stall:
4184 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/8514d1e63c08be93964677b91cbb68dbeeba7c9f'
4185 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 19: Framework stall:
4186 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/3bc51b30677c107b919f54b2ae7723d440768f39'
4187 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 20: Framework stall:
4188 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/755d1e89072d7e83409b262b85ba39a4066fa217'
4189 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 21: Framework stall:
4190 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/2e3012caf0bf6e40504e9d2076ba796548713c47'
4191 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 22: Framework stall:
4192 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/558854ab115fe87591da9d803243c2c41749596d'
4193 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 23: Framework stall:
4194 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/588380daca3b7f47e15b7b75981482307f691cb8'
4195 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 24: Framework stall:
4196 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/45561808630cf63e5fabc664024a883a40cf549e'
4197 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 25: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4198 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/01430e10895ed15cc1828f3f3bda9a05f5934c23'
4199 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 26: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4200 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/a78d97ab59316b2a4d78186641a4ca6dcf6588e0'
4201 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 27: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4202 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/1901e73dde0c3ef53a9c0a2bb6f7fb3b21d95340'
4203 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 28: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4204 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/f8f7bcb36217b36949246494c80bd4dc32b68c50'
4205 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 29: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4206 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/bd05fdd495e9f7fa29fe83bda0a8bfbfbe1cff85'
4207 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 30: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4208 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/40845d554a27e5aeafaab6c842ded0f50eb14c3b'
4209 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 31: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4210 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/acde27d72c186378d5a3683cf24af2c802c6b02f'
4211 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 32: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4212 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/a90358ceb066b8ec36cc287613940785930f39fb'
4213 | 2025-07-03T21:21:56.536Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| 33: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4214 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.688 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/dcded306a45a74a9bd3b03f4f076d53026eea45c'
4215 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.688 INFO E\| 34: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4216 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/f20710db6bf396974a2e355ad0e368fc91f3dd22'
4217 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.688 INFO E\| 35: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4218 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/6ae5e6cf99e1bbe3d1d29c7bb4db39baac750149'
4219 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.688 INFO E\| 36: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4220 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/1b40f28eb54e6e5b620e5c3172ae7a971698ab8a'
4221 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.688 INFO E\| 37: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4222 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.688 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/4509361c560a9c31a64e4dfe97b8032e5855396b'
4223 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.688 INFO E\| 38: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4224 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/a2f0d3acf767af1c102ff6970aebe8b281d9bd35'
4225 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| 39: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4226 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/6cddc9abe6379892a3be666d38203e8edab316c0'
4227 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| 40: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4228 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/b71c1309636d896ef83ae77f5c415448edcfb38f'
4229 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| 41: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4230 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/7378fb226d689c833c4f527933b9027580364b06'
4231 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| 42: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4232 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/96cdf31dc6a3a588c119ac051032a89a1a31fb98'
4233 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| 43: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4234 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/9d625b307167a50178e860a8431afd597fb4e856'
4235 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| 44: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4236 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/b7ca23a84588f12ff9e26c3ab5453f36399fdfb4'
4237 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| 45: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4238 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/755d1e89072d7e83409b262b85ba39a4066fa217' (happened 2 times)
4239 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| 46: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4240 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/3bc51b30677c107b919f54b2ae7723d440768f39' (happened 2 times)
4241 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| 47: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4242 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/8514d1e63c08be93964677b91cbb68dbeeba7c9f' (happened 2 times)
4243 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| 48: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4244 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/b30e1f5d1f3894cf829d3112bdaed11869eb213d' (happened 2 times)
4245 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| 49: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4246 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/efca3b5f1d84a26f6c810558c3186d574db47625' (happened 2 times)
4247 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| 50: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4248 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/420afd281ce3e04e0d4e81234a6d714dc8c84f97' (happened 2 times)
4249 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| 51: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4250 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/02eaa693b5b4b79bd048c40e7ca2882a937b0cee' (happened 2 times)
4251 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| 52: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4252 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/3883f8d860d0179c4ae0cbbff85aed28c722b107' (happened 2 times)
4253 | 2025-07-03T21:21:56.537Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| 53: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4254 | 2025-07-03T21:21:56.542Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/ca8ce9ea77a45d1a2bbadd179a70b5fadf2c0d6f' (happened 2 times)
4255 | 2025-07-03T21:21:56.542Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| 54: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4256 | 2025-07-03T21:21:56.542Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/4ec8422d11086f58a810326f57f01b070da9b4f8' (happened 2 times)
4257 | 2025-07-03T21:21:56.542Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| 55: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4258 | 2025-07-03T21:21:56.542Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/1980e52d40c1348e19db6914625032d58ec0f4d2' (happened 2 times)
4259 | 2025-07-03T21:21:56.542Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| 56: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4260 | 2025-07-03T21:21:56.542Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/87a5017d1d75e354f6f797338f1e39dccfd6a6e1' (happened 2 times)
4261 | 2025-07-03T21:21:56.542Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| 57: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4262 | 2025-07-03T21:21:56.542Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/b3cefca89be497b97fe1a8c397d9df3a81232469' (happened 2 times)
4263 | 2025-07-03T21:21:56.542Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| 58: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4264 | 2025-07-03T21:21:56.542Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/239b8357c815fe07d408432a66e2ddd7acb6fb1f' (happened 2 times)
4265 | 2025-07-03T21:21:56.542Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| 59: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4266 | 2025-07-03T21:21:56.542Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/6e1babf526ebc0712f3bf37df931791d24df1ba1' (happened 2 times)
4267 | 2025-07-03T21:21:56.542Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| 60: Framework error: code: E_OPERATION_TIMEOUTED (28) reason: Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
4268 | 2025-07-03T21:21:56.542Z | [      host-image] Jul 03 21:21:54.689 INFO O\| Jul 03 21:21:54.689 INFO E\| URL: 'https://pkg.oxide.computer/helios/2/dev/helios-dev/file/1/1c7057316b39e5e8dca7bee64b47838e1edd28a7' (happened 2 times)
4269 | 2025-07-03T21:22:01.182Z | [      host-image] Jul 03 21:21:59.343 INFO O\| Jul 03 21:21:59.343 CRIT fatal error: exec ["/usr/bin/pkg", "-R", "/rpool/images/build/host/work/sled/ramdisk", "install", "/developer/debug/mdb", "/system/kernel/dtrace/providers", "/system/network", "/system/microcode/oxide", "/driver/network/cxgbe", "/driver/network/e1000g", "/driver/network/igb", "/driver/network/ixgbe", "/driver/network/rge", "/driver/storage/nvme", "/developer/macro/cpp", "/system/library/gcc-runtime", "/system/library/g++-runtime", "/developer/linker", "/diagnostic/diskinfo", "/diagnostic/pci", "/diagnostic/cpu-counters", "/system/data/hardware-registry", "/system/extended-system-utilities", "/web/curl", "/text/column", "/text/less", "/text/looker", "/ooce/util/jq", "/system/watch", "/editor/vim", "/terminal/resize", "/system/data/urxvt-terminfo", "/network/netcat", "/network/rsync", "/network/snoop", "/network/overwatch", "/compress/gzip", "/locale/en", "/system/network/routing", "/driver/cpu/amd/zen", "/driver/gpio/amd/zen", "/system/gpio", "/system/bhyve", "/system/library/bhyve", "/service/network/chrony", "/driver/misc/tofino", "/library/expat", "/system/library/pcap", "/network/dns/bind", "/network/openssh-server", "oxide/platform-identity-cacerts", "/consolidation/oxide/omicron-release-incorporation@15.0.0.0"]: failed ExitStatus(unix_wait_status(256))
4270 | 2025-07-03T21:22:01.186Z | [      host-image] Error: exec ["pfexec", "/work/oxidecomputer/helios/projects/image-builder/target/debug/image-builder", "build", "-d", "rpool/images/build/host", "-g", "sled", "-T", "/work/oxidecomputer/helios/image/templates", "-E", "/tmp/.tmpH7P5dx/proto/host/root", "-F", "genproto=/work/oxidecomputer/helios/tmp/image.host/genproto.json", "-E", "/work/oxidecomputer/helios/tmp/image.host/omicron1", "-E", "/work/oxidecomputer/helios/projects", "-F", "publisher_0_name=helios-dev", "-F", "publisher_0_url+=file:///work/incorporation.p5p", "-F", "publisher_0_url+=https://pkg.oxide.computer/helios/2/dev/", "-F", "baud=3000000", "-F", "omicron1", "-F", "optever=0.37.386", "-F", "extra_packages+=/consolidation/oxide/omicron-release-incorporation@15.0.0.0", "-n", "ramdisk-01-os", "--fullreset"]: failed ExitStatus(unix_wait_status(256))
4271 | 2025-07-03T21:22:01.192Z | [      host-image]
4272 | 2025-07-03T21:22:01.192Z | [      host-image] real    13:52.993703052
4273 | 2025-07-03T21:22:01.196Z | [      host-image] user     7:39.934908278
4274 | 2025-07-03T21:22:01.196Z | [      host-image] sys        21.222766178
4275 | 2025-07-03T21:22:01.196Z | [      host-image] trap        0.412329212
4276 | 2025-07-03T21:22:01.196Z | [      host-image] tflt        0.043762512
4277 | 2025-07-03T21:22:01.196Z | [      host-image] dflt        0.019319915
4278 | 2025-07-03T21:22:01.196Z | [      host-image] kflt        0.049850294
4279 | 2025-07-03T21:22:01.196Z | [      host-image] lock    13:51.816398748
4280 | 2025-07-03T21:22:01.196Z | [      host-image] slp   1:26:08.427761908
4281 | 2025-07-03T21:22:01.196Z | [      host-image] lat      2:52.570424359
4282 | 2025-07-03T21:22:01.196Z | [      host-image] stop        0.002889108
4283 | 2025-07-03T21:22:01.196Z | Jul 03 21:21:59.355 ERRO [host-image] process exited with exit status: 1 (833.003366732s)
4284 | 2025-07-03T21:22:01.533Z | Error: command `IMAGE_DATASET=rpool/images/build/host ptime -m /work/oxidecomputer/helios/helios-build experiment-image -o /work/os-host -F 'optever=0.37.386' -P /tmp/.tmpH7P5dx/proto/host/root -N 'ci 3f50db7/${os_short_commit} 2025-07-03 20:41' -s host -B -F 'extra_packages+=/consolidation/oxide/omicron-release-incorporation@15.0.0.0' -p 'helios-dev=file:///work/incorporation.p5p' -p 'helios-dev=https://pkg.oxide.computer/helios/2/dev/'` exited with exit status: 1
4285 | 2025-07-03T21:22:01.601Z | process exited: duration 3098621 ms, exit code 1
 

[bnaecker]

I saw this or something very similar on #9103, here