Skip to content

[sled-agent] route SIGCHLD to dedicated signal-handling thread #9982

Merged
hawkw merged 5 commits intomainfrom
eliza/sled-agent-signal-thread
Mar 12, 2026
Merged

[sled-agent] route SIGCHLD to dedicated signal-handling thread #9982
hawkw merged 5 commits intomainfrom
eliza/sled-agent-signal-thread

Conversation

@hawkw
Copy link
Member

@hawkw hawkw commented Mar 5, 2026 •
edited
Loading

As described in #9849, receiving a handled Unix signal on an application
thread interrupts whatever that thread is presently doing, which
interferes with IPCC communication (see oxidecomputer/stlouis#922).
Since sled-agent stends to spawn a large number of child processes,
many of which are short-lived, and it uses tokio::process to manage
these children, we receive a lot of SIGCHLDs when our child processes
exit. These get delivered to any arbitrary thread in the process, which
can mess up IPCC stuff.

Therefore, this branch changes sled-agent to use the new
OxideBuilder::signal_thread API added in
oxidecomputer/oxide-tokio-rt#4 to set up a dedicated thread outside the
runtime, which will block in sigsuspend in a loop, and mask out
SIGCHLD on all other threads in the process, ensuring that receiving
a SIGCHLD doesn't interfere with other operations.

This required updating oxide-tokio-rt to 0.1.3 in order to pick up the
new API. This in turn also required updating our dependency on the nix
crate from 0.30 to 0.31, as I used the latest version in
oxide-tokio-rt and it's part of the public API for the new feature.
The nix update is easy for us as the only breaking change here is the
removal of Eq and PartialEq implementations for SigHandler, which
we are not using --- see their changelog.

Closes #9849

hawkw added 2 commits March 5, 2026 09:42
@hawkw
Update oxide-tokio-rt to 0.1.3, nix to 0.31
7c46d7f 
This is necessary to pick up oxidecomputer/oxide-tokio-rt#4, which adds
the API for configuring a dedicated signal-handling thread outside the
Tokio runtime. That API uses the latest version of `nix`, which is 0.31.
We presently depend on 0.30. The `nix` update is easy for us as the only
breaking change here is the removal of `Eq` and `PartialEq`
implementations for `SigHandler`, which we are not using --- see [their
changelog][1].

[1]:
https://github.com/nix-rust/nix/blob/bf1d0e9707189422f546e398594fa1a51a772d9d/CHANGELOG.md#0310---2026-01-22
@hawkw
[sled-agent] route SIGCHLD to dedicated signal-handling thread
e7f1c1d 
As described in #9849, receiving a handled Unix signal on an application
thread interrupts whatever that thread is presently doing, which
interferes with IPCC communication (see oxidecomputer/stlouis#922).
Since `sled-agent` stends to spawn a large number of child processes,
many of which are short-lived, and it uses `tokio::process` to manage
these children, we receive a lot of `SIGCHLD`s when our child processes
exit. These get delivered to any arbitrary thread in the process, which
can mess up IPCC stuff. Therefore, use the new
`OxideBuilder::signal_thread` API added in
oxidecomputer/oxide-tokio-rt#4 to set up a dedicated thread outside the
runtime, which will block in `sigsuspend` in a loop, and mask out
`SIGCHLD` on all other threads in the process.
@hawkw
Copy link
Member Author

hawkw commented Mar 5, 2026

@dancrossnyc, @jgallagher et al., I don't suppose there's a procedure to reproduce the IPCC issues? I'd like to be able to test that this change actually resolves the problem, if possible.

@dancrossnyc
Copy link
Contributor

dancrossnyc commented Mar 6, 2026

@dancrossnyc, @jgallagher et al., I don't suppose there's a procedure to reproduce the IPCC issues? I'd like to be able to test that this change actually resolves the problem, if possible.

The variant I've been using is available as a pull request on John's reproducer repo:
oxidecomputer/john-ipcc-signals#1

But that's not testing against omicron proper.

To probe the effect of the oxide-tokio-rt change within omicron, I'd probably instrument it to spawn a tokio task that loops and, at some frequency (say, 20 Hz), fork's and immediately exits in the child, and see if it impacts IPCC flows by looking at the kernel IPCC debug message buffer; if IPCC send and/or receive has been interrupted by signal delivery, you should see a notice.

@hawkw
Copy link
Member Author

hawkw commented Mar 9, 2026

okay, i _think) that this means we are doing IPCC successfuly whilst the code I added in 1c8a990 is getting loads of SIGCHLD, but will leave it there for a bit to keep poking at it:

eliza@castle ~ $ pilot -r london host ls
INTERFACE    IP                        SERIAL      IMAGE
london_host0 fe80::eaea:6aff:fe09:8567 BRM42220030 ci 1c8a990/a064bad 2026-03-09 18:20
london_host1 fe80::eaea:6aff:fe09:865a BRM42220036 ci 1c8a990/a064bad 2026-03-09 18:20
eliza@castle ~ $ pilot -r london host login BRM42220036

    #####
   ##   ##
  ##   # ##  ##   ##
  ##  #  ##   ## ##     Oxide Computer Company
  ## #   ##    ###      Engineering
   ##   ##    ## ##
    #####    ##   ##    Compute Sled

BRM42220036 # mdb -ke ::ipcc_dbgmsg
1986 Dec 28 00:00:41 Using device '/devices/huashan@0,0/dwu@1:0,cu'
1986 Dec 28 00:00:41 Using SP interrupt DPIO '/dev/dpio/ipcc-sp-intr'
1986 Dec 28 00:00:41 Popping module ttcompat
1986 Dec 28 00:00:41 Popping module ldterm
1986 Dec 28 00:00:41
------> Sending IPCC command 0x4 (IDENT), attempt 1/10
1986 Dec 28 00:00:41      OUT  00:  cc19de01 01000000 66000000 00000000 | ........f.......
1986 Dec 28 00:00:41      OUT  10:  0431f2                              | .1.
1986 Dec 28 00:00:41 COBS OUT  00:  06cc19de 01010101 02660101 01010101 | .........f......
1986 Dec 28 00:00:41 COBS OUT  10:  040431f2                            | ..1.
1986 Dec 28 00:00:41  COBS IN  00:  06cc19de 01010101 02660101 0101010e | .........f......
1986 Dec 28 00:00:41  COBS IN  10:  80043931 332d3030 30303031 39010101 | ..913-0000019...
1986 Dec 28 00:00:41  COBS IN  20:  01010101 01010101 01010101 01010101 | ................
1986 Dec 28 00:00:41  COBS IN  30:  01010101 01010101 01010101 01010101 | ................
1986 Dec 28 00:00:41  COBS IN  40:  01010101 02060101 0c42524d 34323232 | .........BRM4222
1986 Dec 28 00:00:41  COBS IN  50:  30303336 01010101 01010101 01010101 | 0036............
1986 Dec 28 00:00:41  COBS IN  60:  01010101 01010101 01010101 01010101 | ................
1986 Dec 28 00:00:41  COBS IN  70:  01010101 01010101 01010103 5444     | ............TD
1986 Dec 28 00:00:41       IN  00:  cc19de01 01000000 66000000 00000080 | ........f.......
1986 Dec 28 00:00:41       IN  10:  04393133 2d303030 30303139 00000000 | .913-0000019....
1986 Dec 28 00:00:41       IN  20:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:41       IN  30:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:41       IN  40:  00000000 06000000 42524d34 32323230 | ........BRM42220
1986 Dec 28 00:00:41       IN  50:  30333600 00000000 00000000 00000000 | 036.............
1986 Dec 28 00:00:41       IN  60:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:41       IN  70:  00000000 00000000 00000054 44       | ...........TD
1986 Dec 28 00:00:41 Additional data length: 0x6a
1986 Dec 28 00:00:41  DATA IN  00:  3931332d 30303030 30313900 00000000 | 913-0000019.....
1986 Dec 28 00:00:41  DATA IN  10:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:41  DATA IN  20:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:41  DATA IN  30:  00000006 00000042 524d3432 32323030 | .......BRM422200
1986 Dec 28 00:00:41  DATA IN  40:  33360000 00000000 00000000 00000000 | 36..............
1986 Dec 28 00:00:41  DATA IN  50:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:41  DATA IN  60:  00000000 00000000 0000              | ..........
1986 Dec 28 00:00:41 Popping module ttcompat
1986 Dec 28 00:00:41 Popping module ldterm
1986 Dec 28 00:00:41
------> Sending IPCC command 0xe (KEYLOOKUP), attempt 1/10
1986 Dec 28 00:00:41 Additional data length: 0x3
1986 Dec 28 00:00:41 DATA OUT  0:  020800                              | ...
1986 Dec 28 00:00:41      OUT  00:  cc19de01 01000000 67000000 00000000 | ........g.......
1986 Dec 28 00:00:41      OUT  10:  0e020800 46d0                       | ....F.
1986 Dec 28 00:00:41 COBS OUT  00:  06cc19de 01010101 02670101 01010101 | .........g......
1986 Dec 28 00:00:41 COBS OUT  10:  040e0208 0346d0                     | .....F.
1986 Dec 28 00:00:41  COBS IN  00:  06cc19de 01010101 02670101 01010103 | .........g......
1986 Dec 28 00:00:41  COBS IN  10:  800a0249 01010101 01010302 cb       | ...I.........
1986 Dec 28 00:00:41       IN  00:  cc19de01 01000000 67000000 00000080 | ........g.......
1986 Dec 28 00:00:41       IN  10:  0a004900 00000000 000002cb          | ..I.........
1986 Dec 28 00:00:41 Additional data length: 0x9
1986 Dec 28 00:00:41  DATA IN  0:  00490000 00000000 00                | .I.......
1986 Dec 28 00:00:42 Popping module ttcompat
1986 Dec 28 00:00:42 Popping module ldterm
1986 Dec 28 00:00:42
------> Sending IPCC command 0x4 (IDENT), attempt 1/10
1986 Dec 28 00:00:42      OUT  00:  cc19de01 01000000 68000000 00000000 | ........h.......
1986 Dec 28 00:00:42      OUT  10:  043305                              | .3.
1986 Dec 28 00:00:42 COBS OUT  00:  06cc19de 01010101 02680101 01010101 | .........h......
1986 Dec 28 00:00:42 COBS OUT  10:  04043305                            | ..3.
1986 Dec 28 00:00:42  COBS IN  00:  06cc19de 01010101 02680101 0101010e | .........h......
1986 Dec 28 00:00:42  COBS IN  10:  80043931 332d3030 30303031 39010101 | ..913-0000019...
1986 Dec 28 00:00:42  COBS IN  20:  01010101 01010101 01010101 01010101 | ................
1986 Dec 28 00:00:42  COBS IN  30:  01010101 01010101 01010101 01010101 | ................
1986 Dec 28 00:00:42  COBS IN  40:  01010101 02060101 0c42524d 34323232 | .........BRM4222
1986 Dec 28 00:00:42  COBS IN  50:  30303336 01010101 01010101 01010101 | 0036............
1986 Dec 28 00:00:42  COBS IN  60:  01010101 01010101 01010101 01010101 | ................
1986 Dec 28 00:00:42  COBS IN  70:  01010101 01010101 01010103 562b     | ............V+
1986 Dec 28 00:00:42       IN  00:  cc19de01 01000000 68000000 00000080 | ........h.......
1986 Dec 28 00:00:42       IN  10:  04393133 2d303030 30303139 00000000 | .913-0000019....
1986 Dec 28 00:00:42       IN  20:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:42       IN  30:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:42       IN  40:  00000000 06000000 42524d34 32323230 | ........BRM42220
1986 Dec 28 00:00:42       IN  50:  30333600 00000000 00000000 00000000 | 036.............
1986 Dec 28 00:00:42       IN  60:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:42       IN  70:  00000000 00000000 00000056 2b       | ...........V+
1986 Dec 28 00:00:42 Additional data length: 0x6a
1986 Dec 28 00:00:42  DATA IN  00:  3931332d 30303030 30313900 00000000 | 913-0000019.....
1986 Dec 28 00:00:42  DATA IN  10:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:42  DATA IN  20:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:42  DATA IN  30:  00000006 00000042 524d3432 32323030 | .......BRM422200
1986 Dec 28 00:00:42  DATA IN  40:  33360000 00000000 00000000 00000000 | 36..............
1986 Dec 28 00:00:42  DATA IN  50:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:42  DATA IN  60:  00000000 00000000 0000              | ..........
1986 Dec 28 00:00:42 Popping module ttcompat
1986 Dec 28 00:00:42 Popping module ldterm
1986 Dec 28 00:00:42
------> Sending IPCC command 0xe (KEYLOOKUP), attempt 1/10
1986 Dec 28 00:00:42 Additional data length: 0x3
1986 Dec 28 00:00:42 DATA OUT  0:  020800                              | ...
1986 Dec 28 00:00:42      OUT  00:  cc19de01 01000000 69000000 00000000 | ........i.......
1986 Dec 28 00:00:42      OUT  10:  0e020800 48e8                       | ....H.
1986 Dec 28 00:00:42 COBS OUT  00:  06cc19de 01010101 02690101 01010101 | .........i......
1986 Dec 28 00:00:42 COBS OUT  10:  040e0208 0348e8                     | .....H.
1986 Dec 28 00:00:42  COBS IN  00:  06cc19de 01010101 02690101 01010103 | .........i......
1986 Dec 28 00:00:42  COBS IN  10:  800a0249 01010101 01010304 ef       | ...I.........
1986 Dec 28 00:00:42       IN  00:  cc19de01 01000000 69000000 00000080 | ........i.......
1986 Dec 28 00:00:42       IN  10:  0a004900 00000000 000004ef          | ..I.........
1986 Dec 28 00:00:42 Additional data length: 0x9
1986 Dec 28 00:00:42  DATA IN  0:  00490000 00000000 00                | .I.......
1986 Dec 28 00:00:44 Popping module ttcompat
1986 Dec 28 00:00:44 Popping module ldterm
1986 Dec 28 00:00:44
------> Sending IPCC command 0x4 (IDENT), attempt 1/10
1986 Dec 28 00:00:44      OUT  00:  cc19de01 01000000 6a000000 00000000 | ........j.......
1986 Dec 28 00:00:44      OUT  10:  043517                              | .5.
1986 Dec 28 00:00:44 COBS OUT  00:  06cc19de 01010101 026a0101 01010101 | .........j......
1986 Dec 28 00:00:44 COBS OUT  10:  04043517                            | ..5.
1986 Dec 28 00:00:44  COBS IN  00:  06cc19de 01010101 026a0101 0101010e | .........j......
1986 Dec 28 00:00:44  COBS IN  10:  80043931 332d3030 30303031 39010101 | ..913-0000019...
1986 Dec 28 00:00:44  COBS IN  20:  01010101 01010101 01010101 01010101 | ................
1986 Dec 28 00:00:44  COBS IN  30:  01010101 01010101 01010101 01010101 | ................
1986 Dec 28 00:00:44  COBS IN  40:  01010101 02060101 0c42524d 34323232 | .........BRM4222
1986 Dec 28 00:00:44  COBS IN  50:  30303336 01010101 01010101 01010101 | 0036............
1986 Dec 28 00:00:44  COBS IN  60:  01010101 01010101 01010101 01010101 | ................
1986 Dec 28 00:00:44  COBS IN  70:  01010101 01010101 01010103 5812     | ............X.
1986 Dec 28 00:00:44       IN  00:  cc19de01 01000000 6a000000 00000080 | ........j.......
1986 Dec 28 00:00:44       IN  10:  04393133 2d303030 30303139 00000000 | .913-0000019....
1986 Dec 28 00:00:44       IN  20:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:44       IN  30:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:44       IN  40:  00000000 06000000 42524d34 32323230 | ........BRM42220
1986 Dec 28 00:00:44       IN  50:  30333600 00000000 00000000 00000000 | 036.............
1986 Dec 28 00:00:44       IN  60:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:44       IN  70:  00000000 00000000 00000058 12       | ...........X.
1986 Dec 28 00:00:44 Additional data length: 0x6a
1986 Dec 28 00:00:44  DATA IN  00:  3931332d 30303030 30313900 00000000 | 913-0000019.....
1986 Dec 28 00:00:44  DATA IN  10:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:44  DATA IN  20:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:44  DATA IN  30:  00000006 00000042 524d3432 32323030 | .......BRM422200
1986 Dec 28 00:00:44  DATA IN  40:  33360000 00000000 00000000 00000000 | 36..............
1986 Dec 28 00:00:44  DATA IN  50:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:44  DATA IN  60:  00000000 00000000 0000              | ..........
1986 Dec 28 00:00:44 Popping module ttcompat
1986 Dec 28 00:00:44 Popping module ldterm
1986 Dec 28 00:00:44
------> Sending IPCC command 0xe (KEYLOOKUP), attempt 1/10
1986 Dec 28 00:00:44 Additional data length: 0x3
1986 Dec 28 00:00:44 DATA OUT  0:  020800                              | ...
1986 Dec 28 00:00:44      OUT  00:  cc19de01 01000000 6b000000 00000000 | ........k.......
1986 Dec 28 00:00:44      OUT  10:  0e020800 4a01                       | ....J.
1986 Dec 28 00:00:44 COBS OUT  00:  06cc19de 01010101 026b0101 01010101 | .........k......
1986 Dec 28 00:00:44 COBS OUT  10:  040e0208 034a01                     | .....J.
1986 Dec 28 00:00:44  COBS IN  00:  06cc19de 01010101 026b0101 01010103 | .........k......
1986 Dec 28 00:00:44  COBS IN  10:  800a0249 01010101 01010306 14       | ...I.........
1986 Dec 28 00:00:44       IN  00:  cc19de01 01000000 6b000000 00000080 | ........k.......
1986 Dec 28 00:00:44       IN  10:  0a004900 00000000 00000614          | ..I.........
1986 Dec 28 00:00:44 Additional data length: 0x9
1986 Dec 28 00:00:44  DATA IN  0:  00490000 00000000 00                | .I.......
1986 Dec 28 00:00:46 Popping module ttcompat
1986 Dec 28 00:00:46 Popping module ldterm
1986 Dec 28 00:00:46
------> Sending IPCC command 0x4 (IDENT), attempt 1/10
1986 Dec 28 00:00:46      OUT  00:  cc19de01 01000000 6c000000 00000000 | ........l.......
1986 Dec 28 00:00:46      OUT  10:  043729                              | .7)
1986 Dec 28 00:00:46 COBS OUT  00:  06cc19de 01010101 026c0101 01010101 | .........l......
1986 Dec 28 00:00:46 COBS OUT  10:  04043729                            | ..7)
1986 Dec 28 00:00:46  COBS IN  00:  06cc19de 01010101 026c0101 0101010e | .........l......
1986 Dec 28 00:00:46  COBS IN  10:  80043931 332d3030 30303031 39010101 | ..913-0000019...
1986 Dec 28 00:00:46  COBS IN  20:  01010101 01010101 01010101 01010101 | ................
1986 Dec 28 00:00:46  COBS IN  30:  01010101 01010101 01010101 01010101 | ................
1986 Dec 28 00:00:46  COBS IN  40:  01010101 02060101 0c42524d 34323232 | .........BRM4222
1986 Dec 28 00:00:46  COBS IN  50:  30303336 01010101 01010101 01010101 | 0036............
1986 Dec 28 00:00:46  COBS IN  60:  01010101 01010101 01010101 01010101 | ................
1986 Dec 28 00:00:46  COBS IN  70:  01010101 01010101 01010103 5af8     | ............Z.
1986 Dec 28 00:00:46       IN  00:  cc19de01 01000000 6c000000 00000080 | ........l.......
1986 Dec 28 00:00:46       IN  10:  04393133 2d303030 30303139 00000000 | .913-0000019....
1986 Dec 28 00:00:46       IN  20:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:46       IN  30:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:46       IN  40:  00000000 06000000 42524d34 32323230 | ........BRM42220
1986 Dec 28 00:00:46       IN  50:  30333600 00000000 00000000 00000000 | 036.............
1986 Dec 28 00:00:46       IN  60:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:46       IN  70:  00000000 00000000 0000005a f8       | ...........Z.
1986 Dec 28 00:00:46 Additional data length: 0x6a
1986 Dec 28 00:00:46  DATA IN  00:  3931332d 30303030 30313900 00000000 | 913-0000019.....
1986 Dec 28 00:00:46  DATA IN  10:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:46  DATA IN  20:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:46  DATA IN  30:  00000006 00000042 524d3432 32323030 | .......BRM422200
1986 Dec 28 00:00:46  DATA IN  40:  33360000 00000000 00000000 00000000 | 36..............
1986 Dec 28 00:00:46  DATA IN  50:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:46  DATA IN  60:  00000000 00000000 0000              | ..........
1986 Dec 28 00:00:46 Popping module ttcompat
1986 Dec 28 00:00:46 Popping module ldterm
1986 Dec 28 00:00:46
------> Sending IPCC command 0xe (KEYLOOKUP), attempt 1/10
1986 Dec 28 00:00:46 Additional data length: 0x3
1986 Dec 28 00:00:46 DATA OUT  0:  020800                              | ...
1986 Dec 28 00:00:46      OUT  00:  cc19de01 01000000 6d000000 00000000 | ........m.......
1986 Dec 28 00:00:46      OUT  10:  0e020800 4c19                       | ....L.
1986 Dec 28 00:00:46 COBS OUT  00:  06cc19de 01010101 026d0101 01010101 | .........m......
1986 Dec 28 00:00:46 COBS OUT  10:  040e0208 034c19                     | .....L.
1986 Dec 28 00:00:46  COBS IN  00:  06cc19de 01010101 026d0101 01010103 | .........m......
1986 Dec 28 00:00:46  COBS IN  10:  800a0249 01010101 01010308 38       | ...I........8
1986 Dec 28 00:00:46       IN  00:  cc19de01 01000000 6d000000 00000080 | ........m.......
1986 Dec 28 00:00:46       IN  10:  0a004900 00000000 00000838          | ..I........8
1986 Dec 28 00:00:46 Additional data length: 0x9
1986 Dec 28 00:00:46  DATA IN  0:  00490000 00000000 00                | .I.......
1986 Dec 28 00:00:47 Popping module ttcompat
1986 Dec 28 00:00:47 Popping module ldterm
1986 Dec 28 00:00:47
------> Sending IPCC command 0x4 (IDENT), attempt 1/10
1986 Dec 28 00:00:47      OUT  00:  cc19de01 01000000 6e000000 00000000 | ........n.......
1986 Dec 28 00:00:47      OUT  10:  04393b                              | .9;
1986 Dec 28 00:00:47 COBS OUT  00:  06cc19de 01010101 026e0101 01010101 | .........n......
1986 Dec 28 00:00:47 COBS OUT  10:  0404393b                            | ..9;
1986 Dec 28 00:00:47  COBS IN  00:  06cc19de 01010101 026e0101 0101010e | .........n......
1986 Dec 28 00:00:47  COBS IN  10:  80043931 332d3030 30303031 39010101 | ..913-0000019...
1986 Dec 28 00:00:47  COBS IN  20:  01010101 01010101 01010101 01010101 | ................
1986 Dec 28 00:00:47  COBS IN  30:  01010101 01010101 01010101 01010101 | ................
1986 Dec 28 00:00:47  COBS IN  40:  01010101 02060101 0c42524d 34323232 | .........BRM4222
1986 Dec 28 00:00:47  COBS IN  50:  30303336 01010101 01010101 01010101 | 0036............
1986 Dec 28 00:00:47  COBS IN  60:  01010101 01010101 01010101 01010101 | ................
1986 Dec 28 00:00:47  COBS IN  70:  01010101 01010101 01010103 5cdf     | ............\.
1986 Dec 28 00:00:47       IN  00:  cc19de01 01000000 6e000000 00000080 | ........n.......
1986 Dec 28 00:00:47       IN  10:  04393133 2d303030 30303139 00000000 | .913-0000019....
1986 Dec 28 00:00:47       IN  20:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:47       IN  30:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:47       IN  40:  00000000 06000000 42524d34 32323230 | ........BRM42220
1986 Dec 28 00:00:47       IN  50:  30333600 00000000 00000000 00000000 | 036.............
1986 Dec 28 00:00:47       IN  60:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:47       IN  70:  00000000 00000000 0000005c df       | ...........\.
1986 Dec 28 00:00:47 Additional data length: 0x6a
1986 Dec 28 00:00:47  DATA IN  00:  3931332d 30303030 30313900 00000000 | 913-0000019.....
1986 Dec 28 00:00:47  DATA IN  10:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:47  DATA IN  20:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:47  DATA IN  30:  00000006 00000042 524d3432 32323030 | .......BRM422200
1986 Dec 28 00:00:47  DATA IN  40:  33360000 00000000 00000000 00000000 | 36..............
1986 Dec 28 00:00:47  DATA IN  50:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:47  DATA IN  60:  00000000 00000000 0000              | ..........
1986 Dec 28 00:00:47 Popping module ttcompat
1986 Dec 28 00:00:47 Popping module ldterm
1986 Dec 28 00:00:47
------> Sending IPCC command 0xe (KEYLOOKUP), attempt 1/10
1986 Dec 28 00:00:47 Additional data length: 0x3
1986 Dec 28 00:00:47 DATA OUT  0:  020800                              | ...
1986 Dec 28 00:00:47      OUT  00:  cc19de01 01000000 6f000000 00000000 | ........o.......
1986 Dec 28 00:00:47      OUT  10:  0e020800 4e31                       | ....N1
1986 Dec 28 00:00:47 COBS OUT  00:  06cc19de 01010101 026f0101 01010101 | .........o......
1986 Dec 28 00:00:47 COBS OUT  10:  040e0208 034e31                     | .....N1
1986 Dec 28 00:00:47  COBS IN  00:  06cc19de 01010101 026f0101 01010103 | .........o......
1986 Dec 28 00:00:47  COBS IN  10:  800a0249 01010101 0101030a 5c       | ...I........\
1986 Dec 28 00:00:47       IN  00:  cc19de01 01000000 6f000000 00000080 | ........o.......
1986 Dec 28 00:00:47       IN  10:  0a004900 00000000 00000a5c          | ..I........\
1986 Dec 28 00:00:47 Additional data length: 0x9
1986 Dec 28 00:00:47  DATA IN  0:  00490000 00000000 00                | .I.......
1986 Dec 28 00:00:53 Popping module ttcompat
1986 Dec 28 00:00:53 Popping module ldterm
1986 Dec 28 00:00:53
------> Sending IPCC command 0xb (ROT), attempt 1/10
1986 Dec 28 00:00:53 Additional data length: 0x9
1986 Dec 28 00:00:53 DATA OUT  0:  00577ea7 01000000 03                | .W~......
1986 Dec 28 00:00:53      OUT  00:  cc19de01 01000000 70000000 00000000 | ........p.......
1986 Dec 28 00:00:53      OUT  10:  0b00577e a7010000 0003c3ce          | ..W~........
1986 Dec 28 00:00:53 COBS OUT  00:  06cc19de 01010101 02700101 01010101 | .........p......
1986 Dec 28 00:00:53 COBS OUT  10:  020b0557 7ea70101 010403c3 ce       | ...W~........
1986 Dec 28 00:00:53  COBS IN  000:  06cc19de 01010101 02700101 01010103 | .........p......
1986 Dec 28 00:00:53  COBS IN  010:  80080557 7ea70101 01d50430 82019e30 | ...W~......0...0
1986 Dec 28 00:00:53  COBS IN  020:  820150a0 03020102 02010230 0506032b | ..P........0...+
1986 Dec 28 00:00:53  COBS IN  030:  65703042 310b3009 06035504 06130255 | ep0B1.0...U....U
1986 Dec 28 00:00:53  COBS IN  040:  53311f30 1d060355 040a0c16 4f786964 | S1.0...U....Oxid
1986 Dec 28 00:00:53  COBS IN  050:  6520436f 6d707574 65722043 6f6d7061 | e Computer Compa
1986 Dec 28 00:00:53  COBS IN  060:  6e793112 30100603 5504030c 09646576 | ny1.0...U....dev
1986 Dec 28 00:00:53  COBS IN  070:  6963652d 69643020 170d3233 30353032 | ice-id0 ..230502
1986 Dec 28 00:00:53  COBS IN  080:  31393439 31315a18 0f393939 39313233 | 194911Z..9999123
1986 Dec 28 00:00:53  COBS IN  090:  31323335 3935395a 3049310b 30090603 | 1235959Z0I1.0...
1986 Dec 28 00:00:53  COBS IN  0a0:  55040613 02555331 1f301d06 0355040a | U....US1.0...U..
1986 Dec 28 00:00:53  COBS IN  0b0:  0c164f78 69646520 436f6d70 75746572 | ..Oxide Computer
1986 Dec 28 00:00:53  COBS IN  0c0:  20436f6d 70616e79 31193017 06035504 |  Company1.0...U.
1986 Dec 28 00:00:53  COBS IN  0d0:  030c1074 72757374 2d71756f 72756d2d | ...trust-quorum-
1986 Dec 28 00:00:53  COBS IN  0e0:  64686530 2a300506 032b6570 0321326b | dhe0*0...+ep.!2k
1986 Dec 28 00:00:53  COBS IN  0f0:  bc3b044e fd47b1ae 9216f59d 0a4821cb | .;.N.G.......H!.
1986 Dec 28 00:00:53       IN  000:  cc19de01 01000000 70000000 00000080 | ........p.......
1986 Dec 28 00:00:53       IN  010:  0800577e a7010000 00043082 019e3082 | ..W~......0...0.
1986 Dec 28 00:00:53       IN  020:  0150a003 02010202 01023005 06032b65 | .P........0...+e
1986 Dec 28 00:00:53       IN  030:  70304231 0b300906 03550406 13025553 | p0B1.0...U....US
1986 Dec 28 00:00:53       IN  040:  311f301d 06035504 0a0c164f 78696465 | 1.0...U....Oxide
1986 Dec 28 00:00:53       IN  050:  20436f6d 70757465 7220436f 6d70616e |  Computer Compan
1986 Dec 28 00:00:53       IN  060:  79311230 10060355 04030c09 64657669 | y1.0...U....devi
1986 Dec 28 00:00:53       IN  070:  63652d69 64302017 0d323330 35303231 | ce-id0 ..2305021
1986 Dec 28 00:00:53       IN  080:  39343931 315a180f 39393939 31323331 | 94911Z..99991231
1986 Dec 28 00:00:53       IN  090:  32333539 35395a30 49310b30 09060355 | 235959Z0I1.0...U
1986 Dec 28 00:00:53       IN  0a0:  04061302 5553311f 301d0603 55040a0c | ....US1.0...U...
1986 Dec 28 00:00:53       IN  0b0:  164f7869 64652043 6f6d7075 74657220 | .Oxide Computer
1986 Dec 28 00:00:53       IN  0c0:  436f6d70 616e7931 19301706 03550403 | Company1.0...U..
1986 Dec 28 00:00:53       IN  0d0:  0c107472 7573742d 71756f72 756d2d64 | ..trust-quorum-d
1986 Dec 28 00:00:53       IN  0e0:  6865302a 30050603 2b657003 21006bbc | he0*0...+ep.!.k.
1986 Dec 28 00:00:53       IN  0f0:  3b044efd 47b1ae92 16f59d0a 4821cb69 | ;.N.G.......H!.i
1986 Dec 28 00:00:53 Additional data length: 0x820
1986 Dec 28 00:00:53  DATA IN  000:  00577ea7 01000000 04308201 9e308201 | .W~......0...0..
1986 Dec 28 00:00:53  DATA IN  010:  50a00302 01020201 02300506 032b6570 | P........0...+ep
1986 Dec 28 00:00:53  DATA IN  020:  3042310b 30090603 55040613 02555331 | 0B1.0...U....US1
1986 Dec 28 00:00:53  DATA IN  030:  1f301d06 0355040a 0c164f78 69646520 | .0...U....Oxide
1986 Dec 28 00:00:53  DATA IN  040:  436f6d70 75746572 20436f6d 70616e79 | Computer Company
1986 Dec 28 00:00:53  DATA IN  050:  31123010 06035504 030c0964 65766963 | 1.0...U....devic
1986 Dec 28 00:00:53  DATA IN  060:  652d6964 3020170d 32333035 30323139 | e-id0 ..23050219
1986 Dec 28 00:00:53  DATA IN  070:  34393131 5a180f39 39393931 32333132 | 4911Z..999912312
1986 Dec 28 00:00:53  DATA IN  080:  33353935 395a3049 310b3009 06035504 | 35959Z0I1.0...U.
1986 Dec 28 00:00:53  DATA IN  090:  06130255 53311f30 1d060355 040a0c16 | ...US1.0...U....
1986 Dec 28 00:00:53  DATA IN  0a0:  4f786964 6520436f 6d707574 65722043 | Oxide Computer C
1986 Dec 28 00:00:53  DATA IN  0b0:  6f6d7061 6e793119 30170603 5504030c | ompany1.0...U...
1986 Dec 28 00:00:53  DATA IN  0c0:  10747275 73742d71 756f7275 6d2d6468 | .trust-quorum-dh
1986 Dec 28 00:00:53  DATA IN  0d0:  65302a30 0506032b 65700321 006bbc3b | e0*0...+ep.!.k.;
1986 Dec 28 00:00:53  DATA IN  0e0:  044efd47 b1ae9216 f59d0a48 21cb6939 | .N.G.......H!.i9
1986 Dec 28 00:00:53  DATA IN  0f0:  8a63979e c5f1c194 bb407935 d3a36230 | .c.......@y5..b0
1986 Dec 28 00:00:53 Popping module ttcompat
1986 Dec 28 00:00:53 Popping module ldterm
1986 Dec 28 00:00:53
------> Sending IPCC command 0xb (ROT), attempt 1/10
1986 Dec 28 00:00:53 Additional data length: 0x29
1986 Dec 28 00:00:53 DATA OUT  00:  00577ea7 01000000 04002cbf aa426eb1 | .W~.......,..Bn.
1986 Dec 28 00:00:53 DATA OUT  10:  cf00a411 85c6fa4c 06c41067 62f24cdb | .......L...gb.L.
1986 Dec 28 00:00:53 DATA OUT  20:  260a8993 319dad42 77                | &...1..Bw
1986 Dec 28 00:00:53      OUT  00:  cc19de01 01000000 71000000 00000000 | ........q.......
1986 Dec 28 00:00:53      OUT  10:  0b00577e a7010000 0004002c bfaa426e | ..W~.......,..Bn
1986 Dec 28 00:00:53      OUT  20:  b1cf00a4 1185c6fa 4c06c410 6762f24c | ........L...gb.L
1986 Dec 28 00:00:53      OUT  30:  db260a89 93319dad 42771b96          | .&...1..Bw..
1986 Dec 28 00:00:53 COBS OUT  00:  06cc19de 01010101 02710101 01010101 | .........q......
1986 Dec 28 00:00:53 COBS OUT  10:  020b0557 7ea70101 01020408 2cbfaa42 | ...W~.......,..B
1986 Dec 28 00:00:53 COBS OUT  20:  6eb1cf1a a41185c6 fa4c06c4 106762f2 | n........L...gb.
1986 Dec 28 00:00:53 COBS OUT  30:  4cdb260a 8993319d ad42771b 96       | L.&...1..Bw..
1986 Dec 28 00:00:53  COBS IN  00:  06cc19de 01010101 02710101 01010103 | .........q......
1986 Dec 28 00:00:53  COBS IN  10:  80080557 7ea70101 01440560 98a87a0f | ...W~....D.`..z.
1986 Dec 28 00:00:53  COBS IN  20:  0f022b2c a1f459d5 70d9c27f d9f4db34 | ..+,..Y.p......4
1986 Dec 28 00:00:53  COBS IN  30:  34103018 843d62bb cc5a2130 1f52e949 | 4.0..=b..Z!0.R.I
1986 Dec 28 00:00:53  COBS IN  40:  87637190 e83a0810 5c615fb4 d44c1e83 | .cq..:..\a_..L..
1986 Dec 28 00:00:53  COBS IN  50:  2c52dd37 844158e6 f2d00db8 0d       | ,R.7.AX......
1986 Dec 28 00:00:53       IN  00:  cc19de01 01000000 71000000 00000080 | ........q.......
1986 Dec 28 00:00:53       IN  10:  0800577e a7010000 00056098 a87a0f0f | ..W~......`..z..
1986 Dec 28 00:00:53       IN  20:  022b2ca1 f459d570 d9c27fd9 f4db3434 | .+,..Y.p......44
1986 Dec 28 00:00:53       IN  30:  10301884 3d62bbcc 5a21301f 52e94987 | .0..=b..Z!0.R.I.
1986 Dec 28 00:00:53       IN  40:  637190e8 3a08105c 615fb4d4 4c1e832c | cq..:..\a_..L..,
1986 Dec 28 00:00:53       IN  50:  52dd3784 4158e6f2 d00db80d          | R.7.AX......
1986 Dec 28 00:00:53 Additional data length: 0x49
1986 Dec 28 00:00:53  DATA IN  00:  00577ea7 01000000 056098a8 7a0f0f02 | .W~......`..z...
1986 Dec 28 00:00:53  DATA IN  10:  2b2ca1f4 59d570d9 c27fd9f4 db343410 | +,..Y.p......44.
1986 Dec 28 00:00:53  DATA IN  20:  3018843d 62bbcc5a 21301f52 e9498763 | 0..=b..Z!0.R.I.c
1986 Dec 28 00:00:53  DATA IN  30:  7190e83a 08105c61 5fb4d44c 1e832c52 | q..:..\a_..L..,R
1986 Dec 28 00:00:53  DATA IN  40:  dd378441 58e6f2d0 0d                | .7.AX....
1986 Dec 28 00:00:54 Popping module ttcompat
1986 Dec 28 00:00:54 Popping module ldterm
1986 Dec 28 00:00:54
------> Sending IPCC command 0xb (ROT), attempt 1/10
1986 Dec 28 00:00:54 Additional data length: 0x9
1986 Dec 28 00:00:54 DATA OUT  0:  00577ea7 01000000 00                | .W~......
1986 Dec 28 00:00:54      OUT  00:  cc19de01 01000000 72000000 00000000 | ........r.......
1986 Dec 28 00:00:54      OUT  10:  0b00577e a7010000 0000c2ef          | ..W~........
1986 Dec 28 00:00:54 COBS OUT  00:  06cc19de 01010101 02720101 01010101 | .........r......
1986 Dec 28 00:00:54 COBS OUT  10:  020b0557 7ea70101 010103c2 ef       | ...W~........
1986 Dec 28 00:00:54  COBS IN  000:  06cc19de 01010101 02720101 01010103 | .........r......
1986 Dec 28 00:00:54  COBS IN  010:  80080557 7ea70101 01ca0130 8201ac30 | ...W~......0...0
1986 Dec 28 00:00:54  COBS IN  020:  82015ea0 03020102 02010130 0506032b | ..^........0...+
1986 Dec 28 00:00:54  COBS IN  030:  65703042 310b3009 06035504 06130255 | ep0B1.0...U....U
1986 Dec 28 00:00:54  COBS IN  040:  53311f30 1d060355 040a0c16 4f786964 | S1.0...U....Oxid
1986 Dec 28 00:00:54  COBS IN  050:  6520436f 6d707574 65722043 6f6d7061 | e Computer Compa
1986 Dec 28 00:00:54  COBS IN  060:  6e793112 30100603 5504030c 09646576 | ny1.0...U....dev
1986 Dec 28 00:00:54  COBS IN  070:  6963652d 69643020 170d3233 30353032 | ice-id0 ..230502
1986 Dec 28 00:00:54  COBS IN  080:  31393439 31315a18 0f393939 39313233 | 194911Z..9999123
1986 Dec 28 00:00:54  COBS IN  090:  31323335 3935395a 303e310b 30090603 | 1235959Z0>1.0...
1986 Dec 28 00:00:54  COBS IN  0a0:  55040613 02555331 1f301d06 0355040a | U....US1.0...U..
1986 Dec 28 00:00:54  COBS IN  0b0:  0c164f78 69646520 436f6d70 75746572 | ..Oxide Computer
1986 Dec 28 00:00:54  COBS IN  0c0:  20436f6d 70616e79 310e300c 06035504 |  Company1.0...U.
1986 Dec 28 00:00:54  COBS IN  0d0:  030c0561 6c696173 302a3005 06032b65 | ...alias0*0...+e
1986 Dec 28 00:00:54  COBS IN  0e0:  70032132 3314e413 1647f131 7bfe4433 | p.!23....G.1{.D3
1986 Dec 28 00:00:54  COBS IN  0f0:  4c1f30f1 59f3f5c2 0d16f6e9 5cdbdb6b | L.0.Y.......\..k
1986 Dec 28 00:00:54       IN  000:  cc19de01 01000000 72000000 00000080 | ........r.......
1986 Dec 28 00:00:54       IN  010:  0800577e a7010000 00013082 01ac3082 | ..W~......0...0.
1986 Dec 28 00:00:54       IN  020:  015ea003 02010202 01013005 06032b65 | .^........0...+e
1986 Dec 28 00:00:54       IN  030:  70304231 0b300906 03550406 13025553 | p0B1.0...U....US
1986 Dec 28 00:00:54       IN  040:  311f301d 06035504 0a0c164f 78696465 | 1.0...U....Oxide
1986 Dec 28 00:00:54       IN  050:  20436f6d 70757465 7220436f 6d70616e |  Computer Compan
1986 Dec 28 00:00:54       IN  060:  79311230 10060355 04030c09 64657669 | y1.0...U....devi
1986 Dec 28 00:00:54       IN  070:  63652d69 64302017 0d323330 35303231 | ce-id0 ..2305021
1986 Dec 28 00:00:54       IN  080:  39343931 315a180f 39393939 31323331 | 94911Z..99991231
1986 Dec 28 00:00:54       IN  090:  32333539 35395a30 3e310b30 09060355 | 235959Z0>1.0...U
1986 Dec 28 00:00:54       IN  0a0:  04061302 5553311f 301d0603 55040a0c | ....US1.0...U...
1986 Dec 28 00:00:54       IN  0b0:  164f7869 64652043 6f6d7075 74657220 | .Oxide Computer
1986 Dec 28 00:00:54       IN  0c0:  436f6d70 616e7931 0e300c06 03550403 | Company1.0...U..
1986 Dec 28 00:00:54       IN  0d0:  0c05616c 69617330 2a300506 032b6570 | ..alias0*0...+ep
1986 Dec 28 00:00:54       IN  0e0:  03210033 14e41316 47f1317b fe44334c | .!.3....G.1{.D3L
1986 Dec 28 00:00:54       IN  0f0:  1f30f159 f3f5c20d 16f6e95c dbdb6bd9 | .0.Y.......\..k.
1986 Dec 28 00:00:54 Additional data length: 0x82e
1986 Dec 28 00:00:54  DATA IN  000:  00577ea7 01000000 01308201 ac308201 | .W~......0...0..
1986 Dec 28 00:00:54  DATA IN  010:  5ea00302 01020201 01300506 032b6570 | ^........0...+ep
1986 Dec 28 00:00:54  DATA IN  020:  3042310b 30090603 55040613 02555331 | 0B1.0...U....US1
1986 Dec 28 00:00:54  DATA IN  030:  1f301d06 0355040a 0c164f78 69646520 | .0...U....Oxide
1986 Dec 28 00:00:54  DATA IN  040:  436f6d70 75746572 20436f6d 70616e79 | Computer Company
1986 Dec 28 00:00:54  DATA IN  050:  31123010 06035504 030c0964 65766963 | 1.0...U....devic
1986 Dec 28 00:00:54  DATA IN  060:  652d6964 3020170d 32333035 30323139 | e-id0 ..23050219
1986 Dec 28 00:00:54  DATA IN  070:  34393131 5a180f39 39393931 32333132 | 4911Z..999912312
1986 Dec 28 00:00:54  DATA IN  080:  33353935 395a303e 310b3009 06035504 | 35959Z0>1.0...U.
1986 Dec 28 00:00:54  DATA IN  090:  06130255 53311f30 1d060355 040a0c16 | ...US1.0...U....
1986 Dec 28 00:00:54  DATA IN  0a0:  4f786964 6520436f 6d707574 65722043 | Oxide Computer C
1986 Dec 28 00:00:54  DATA IN  0b0:  6f6d7061 6e79310e 300c0603 5504030c | ompany1.0...U...
1986 Dec 28 00:00:54  DATA IN  0c0:  05616c69 6173302a 30050603 2b657003 | .alias0*0...+ep.
1986 Dec 28 00:00:54  DATA IN  0d0:  21003314 e4131647 f1317bfe 44334c1f | !.3....G.1{.D3L.
1986 Dec 28 00:00:54  DATA IN  0e0:  30f159f3 f5c20d16 f6e95cdb db6bd990 | 0.Y.......\..k..
1986 Dec 28 00:00:54  DATA IN  0f0:  b4f0a37b 3079300c 0603551d 130101ff | ...{0y0...U.....
1986 Dec 28 00:00:54 Popping module ttcompat
1986 Dec 28 00:00:54 Popping module ldterm
1986 Dec 28 00:00:54
------> Sending IPCC command 0xb (ROT), attempt 1/10
1986 Dec 28 00:00:54 Additional data length: 0x9
1986 Dec 28 00:00:54 DATA OUT  0:  00577ea7 01000000 01                | .W~......
1986 Dec 28 00:00:54      OUT  00:  cc19de01 01000000 73000000 00000000 | ........s.......
1986 Dec 28 00:00:54      OUT  10:  0b00577e a7010000 0001c403          | ..W~........
1986 Dec 28 00:00:54 COBS OUT  00:  06cc19de 01010101 02730101 01010101 | .........s......
1986 Dec 28 00:00:54 COBS OUT  10:  020b0557 7ea70101 010401c4 03       | ...W~........
1986 Dec 28 00:00:54  COBS IN  000:  06cc19de 01010101 02730101 01010103 | .........s......
1986 Dec 28 00:00:54  COBS IN  010:  80080557 7ea70101 01030201 01010121 | ...W~..........!
1986 Dec 28 00:00:54  COBS IN  020:  492c023c 121797aa 3aa49ca8 ec37f35f | I,.<....:....7._
1986 Dec 28 00:00:54  COBS IN  030:  c2adbcfc a3ff0c48 42526374 7edcca58 | .......HBRct~..X
1986 Dec 28 00:00:54  COBS IN  040:  01010101 01010101 01010101 01010101 | ................
1986 Dec 28 00:00:54  COBS IN  050:  01010101 01010101 01010101 01010101 | ................
1986 Dec 28 00:00:54  COBS IN  060:  01010101 01010101 01010101 01010101 | ................
1986 Dec 28 00:00:54  COBS IN  070:  01010101 01010101 01010101 01010101 | ................
1986 Dec 28 00:00:54  COBS IN  080:  01010101 01010101 01010101 01010101 | ................
1986 Dec 28 00:00:54  COBS IN  090:  01010101 01010101 01010101 01010101 | ................
1986 Dec 28 00:00:54  COBS IN  0a0:  01010101 01010101 01010101 01010101 | ................
1986 Dec 28 00:00:54  COBS IN  0b0:  01010101 01010101 01010101 01010101 | ................
1986 Dec 28 00:00:54  COBS IN  0c0:  01010101 01010101 01010101 01010101 | ................
1986 Dec 28 00:00:54  COBS IN  0d0:  01010101 01010101 01010101 01010101 | ................
1986 Dec 28 00:00:54  COBS IN  0e0:  01010101 01010101 01010101 01010101 | ................
1986 Dec 28 00:00:54  COBS IN  0f0:  01010101 01010101 01010101 01010101 | ................
1986 Dec 28 00:00:54       IN  000:  cc19de01 01000000 73000000 00000080 | ........s.......
1986 Dec 28 00:00:54       IN  010:  0800577e a7010000 00020100 00000049 | ..W~...........I
1986 Dec 28 00:00:54       IN  020:  2c023c12 1797aa3a a49ca8ec 37f35fc2 | ,.<....:....7._.
1986 Dec 28 00:00:54       IN  030:  adbcfca3 ff0c4842 5263747e dcca5800 | ......HBRct~..X.
1986 Dec 28 00:00:54       IN  040:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:54       IN  050:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:54       IN  060:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:54       IN  070:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:54       IN  080:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:54       IN  090:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:54       IN  0a0:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:54       IN  0b0:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:54       IN  0c0:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:54       IN  0d0:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:54       IN  0e0:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:54       IN  0f0:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:54 Additional data length: 0x21d
1986 Dec 28 00:00:54  DATA IN  000:  00577ea7 01000000 02010000 0000492c | .W~...........I,
1986 Dec 28 00:00:54  DATA IN  010:  023c1217 97aa3aa4 9ca8ec37 f35fc2ad | .<....:....7._..
1986 Dec 28 00:00:54  DATA IN  020:  bcfca3ff 0c484252 63747edc ca580000 | .....HBRct~..X..
1986 Dec 28 00:00:54  DATA IN  030:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:54  DATA IN  040:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:54  DATA IN  050:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:54  DATA IN  060:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:54  DATA IN  070:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:54  DATA IN  080:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:54  DATA IN  090:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:54  DATA IN  0a0:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:54  DATA IN  0b0:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:54  DATA IN  0c0:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:54  DATA IN  0d0:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:54  DATA IN  0e0:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:54  DATA IN  0f0:  00000000 00000000 00000000 00000000 | ................
1986 Dec 28 00:00:54 Popping module ttcompat
1986 Dec 28 00:00:54 Popping module ldterm
1986 Dec 28 00:00:54
------> Sending IPCC command 0xb (ROT), attempt 1/10
1986 Dec 28 00:00:54 Additional data length: 0x29
1986 Dec 28 00:00:54 DATA OUT  00:  00577ea7 01000000 02eaa822 6f4422f8 | .W~........"oD".
1986 Dec 28 00:00:54 DATA OUT  10:  7e39d92f 09765e87 45bf5697 61df39c2 | ~9./.v^.E.V.a.9.
1986 Dec 28 00:00:54 DATA OUT  20:  502515db f78e94f0 d6                | P%.......
1986 Dec 28 00:00:54      OUT  00:  cc19de01 01000000 74000000 00000000 | ........t.......
1986 Dec 28 00:00:54      OUT  10:  0b00577e a7010000 0002eaa8 226f4422 | ..W~........"oD"
1986 Dec 28 00:00:54      OUT  20:  f87e39d9 2f09765e 8745bf56 9761df39 | .~9./.v^.E.V.a.9
1986 Dec 28 00:00:54      OUT  30:  c2502515 dbf78e94 f0d6ea42          | .P%........B
1986 Dec 28 00:00:54 COBS OUT  00:  06cc19de 01010101 02740101 01010101 | .........t......
1986 Dec 28 00:00:54 COBS OUT  10:  020b0557 7ea70101 012402ea a8226f44 | ...W~....$..."oD
1986 Dec 28 00:00:54 COBS OUT  20:  22f87e39 d92f0976 5e8745bf 569761df | ".~9./.v^.E.V.a.
1986 Dec 28 00:00:54 COBS OUT  30:  39c25025 15dbf78e 94f0d6ea 42       | 9.P%........B
1986 Dec 28 00:00:54  COBS IN  00:  06cc19de 01010101 02740101 01010103 | .........t......
1986 Dec 28 00:00:54  COBS IN  10:  80080557 7ea70101 01020325 b2314a6b | ...W~......%.1Jk
1986 Dec 28 00:00:54  COBS IN  20:  10ce6744 8c6c77de 447014b8 ed597322 | ..gD.lw.Dp...Ys"
1986 Dec 28 00:00:54  COBS IN  30:  c6690152 9c22f6ac d20a2a98 34c5752e | .i.R."....*.4.u.
1986 Dec 28 00:00:54  COBS IN  40:  1ee56a76 bb37a0c2 3b885888 34051f07 | ..jv.7..;.X.4...
1986 Dec 28 00:00:54  COBS IN  50:  d37d16e8 c493cae0 cf305301 043f     | .}.......0S..?
1986 Dec 28 00:00:54       IN  00:  cc19de01 01000000 74000000 00000080 | ........t.......
1986 Dec 28 00:00:54       IN  10:  0800577e a7010000 000300b2 314a6b10 | ..W~........1Jk.
1986 Dec 28 00:00:54       IN  20:  ce67448c 6c77de44 7014b8ed 597322c6 | .gD.lw.Dp...Ys".
1986 Dec 28 00:00:54       IN  30:  6901529c 22f6acd2 0a2a9834 c5752e00 | i.R."....*.4.u..
1986 Dec 28 00:00:54       IN  40:  e56a76bb 37a0c23b 88588834 051f07d3 | .jv.7..;.X.4....
1986 Dec 28 00:00:54       IN  50:  7d16e8c4 93cae0cf 30530104 3f       | }.......0S..?
1986 Dec 28 00:00:54 Additional data length: 0x4a
1986 Dec 28 00:00:54  DATA IN  00:  00577ea7 01000000 0300b231 4a6b10ce | .W~........1Jk..
1986 Dec 28 00:00:54  DATA IN  10:  67448c6c 77de4470 14b8ed59 7322c669 | gD.lw.Dp...Ys".i
1986 Dec 28 00:00:54  DATA IN  20:  01529c22 f6acd20a 2a9834c5 752e00e5 | .R."....*.4.u...
1986 Dec 28 00:00:54  DATA IN  30:  6a76bb37 a0c23b88 58883405 1f07d37d | jv.7..;.X.4....}
1986 Dec 28 00:00:54  DATA IN  40:  16e8c493 cae0cf30 5301              | .......0S.
BRM42220036 # svcs
STATE          STIME    FMRI
legacy_run      0:00:07 lrc:/etc/rc2_d/S20sysetup
legacy_run      0:00:07 lrc:/etc/rc2_d/S89PRESERVE
online          0:00:02 svc:/system/svc/restarter:default
online          0:00:02 svc:/system/early-manifest-import:default
online          0:00:02 svc:/network/datalink-management:default
online          0:00:02 svc:/system/filesystem/root:default
online          0:00:02 svc:/network/ip-interface-management:default
online          0:00:03 svc:/system/cryptosvc:default
online          0:00:03 svc:/system/boot-archive:default
online          0:00:03 svc:/system/scheduler:default
online          0:00:03 svc:/network/loopback:default
online          0:00:03 svc:/system/filesystem/usr:default
online          0:00:03 svc:/system/keymap:default
online          0:00:03 svc:/system/pfexec:default
online          0:00:03 svc:/system/auditset:default
online          0:00:04 svc:/system/device/local:default
online          0:00:04 svc:/milestone/devices:default
online          0:00:04 svc:/system/filesystem/minimal:default
online          0:00:04 svc:/system/resource-mgmt:default
online          0:00:04 svc:/system/rmtmpfiles:default
online          0:00:04 svc:/system/zones-monitoring:default
online          0:00:04 svc:/system/sysevent:default
online          0:00:04 svc:/system/ipcc:default
online          0:00:04 svc:/network/physical:default
online          0:00:04 svc:/site/compliance/hostname:default
online          0:00:04 svc:/system/logadm-upgrade:default
online          0:00:04 svc:/oxide/net-setup:e1000g
online          0:00:04 svc:/site/postboot:default
online          0:00:04 svc:/oxide/net-setup:ixgbe
online          0:00:05 svc:/milestone/network:default
online          0:00:05 svc:/system/identity:domain
online          0:00:05 svc:/oxide/net-setup:igb
online          0:00:05 svc:/system/identity:node
online          0:00:05 svc:/network/initial:default
online          0:00:05 svc:/system/picl:default
online          0:00:05 svc:/network/service:default
online          0:00:05 svc:/network/netmask:default
online          0:00:05 svc:/network/dns/client:default
online          0:00:05 svc:/milestone/name-services:default
online          0:00:06 svc:/system/rbac:default
online          0:00:06 svc:/system/name-service-cache:default
online          0:00:07 svc:/system/manifest-import:default
online          0:00:07 svc:/network/routing-setup:default
online          0:00:07 svc:/milestone/single-user:default
online          0:00:07 svc:/milestone/sysconfig:default
online          0:00:07 svc:/system/utmp:default
online          0:00:07 svc:/system/console-login:default
online          0:00:07 svc:/system/filesystem/local:default
online          0:00:07 svc:/system/sac:default
online          0:00:07 svc:/system/boot-archive-update:default
online          0:00:07 svc:/system/cron:default
online          0:00:07 svc:/system/system-log:default
online          0:00:07 svc:/system/hotplug:default
online          0:00:07 svc:/system/dumpadm:default
online          0:00:07 svc:/milestone/multi-user:default
online          0:00:07 svc:/system/boot-config:default
online          0:00:08 svc:/system/update-man-index:default
online          0:00:09 svc:/network/ssh:default
online          0:00:09 svc:/milestone/multi-user-server:default
online          0:00:09 svc:/site/compliance/beacon:default
online          0:00:10 svc:/system/fmd:default
online          0:00:17 svc:/network/routing/ndp:default
online          0:00:43 svc:/system/t6init:default
online          0:00:43 svc:/oxide/sled-agent:default
online          0:00:44 svc:/oxide/mg-ddm:default
online          0:00:44 svc:/oxide/pumpkind:default
online          0:00:45 svc:/network/ipv6-forwarding:default
online          0:00:47 svc:/system/coreadm:default

@hawkw
get rid of test dingus
78d84f2 
this reverts commit 1c8a990
@hawkw hawkw requested a review from jgallagher March 11, 2026 16:57
@hawkw
Copy link
Member Author

hawkw commented Mar 11, 2026

As far as reviewing this goes, I'd mainly like someone more knowledgeable about the IPCC stuff than I to cross-check the manual testing I did in #9982 (comment) --- is this sufficient to confirm the fix or is there anything else I ought to do? I've reverted the test code I added to generate a bunch of SIGCHLDs, in order to prep this change to merge, but I've left it in the Git history here so that it can be reviewed as well: 1c8a990

@jgallagher
Copy link
Contributor

jgallagher commented Mar 12, 2026

As far as reviewing this goes, I'd mainly like someone more knowledgeable about the IPCC stuff than I to cross-check the manual testing I did in #9982 (comment) --- is this sufficient to confirm the fix or is there anything else I ought to do?

This looks sufficient to me; thanks for picking this up.

@hawkw hawkw merged commit d3b6e92 into main Mar 12, 2026
18 checks passed
@hawkw hawkw deleted the eliza/sled-agent-signal-thread branch March 12, 2026 20:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jgallagher jgallagher jgallagher approved these changes

@davepacheco davepacheco Awaiting requested review from davepacheco

@dancrossnyc dancrossnyc Awaiting requested review from dancrossnyc

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

sled-agent could handle Unix signals better

4 participants

@hawkw @dancrossnyc @jgallagher @davepacheco