lots of Crucible test failures due to timeouts

[davepacheco]

I'm running the test suite locally for the first time in a while and ran into a log of test Crucible-related test failures that I'd expect to be unrelated to my changes:

$ cargo nextest run --no-fail-fast
...
────────────
     Summary [1935.375s] 2194 tests run: 2183 passed (95 slow), 11 failed, 3 skipped
        FAIL [  53.948s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_replacement_state_replacement_done
        FAIL [  81.001s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_replacement_state_requested
        FAIL [  53.394s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_replacement_state_running_on_pantry
        FAIL [  53.295s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_state_requested
        FAIL [  52.203s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_state_running
        FAIL [  51.029s] omicron-nexus::test_all integration_tests::disks::test_disk_expunge
        FAIL [  50.192s] omicron-nexus::test_all integration_tests::disks::test_disk_slot_assignment
        FAIL [  51.146s] omicron-nexus::test_all integration_tests::volume_management::test_delete_image_order_5
        FAIL [  72.444s] omicron-nexus::test_all integration_tests::volume_management::test_read_only_region_reference_counting
        FAIL [  47.413s] omicron-nexus::test_all integration_tests::volume_management::test_read_only_region_reference_usage_sanity
        FAIL [  11.927s] omicron-omdb::test_all_output test_omdb_usage_errors
error: test run failed

(the omdb one is definitely mine)

This is all on commit ad09008 (which is far from "main", just not in ways I'd expect to affect this). I will double check whether this happens in CI and whether I see it on "main" when running locally too.

[davepacheco]

Fuller output (the full output doesn't fit in a GitHub comment):

 Nextest run ID fdcf9420-fd7e-4a90-8bee-b65b92125695 with nextest profile: default
    Starting 2194 tests across 211 binaries (3 tests and 5 binaries skipped, including 5 binaries via profile.default.default-filter)
             [ 00:00:00]                                            0/2197:             SETUP [      1/2] crdb-seed: cargo run -p crdb-seed --profile test
    Finished `test` profile [unoptimized + debuginfo] target(s) in 1.69s
     Running `target/debug/crdb-seed`
Jul 31 17:38:51.962 INFO Using existing CRDB seed tarball: `/dangerzone/omicron_tmp/crdb-base-dap/d48510f3250500ceffceaf55f81a712083af2ccc88348d5987c5745269c9b155.tar`
  SETUP PASS [   1.936s] crdb-seed: cargo run -p crdb-seed --profile test
             [ 00:00:01]                                            0/2197:             SETUP [      2/2] clickhouse-cluster: cargo run -p clickhouse-cluster-dev
   Compiling clickhouse-admin-test-utils v0.1.0 (/home/dap/omicron-work/clickhouse-admin/test-utils)
   Compiling oximeter-test-utils v0.1.0 (/home/dap/omicron-work/oximeter/test-utils)
   Compiling clickhouse-cluster-dev v0.1.0 (/home/dap/omicron-work/dev-tools/clickhouse-cluster-dev)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 7.50s
     Running `target/debug/clickhouse-cluster-dev`
Jul 31 17:38:59.686 INFO Setting up a ClickHouse cluster
Deploying keeper: /dangerzone/omicron_tmp/clickward_test/keeper-3
Deploying keeper: /dangerzone/omicron_tmp/clickward_test/keeper-2
Deploying keeper: /dangerzone/omicron_tmp/clickward_test/keeper-1
Deploying clickhouse server: /dangerzone/omicron_tmp/clickward_test/clickhouse-1
Deploying clickhouse server: /dangerzone/omicron_tmp/clickward_test/clickhouse-2
Jul 31 17:39:00.718 INFO ClickHouse server ready
Jul 31 17:39:00.719 INFO ClickHouse server ready
Jul 31 17:39:02.067 INFO Keepers ready: [KeeperId(1), KeeperId(2), KeeperId(3)]
  SETUP PASS [  10.107s] clickhouse-cluster: cargo run -p clickhouse-cluster-dev
        PASS [  25.002s] omicron-nexus app::sagas::image_create::test::test_action_failure_can_unwind_idempotently
        PASS [  25.495s] omicron-nexus app::sagas::image_create::test::test_action_failure_can_unwind
        PASS [  36.331s] omicron-nexus app::sagas::region_replacement_start::test::test_action_failure_can_unwind_idempotently
...
        PASS [  25.858s] omicron-nexus::test_all integration_tests::console_api::test_session_idle_timeout_deletes_session
        PASS [  21.247s] omicron-nexus::test_all integration_tests::console_api::test_unauthed_console_pages
        PASS [  29.231s] omicron-nexus::test_all integration_tests::console_api::test_sessions
        FAIL [  53.948s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_replacement_state_replacement_done
  stdout ───

    running 1 test
    test integration_tests::crucible_replacements::test_delete_volume_region_replacement_state_replacement_done ... FAILED

    failures:

    failures:
        integration_tests::crucible_replacements::test_delete_volume_region_replacement_state_replacement_done

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 454 filtered out; finished in 53.44s
    
  stderr ───
    log file: /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_replacement_state_replacement_done.13752.0.log
    note: configured to log to "/dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_replacement_state_replacement_done.13752.0.log"
    DB URL: postgresql://root@[::1]:37880/omicron?sslmode=disable
    DB address: [::1]:37880
    log file: /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_replacement_state_replacement_done.13752.2.log
    note: configured to log to "/dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_replacement_state_replacement_done.13752.2.log"
    log file: /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_replacement_state_replacement_done.13752.3.log
    note: configured to log to "/dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_replacement_state_replacement_done.13752.3.log"

    thread 'integration_tests::crucible_replacements::test_delete_volume_region_replacement_state_replacement_done' panicked at /home/dap/omicron-work/nexus/test-utils/src/resource_helpers.rs:1697:10:
    expected to find inventory collection: TimedOut(30.1036941s)
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
    WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: temporary directory leaked: "/dangerzone/omicron_tmp/.tmp7AEa03"
        If you would like to access the database for debugging, run the following:

        # Run the database
        cargo xtask db-dev run --no-populate --store-dir "/dangerzone/omicron_tmp/.tmp7AEa03/data"
        # Access the database. Note the port may change if you run multiple databases.
        cockroach sql --host=localhost:32221 --insecure
    WARN: dropped ClickHouse process without cleaning it up first (there may still be a child process running (PID 13777) and a temporary directory leaked, /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_replacement_state_replacement_done.13752.1-clickhouse-Z8zRQB)
    failed to clean up ClickHouse data dir:
    - /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_replacement_state_replacement_done.13752.1-clickhouse-Z8zRQB: File exists (os error 17)
    WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: dendrite temporary directory leaked: /dangerzone/omicron_tmp/.tmp2hO1eK
    WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: dendrite temporary directory leaked: /dangerzone/omicron_tmp/.tmp2wFyeI
    WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: mgd temporary directory leaked: /dangerzone/omicron_tmp/.tmprNA9HY
    WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: mgd temporary directory leaked: /dangerzone/omicron_tmp/.tmph1AsCW

        FAIL [  53.394s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_replacement_state_running_on_pantry
  stdout ───

    running 1 test
    test integration_tests::crucible_replacements::test_delete_volume_region_replacement_state_running_on_pantry ... FAILED

    failures:

    failures:
        integration_tests::crucible_replacements::test_delete_volume_region_replacement_state_running_on_pantry

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 454 filtered out; finished in 52.86s
    
  stderr ───
    log file: /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_replacement_state_running_on_pantry.13783.0.log
    note: configured to log to "/dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_replacement_state_running_on_pantry.13783.0.log"
    DB URL: postgresql://root@[::1]:64958/omicron?sslmode=disable
    DB address: [::1]:64958
    log file: /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_replacement_state_running_on_pantry.13783.2.log
    note: configured to log to "/dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_replacement_state_running_on_pantry.13783.2.log"
    log file: /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_replacement_state_running_on_pantry.13783.3.log
    note: configured to log to "/dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_replacement_state_running_on_pantry.13783.3.log"

    thread 'integration_tests::crucible_replacements::test_delete_volume_region_replacement_state_running_on_pantry' panicked at /home/dap/omicron-work/nexus/test-utils/src/resource_helpers.rs:1697:10:
    expected to find inventory collection: TimedOut(30.024559711s)
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
    WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: temporary directory leaked: "/dangerzone/omicron_tmp/.tmpq9CAix"
        If you would like to access the database for debugging, run the following:

        # Run the database
        cargo xtask db-dev run --no-populate --store-dir "/dangerzone/omicron_tmp/.tmpq9CAix/data"
        # Access the database. Note the port may change if you run multiple databases.
        cockroach sql --host=localhost:32221 --insecure
    WARN: dropped ClickHouse process without cleaning it up first (there may still be a child process running (PID 13837) and a temporary directory leaked, /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_replacement_state_running_on_pantry.13783.1-clickhouse-SkvMbp)
    failed to clean up ClickHouse data dir:
    - /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_replacement_state_running_on_pantry.13783.1-clickhouse-SkvMbp: File exists (os error 17)
    WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: dendrite temporary directory leaked: /dangerzone/omicron_tmp/.tmpM68Js2
    WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: dendrite temporary directory leaked: /dangerzone/omicron_tmp/.tmpr4W4zw
    WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: mgd temporary directory leaked: /dangerzone/omicron_tmp/.tmpMkrmM9
    WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: mgd temporary directory leaked: /dangerzone/omicron_tmp/.tmpkaaSDA

        PASS [  55.657s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_replacement_state_running
        FAIL [  53.295s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_state_requested
  stdout ───

    running 1 test
    test integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_state_requested ... FAILED

    failures:

    failures:
        integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_state_requested

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 454 filtered out; finished in 52.78s
    
  stderr ───
    log file: /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_snapshot_replacement_state_requested.13863.0.log
    note: configured to log to "/dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_snapshot_replacement_state_requested.13863.0.log"
    DB URL: postgresql://root@[::1]:48263/omicron?sslmode=disable
    DB address: [::1]:48263
    log file: /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_snapshot_replacement_state_requested.13863.2.log
    note: configured to log to "/dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_snapshot_replacement_state_requested.13863.2.log"
    log file: /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_snapshot_replacement_state_requested.13863.3.log
    note: configured to log to "/dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_snapshot_replacement_state_requested.13863.3.log"

    thread 'integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_state_requested' panicked at /home/dap/omicron-work/nexus/test-utils/src/resource_helpers.rs:1697:10:
    expected to find inventory collection: TimedOut(30.166264777s)
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
    WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: temporary directory leaked: "/dangerzone/omicron_tmp/.tmprWFLiq"
        If you would like to access the database for debugging, run the following:

        # Run the database
        cargo xtask db-dev run --no-populate --store-dir "/dangerzone/omicron_tmp/.tmprWFLiq/data"
        # Access the database. Note the port may change if you run multiple databases.
        cockroach sql --host=localhost:32221 --insecure
    WARN: dropped ClickHouse process without cleaning it up first (there may still be a child process running (PID 13884) and a temporary directory leaked, /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_snapshot_replacement_state_requested.13863.1-clickhouse-MFVN9V)
    failed to clean up ClickHouse data dir:
    - /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_snapshot_replacement_state_requested.13863.1-clickhouse-MFVN9V: File exists (os error 17)
    WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: dendrite temporary directory leaked: /dangerzone/omicron_tmp/.tmpT6eOSp
    WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: dendrite temporary directory leaked: /dangerzone/omicron_tmp/.tmpgOSSt1
    WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: mgd temporary directory leaked: /dangerzone/omicron_tmp/.tmpGdQTZp
    WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: mgd temporary directory leaked: /dangerzone/omicron_tmp/.tmp4s2Wnx

        PASS [  44.903s] omicron-nexus::test_all integration_tests::crucible_replacements::test_region_replacement_does_not_create_freed_region
        SLOW [> 60.000s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_replacement_state_requested
        SLOW [> 60.000s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_state_replacement_done
        FAIL [  52.203s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_state_running
  stdout ───

    running 1 test
    test integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_state_running ... FAILED

    failures:

    failures:
        integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_state_running

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 454 filtered out; finished in 51.66s
    
  stderr ───
    log file: /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_snapshot_replacement_state_running.13920.0.log
    note: configured to log to "/dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_snapshot_replacement_state_running.13920.0.log"
    DB URL: postgresql://root@[::1]:55349/omicron?sslmode=disable
    DB address: [::1]:55349
    log file: /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_snapshot_replacement_state_running.13920.2.log
    note: configured to log to "/dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_snapshot_replacement_state_running.13920.2.log"
    log file: /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_snapshot_replacement_state_running.13920.3.log
    note: configured to log to "/dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_snapshot_replacement_state_running.13920.3.log"

    thread 'integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_state_running' panicked at /home/dap/omicron-work/nexus/test-utils/src/resource_helpers.rs:1697:10:
    expected to find inventory collection: TimedOut(30.402357595s)
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
    WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: temporary directory leaked: "/dangerzone/omicron_tmp/.tmp9tMKhB"
        If you would like to access the database for debugging, run the following:

        # Run the database
        cargo xtask db-dev run --no-populate --store-dir "/dangerzone/omicron_tmp/.tmp9tMKhB/data"
        # Access the database. Note the port may change if you run multiple databases.
        cockroach sql --host=localhost:32221 --insecure
    WARN: dropped ClickHouse process without cleaning it up first (there may still be a child process running (PID 13947) and a temporary directory leaked, /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_snapshot_replacement_state_running.13920.1-clickhouse-pbnD37)
    failed to clean up ClickHouse data dir:
    - /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_snapshot_replacement_state_running.13920.1-clickhouse-pbnD37: File exists (os error 17)
    WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: dendrite temporary directory leaked: /dangerzone/omicron_tmp/.tmpDmT57G
    WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: dendrite temporary directory leaked: /dangerzone/omicron_tmp/.tmpbCISMV
    WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: mgd temporary directory leaked: /dangerzone/omicron_tmp/.tmpKjoSwz
    WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: mgd temporary directory leaked: /dangerzone/omicron_tmp/.tmpTIXoCG

        SLOW [> 60.000s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_state_requested_2
        SLOW [> 60.000s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_state_requested_3
        SLOW [> 60.000s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_step
        SLOW [> 60.000s] omicron-nexus::test_all integration_tests::crucible_replacements::test_racing_replacements_for_soft_deleted_disk_volume
        SLOW [> 60.000s] omicron-nexus::test_all integration_tests::crucible_replacements::test_read_only_replacement_sanity
        SLOW [> 60.000s] omicron-nexus::test_all integration_tests::crucible_replacements::test_region_replacement_triple_sanity
        SLOW [> 60.000s] omicron-nexus::test_all integration_tests::crucible_replacements::test_region_replacement_triple_sanity_2
        PASS [  19.746s] omicron-nexus::test_all integration_tests::demo_saga::test_demo_saga
        FAIL [  81.001s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_replacement_state_requested
  stdout ───

    running 1 test
    test integration_tests::crucible_replacements::test_delete_volume_region_replacement_state_requested has been running for over 60 seconds
    test integration_tests::crucible_replacements::test_delete_volume_region_replacement_state_requested ... FAILED

    failures:

    failures:
        integration_tests::crucible_replacements::test_delete_volume_region_replacement_state_requested

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 454 filtered out; finished in 80.44s
    
  stderr ───
    log file: /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_replacement_state_requested.13760.0.log
    note: configured to log to "/dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_replacement_state_requested.13760.0.log"
    DB URL: postgresql://root@[::1]:45288/omicron?sslmode=disable
    DB address: [::1]:45288
    log file: /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_replacement_state_requested.13760.2.log
    note: configured to log to "/dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_replacement_state_requested.13760.2.log"
    log file: /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_replacement_state_requested.13760.3.log
    note: configured to log to "/dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_replacement_state_requested.13760.3.log"

    thread 'integration_tests::crucible_replacements::test_delete_volume_region_replacement_state_requested' panicked at /home/dap/omicron-work/nexus/test-utils/src/resource_helpers.rs:1697:10:
    expected to find inventory collection: TimedOut(30.07450617s)
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
    WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: temporary directory leaked: "/dangerzone/omicron_tmp/.tmpsnrwIF"
        If you would like to access the database for debugging, run the following:

        # Run the database
        cargo xtask db-dev run --no-populate --store-dir "/dangerzone/omicron_tmp/.tmpsnrwIF/data"
        # Access the database. Note the port may change if you run multiple databases.
        cockroach sql --host=localhost:32221 --insecure
    WARN: dropped ClickHouse process without cleaning it up first (there may still be a child process running (PID 13788) and a temporary directory leaked, /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_replacement_state_requested.13760.1-clickhouse-e4oF6V)
    failed to clean up ClickHouse data dir:
    - /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_volume_region_replacement_state_requested.13760.1-clickhouse-e4oF6V: File exists (os error 17)
    WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: dendrite temporary directory leaked: /dangerzone/omicron_tmp/.tmpiimddi
    WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: dendrite temporary directory leaked: /dangerzone/omicron_tmp/.tmpbE8fWj
    WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: mgd temporary directory leaked: /dangerzone/omicron_tmp/.tmpAKhljY
    WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: mgd temporary directory leaked: /dangerzone/omicron_tmp/.tmpyxUDce

        SLOW [> 60.000s] omicron-nexus::test_all integration_tests::crucible_replacements::test_region_snapshot_replacement_step_after_rop_remove
        PASS [  86.271s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_state_replacement_done
        PASS [  84.603s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_state_requested_2
        PASS [  81.261s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_step
        PASS [  72.358s] omicron-nexus::test_all integration_tests::crucible_replacements::test_region_snapshot_replacement_step_after_rop_remove
        PASS [  84.955s] omicron-nexus::test_all integration_tests::crucible_replacements::test_racing_replacements_for_soft_deleted_disk_volume
        PASS [  91.058s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_state_requested_3
        PASS [  21.013s] omicron-nexus::test_all integration_tests::device_auth::test_device_auth_flow
        PASS [  37.940s] omicron-nexus::test_all integration_tests::device_auth::test_admin_logout_deletes_tokens_and_sessions
        PASS [  24.836s] omicron-nexus::test_all integration_tests::device_auth::test_device_token_expiration
        PASS [  18.173s] omicron-nexus::test_all integration_tests::device_auth::test_device_token_request_ttl_invalid
        SLOW [> 60.000s] omicron-nexus::test_all integration_tests::crucible_replacements::test_region_snapshot_replacement_step_after_rop_remove_target_gone
        PASS [  23.944s] omicron-nexus::test_all integration_tests::device_auth::test_device_token_request_ttl
        SLOW [> 60.000s] omicron-nexus::test_all integration_tests::crucible_replacements::test_replacement_sanity
        SLOW [> 60.000s] omicron-nexus::test_all integration_tests::crucible_replacements::test_replacement_sanity_twice
        SLOW [> 60.000s] omicron-nexus::test_all integration_tests::crucible_replacements::test_replacement_sanity_twice_after_snapshot_delete
        PASS [ 113.984s] omicron-nexus::test_all integration_tests::crucible_replacements::test_read_only_replacement_sanity
        PASS [  79.574s] omicron-nexus::test_all integration_tests::crucible_replacements::test_region_snapshot_replacement_step_after_rop_remove_target_gone
        PASS [  36.105s] omicron-nexus::test_all integration_tests::disks::test_disk_create_disk_that_already_exists_fails
        SLOW [>120.000s] omicron-nexus::test_all integration_tests::crucible_replacements::test_region_replacement_triple_sanity
        SLOW [>120.000s] omicron-nexus::test_all integration_tests::crucible_replacements::test_region_replacement_triple_sanity_2
        PASS [  82.650s] omicron-nexus::test_all integration_tests::crucible_replacements::test_replacement_sanity
        PASS [  46.177s] omicron-nexus::test_all integration_tests::disks::test_disk_creation_region_requested_then_started
        PASS [ 132.964s] omicron-nexus::test_all integration_tests::crucible_replacements::test_region_replacement_triple_sanity
        PASS [  50.139s] omicron-nexus::test_all integration_tests::disks::test_disk_create_for_importing
        SLOW [> 60.000s] omicron-nexus::test_all integration_tests::device_auth::test_session_list_excludes_expired
        SLOW [> 60.000s] omicron-nexus::test_all integration_tests::disks::test_disk_backed_by_multiple_region_sets
        PASS [ 100.086s] omicron-nexus::test_all integration_tests::crucible_replacements::test_replacement_sanity_twice
        FAIL [  51.029s] omicron-nexus::test_all integration_tests::disks::test_disk_expunge
  stdout ───

    running 1 test
    test integration_tests::disks::test_disk_expunge ... FAILED

    failures:

    failures:
        integration_tests::disks::test_disk_expunge

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 454 filtered out; finished in 50.40s
    
  stderr ───
    log file: /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_disk_expunge.14434.0.log
    note: configured to log to "/dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_disk_expunge.14434.0.log"
    DB URL: postgresql://root@[::1]:46186/omicron?sslmode=disable
    DB address: [::1]:46186
    log file: /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_disk_expunge.14434.2.log
    note: configured to log to "/dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_disk_expunge.14434.2.log"

    thread 'integration_tests::disks::test_disk_expunge' panicked at /home/dap/omicron-work/nexus/test-utils/src/resource_helpers.rs:1697:10:
    expected to find inventory collection: TimedOut(30.077171793s)
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
    WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: temporary directory leaked: "/dangerzone/omicron_tmp/.tmpHsE18p"
        If you would like to access the database for debugging, run the following:

        # Run the database
        cargo xtask db-dev run --no-populate --store-dir "/dangerzone/omicron_tmp/.tmpHsE18p/data"
        # Access the database. Note the port may change if you run multiple databases.
        cockroach sql --host=localhost:32221 --insecure
    WARN: dropped ClickHouse process without cleaning it up first (there may still be a child process running (PID 14443) and a temporary directory leaked, /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_disk_expunge.14434.1-clickhouse-gFU6Gs)
    failed to clean up ClickHouse data dir:
    - /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_disk_expunge.14434.1-clickhouse-gFU6Gs: File exists (os error 17)
    WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: dendrite temporary directory leaked: /dangerzone/omicron_tmp/.tmpCRvQAD
    WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: mgd temporary directory leaked: /dangerzone/omicron_tmp/.tmpnsqmsj

        PASS [ 140.947s] omicron-nexus::test_all integration_tests::crucible_replacements::test_region_replacement_triple_sanity_2
        SLOW [> 60.000s] omicron-nexus::test_all integration_tests::disks::test_disk_create_attach_detach_delete
        PASS [  64.530s] omicron-nexus::test_all integration_tests::disks::test_disk_backed_by_multiple_region_sets
        PASS [  54.610s] omicron-nexus::test_all integration_tests::disks::test_disk_invalid_block_size_rejected
        PASS [  71.568s] omicron-nexus::test_all integration_tests::device_auth::test_session_list_excludes_expired
        PASS [  21.432s] omicron-nexus::test_all integration_tests::disks::test_disk_reject_total_size_greater_than_max_disk_size_bytes
        PASS [  22.051s] omicron-nexus::test_all integration_tests::disks::test_disk_reject_total_size_less_than_min_disk_size_bytes
        PASS [  36.926s] omicron-nexus::test_all integration_tests::disks::test_disk_not_found_before_creation
        SLOW [> 60.000s] omicron-nexus::test_all integration_tests::disks::test_disk_metrics
        PASS [  77.874s] omicron-nexus::test_all integration_tests::disks::test_disk_create_attach_detach_delete
        PASS [  26.426s] omicron-nexus::test_all integration_tests::disks::test_disk_reject_total_size_not_divisible_by_block_size
        PASS [  21.615s] omicron-nexus::test_all integration_tests::disks::test_disk_reject_total_size_not_divisible_by_min_disk_size
        SLOW [>120.000s] omicron-nexus::test_all integration_tests::crucible_replacements::test_replacement_sanity_twice_after_snapshot_delete
        PASS [  51.420s] omicron-nexus::test_all integration_tests::disks::test_disk_metrics_paginated
        PASS [  44.802s] omicron-nexus::test_all integration_tests::disks::test_disk_region_creation_failure
        PASS [  74.253s] omicron-nexus::test_all integration_tests::disks::test_disk_metrics
        PASS [  58.205s] omicron-nexus::test_all integration_tests::disks::test_disk_move_between_instances
        PASS [ 131.783s] omicron-nexus::test_all integration_tests::crucible_replacements::test_replacement_sanity_twice_after_snapshot_delete
        PASS [  34.978s] omicron-nexus::test_all integration_tests::disks::test_disk_size_accounting
        PASS [  29.961s] omicron-nexus::test_all integration_tests::disks::test_disk_virtual_provisioning_collection_failed_delete
        PASS [  31.548s] omicron-nexus::test_all integration_tests::disks::test_disk_virtual_provisioning_collection
        PASS [  37.934s] omicron-nexus::test_all integration_tests::disks::test_disk_too_big
        PASS [  28.441s] omicron-nexus::test_all integration_tests::disks::test_phantom_disk_rename
        PASS [  36.143s] omicron-nexus::test_all integration_tests::disks::test_do_not_provision_on_dataset_not_enough
        FAIL [  50.192s] omicron-nexus::test_all integration_tests::disks::test_disk_slot_assignment
  stdout ───

    running 1 test
    test integration_tests::disks::test_disk_slot_assignment ... FAILED

    failures:

    failures:
        integration_tests::disks::test_disk_slot_assignment

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 454 filtered out; finished in 49.65s
    
  stderr ───
    log file: /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_disk_slot_assignment.14683.0.log
    note: configured to log to "/dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_disk_slot_assignment.14683.0.log"
    DB URL: postgresql://root@[::1]:38768/omicron?sslmode=disable
    DB address: [::1]:38768
    log file: /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_disk_slot_assignment.14683.2.log
    note: configured to log to "/dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_disk_slot_assignment.14683.2.log"

    thread 'integration_tests::disks::test_disk_slot_assignment' panicked at /home/dap/omicron-work/nexus/test-utils/src/resource_helpers.rs:1697:10:
    expected to find inventory collection: TimedOut(30.097208297s)
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
    WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: temporary directory leaked: "/dangerzone/omicron_tmp/.tmpsynP5N"
        If you would like to access the database for debugging, run the following:

        # Run the database
        cargo xtask db-dev run --no-populate --store-dir "/dangerzone/omicron_tmp/.tmpsynP5N/data"
        # Access the database. Note the port may change if you run multiple databases.
        cockroach sql --host=localhost:32221 --insecure
    WARN: dropped ClickHouse process without cleaning it up first (there may still be a child process running (PID 14722) and a temporary directory leaked, /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_disk_slot_assignment.14683.1-clickhouse-zURa9n)
    failed to clean up ClickHouse data dir:
    - /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_disk_slot_assignment.14683.1-clickhouse-zURa9n: File exists (os error 17)
    WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: dendrite temporary directory leaked: /dangerzone/omicron_tmp/.tmpBkhUMt
    WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: mgd temporary directory leaked: /dangerzone/omicron_tmp/.tmpsal0dI

        PASS [  23.712s] omicron-nexus::test_all integration_tests::disks::test_single_region_allocate
        PASS [  30.297s] omicron-nexus::test_all integration_tests::disks::test_region_allocation_strategy_random_is_idempotent_arbitrary
        PASS [  37.705s] omicron-nexus::test_all integration_tests::disks::test_no_halt_disk_delete_one_region_on_expunged_agent
        PASS [  41.210s] omicron-nexus::test_all integration_tests::disks::test_project_delete_disk_no_auth_idempotent
...
        PASS [  45.766s] omicron-nexus::test_all integration_tests::volume_management::test_delete_image_order_1
        PASS [  40.849s] omicron-nexus::test_all integration_tests::volume_management::test_delete_image_order_6
        PASS [  40.221s] omicron-nexus::test_all integration_tests::volume_management::test_delete_snapshot_then_disk
        FAIL [  51.146s] omicron-nexus::test_all integration_tests::volume_management::test_delete_image_order_5
  stdout ───

    running 1 test
    test integration_tests::volume_management::test_delete_image_order_5 ... FAILED

    failures:

    failures:
        integration_tests::volume_management::test_delete_image_order_5

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 454 filtered out; finished in 50.59s
    
  stderr ───
    log file: /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_image_order_5.24548.0.log
    note: configured to log to "/dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_image_order_5.24548.0.log"
    DB URL: postgresql://root@[::1]:57056/omicron?sslmode=disable
    DB address: [::1]:57056
    log file: /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_image_order_5.24548.2.log
    note: configured to log to "/dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_image_order_5.24548.2.log"

    thread 'integration_tests::volume_management::test_delete_image_order_5' panicked at /home/dap/omicron-work/nexus/test-utils/src/resource_helpers.rs:1697:10:
    expected to find inventory collection: TimedOut(30.021913288s)
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
    WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: temporary directory leaked: "/dangerzone/omicron_tmp/.tmp7ELN14"
        If you would like to access the database for debugging, run the following:

        # Run the database
        cargo xtask db-dev run --no-populate --store-dir "/dangerzone/omicron_tmp/.tmp7ELN14/data"
        # Access the database. Note the port may change if you run multiple databases.
        cockroach sql --host=localhost:32221 --insecure
    WARN: dropped ClickHouse process without cleaning it up first (there may still be a child process running (PID 24568) and a temporary directory leaked, /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_image_order_5.24548.1-clickhouse-3k9uBe)
    failed to clean up ClickHouse data dir:
    - /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_delete_image_order_5.24548.1-clickhouse-3k9uBe: File exists (os error 17)
    WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: dendrite temporary directory leaked: /dangerzone/omicron_tmp/.tmpGAJeYD
    WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: mgd temporary directory leaked: /dangerzone/omicron_tmp/.tmp2Iulkr

        PASS [  55.511s] omicron-nexus::test_all integration_tests::volume_management::test_delete_image_order_3
        PASS [  30.071s] omicron-nexus::test_all integration_tests::volume_management::test_keep_your_targets_straight
        PASS [  41.218s] omicron-nexus::test_all integration_tests::volume_management::test_migrate_to_ref_count_with_records
        PASS [  39.401s] omicron-nexus::test_all integration_tests::volume_management::test_migrate_to_ref_count_with_records_region_snapshot_deleting
        SLOW [> 60.000s] omicron-nexus::test_all integration_tests::volume_management::test_double_layer_snapshot_with_read_only_region_delete_2
        PASS [  41.899s] omicron-nexus::test_all integration_tests::volume_management::test_multiple_disks_multiple_snapshots_order_1
        PASS [  62.382s] omicron-nexus::test_all integration_tests::volume_management::test_double_layer_snapshot_with_read_only_region_delete_2
        SLOW [> 60.000s] omicron-nexus::test_all integration_tests::volume_management::test_double_layer_with_read_only_region_delete
        PASS [  62.939s] omicron-nexus::test_all integration_tests::volume_management::test_double_layer_with_read_only_region_delete
        PASS [  47.785s] omicron-nexus::test_all integration_tests::volume_management::test_migrate_to_ref_count_with_records_soft_delete_volume
        PASS [  43.283s] omicron-nexus::test_all integration_tests::volume_management::test_multiple_layers_of_snapshots_delete_all_snapshots_first
        PASS [  55.159s] omicron-nexus::test_all integration_tests::volume_management::test_multiple_disks_multiple_snapshots_order_2
        PASS [ 114.815s] omicron-nexus::test_all integration_tests::unauthorized::test_unauthorized
        SLOW [> 60.000s] omicron-nexus::test_all integration_tests::volume_management::test_multiple_layers_of_snapshots_delete_all_disks_first
        PASS [  60.717s] omicron-nexus::test_all integration_tests::volume_management::test_multiple_layers_of_snapshots_delete_all_disks_first
        PASS [  51.344s] omicron-nexus::test_all integration_tests::volume_management::test_multiple_snapshots
        PASS [  59.758s] omicron-nexus::test_all integration_tests::volume_management::test_multiple_layers_of_snapshots_random_delete_order
        PASS [  30.351s] omicron-nexus::test_all integration_tests::volume_management::test_read_only_region_reference_sanity_multi
        PASS [  57.401s] omicron-nexus::test_all integration_tests::volume_management::test_no_zombie_region_snapshots
        PASS [  27.410s] omicron-nexus::test_all integration_tests::volume_management::test_snapshot_create_saga_unwinds_correctly
        SLOW [> 60.000s] omicron-nexus::test_all integration_tests::volume_management::test_no_zombie_read_only_regions
        SLOW [> 60.000s] omicron-nexus::test_all integration_tests::volume_management::test_no_zombie_read_write_regions
        PASS [  62.183s] omicron-nexus::test_all integration_tests::volume_management::test_no_zombie_read_only_regions
        SLOW [> 60.000s] omicron-nexus::test_all integration_tests::volume_management::test_proper_region_sled_redundancy
        PASS [  15.888s] omicron-nexus::test_all integration_tests::volume_management::test_upstairs_notify_downstairs_client_stop_request
        PASS [  15.715s] omicron-nexus::test_all integration_tests::volume_management::test_upstairs_notify_downstairs_client_stops
        PASS [  16.640s] omicron-nexus::test_all integration_tests::volume_management::test_upstairs_repair_different_upstairs_retry
        PASS [  18.283s] omicron-nexus::test_all integration_tests::volume_management::test_upstairs_repair_different_upstairs_retry_interrupted
        PASS [  48.731s] omicron-nexus::test_all integration_tests::volume_management::test_read_only_region_reference_sanity_rop
        PASS [  73.272s] omicron-nexus::test_all integration_tests::volume_management::test_no_zombie_read_write_regions
        FAIL [  47.413s] omicron-nexus::test_all integration_tests::volume_management::test_read_only_region_reference_usage_sanity
  stdout ───

    running 1 test
    test integration_tests::volume_management::test_read_only_region_reference_usage_sanity ... FAILED

    failures:

    failures:
        integration_tests::volume_management::test_read_only_region_reference_usage_sanity

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 454 filtered out; finished in 46.86s
    
  stderr ───
    log file: /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_read_only_region_reference_usage_sanity.25058.0.log
    note: configured to log to "/dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_read_only_region_reference_usage_sanity.25058.0.log"
    DB URL: postgresql://root@[::1]:58074/omicron?sslmode=disable
    DB address: [::1]:58074
    log file: /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_read_only_region_reference_usage_sanity.25058.2.log
    note: configured to log to "/dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_read_only_region_reference_usage_sanity.25058.2.log"

    thread 'integration_tests::volume_management::test_read_only_region_reference_usage_sanity' panicked at /home/dap/omicron-work/nexus/test-utils/src/resource_helpers.rs:1697:10:
    expected to find inventory collection: TimedOut(30.042680838s)
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
    WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: temporary directory leaked: "/dangerzone/omicron_tmp/.tmpjZ9GXT"
        If you would like to access the database for debugging, run the following:

        # Run the database
        cargo xtask db-dev run --no-populate --store-dir "/dangerzone/omicron_tmp/.tmpjZ9GXT/data"
        # Access the database. Note the port may change if you run multiple databases.
        cockroach sql --host=localhost:32221 --insecure
    WARN: dropped ClickHouse process without cleaning it up first (there may still be a child process running (PID 25078) and a temporary directory leaked, /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_read_only_region_reference_usage_sanity.25058.1-clickhouse-xBzca7)
    failed to clean up ClickHouse data dir:
    - /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_read_only_region_reference_usage_sanity.25058.1-clickhouse-xBzca7: File exists (os error 17)
    WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: dendrite temporary directory leaked: /dangerzone/omicron_tmp/.tmpsoFVjW
    WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: mgd temporary directory leaked: /dangerzone/omicron_tmp/.tmpsW9j5D

        SLOW [> 60.000s] omicron-nexus::test_all integration_tests::volume_management::test_read_only_region_reference_counting
        PASS [  33.209s] omicron-nexus::test_all integration_tests::volume_management::test_snapshot_then_delete_disk
        PASS [  19.545s] omicron-nexus::test_all integration_tests::volume_management::test_upstairs_repair_notify_different_finish_status
        SLOW [> 60.000s] omicron-nexus::test_all integration_tests::volume_management::test_read_only_region_reference_counting_layers
        PASS [  19.583s] omicron-nexus::test_all integration_tests::volume_management::test_upstairs_repair_notify_idempotent
        PASS [  42.056s] omicron-nexus::test_all integration_tests::volume_management::test_snapshot_prevents_other_disk
        PASS [  19.775s] omicron-nexus::test_all integration_tests::volume_management::test_upstairs_repair_reject_submit_progress_when_no_repair
        PASS [  20.227s] omicron-nexus::test_all integration_tests::volume_management::test_upstairs_repair_repair_id_and_type_conflict
        PASS [  57.371s] omicron-nexus::test_all integration_tests::volume_management::test_read_only_region_reference_sanity_rop_multi
        PASS [  21.751s] omicron-nexus::test_all integration_tests::volume_management::test_upstairs_repair_same_upstairs_retry
        FAIL [  72.444s] omicron-nexus::test_all integration_tests::volume_management::test_read_only_region_reference_counting
  stdout ───

    running 1 test
    test integration_tests::volume_management::test_read_only_region_reference_counting has been running for over 60 seconds
    test integration_tests::volume_management::test_read_only_region_reference_counting ... FAILED

    failures:

    failures:
        integration_tests::volume_management::test_read_only_region_reference_counting

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 454 filtered out; finished in 71.94s
    
  stderr ───
    log file: /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_read_only_region_reference_counting.24972.0.log
    note: configured to log to "/dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_read_only_region_reference_counting.24972.0.log"
    DB URL: postgresql://root@[::1]:61436/omicron?sslmode=disable
    DB address: [::1]:61436
    log file: /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_read_only_region_reference_counting.24972.2.log
    note: configured to log to "/dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_read_only_region_reference_counting.24972.2.log"
    log file: /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_read_only_region_reference_counting.24972.3.log
    note: configured to log to "/dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_read_only_region_reference_counting.24972.3.log"

    thread 'integration_tests::volume_management::test_read_only_region_reference_counting' panicked at /home/dap/omicron-work/nexus/test-utils/src/resource_helpers.rs:1697:10:
    expected to find inventory collection: TimedOut(30.020343641s)
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
    WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: temporary directory leaked: "/dangerzone/omicron_tmp/.tmp32F0hD"
        If you would like to access the database for debugging, run the following:

        # Run the database
        cargo xtask db-dev run --no-populate --store-dir "/dangerzone/omicron_tmp/.tmp32F0hD/data"
        # Access the database. Note the port may change if you run multiple databases.
        cockroach sql --host=localhost:32221 --insecure
    WARN: dropped ClickHouse process without cleaning it up first (there may still be a child process running (PID 24980) and a temporary directory leaked, /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_read_only_region_reference_counting.24972.1-clickhouse-9vLW6x)
    failed to clean up ClickHouse data dir:
    - /dangerzone/omicron_tmp/test_all-09f39310580ee44c-test_read_only_region_reference_counting.24972.1-clickhouse-9vLW6x: File exists (os error 17)
    WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: dendrite temporary directory leaked: /dangerzone/omicron_tmp/.tmprXkUUC
    WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: dendrite temporary directory leaked: /dangerzone/omicron_tmp/.tmpDO3Az6
    WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: mgd temporary directory leaked: /dangerzone/omicron_tmp/.tmpYRduQQ
    WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: mgd temporary directory leaked: /dangerzone/omicron_tmp/.tmpZpLngp

        PASS [  25.436s] omicron-nexus::test_all integration_tests::volume_management::test_upstairs_repair_submit_progress
...
        PASS [   0.021s] omicron-passwords test::test_password_constraints
        FAIL [  11.927s] omicron-omdb::test_all_output test_omdb_usage_errors
  stdout ───

    running 1 test
    @@ -847,10 +847,11 @@
       blueprints            interact with blueprints
       chicken-switches      interact with reconfigurator chicken switches
       clickhouse-policy     interact with clickhouse policy
       mgs-updates           print information about pending MGS updates
       oximeter-read-policy  interact with oximeter read policy
    +  quiesce               view or modify the quiesce status
       sagas                 view sagas, create and complete demo sagas
       sleds                 interact with sleds
       support-bundles       interact with support bundles [aliases: sb]
       update-status         show running artifact versions
       help                  Print this message or the help of the given subcommand(s)

    test test_omdb_usage_errors ... FAILED

    failures:

    failures:
        test_omdb_usage_errors

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 2 filtered out; finished in 11.87s
    
  stderr ───

    thread 'test_omdb_usage_errors' panicked at dev-tools/omdb/tests/test_all_output.rs:129:5:
    assertion failed: string doesn't match the contents of file: "tests/usage_errors.out" see diffset above
                    set EXPECTORATE=overwrite if these changes are intentional
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        PASS [  20.354s] omicron-nexus::test_all integration_tests::webhooks::test_subscription_remove
        PASS [   0.044s] omicron-passwords test::test_stable
...
        PASS [  38.121s] sled-agent-config-reconciler dataset_serialization_task::illumos_tests::ensure_many_datasets
        PASS [  35.993s] sled-storage manager::tests::ensure_many_datasets
────────────
     Summary [1935.375s] 2194 tests run: 2183 passed (95 slow), 11 failed, 3 skipped
        FAIL [  53.948s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_replacement_state_replacement_done
        FAIL [  81.001s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_replacement_state_requested
        FAIL [  53.394s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_replacement_state_running_on_pantry
        FAIL [  53.295s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_state_requested
        FAIL [  52.203s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_state_running
        FAIL [  51.029s] omicron-nexus::test_all integration_tests::disks::test_disk_expunge
        FAIL [  50.192s] omicron-nexus::test_all integration_tests::disks::test_disk_slot_assignment
        FAIL [  51.146s] omicron-nexus::test_all integration_tests::volume_management::test_delete_image_order_5
        FAIL [  72.444s] omicron-nexus::test_all integration_tests::volume_management::test_read_only_region_reference_counting
        FAIL [  47.413s] omicron-nexus::test_all integration_tests::volume_management::test_read_only_region_reference_usage_sanity
        FAIL [  11.927s] omicron-omdb::test_all_output test_omdb_usage_errors
error: test run failed

[leftwo]

It seems like all the crucible tests are failing with this:

expected to find inventory collection: TimedOut(30.020343641s)

At least, I can find that output in all the failing tests.

[davepacheco]

On "main" (commit 76fb94), I got a similar result:

────────────
     Summary [1673.675s] 2192 tests run: 2178 passed (54 slow), 14 failed, 3 skipped
        FAIL [  55.925s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_replacement_state_running
        FAIL [  52.915s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_replacement_state_running_on_pantry
        FAIL [  51.768s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_state_replacement_done
        FAIL [  51.374s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_state_requested
        FAIL [  51.517s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_state_requested_2
        FAIL [  49.207s] omicron-nexus::test_all integration_tests::crucible_replacements::test_racing_replacements_for_soft_deleted_disk_volume
        FAIL [  49.257s] omicron-nexus::test_all integration_tests::crucible_replacements::test_read_only_replacement_sanity
        FAIL [  49.648s] omicron-nexus::test_all integration_tests::crucible_replacements::test_region_replacement_triple_sanity
        FAIL [  74.268s] omicron-nexus::test_all integration_tests::crucible_replacements::test_region_replacement_triple_sanity_2
        FAIL [  49.862s] omicron-nexus::test_all integration_tests::crucible_replacements::test_replacement_sanity_twice
        FAIL [  45.417s] omicron-nexus::test_all integration_tests::disks::test_disk_region_creation_failure
        FAIL [  48.511s] omicron-nexus::test_all integration_tests::instances::test_cannot_attach_nine_disks_to_instance
        FAIL [  71.073s] omicron-nexus::test_all integration_tests::volume_management::test_migrate_to_ref_count_with_records
        FAIL [  47.345s] omicron-nexus::test_all integration_tests::volume_management::test_multiple_layers_of_snapshots_delete_all_snapshots_first
error: test run failed

They're all the same timeout trying to find an inventory collection.

[davepacheco]

Similar errors were reported in #8568, though that issue also talks about some other errors.

[andrewjstone]

Another run produced a few errors: https://github.com/oxidecomputer/omicron/pull/8878/checks?check_run_id=48935034794

Summary [5598.505s] 2208 tests run: 2205 passed (167 slow), 3 failed, 3 skipped
--
9296 | 2025-08-26T19:50:38.544Z | FAIL [ 173.854s] omicron-nexus::test_all integration_tests::crucible_replacements::test_region_replacement_triple_sanity_2
9297 | 2025-08-26T19:50:38.544Z | FAIL [  67.612s] omicron-nexus::test_all integration_tests::crucible_replacements::test_replacement_sanity
9298 | 2025-08-26T19:50:38.544Z | FAIL [  59.851s] omicron-omdb::test_all_output test_omdb_success_cases
9299 | 2025-08-26T19:50:38.628Z | error: test run failed

Summary [5598.505s] 2208 tests run: 2205 passed (167 slow), 3 failed, 3 skipped
[9296](https://buildomat.eng.oxide.computer/wg/0/details/01K3H9XRPGF7BNGWH4NHQM4Z3S/M8Iq0v43RCGitgB0ciOhKSSAiAKjOjKeCyz5jtHxeHY6EjZF/01K3KQ0Q21ZEPA5AR3RXHZY1TT#S9296)	2025-08-26T19:50:38.544Z	        FAIL [ 173.854s] omicron-nexus::test_all integration_tests::crucible_replacements::test_region_replacement_triple_sanity_2
[9297](https://buildomat.eng.oxide.computer/wg/0/details/01K3H9XRPGF7BNGWH4NHQM4Z3S/M8Iq0v43RCGitgB0ciOhKSSAiAKjOjKeCyz5jtHxeHY6EjZF/01K3KQ0Q21ZEPA5AR3RXHZY1TT#S9297)	2025-08-26T19:50:38.544Z	        FAIL [  67.612s] omicron-nexus::test_all integration_tests::crucible_replacements::test_replacement_sanity
[9298](https://buildomat.eng.oxide.computer/wg/0/details/01K3H9XRPGF7BNGWH4NHQM4Z3S/M8Iq0v43RCGitgB0ciOhKSSAiAKjOjKeCyz5jtHxeHY6EjZF/01K3KQ0Q21ZEPA5AR3RXHZY1TT#S9298)	2025-08-26T19:50:38.544Z	        FAIL [  59.851s] omicron-omdb::test_all_output test_omdb_success_cases
[9299](https://buildomat.eng.oxide.computer/wg/0/details/01K3H9XRPGF7BNGWH4NHQM4Z3S/M8Iq0v43RCGitgB0ciOhKSSAiAKjOjKeCyz5jtHxeHY6EjZF/01K3KQ0Q21ZEPA5AR3RXHZY1TT#S9299)	2025-08-26T19:50:38.628Z	error: test run failed

[andrewjstone]

And on a separate PR: https://github.com/oxidecomputer/omicron/pull/8682/checks?check_run_id=48935048199

Summary [5577.333s] 2211 tests run: 2209 passed (158 slow), 2 failed, 3 skipped
--
9174 | 2025-08-26T19:48:17.738Z | FAIL [ 164.099s] omicron-nexus::test_all integration_tests::crucible_replacements::test_region_replacement_triple_sanity_2
9175 | 2025-08-26T19:48:17.738Z | FAIL [  64.334s] omicron-omdb::test_all_output test_omdb_success_cases
9176 | 2025-08-26T19:48:17.830Z | error: test run failed

Summary [5577.333s] 2211 tests run: 2209 passed (158 slow), 2 failed, 3 skipped
[9174](https://buildomat.eng.oxide.computer/wg/0/details/01K3KCXZBHVKV4D0YG7NXF0SNH/Ksnk5OpVs1IARUt4aGsLU4gKI7Wg8ulDVUNU3ewSypsCiMGU/01K3KQ116DHHA13J4MC3SHCKZC#S9174)	2025-08-26T19:48:17.738Z	        FAIL [ 164.099s] omicron-nexus::test_all integration_tests::crucible_replacements::test_region_replacement_triple_sanity_2
[9175](https://buildomat.eng.oxide.computer/wg/0/details/01K3KCXZBHVKV4D0YG7NXF0SNH/Ksnk5OpVs1IARUt4aGsLU4gKI7Wg8ulDVUNU3ewSypsCiMGU/01K3KQ116DHHA13J4MC3SHCKZC#S9175)	2025-08-26T19:48:17.738Z	        FAIL [  64.334s] omicron-omdb::test_all_output test_omdb_success_cases
[9176](https://buildomat.eng.oxide.computer/wg/0/details/01K3KCXZBHVKV4D0YG7NXF0SNH/Ksnk5OpVs1IARUt4aGsLU4gKI7Wg8ulDVUNU3ewSypsCiMGU/01K3KQ116DHHA13J4MC3SHCKZC#S9176)	2025-08-26T19:48:17.830Z	error: test run failed

[jmpesp]

I wonder if these issues are related to the _rst_sent_rate issue. I haven't dug into all the recent ones but whenever I have, it's due to large delays between the saga steps.

[smklein]

I wonder if these issues are related to the _rst_sent_rate issue. I haven't dug into all the recent ones but whenever I have, it's due to large delays between the saga steps.

For context, this is #8762

For the rst_sent_rate to be the issue here, a few things must be true:

1. The test must be running on Helios
2. We must be making IP requests to a port on localhost, for which no one is listening. This happens during inventory collection.

I've seen this failure happen on #8756 (comment), with the comment:

expected to find inventory collection: TimedOut(30.074112585s)

Is that happening here? These look like Helios tests, but I'm not seeing the same inventory-related error message in @andrewjstone 's most recent logs.

[smklein]

Just saw test_delete_volume_region_snapshot_replacement_state_requested fail on a Helios machine. Not seeing that expected to find inventory collection: TimedOut message anywhere in the logs.

Run: https://github.com/oxidecomputer/omicron/pull/8931/checks?check_run_id=49050162164
Logs: https://buildomat.eng.oxide.computer/wg/0/details/01K3PY1PTHZXFT93AM4P0R24VY/Cw2PvWC9l1Zq41udFUCHdS61i41n3OSzOqWfb6yBXOvZkByE/01K3PY3RFKBNTRQ0Y1NJVGHFN9

Although this is a test timeout, I'm seeing the following assertion failure in the output:

2025-08-28T01:08:17.965Z	    thread 'integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_state_requested' panicked at nexus/tests/integration_tests/crucible_replacements.rs:1501:13:
2025-08-28T01:08:17.965Z	    assertion failed: volumes.is_empty()

Even if the rst_sent_rate issue was in play - which I don't think we have evidence for, at the moment - it would only cause delays, and would not be the cause of an assertion failure like this.

[smklein]

This assertion failure looks like it's coming from async fn finish_test, which is trying to "Assert no volumes are referencing the snapshot address".

This is asserting that volumes is empty, but it's not. It contains the following, which is returned by find_volumes_referencing_socket_addr:

[
  Volume { 
    identity: VolumeIdentity {
       id: 5a4f4a30-1e46-4e39-94b7-8351052d585b (volume), time_created: 2025-08-28T01:07:47.712864Z, time_modified: 2025-08-28T01:07:47.712864Z
     },
     time_deleted: Some(2025-08-28T01:08:01.962750Z),
     rcgen: Generation(Generation(1)),
     data: "{\"type\":\"volume\",\"id\":\"28f4dfe4-be30-4b2b-98c4-70c37ef0151b\",\"block_size\":512,\"sub_volumes\":[{\"type\":\"region\",\"block_size\":512,\"blocks_per_extent\":131072,\"extent_count\":16,\"opts\":{\"id\":\"cc5c23d2-3838-4f91-bd91-66ba39e5e3ab\",\"target\":[\"[::1]:4002\",\"[::1]:2001\",\"[::1]:1002\"],\"lossy\":false,\"flush_timeout\":null,\"key\":\"hPirEnXLVHd7hdwBIvJn6nnPfqd4cWMsrUUQw4AOkTk=\",\"cert_pem\":null,\"key_pem\":null,\"root_cert_pem\":null,\"control\":null,\"read_only\":true},\"gen\":3}],\"read_only_parent\":null}", resources_to_clean_up: Some("{\"V3\":{\"regions\":[],\"region_snapshots\":[]}}") 
  }
]

[bnaecker]

I hit a similar timeout in test_delete_volume_region_replacement_state_replacement_done here, on #9103. It's another example of a timeout doing something other than inventory collection. Here's the actual fatal error:

thread 'integration_tests::crucible_replacements::test_delete_volume_region_replacement_state_replacement_done' panicked at nexus/tests/integration_tests/crucible_replacements.rs:486:14:
request transitioned to expected state: TimedOut(260.015816019s)

[jgallagher]

I hit the same thing as Ben on #9149 (https://buildomat.eng.oxide.computer/wg/0/details/01K6NWH91ZECNBVV8Q4JZM3PHB/yhiuubh7eUajYdiOUVB2OtOlyGac1ArD4m9q3kFkySmCUduh/01K6NWHS7EFVE59ACQGR8T9FC7):

thread 'integration_tests::crucible_replacements::test_delete_volume_region_replacement_state_replacement_done' panicked at nexus/tests/integration_tests/crucible_replacements.rs:486:14:
request transitioned to expected state: TimedOut(260.002503813s)