[sim] Optionally enable health monitor

[karencfv]

Adds the ability to enable the sled agent health monitor on simulated systems. This is and will be very useful for various types of testing.

Disabled:

$ cargo xtask omicron-dev run-all
<...>
omicron-dev: sled agent API:         http://[::1]:58125
<...>

$ curl -H "api-version: 12.0.0"  http://[::1]:58125/inventory | jq
<...>
  "health_monitor": {
    "smf_services_in_maintenance": {
      "ok": {
        "services": [],
        "errors": [],
        "time_of_status": null
      }
    }
  }
}

Enabled:

$ cargo xtask omicron-dev run-all --enable-sled-agent-health-monitor
<...>
omicron-dev: sled agent API:         http://[::1]:33388
<...>

$ curl -H "api-version: 12.0.0"  http://[::1]:33388/inventory | jq
<...>
  "health_monitor": {
    "smf_services_in_maintenance": {
      "ok": {
        "services": [
          {
            "fmri": "svc:/site/fake-service2:default",
            "zone": "global"
          }
        ],
        "errors": [],
        "time_of_status": "2026-01-13T02:44:01.560645575Z"
      }
    }
  }
}

Closes: #9517

[davepacheco]

Cool. Does this cause the simulated sled agent to look at the actual SMF state wherever it's running? Wouldn't it be more useful to allow the reported state to be customized directly?

[karencfv]

Does this cause the simulated sled agent to look at the actual SMF state wherever it's running?

Yes, but that was the use case I've been having 😄.

Wouldn't it be more useful to allow the reported state to be customized directly?

That would be really useful too. I wonder if there is a possibility to have one or the other. Do you think it' would be relatively straightforward to do that?