should `instance-stop` *really* move instances to failed when they are discovered to already be gone?

[hawkw]

The PUT /instances/{instance}/stop API will send a request to sled-agent asking it to terminate a running instance. If sled-agent responds with something indicating that it actually didn't know about that instance in the first place, Nexus will then transition it to Failed:

omicron/nexus/src/app/instance.rs

Lines 755 to 775 in 0640bb2

	if let Err(e) = self
	.instance_request_state(
	opctx,
	state.instance(),
	state.vmm(),
	InstanceStateChangeRequest::Stop,
	)
	.await
	{
	if let (InstanceStateChangeError::SledAgent(inner), Some(vmm)) =
	(&e, state.vmm())
	{
	if inner.vmm_gone() {
	let _ = self
	.mark_vmm_failed(opctx, authz_instance, vmm, inner)
	.await;
	}
	}
	return Err(e);
	}

This will not happen when the reason the instance is gone is because another concurrent attempt to stop it has succeeded, because the racing stop attempt will have advanced the VMM's generation number whilst moving it to Destroyed, so we won't mark it as Failed. However, in the event of a sled-agent crash, we may encounter an already-gone VMM here, and may move it to Failed.

This seems a bit wacky to me, since Failed instances (which is what the instance will eventually become as a result of its VMM being marked Failed) are eligible to be auto-restarted, while Stopped instances are not --- because the user actually wanted them to be stopped. And, in this case, the user is expressing intent to have an instance stop running, and we just happened to discover that we had already anticipated their desire to stop it and went ahead and stopped it for them before they even asked us to. Admittedly, we weren't supposed to have done that! But, in this case, the requested state is "instance is not running", and it's not running, so it seems a bit unfortunate to go "oh no, i was supposed to make the instance not be running, and when i tried to do that, i discovered that it was not running because we made a mistake, so now i'm actually going to...make it be running again?"

Imagine a scenario where a user goes to stop an instance so that they can change its boot disk or something, and while doing so, we discover that sled-agent has crashed and the instance isn't there. Moving it to Failed results in the instance being restarted, so now the user has to stop the instance a second time before they can actually do what they were trying to do originally.

Maybe we should just always move the instance to stopped when such an error is encountered by an instance-stop attempt. Obviously we would still go to Failed when attempting to do other things to the instance.

[faithanalog]

I don't think the instance should ever turn on if i told it to be stopped. If "failed" is useful information for the user in this case, perhaps a "failed but don't auto restart" would be a useful backend state to create.

But I'm not sure if it is. What would I do if I told my machine to stop, and it said "failed"?

I might think "maybe it failed to stop. is it still running? well I can't get to it on the network, so at least services don't seem to be running, or maybe it tore down networking. ???"

Or I might think "well it's not running. but did I break something in the backend I need to worry about?"

[gjcolombo]

I agree with pretty much all of the above.

The piece of information that gets lost by transitioning to Stopped (and not Failed) when a VMM is gone is the unexpectedness of the VMM's absence. The tradeoff is that (by default) a Failed instance is eligible to be restarted automatically, and I agree that this doesn't make very much sense if we know for certain that the user intends for the VM not to be running, which is the case in the context of a Stop API call. It seems way more sensible to drop the "oops it didn't reach the end of the ride" bit than to force a user to do more work to keep an instance from restarting.¹

---

In terms of what would need to change in Nexus to do this: I think some care is required because an instance-stop API call can race with the instance watcher to determine the final state of a missing VMM:

1. Stop API observes VMM record for missing VMM at generation 10
2. Instance watcher task reads VMM record for missing VMM at generation 10
3. Watcher calls sled agent, gets back "not here"
4. Watcher advances VMM to Failed at generation 11
5. Stop API tries to advance VMM to Destroyed, can't do so because the VMM is already at generation 11

I'd have to think about what to do in this case--does the Stop API call fail (with a 409 or something like that), or does it try to advance the VMM record to Destroyed at generation 12 and then succeed? If the latter, what if the reincarnation task gets to the instance in the meantime and restarts it? Then you have a Stop request that returns a Running instance... It might be simplest just to fail if the stop API was not able to take some meaningful action to drive the instance's state to Stopped.

---

Assuming these practical details sound OK, I'm on board with this conception of the purpose of instance_stop: the point is to drive the system toward a state where the instance has no active VMM and is not eligible to be automatically started, and observing that there's no active VMM (and successfully commandeering its state machine if needed) achieves that goal. If we really want we can always add a separate mechanism later to report that the instance didn't keep its limbs inside the state machine before it came to a complete stop.

Footnotes

1. This is particularly true today because the stop API pulls power from the guest. The distinction is maybe more meaningful for VMs that users shut down from within the guest--in that case Failed is more indicative of a guest that may not have gone through a clean shutdown sequence. ↩

[askfongjojo]

I've noticed a race situation on rack2 which has omicron commit 5fd1c35 (hit it 3 times so far out of a few hundreds stop instance requests). It appears that Nexus declares the instance failed as soon as it receives a 404 for vmm GET request. Here is an example:

23:06:55.042Z Nexus received a vmm_state of "Stopping" (Gen 5) from sled-agent
23:06:55.043Z Sled-agent pushed another state update, still "Stopping" (Gen 6)
23:06:55.063Z Sled-agent pushed another state update, still "Stopping" (Gen 7)
23:06:55.787Z Sled-agent noticed propolis zone was destroyed
23:07:01.284Z Instance watcher got a 404 for vmm state GET API response
23:07:01.284Z Nexus issued a state update to "Failed" state (Gen 8)
23:07:01.802Z Sled-agent pushed the "Destroyed" state update (Gen 9)

After this, a reincarnation saga was kicked off to restart the instance, essentially rolling back what the user was trying to do. This is especially problematic when the stop request is meant to prepare a VM for modification or termination, resulting in failures of subsequent steps.

The problem here is that it sometimes takes seconds for sled-agent to finish creating the zone bundle if the sled-agent is busy serving multiple requests or if the zone has been running for some time and accumulated a lot of logs (in this case, the instance only lived for a few minutes so there shouldn't be much to collect).

Here are the relevant database records and log lines:

root@[fd00:1122:3344:105::3]:32221/omicron> select * from vmm where instance_id = '018a2717-8d34-4b74-b8fd-0d666aee5d03';
                   id                  |         time_created          |         time_deleted          |             instance_id              |      time_state_updated       | state_generation |               sled_id                |        propolis_ip        | propolis_port |   state
---------------------------------------+-------------------------------+-------------------------------+--------------------------------------+-------------------------------+------------------+--------------------------------------+---------------------------+---------------+------------
  82186f75-3cd7-4498-a861-a1313ca4dba3 | 2025-04-01 23:07:02.537643+00 | NULL                          | 018a2717-8d34-4b74-b8fd-0d666aee5d03 | 2025-04-01 23:07:14.156769+00 |                4 | 0c7011f7-a4bf-4daf-90cc-1c2410103300 | fd00:1122:3344:104::1:60c |         12400 | running
  904183f8-bd4e-4637-9216-4f4aa07ac2be | 2025-04-01 23:03:06.407413+00 | 2025-04-01 23:07:01.941629+00 | 018a2717-8d34-4b74-b8fd-0d666aee5d03 | 2025-04-01 23:07:01.296805+00 |                9 | f15774c1-b8e5-434f-a493-ec43f96cba06 | fd00:1122:3344:105::1:979 |         12400 | destroyed
(2 rows)

23:06:23.475Z INFO 65a11c18-7f59-41ac-b9e7-680627f996e7 (ServerContext): received new VMM runtime state from sled agent
    background_task = instance_watcher
    file = nexus/src/app/instance.rs:2168
    instance_id = 018a2717-8d34-4b74-b8fd-0d666aee5d03
    migration_state = Migrations { migration_in: None, migration_out: None }
    propolis_id = 904183f8-bd4e-4637-9216-4f4aa07ac2be
    sled_id = f15774c1-b8e5-434f-a493-ec43f96cba06
    vmm_id = 904183f8-bd4e-4637-9216-4f4aa07ac2be
    vmm_state = VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2025-04-01T23:03:23.798893171Z }
23:06:23.475Z DEBG 65a11c18-7f59-41ac-b9e7-680627f996e7 (ServerContext): roles
    background_task = instance_watcher
    instance_id = 018a2717-8d34-4b74-b8fd-0d666aee5d03
    roles = RoleSet { roles: {} }
    sled_id = f15774c1-b8e5-434f-a493-ec43f96cba06
    vmm_id = 904183f8-bd4e-4637-9216-4f4aa07ac2be
23:06:23.475Z DEBG 65a11c18-7f59-41ac-b9e7-680627f996e7 (ServerContext): authorize result
    action = Query
    actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. })
    background_task = instance_watcher
    instance_id = 018a2717-8d34-4b74-b8fd-0d666aee5d03
    resource = Database
    result = Ok(())
    sled_id = f15774c1-b8e5-434f-a493-ec43f96cba06
    vmm_id = 904183f8-bd4e-4637-9216-4f4aa07ac2be
23:06:54.975Z DEBG 65a11c18-7f59-41ac-b9e7-680627f996e7 (dropshot_external): authorize result
    action = Modify
    actor = Some(Actor::SiloUser { silo_user_id: 648bcf81-e2ab-4017-af45-7bafb728bef3, silo_id: c7dd16b8-255e-4300-8e52-7396a72caf3e, .. })
    actor_id = 648bcf81-e2ab-4017-af45-7bafb728bef3
    authenticated = true
    local_addr = 172.30.2.5:443
    method = POST
    remote_addr = 172.20.17.42:51800
    req_id = a6695499-963e-4f7d-a995-bbd102547024
    resource = Instance { parent: Project { parent: Silo { parent: Fleet, key: c7dd16b8-255e-4300-8e52-7396a72caf3e, lookup_type: ById(c7dd16b8-255e-4300-8e52-7396a72caf3e) }, key: be6cc6d9-8b58-437f-ad95-c27d7c38bbcf, lookup_type: ByName("bob") }, key: 018a2717-8d34-4b74-b8fd-0d666aee5d03, lookup_type: ByName("app-14") }
    result = Ok(())
    uri = /v1/instances/app-14/stop?project=bob
23:06:54.994Z DEBG 65a11c18-7f59-41ac-b9e7-680627f996e7 (dropshot_external): authorize result
    action = Read
    actor = Some(Actor::SiloUser { silo_user_id: 648bcf81-e2ab-4017-af45-7bafb728bef3, silo_id: c7dd16b8-255e-4300-8e52-7396a72caf3e, .. })
    actor_id = 648bcf81-e2ab-4017-af45-7bafb728bef3
    authenticated = true
    local_addr = 172.30.2.5:443
    method = POST
    remote_addr = 172.20.17.42:51800
    req_id = a6695499-963e-4f7d-a995-bbd102547024
    resource = Instance { parent: Project { parent: Silo { parent: Fleet, key: c7dd16b8-255e-4300-8e52-7396a72caf3e, lookup_type: ById(c7dd16b8-255e-4300-8e52-7396a72caf3e) }, key: be6cc6d9-8b58-437f-ad95-c27d7c38bbcf, lookup_type: ByName("bob") }, key: 018a2717-8d34-4b74-b8fd-0d666aee5d03, lookup_type: ByName("app-14") }
    result = Ok(())
    uri = /v1/instances/app-14/stop?project=bob
23:06:55.041Z DEBG 65a11c18-7f59-41ac-b9e7-680627f996e7 (ServerContext): client request
    SledAgent = f15774c1-b8e5-434f-a493-ec43f96cba06
    body = Some(Body)
    method = PUT
    uri = http://[fd00:1122:3344:105::1]:12345/vmms/904183f8-bd4e-4637-9216-4f4aa07ac2be/state
23:06:55.042Z DEBG 65a11c18-7f59-41ac-b9e7-680627f996e7 (ServerContext): client response
    SledAgent = f15774c1-b8e5-434f-a493-ec43f96cba06
    result = Ok(Response { url: "http://[fd00:1122:3344:105::1]:12345/vmms/904183f8-bd4e-4637-9216-4f4aa07ac2be/state", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac35aa53-f8d9-48de-964d-87eeef6c8142", "content-length": "151", "date": "Tue, 01 Apr 2025 23:06:54 GMT"} })
23:06:55.042Z INFO 65a11c18-7f59-41ac-b9e7-680627f996e7 (dropshot_external): received new VMM runtime state from sled agent
    actor_id = 648bcf81-e2ab-4017-af45-7bafb728bef3
    authenticated = true
    file = nexus/src/app/instance.rs:2168
    local_addr = 172.30.2.5:443
    method = POST
    migration_state = Migrations { migration_in: None, migration_out: None }
    propolis_id = 904183f8-bd4e-4637-9216-4f4aa07ac2be
    remote_addr = 172.20.17.42:51800
    req_id = a6695499-963e-4f7d-a995-bbd102547024
    uri = /v1/instances/app-14/stop?project=bob
    vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2025-04-01T23:06:55.042652679Z }
23:06:55.072Z DEBG 65a11c18-7f59-41ac-b9e7-680627f996e7 (dropshot_external): authorize result
    action = Read
    actor = Some(Actor::SiloUser { silo_user_id: 648bcf81-e2ab-4017-af45-7bafb728bef3, silo_id: c7dd16b8-255e-4300-8e52-7396a72caf3e, .. })
    actor_id = 648bcf81-e2ab-4017-af45-7bafb728bef3
    authenticated = true
    local_addr = 172.30.2.5:443
    method = POST
    remote_addr = 172.20.17.42:51800
    req_id = a6695499-963e-4f7d-a995-bbd102547024
    resource = Instance { parent: Project { parent: Silo { parent: Fleet, key: c7dd16b8-255e-4300-8e52-7396a72caf3e, lookup_type: ById(c7dd16b8-255e-4300-8e52-7396a72caf3e) }, key: be6cc6d9-8b58-437f-ad95-c27d7c38bbcf, lookup_type: ByName("bob") }, key: 018a2717-8d34-4b74-b8fd-0d666aee5d03, lookup_type: ByName("app-14") }
    result = Ok(())
    uri = /v1/instances/app-14/stop?project=bob
23:07:01.268Z DEBG 65a11c18-7f59-41ac-b9e7-680627f996e7 (ServerContext): client request
    SledAgent = f15774c1-b8e5-434f-a493-ec43f96cba06
    background_task = instance_watcher
    body = None
    method = GET
    uri = http://[fd00:1122:3344:105::1]:12345/vmms/904183f8-bd4e-4637-9216-4f4aa07ac2be/state
23:07:01.284Z DEBG 65a11c18-7f59-41ac-b9e7-680627f996e7 (ServerContext): client response
    SledAgent = f15774c1-b8e5-434f-a493-ec43f96cba06
    background_task = instance_watcher
    result = Ok(Response { url: "http://[fd00:1122:3344:105::1]:12345/vmms/904183f8-bd4e-4637-9216-4f4aa07ac2be/state", status: 404, headers: {"content-type": "application/json", "x-request-id": "1c7da6a2-8381-48cd-af2e-2fad22d8ef25", "content-length": "120", "date": "Tue, 01 Apr 2025 23:07:01 GMT"} })
23:07:01.284Z INFO 65a11c18-7f59-41ac-b9e7-680627f996e7 (ServerContext): sled-agent error indicates that this instance's VMM has failed!
    background_task = instance_watcher
    error = Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "1c7da6a2-8381-48cd-af2e-2fad22d8ef25", "content-length": "120", "date": "Tue, 01 Apr 2025 23:07:01 GMT"}; value: Error { error_code: Some("NO_SUCH_INSTANCE"), message: "Not Found", request_id: "1c7da6a2-8381-48cd-af2e-2fad22d8ef25" }
    file = nexus/src/app/background/tasks/instance_watcher.rs:161
    instance_id = 018a2717-8d34-4b74-b8fd-0d666aee5d03
    sled_id = f15774c1-b8e5-434f-a493-ec43f96cba06
    vmm_id = 904183f8-bd4e-4637-9216-4f4aa07ac2be
23:07:01.284Z DEBG 65a11c18-7f59-41ac-b9e7-680627f996e7 (ServerContext): updating instance state
    background_task = instance_watcher
    instance_id = 018a2717-8d34-4b74-b8fd-0d666aee5d03
    sled_id = f15774c1-b8e5-434f-a493-ec43f96cba06
    state = SledVmmState { vmm_state: VmmRuntimeState { state: Failed, gen: Generation(8), time_updated: 2025-04-01T23:07:01.284770298Z }, migration_in: None, migration_out: None }
    vmm_id = 904183f8-bd4e-4637-9216-4f4aa07ac2be
23:07:01.284Z INFO 65a11c18-7f59-41ac-b9e7-680627f996e7 (ServerContext): received new VMM runtime state from sled agent
    background_task = instance_watcher
    file = nexus/src/app/instance.rs:2168
    instance_id = 018a2717-8d34-4b74-b8fd-0d666aee5d03
    migration_state = Migrations { migration_in: None, migration_out: None }
    propolis_id = 904183f8-bd4e-4637-9216-4f4aa07ac2be
    sled_id = f15774c1-b8e5-434f-a493-ec43f96cba06
    vmm_id = 904183f8-bd4e-4637-9216-4f4aa07ac2be
    vmm_state = VmmRuntimeState { state: Failed, gen: Generation(8), time_updated: 2025-04-01T23:07:01.284770298Z }
23:07:01.284Z DEBG 65a11c18-7f59-41ac-b9e7-680627f996e7 (ServerContext): roles
    background_task = instance_watcher
    instance_id = 018a2717-8d34-4b74-b8fd-0d666aee5d03
    roles = RoleSet { roles: {} }
    sled_id = f15774c1-b8e5-434f-a493-ec43f96cba06
    vmm_id = 904183f8-bd4e-4637-9216-4f4aa07ac2be
23:07:01.285Z DEBG 65a11c18-7f59-41ac-b9e7-680627f996e7 (ServerContext): authorize result
    action = Query
    actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. })
    background_task = instance_watcher
    instance_id = 018a2717-8d34-4b74-b8fd-0d666aee5d03
    resource = Database
    result = Ok(())
    sled_id = f15774c1-b8e5-434f-a493-ec43f96cba06
    vmm_id = 904183f8-bd4e-4637-9216-4f4aa07ac2be
23:07:01.295Z DEBG 65a11c18-7f59-41ac-b9e7-680627f996e7 (ServerContext): new VMM runtime state from sled agent requires an instance-update saga
    background_task = instance_watcher
    instance_id = 018a2717-8d34-4b74-b8fd-0d666aee5d03
    migration_in_needs_update = false
    migration_out_needs_update = false
    propolis_id = 904183f8-bd4e-4637-9216-4f4aa07ac2be
    sled_id = f15774c1-b8e5-434f-a493-ec43f96cba06
    vmm_id = 904183f8-bd4e-4637-9216-4f4aa07ac2be
    vmm_needs_update = true

Here are the related sled-agent log lines:

23:06:55.042Z INFO SledAgent (InstanceManager): updated state after observing Propolis state change
    file = sled-agent/src/instance.rs:926
    instance_id = 018a2717-8d34-4b74-b8fd-0d666aee5d03
    new_vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(6), time_updated: 2025-04-01T23:06:55.042887625Z }
    propolis_id = 904183f8-bd4e-4637-9216-4f4aa07ac2be
23:06:55.043Z INFO SledAgent (InstanceManager): Publishing instance state update to Nexus
    file = sled-agent/src/instance.rs:823
    instance_id = 018a2717-8d34-4b74-b8fd-0d666aee5d03
    propolis_id = 904183f8-bd4e-4637-9216-4f4aa07ac2be
    state = SledVmmState { vmm_state: VmmRuntimeState { state: Stopping, gen: Generation(6), time_updated: 2025-04-01T23:06:55.042887625Z }, migration_in: None, migration_out: None }
23:06:55.063Z INFO SledAgent (InstanceManager): updated state after observing Propolis state change
    file = sled-agent/src/instance.rs:926
    instance_id = 018a2717-8d34-4b74-b8fd-0d666aee5d03
    new_vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(7), time_updated: 2025-04-01T23:06:55.063487678Z }
    propolis_id = 904183f8-bd4e-4637-9216-4f4aa07ac2be
23:06:55.063Z INFO SledAgent (InstanceManager): Publishing instance state update to Nexus
    file = sled-agent/src/instance.rs:823
    instance_id = 018a2717-8d34-4b74-b8fd-0d666aee5d03
    propolis_id = 904183f8-bd4e-4637-9216-4f4aa07ac2be
    state = SledVmmState { vmm_state: VmmRuntimeState { state: Stopping, gen: Generation(7), time_updated: 2025-04-01T23:06:55.063487678Z }, migration_in: None, migration_out: None }
23:06:55.787Z INFO SledAgent (InstanceManager): updated state after observing Propolis state change
    file = sled-agent/src/instance.rs:926
    instance_id = 018a2717-8d34-4b74-b8fd-0d666aee5d03
    new_vmm_state = VmmRuntimeState { state: Destroyed, gen: Generation(8), time_updated: 2025-04-01T23:06:55.786979812Z }
    propolis_id = 904183f8-bd4e-4637-9216-4f4aa07ac2be
23:07:01.296Z INFO SledAgent (InstanceManager): Publishing instance state update to Nexus
    file = sled-agent/src/instance.rs:823
    instance_id = 018a2717-8d34-4b74-b8fd-0d666aee5d03
    propolis_id = 904183f8-bd4e-4637-9216-4f4aa07ac2be
    state = SledVmmState { vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(9), time_updated: 2025-04-01T23:07:01.296805485Z }, migration_in: None, migration_out: None }
23:07:01.802Z INFO SledAgent (InstanceManager): Publishing instance state update to Nexus
    file = sled-agent/src/instance.rs:823
    instance_id = 018a2717-8d34-4b74-b8fd-0d666aee5d03
    propolis_id = 904183f8-bd4e-4637-9216-4f4aa07ac2be
    state = SledVmmState { vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(9), time_updated: 2025-04-01T23:07:01.296805485Z }, migration_in: None, migration_out: None }

[askfongjojo]

The fix for #7563 was already on rack2 since 3/20. It's unclear why I'm seeing the race issue only after the 3/27 update. It's tricky to tell between disconnected propolis due to a panic or an intentional teardown. I wonder if excluding vmm's with last reported state = "Stopping" may be a safer choice?

I've uploaded the complete nexus and sled-agent logs to /staff/core/omicron-6809 if there is interest in digging into what exactly sled-agent was doing between 23:06:55.787Z and 23:07:01.802Z. (The zone bundle step actually took about 2.3 seconds and sled-agent was serving other zone GET requests and network PUT/DELETE requests in between the events.)

[gjcolombo]

I don't think Propolis panicked; the sled agent logs show the sequence of states I'd expect to see from a Propolis that's shutting down normally.

The VMM shutdown code has grown over time into a bit of a maze of twisty little passages, all alike (e.g. there are multiple routines and channels with some variation on the name terminate), and it could probably stand to be refactored and cleaned up. Here's how I understand what's happening, though:

1. Propolis reports the Destroyed state to sled agent
2. apply_propolis_observation returns InstanceAction::Destroy to InstanceRunner::observe_state
3. This calls InstanceRunner::terminate on the current runner
4. That calls InstanceRunner::terminate_inner, which then calls InstanceTicket::deregister
5. InstanceTicket::deregister sends a message back to the InstanceManagerRunner informing it that the VMM has terminated
6. terminate_inner does its other work (e.g. zone bundle collection, zone teardown) and then returns
7. InstanceRunner::terminate sets self.should_terminate to true
8. Control returns to InstanceRunner::run, which calls self.publish_state_to_nexus, which informs Nexus that the VMM is Destroyed

To my mind there are two problems here, one a mechanical issue in sled agent and one a more nuanced TOCTTOU problem in Nexus.

The sled agent problem is that once InstanceTicket::deregister is called in step 5, the instance manager is free to remove the relevant VMM from its tracking table, even though its final Destroyed state hasn't been sent to Nexus yet. If the instance watcher decides to check on the instance once the VMM is removed, it will get a "VMM gone" status and conclude the instance has failed. If sled agent told Nexus about the VMM's death first, and then removed it from the table, the window would be much smaller. (Note, however, that shutdown operations still have to be ordered carefully: once Nexus learns the VMM is Destroyed, it can free its resource allocation, so the reservoir charges on the sled need to be gone by that point.)

The Nexus problem is that, even once this window is closed, it is possible for a task like the instance watcher (or the instance stop API) to read an instance record, see that the VMM is not Destroyed, decide to call sled agent, and then not actually make that call until sled agent has published the Destroyed state and removed its table entry. This is the sort of thing that we need a "desired instance disposition" value (or some other mechanism) to deal with.

I'm unsure why this problem would have become more common in recent builds. I'll do some history diving to see if I can figure it out.

[gjcolombo]

(Note, however, that shutdown operations still have to be ordered carefully: once Nexus learns the VMM is Destroyed, it can free its resource allocation, so the reservoir charges on the sled need to be gone by that point.)

A related note here: currently it appears that propolis-server does not guarantee that when a VMM is Destroyed its kernel VMM is actually destroyed. I believe the problem is a handle leak and will file an issue in the Propolis repo about it.

The reason sled agent might care is that, if "Propolis VMM destroyed" entailed "reservoir is freed," sled agent could report the Destroyed state to Nexus as soon as it received it from Propolis, then finish all its other zone bundling tasks asynchronously. As it stands, sled agent needs to tear down the zone (or at least the Propolis process in it) before it can accurately send a Destroyed transition to Nexus.

Without changing Propolis's behavior we'll need to account for this in whatever final sequence of VM-destruction operations we come up with.