server: extend API request queue's memory of prior requests (#880)

Today the server's API request queue decides whether to accept or reject
incoming requests by storing a fixed disposition for each kind of
request and adjusting those dispositions as new requests are queued or
as a VM's state changes. This is not always enough information to
maintain the proper dispositions. Consider the following example:

1. A VM begins to start. Reboot requests are denied in this state (since
   a VM that hasn't booted yet can't be rebooted).
2. A caller queues a request to stop the VM after it starts. All
   subsequent reboot requests should now be denied because the VM will
   halt before they can be serviced.
3. The VM successfully starts. The correct reboot disposition is still
   "deny," because there's a pending stop request, but there's no way to
   discern this from the prior reboot disposition (it was also "deny"
   before the stop request was queued!).

To address this sort of problem, and to pave the way for better handling
of Crucible VCR replacements during instance start, refactor the queue
as follows:

- Remember what kinds of outstanding requests have not been processed
  and dispose of requests based on this state and the queue's notion
  of the instance's overall state.
- Break external API requests into "state change" and "component change"
  requests. This will allow them to be dequeued independently when the
  state driver wants to handle component changes without changing the
  order of state change requests.
- Tweak the language the state driver uses to communicate with the
  queue: instead of saying "the VM is now in state X," the driver says
  "I handled (or failed to handle) a request of type Y."

Tests: cargo test, PHD.

Author: gjcolombo

Cargo.lock

@@ -73,6 +73,7 @@ ring.workspace = true
 slog = { workspace = true, features = [ "max_level_trace", "release_max_level_debug" ] }
 expectorate.workspace = true
 mockall.workspace = true
+proptest.workspace = true
 
 [features]
 default = []
@@ -83,7 +84,6 @@ omicron-build = ["propolis/omicron-build"]
 
 # Falcon builds require corresponding bits turned on in the dependency libs
 falcon = ["propolis/falcon"]
-
 # Testing necessitates injecting failures which should hopefully be rare or even
 # never occur on real otherwise-unperturbed systems. We conditionally compile
 # code supporting failure injection to avoid the risk of somehow injecting

bin/propolis-server/Cargo.toml

@@ -63,9 +63,9 @@ impl ActiveVm {
 
         self.state_driver_queue
             .queue_external_request(match requested {
-                InstanceStateRequested::Run => ExternalRequest::Start,
-                InstanceStateRequested::Stop => ExternalRequest::Stop,
-                InstanceStateRequested::Reboot => ExternalRequest::Reboot,
+                InstanceStateRequested::Run => ExternalRequest::start(),
+                InstanceStateRequested::Stop => ExternalRequest::stop(),
+                InstanceStateRequested::Reboot => ExternalRequest::reboot(),
             })
             .map_err(Into::into)
     }
@@ -79,10 +79,7 @@ impl ActiveVm {
         websock: dropshot::WebsocketConnection,
     ) -> Result<(), VmError> {
         Ok(self.state_driver_queue.queue_external_request(
-            ExternalRequest::MigrateAsSource {
-                migration_id,
-                websock: websock.into(),
-            },
+            ExternalRequest::migrate_as_source(migration_id, websock),
         )?)
     }
 
@@ -107,11 +104,11 @@ impl ActiveVm {
     ) -> Result<(), VmError> {
         self.state_driver_queue
             .queue_external_request(
-                ExternalRequest::ReconfigureCrucibleVolume {
+                ExternalRequest::reconfigure_crucible_volume(
                     backend_id,
                     new_vcr_json,
                     result_tx,
-                },
+                ),
             )
             .map_err(Into::into)
     }