server: return better HTTP errors when not ensured

Currently, if the `propolis-server` APIs that require a running instance
are called when the server has not received an `InstanceEnsure` request,
we return a 404. Sled-agent and Nexus handle this status code by
assuming *they* have done something wrong, such as attempting to call an
API route that the server doesn't support. This results in issues where
sled-agent assumes that it's the one that made a mistake when it tries
to call APIs on a `propolis-server` that has lost its VM instance due to
being restarted after a panic.

To make it easier for callers distinguish between errors that indicate
"this server doesn't know about a running instance" and errors that
indicate "you attempted to call an API route that I don't know about",
I've changed the status code returned by `propolis-server`s that haven't
created an instance to HTTP [424 Failed Dependency]. The best thing
about returning this status code is that it's "not 404", but it does
arguably have the most correct semantics for this specific error:

> The HTTP 424 Failed Dependency client error response code indicates
> that the method could not be performed on the resource because the
> requested action depended on another action, and that action failed.

On the other hand, this is kind of a weird status code that's mostly
used by WebDAV servers, so I'm open to using something else. We could
alternatively return 410 Gone in these cases, but that status is already
returned by servers whose VM instance has been deleted. I thought it
might be nice to be able to distinguish between "not ensured" and
"ensured, and then deleted"...but we could switch to 410 Gone here, too.

[424 Failed Dependency]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/424

Author: hawkw

bin/propolis-server/src/lib/server.rs

@@ -263,12 +263,7 @@ impl DropshotEndpointContext {
             self.services.vm.lock().await,
             VmControllerState::as_controller,
         )
-        .map_err(|_| {
-            HttpError::for_not_found(
-                None,
-                "Server not initialized (no instance)".to_string(),
-            )
-        })
+        .map_err(|_| no_instance_error())
     }
 }
 
@@ -647,10 +642,7 @@ async fn instance_get_common(
 ) -> Result<(api::Instance, VersionedInstanceSpec), HttpError> {
     let ctx = rqctx.context();
     match &*ctx.services.vm.lock().await {
-        VmControllerState::NotCreated => Err(HttpError::for_not_found(
-            None,
-            "Server not initialized (no instance)".to_string(),
-        )),
+        VmControllerState::NotCreated => Err(no_instance_error()),
         VmControllerState::Created(vm) => {
             Ok((
                 api::Instance {
@@ -724,10 +716,7 @@ async fn instance_state_monitor(
         let vm_state = ctx.services.vm.lock().await;
         match &*vm_state {
             VmControllerState::NotCreated => {
-                return Err(HttpError::for_not_found(
-                    None,
-                    "Server not initialized (no instance)".to_string(),
-                ));
+                return Err(no_instance_error());
             }
             VmControllerState::Created(vm) => vm.state_watcher().clone(),
             VmControllerState::Destroyed { state_watcher, .. } => {
@@ -747,12 +736,13 @@ async fn instance_state_monitor(
         // means it will never reach the number the client wants it to reach.
         // Inform the client of this condition so it doesn't wait forever.
         state_watcher.changed().await.map_err(|_| {
-            HttpError::for_status(
-                Some(format!(
+            HttpError::for_client_error(
+                Some(NO_INSTANCE.to_string()),
+                http::status::StatusCode::GONE,
+                format!(
                     "No instance present; will never reach generation {}",
                     gen
-                )),
-                http::status::StatusCode::GONE,
+                ),
             )
         })?;
     }
@@ -1065,6 +1055,16 @@ pub fn api() -> ApiDescription<Arc<DropshotEndpointContext>> {
     api
 }
 
+const NO_INSTANCE: &str = "NO_INSTANCE";
+
+fn no_instance_error() -> HttpError {
+    HttpError::for_client_error(
+        Some(NO_INSTANCE.to_string()),
+        http::StatusCode::FAILED_DEPENDENCY,
+        "Server not initialized (no instance)".to_string(),
+    )
+}
+
 #[cfg(test)]
 mod tests {
     #[test]