Select a random Nexus from DNS (#1622)

When a sled drops off the network, there's a delay before an expunged
Nexus is no longer returned in the list of records (due to a blueprint
update). Crucible will try a few times to send high priority messages
but will eventually give up.

If the expunged Nexus is the first in the list of DNS records returned,
this unfortunately won't change: the DNS server does not randomize the
returned records. Crucible's notification will be continuously sent to
the same expunged Nexus and won't arrive.

If the "started" message for a repair or reconciliation is not received
by a Nexus, it will drop progress and finish messages with the same
upstairs ID. This can cause stuck repairs if Nexus can't poll for a
repair's progress for some reason.

Author: jmpesp

upstairs/src/notify.rs

@@ -5,6 +5,7 @@
 //! Nexus-flavored types internally.
 
 use chrono::{DateTime, Utc};
+use rand::prelude::SliceRandom;
 use slog::{debug, error, info, o, warn, Logger};
 use std::net::{Ipv6Addr, SocketAddr};
 use tokio::sync::mpsc;
@@ -425,8 +426,21 @@ pub(crate) async fn get_nexus_client(
     };
 
     let nexus_address =
-        match resolver.lookup_socket_v6(ServiceName::Nexus).await {
-            Ok(addr) => addr,
+        match resolver.lookup_all_socket_v6(ServiceName::Nexus).await {
+            Ok(addrs) => {
+                if addrs.is_empty() {
+                    error!(log, "no Nexus addresses returned!");
+                    return None;
+                }
+
+                let Some(addr) = addrs.choose(&mut rand::thread_rng()) else {
+                    error!(log, "somehow, choose failed!");
+                    return None;
+                };
+
+                *addr
+            }
+
             Err(e) => {
                 error!(log, "lookup Nexus address failed: {e}");
                 return None;