build universal macOS binaries on arm64 runners

[iliana]

No description provided.

[iliana]

I ran a very unscientific experiment (sample size = 1) to see if the universal binary improves Omicron test runtime on macOS.

Based on past reading I believe that CockroachDB (like most databases) is trying to be safe with writing to the filesystem, and that full fsync on Apple silicon hardware is going to be oppressively slow. So I ran these tests with a ramdisk to ensure that these binaries were being tested on their merits and not my disk:

$ hdiutil attach -nomount ram://1048576 # 512 MiB
/dev/disk5
$ diskutil erasevolume HFS+ ramdisk /dev/disk5
Started erase on disk5
Unmounting disk
Erasing
Initialized /dev/rdisk5 as a 512 MB case-insensitive HFS Plus volume
Mounting disk
Finished erase on disk5 (ramdisk)
$ touch /Volumes/ramdisk/.metadata_never_index
$ export TMPDIR=/Volumes/ramdisk

These tests were run on a stock Mac Studio (2022), Apple M1 Max, 32 GB memory, 512 GB SSD.

For each test I used oxidecomputer/omicron@122eccf, running the nexus-db-queries crate tests. Before this I ran cargo nextest test -p omicron-omdb -- test_omdb_env_settings and deleted the /Volumes/ramdisk/crdb-base-iliana directory as I noticed the first time I ran the tests it was slower.

amd64-only binary (76e176e):

Summary [ 118.631s] 231 tests run: 231 passed, 1 skipped

Nov 04 18:02:28.995 INFO cockroach pid: 58478
Nov 04 18:02:28.995 INFO cockroach listen URL: postgresql://root@[::1]:51432/omicron?sslmode=disable
Nov 04 18:02:28.995 INFO cockroach: populating
Nov 04 18:02:30.453 INFO cockroach: populated
Nov 04 18:02:33.496 INFO Created CRDB seed tarball: `/Volumes/ramdisk/crdb-base-iliana/b0167e6136cac141e08a5dc4564e8eaae50be5cbb44fd23d1c7f4cb2f6780222.tar`
  SETUP PASS [      1/1] crdb-seed: cargo run -p crdb-seed --profile test
        PASS [   3.334s] nexus-db-queries db::collection_attach::test::test_attach_missing_collection_fails
        PASS [   0.016s] nexus-db-queries db::collection_attach::test::test_verify_query
        PASS [   3.392s] nexus-db-queries db::collection_attach::test::test_attach_once_synchronous
        PASS [   3.393s] nexus-db-queries db::collection_attach::test::test_attach_beyond_capacity_fails
        PASS [   3.473s] nexus-db-queries db::collection_attach::test::test_attach_with_filters
        PASS [   3.534s] nexus-db-queries db::collection_attach::test::test_attach_deleted_resource_fails
        PASS [   3.533s] nexus-db-queries db::collection_attach::test::test_attach_once
        PASS [   3.536s] nexus-db-queries db::collection_attach::test::test_attach_while_already_attached
        PASS [   0.013s] nexus-db-queries db::collection_detach::test::test_verify_query
        PASS [   3.570s] nexus-db-queries db::collection_attach::test::test_attach_without_update_filter
        PASS [   3.624s] nexus-db-queries db::collection_attach::test::test_attach_multiple_times
        PASS [   3.628s] nexus-db-queries db::collection_attach::test::test_attach_missing_resource_fails
        PASS [   3.662s] nexus-db-queries db::collection_detach::test::test_detach_missing_resource_fails
        PASS [   3.707s] nexus-db-queries db::collection_detach::test::test_detach_deleted_resource_fails
        PASS [   3.666s] nexus-db-queries db::collection_detach::test::test_detach_missing_collection_fails
        PASS [   3.658s] nexus-db-queries db::collection_detach::test::test_detach_once
        PASS [   3.634s] nexus-db-queries db::collection_detach::test::test_detach_without_update_filter
        PASS [   0.015s] nexus-db-queries db::collection_detach_many::test::test_verify_query
        PASS [   3.581s] nexus-db-queries db::collection_detach_many::test::test_detach_missing_collection_fails
        PASS [   3.683s] nexus-db-queries db::collection_detach_many::test::test_detach_filter_collection
        PASS [   3.709s] nexus-db-queries db::collection_detach_many::test::test_detach_deleted_resource
        PASS [   0.013s] nexus-db-queries db::collection_insert::test::test_verify_query
        PASS [   0.013s] nexus-db-queries db::column_walker::test::test_all_columns_with_prefix
        PASS [   0.013s] nexus-db-queries db::column_walker::test::test_walk_table
        PASS [   0.019s] nexus-db-queries db::column_walker::test::test_walk_columns
        PASS [   3.698s] nexus-db-queries db::collection_detach_many::test::test_detach_many
        PASS [   3.866s] nexus-db-queries db::collection_detach::test::test_detach_while_already_detached
        PASS [   2.965s] nexus-db-queries db::collection_insert::test::test_collection_not_present
        PASS [   3.187s] nexus-db-queries db::collection_insert::test::test_collection_present
        PASS [   3.507s] nexus-db-queries db::collection_detach_many::test::test_detach_missing_resource_succeeds
        PASS [   3.524s] nexus-db-queries db::collection_detach_many::test::test_detach_once
        PASS [   3.638s] nexus-db-queries db::collection_detach_many::test::test_detach_once_synchronous
        PASS [   3.656s] nexus-db-queries db::collection_detach_many::test::test_detach_while_already_detached
        PASS [   4.232s] nexus-db-queries db::datastore::allow_list::tests::test_allowed_source_ip_database_ops
        PASS [   4.257s] nexus-db-queries db::datastore::bgp::tests::test_delete_bgp_config_and_announce_set_by_name
        PASS [   4.241s] nexus-db-queries db::datastore::clickhouse_policy::tests::test_clickhouse_policy_basic
        PASS [   4.253s] nexus-db-queries db::datastore::cockroachdb_node_id::tests::test_cockroachdb_node_id
        PASS [   2.596s] nexus-db-queries db::datastore::db_metadata::test::ensure_schema_is_current_version
        PASS [   2.890s] nexus-db-queries db::datastore::db_metadata::test::schema_version_subcomponents_save_progress
        PASS [   3.487s] nexus-db-queries db::datastore::db_metadata::test::concurrent_nexus_instances_only_move_forward
        PASS [   4.599s] nexus-db-queries db::datastore::cockroachdb_settings::test::test_preserve_downgrade
        PASS [   4.629s] nexus-db-queries db::datastore::dataset::test::test_insert_if_not_exists
        PASS [   4.705s] nexus-db-queries db::datastore::dataset::test::test_upsert_and_delete_while_blueprint_changes
        PASS [   0.014s] nexus-db-queries db::datastore::dns::test::test_dns_builder_basic
        PASS [   4.030s] nexus-db-queries db::datastore::deployment::tests::test_empty_blueprint
        PASS [   4.314s] nexus-db-queries db::datastore::deployment::external_networking::tests::test_deallocate_external_networking
        PASS [   4.467s] nexus-db-queries db::datastore::deployment::external_networking::tests::test_allocate_external_networking
        PASS [   4.680s] nexus-db-queries db::datastore::deployment::tests::test_ensure_external_networking_bails_on_bad_target
        PASS [   5.449s] nexus-db-queries db::datastore::deployment::tests::test_ensure_external_networking_works_with_good_target
        PASS [   4.931s] nexus-db-queries db::datastore::deployment::tests::test_set_target
        PASS [   4.106s] nexus-db-queries db::datastore::disk::tests::test_undelete_disk_set_faulted_idempotent
        PASS [   3.910s] nexus-db-queries db::datastore::dns::test::test_dns_uniqueness
        PASS [   5.539s] nexus-db-queries db::datastore::deployment::tests::test_representative_blueprint
        PASS [   4.715s] nexus-db-queries db::datastore::deployment::tests::test_set_target_enabled
        PASS [   4.329s] nexus-db-queries db::datastore::dns::test::test_dns_update_from_version
        PASS [   4.579s] nexus-db-queries db::datastore::dns::test::test_read_dns_config_basic
        PASS [   4.755s] nexus-db-queries db::datastore::dns::test::test_dns_update_incremental
        PASS [   5.639s] nexus-db-queries db::datastore::dns::test::test_read_dns_config_complex
        PASS [   4.591s] nexus-db-queries db::datastore::dns::test::test_read_dns_config_uninitialized
        PASS [   4.245s] nexus-db-queries db::datastore::instance::tests::test_instance_commit_update_is_idempotent
        PASS [   4.438s] nexus-db-queries db::datastore::instance::tests::test_instance_fetch_all
        PASS [   4.759s] nexus-db-queries db::datastore::external_ip::tests::test_service_ip_list
        PASS [   4.450s] nexus-db-queries db::datastore::instance::tests::test_instance_set_migration_ids
        PASS [   4.217s] nexus-db-queries db::datastore::instance::tests::test_instance_update_invalidated_while_locked
        PASS [   5.542s] nexus-db-queries db::datastore::instance::tests::test_instance_and_vmm_list_by_sled_agent
        PASS [   4.911s] nexus-db-queries db::datastore::instance::tests::test_instance_updater_acquires_lock
        PASS [   4.910s] nexus-db-queries db::datastore::instance::tests::test_instance_updater_cant_unlock_someone_elses_instance_
        PASS [   5.144s] nexus-db-queries db::datastore::instance::tests::test_instance_updater_lock_is_idempotent
        PASS [   4.148s] nexus-db-queries db::datastore::inventory::test::test_find_hw_baseboard_id_missing_returns_not_found
        PASS [   4.362s] nexus-db-queries db::datastore::instance::tests::test_unlocking_a_deleted_instance_is_okay
        PASS [   4.229s] nexus-db-queries db::datastore::inventory::test::test_representative_collection_populates_database
        PASS [   4.550s] nexus-db-queries db::datastore::inventory::test::test_inventory_deletion
        PASS [   4.176s] nexus-db-queries db::datastore::ip_pool::test::test_default_ip_pools
        PASS [   4.588s] nexus-db-queries db::datastore::ip_pool::test::test_internal_ip_pool
        PASS [   5.938s] nexus-db-queries db::datastore::inventory::test::test_inventory_insert
        PASS [   5.648s] nexus-db-queries db::datastore::ip_pool::test::test_ip_pool_utilization
        PASS [   4.379s] nexus-db-queries db::datastore::ipv4_nat_entry::test::ipv4_nat_sync_service_zones
        PASS [   4.447s] nexus-db-queries db::datastore::ipv4_nat_entry::test::nat_version_tracking
        PASS [   4.452s] nexus-db-queries db::datastore::ipv4_nat_entry::test::table_allows_unique_active_multiple_deleted
        PASS [   4.238s] nexus-db-queries db::datastore::migration::tests::test_migration_query_by_instance
        PASS [   4.144s] nexus-db-queries db::datastore::oximeter::tests::test_oximeter_expunge
        PASS [   6.939s] nexus-db-queries db::datastore::ipv4_nat_entry::test::ipv4_nat_changeset
        PASS [   4.557s] nexus-db-queries db::datastore::network_interface::tests::test_service_network_interfaces_list
        PASS [   5.659s] nexus-db-queries db::datastore::oximeter::tests::test_oximeter_reassign_fails_if_no_collectors
        PASS [   4.947s] nexus-db-queries db::datastore::oximeter::tests::test_producer_endpoint_reassigns_if_oximeter_expunged
        PASS [   4.308s] nexus-db-queries db::datastore::oximeter::tests::test_producers_list_expired
        PASS [   4.453s] nexus-db-queries db::datastore::physical_disk::test::physical_disk_cannot_insert_to_expunged_sled
        PASS [   4.302s] nexus-db-queries db::datastore::physical_disk::test::physical_disk_deletion_idempotency
        PASS [   4.486s] nexus-db-queries db::datastore::physical_disk::test::physical_disk_insert_different_disks
        PASS [   4.679s] nexus-db-queries db::datastore::physical_disk::test::physical_disk_insert_delete_reupsert_new_sled
        PASS [   5.277s] nexus-db-queries db::datastore::physical_disk::test::physical_disk_insert_reupsert_new_sled
        PASS [   6.118s] nexus-db-queries db::datastore::physical_disk::test::physical_disk_insert_same_uuid_collides
        PASS [   4.631s] nexus-db-queries db::datastore::rack::test::rack_set_initialized_missing_service_pool_ip_throws_error
        PASS [   0.014s] nexus-db-queries db::datastore::region::test::test_extent_count
        PASS [   4.387s] nexus-db-queries db::datastore::rack::test::rack_set_initialized_overlapping_ips_throws_error
        PASS [   4.767s] nexus-db-queries db::datastore::rack::test::rack_set_initialized_empty
        PASS [   5.377s] nexus-db-queries db::datastore::physical_disk::test::physical_disk_uninitialized_list
        PASS [   4.921s] nexus-db-queries db::datastore::rack::test::rack_set_initialized_with_many_nexus_services
        PASS [   5.889s] nexus-db-queries db::datastore::rack::test::allocate_sled_underlay_subnet_octets
        PASS [  12.860s] nexus-db-queries db::datastore::oximeter::tests::test_oximeter_reassigns_randomly
        PASS [   6.830s] nexus-db-queries db::datastore::rack::test::rack_set_initialized_with_services
        PASS [   4.541s] nexus-db-queries db::datastore::region_replacement::test::test_replacement_done_in_middle_of_drive_saga
        PASS [   4.870s] nexus-db-queries db::datastore::rack::test::rack_sled_subnet_allocations
        PASS [   4.588s] nexus-db-queries db::datastore::region_replacement::test::test_one_replacement_per_volume
        PASS [   4.637s] nexus-db-queries db::datastore::region_replacement::test::test_replacement_done_in_middle_of_finish_saga
        PASS [   4.757s] nexus-db-queries db::datastore::region_snapshot_replacement::test::count_replacement_steps
        PASS [   4.220s] nexus-db-queries db::datastore::region_snapshot_replacement::test::region_snapshot_replacement_step_conflict
        PASS [   4.795s] nexus-db-queries db::datastore::region_snapshot_replacement::test::region_snapshot_replacement_step_conflict_with_region_replacement
        PASS [   4.954s] nexus-db-queries db::datastore::region_snapshot_replacement::test::region_snapshot_replacement_step_gc
        PASS [   4.866s] nexus-db-queries db::datastore::region_snapshot_replacement::test::test_one_replacement_per_volume
        PASS [   4.350s] nexus-db-queries db::datastore::saga::test::test_create_event_idempotent
        PASS [   4.086s] nexus-db-queries db::datastore::saga::test::test_list_no_unfinished_nodes
        PASS [   4.492s] nexus-db-queries db::datastore::region_snapshot_replacement::test::unique_region_snapshot_replacement_step_per_volume
        PASS [   4.600s] nexus-db-queries db::datastore::region_snapshot_replacement::test::test_one_replacement_per_volume_conflict_with_region
        PASS [   4.555s] nexus-db-queries db::datastore::saga::test::test_list_candidate_sagas
        PASS [   4.518s] nexus-db-queries db::datastore::saga::test::test_list_unfinished_nodes
        PASS [  19.759s] nexus-db-queries db::datastore::oximeter::tests::test_producer_endpoint_upsert_rejects_expunged_oximeters
        PASS [   4.724s] nexus-db-queries db::datastore::saga::test::test_saga_reassignment
        PASS [   5.318s] nexus-db-queries db::datastore::saga::test::test_update_state_idempotent
        PASS [   4.641s] nexus-db-queries db::datastore::sled::test::test_sled_expungement_also_expunges_disks
        PASS [   4.549s] nexus-db-queries db::datastore::sled::test::upsert_sled_doesnt_update_decommissioned
        PASS [   4.491s] nexus-db-queries db::datastore::sled::test::upsert_sled_updates_fails_with_stale_sled_agent_gen
        PASS [   5.398s] nexus-db-queries db::datastore::sled::test::sled_list_batch
        PASS [   4.927s] nexus-db-queries db::datastore::sled::test::sled_reservation_create_non_provisionable
        PASS [   4.604s] nexus-db-queries db::datastore::sled::test::upsert_sled_updates_hardware
        PASS [   4.319s] nexus-db-queries db::datastore::switch_port::test::test_bgp_boundary_switches
        PASS [   5.152s] nexus-db-queries db::datastore::test::test_deallocate_external_ip_by_instance_id_is_idempotent
        PASS [   7.572s] nexus-db-queries db::datastore::sled::test::test_sled_transitions
        PASS [   5.827s] nexus-db-queries db::datastore::test::test_deallocate_external_ip_is_idempotent
        PASS [   4.123s] nexus-db-queries db::datastore::test::test_queries_do_not_require_full_table_scan
        PASS [   4.233s] nexus-db-queries db::datastore::test::test_project_creation
        PASS [   4.474s] nexus-db-queries db::datastore::test::test_external_ip_check_constraints
        PASS [   4.423s] nexus-db-queries db::datastore::test::test_rack_initialize_is_idempotent
        PASS [   4.425s] nexus-db-queries db::datastore::test::test_region_allocation_not_enough_zpools
        PASS [   4.632s] nexus-db-queries db::datastore::test::test_region_allocation_only_considers_disks_in_service
        PASS [   6.511s] nexus-db-queries db::datastore::test::test_region_allocation_only_operates_on_zpools_in_inventory
        PASS [   4.114s] nexus-db-queries db::datastore::test::test_session_methods
        PASS [   4.047s] nexus-db-queries db::datastore::test::test_sled_ipv6_address_allocation
        PASS [   8.726s] nexus-db-queries db::datastore::test::test_region_allocation_is_idempotent
        PASS [   4.164s] nexus-db-queries db::datastore::test::test_ssh_keys
        PASS [   5.127s] nexus-db-queries db::datastore::test::test_table_scan
        PASS [   8.494s] nexus-db-queries db::datastore::test::test_region_allocation_strat_random
        PASS [   9.401s] nexus-db-queries db::datastore::test::test_region_allocation_out_of_space_fails
        PASS [   0.013s] nexus-db-queries db::datastore::volume::tests::test_read_only_target_in_vcr
        PASS [   0.014s] nexus-db-queries db::datastore::volume::tests::test_replace_read_only_target_in_vcr
        PASS [   8.457s] nexus-db-queries db::datastore::test::test_region_allocation_strat_random_with_distinct_sleds_fails
        PASS [   8.943s] nexus-db-queries db::datastore::test::test_region_allocation_strat_random_with_distinct_sleds
        PASS [   4.822s] nexus-db-queries db::datastore::virtual_provisioning_collection::test::test_instance_create_and_delete_twice
        PASS [   4.929s] nexus-db-queries db::datastore::virtual_provisioning_collection::test::test_instance_create_and_delete
        PASS [   4.786s] nexus-db-queries db::datastore::virtual_provisioning_collection::test::test_storage_create_and_delete
        PASS [   4.642s] nexus-db-queries db::datastore::virtual_provisioning_collection::test::test_storage_create_and_delete_twice
        PASS [   4.665s] nexus-db-queries db::datastore::vmm::tests::test_vmm_and_migration_update_runtime
        PASS [   5.463s] nexus-db-queries db::datastore::volume::tests::test_deserialize_old_crucible_resources
        PASS [   4.828s] nexus-db-queries db::datastore::volume::tests::test_find_volumes_referencing_socket_addr
        PASS [   4.585s] nexus-db-queries db::datastore::volume::tests::test_volume_replace_region
        PASS [   4.495s] nexus-db-queries db::datastore::volume::tests::test_volume_replace_snapshot
        PASS [   4.218s] nexus-db-queries db::datastore::volume_repair::test::volume_lock_conflict_error_returned
        PASS [   4.303s] nexus-db-queries db::datastore::vpc::tests::test_project_create_vpc_retries
        PASS [   4.316s] nexus-db-queries db::datastore::vpc::tests::test_project_create_vpc_raw_returns_none_on_vni_exhaustion
        PASS [   4.754s] nexus-db-queries db::datastore::vpc::tests::test_vpc_router_rule_instance_resolve
        PASS [   5.294s] nexus-db-queries db::datastore::vpc::tests::test_vpc_resolve_to_sleds_uses_current_target_blueprint
        PASS [   0.012s] nexus-db-queries db::pagination::test::test_paginator
        PASS [   5.257s] nexus-db-queries db::datastore::vpc::tests::test_vpc_system_router_sync_to_subnets
        PASS [   3.558s] nexus-db-queries db::explain::test::test_explain_async
        PASS [   2.596s] nexus-db-queries db::explain::test::test_explain_full_table_scan
        PASS [   3.106s] nexus-db-queries db::pagination::test::test_paginated_multicolumn_ascending
        PASS [   2.918s] nexus-db-queries db::pagination::test::test_paginated_single_column_ascending
        PASS [   3.356s] nexus-db-queries db::pagination::test::test_paginated_multicolumn_descending
        PASS [   3.049s] nexus-db-queries db::pagination::test::test_paginated_multicolumn_works_with_joins
        PASS [   3.823s] nexus-db-queries db::lookup::test::test_lookup
        PASS [   3.228s] nexus-db-queries db::pagination::test::test_paginated_single_column_descending
        PASS [   5.352s] nexus-db-queries db::queries::external_ip::tests::test_ensure_pool_exhaustion_does_not_use_other_pool
        PASS [   4.784s] nexus-db-queries db::queries::external_ip::tests::test_external_ip_allocate_omicron_zone_out_of_range
        PASS [   0.013s] nexus-db-queries db::queries::network_interface::tests::test_first_available_address
        PASS [   5.483s] nexus-db-queries db::queries::external_ip::tests::test_ephemeral_and_snat_ips_do_not_overlap
        PASS [   5.162s] nexus-db-queries db::queries::external_ip::tests::test_external_ip_allocate_omicron_zone_is_idempotent
        PASS [   4.544s] nexus-db-queries db::queries::external_ip::tests::test_insert_external_ip_is_idempotent
        PASS [   4.437s] nexus-db-queries db::queries::external_ip::tests::test_next_external_ip_allocation_and_exhaustion
        PASS [   4.412s] nexus-db-queries db::queries::external_ip::tests::test_next_external_ip_is_restricted_to_pools
        PASS [   4.423s] nexus-db-queries db::queries::external_ip::tests::test_next_external_ip_out_of_order_allocation_ok
        PASS [   4.503s] nexus-db-queries db::queries::external_ip::tests::test_next_external_ip_with_ephemeral_takes_whole_port_range
        PASS [   5.040s] nexus-db-queries db::queries::network_interface::tests::test_delete_service_is_idempotent
        PASS [   4.710s] nexus-db-queries db::queries::network_interface::tests::test_insert_multiple_vpc_subnets_succeeds
        PASS [   4.767s] nexus-db-queries db::queries::network_interface::tests::test_insert_multiple_vpcs_fails
        PASS [   4.649s] nexus-db-queries db::queries::network_interface::tests::test_insert_no_instance_fails
        PASS [   4.203s] nexus-db-queries db::queries::network_interface::tests::test_insert_request_mac
        PASS [   4.180s] nexus-db-queries db::queries::network_interface::tests::test_insert_request_same_mac_fails
        PASS [   4.506s] nexus-db-queries db::queries::network_interface::tests::test_insert_request_exact_ip
        PASS [   0.012s] nexus-db-queries db::queries::network_interface::tests::test_last_available_address
        PASS [   4.468s] nexus-db-queries db::queries::network_interface::tests::test_insert_request_same_ip_fails
        PASS [   4.290s] nexus-db-queries db::queries::network_interface::tests::test_insert_request_same_slot_fails
        PASS [   5.806s] nexus-db-queries db::queries::network_interface::tests::test_detect_ip_exhaustion
        PASS [   3.178s] nexus-db-queries db::queries::next_item::tests::test_explain_next_item_self_joined
        PASS [   3.382s] nexus-db-queries db::queries::next_item::tests::test_next_item_query_is_ordered_by_indices
        PASS [   6.405s] nexus-db-queries db::queries::network_interface::tests::test_insert_request_slot
        PASS [   0.014s] nexus-db-queries db::queries::next_item::tests::test_shift_indices
        PASS [   3.300s] nexus-db-queries db::queries::next_item::tests::test_next_item_self_joined
        PASS [   4.481s] nexus-db-queries db::queries::network_interface::tests::test_insert_same_interface_fails
        PASS [   4.482s] nexus-db-queries db::queries::network_interface::tests::test_insert_same_vpc_subnet_fails
        PASS [   4.597s] nexus-db-queries db::queries::network_interface::tests::test_insert_running_instance_fails
        PASS [   0.777s] nexus-db-queries db::queries::oximeter::test::expectorate_query_reassign_producers
        PASS [   4.570s] nexus-db-queries db::queries::network_interface::tests::test_insert_sequential_ip_allocation
        PASS [   4.520s] nexus-db-queries db::queries::network_interface::tests::test_insert_with_duplicate_name_fails
        PASS [   0.703s] nexus-db-queries db::queries::oximeter::test::expectorate_query_upsert_producer
        PASS [   4.614s] nexus-db-queries db::queries::network_interface::tests::test_limit_number_of_interfaces_per_instance_query
        PASS [   0.469s] nexus-db-queries db::queries::virtual_provisioning_collection_update::test::expectorate_query_delete_instance
        PASS [   0.491s] nexus-db-queries db::queries::virtual_provisioning_collection_update::test::expectorate_query_delete_storage
        PASS [   0.480s] nexus-db-queries db::queries::virtual_provisioning_collection_update::test::expectorate_query_insert_instance
        PASS [   0.549s] nexus-db-queries db::queries::virtual_provisioning_collection_update::test::expectorate_query_insert_storage
        PASS [   3.185s] nexus-db-queries db::queries::next_item::tests::test_next_item_self_joined_with_gaps
        PASS [   0.038s] nexus-db-queries db::queries::vpc::tests::test_vni_search_iter_full_count
        PASS [   0.011s] nexus-db-queries db::queries::vpc::tests::test_vni_search_iter_steps
        PASS [   2.347s] nexus-db-queries db::queries::oximeter::test::explainable_reassign_producers
        PASS [   0.012s] nexus-db-queries db::queries::vpc::tests::test_vni_search_iter_wrapping
        PASS [   0.011s] nexus-db-queries db::queries::vpc::tests::test_vni_shift_no_wrapping
        PASS [   0.011s] nexus-db-queries db::queries::vpc::tests::test_vni_shift_with_wrapping
        PASS [   3.093s] nexus-db-queries db::queries::next_item::tests::test_wrapping_next_item_query
        PASS [   2.517s] nexus-db-queries db::queries::oximeter::test::explainable_upsert_producer
        PASS [   2.426s] nexus-db-queries db::queries::region_allocation::test::explainable
        PASS [   2.581s] nexus-db-queries db::queries::region_allocation::test::expectorate_query
        PASS [   2.235s] nexus-db-queries db::queries::virtual_provisioning_collection_update::test::explain_delete_storage
        PASS [   2.290s] nexus-db-queries db::queries::virtual_provisioning_collection_update::test::explain_delete_instance
        PASS [   2.454s] nexus-db-queries db::queries::virtual_provisioning_collection_update::test::explain_insert_instance
        PASS [   2.782s] nexus-db-queries db::queries::virtual_provisioning_collection_update::test::explain_insert_storage
        PASS [   2.502s] nexus-db-queries db::queries::vpc_subnet::test::explain_insert_query
        PASS [   2.553s] nexus-db-queries db::queries::vpc_subnet::test::test_insert_vpc_subnet_query_is_idempotent
        PASS [   2.643s] nexus-db-queries db::queries::vpc_subnet::test::test_insert_vpc_subnet_query
        PASS [   3.037s] nexus-db-queries transaction_retry::test::test_transaction_rollback_produces_no_samples
        PASS [   3.504s] nexus-db-queries transaction_retry::test::test_transaction_retry_produces_samples
        PASS [   4.577s] nexus-db-queries policy_test::test_conferred_roles
        PASS [  21.396s] nexus-db-queries policy_test::test_iam_roles_behavior
------------
     Summary [ 118.631s] 231 tests run: 231 passed, 1 skipped

Universal binary:

Omicron diff to use binary from this PR

diff --git a/tools/cockroachdb_checksums b/tools/cockroachdb_checksums
index 9b5dd6242..37993e238 100644
--- a/tools/cockroachdb_checksums
+++ b/tools/cockroachdb_checksums
@@ -1,4 +1,4 @@
-COCKROACH_COMMIT="76e176e260b7ba2aba9a5f5cec4c35c766dbde1a"
-CIDL_SHA256_DARWIN="e881be74c76d6e08d87307ccdeab55f79bf13e7e914e5c7422e5b8b03140c9c9"
+COCKROACH_COMMIT="40709036f487931ce8042f3b6dcf4c4a22ed5cf5"
+CIDL_SHA256_DARWIN="c1f9879a2d857e73dad3b63f6a15d87101c05e6291002c5462ae05d7f4705945"
 CIDL_SHA256_LINUX="760a696dced1cb17f36ac847d5315e13b71f72295c350763d7885457503196ec"
 CIDL_SHA256_ILLUMOS="9037b4d66bbceb8c7d8bdd8d7dadc5f9d94d59a6fee1fd478a1f7201213d7daa"
diff --git a/tools/cockroachdb_version b/tools/cockroachdb_version
index 3cc8784e1..662e6871e 100644
--- a/tools/cockroachdb_version
+++ b/tools/cockroachdb_version
@@ -1 +1 @@
-v22.1.22-27-g76e176e260
+v22.1.22-30-g40709036f4

Summary [ 70.486s] 231 tests run: 231 passed, 1 skipped

Nov 04 18:06:54.864 INFO cockroach pid: 60702
Nov 04 18:06:54.864 INFO cockroach listen URL: postgresql://root@[::1]:55686/omicron?sslmode=disable
Nov 04 18:06:54.864 INFO cockroach: populating
Nov 04 18:06:55.836 INFO cockroach: populated
Nov 04 18:06:58.676 INFO Created CRDB seed tarball: `/Volumes/ramdisk/crdb-base-iliana/a2021fecc999c8eadc57f073f41184a691328c578c628218c2afdd8a1721397b.tar`
  SETUP PASS [      1/1] crdb-seed: cargo run -p crdb-seed --profile test
        PASS [   1.861s] nexus-db-queries db::collection_attach::test::test_attach_with_filters
        PASS [   1.864s] nexus-db-queries db::collection_attach::test::test_attach_missing_resource_fails
        PASS [   0.016s] nexus-db-queries db::collection_attach::test::test_verify_query
        PASS [   1.885s] nexus-db-queries db::collection_attach::test::test_attach_missing_collection_fails
        PASS [   1.907s] nexus-db-queries db::collection_attach::test::test_attach_once
        PASS [   1.924s] nexus-db-queries db::collection_attach::test::test_attach_without_update_filter
        PASS [   1.931s] nexus-db-queries db::collection_attach::test::test_attach_deleted_resource_fails
        PASS [   1.938s] nexus-db-queries db::collection_attach::test::test_attach_once_synchronous
        PASS [   0.012s] nexus-db-queries db::collection_detach::test::test_verify_query
        PASS [   1.984s] nexus-db-queries db::collection_attach::test::test_attach_beyond_capacity_fails
        PASS [   2.001s] nexus-db-queries db::collection_attach::test::test_attach_multiple_times
        PASS [   2.072s] nexus-db-queries db::collection_attach::test::test_attach_while_already_attached
        PASS [   1.294s] nexus-db-queries db::collection_detach::test::test_detach_missing_resource_fails
        PASS [   1.462s] nexus-db-queries db::collection_detach::test::test_detach_deleted_resource_fails
        PASS [   1.594s] nexus-db-queries db::collection_detach::test::test_detach_once
        PASS [   1.596s] nexus-db-queries db::collection_detach::test::test_detach_while_already_detached
        PASS [   1.547s] nexus-db-queries db::collection_detach_many::test::test_detach_filter_collection
        PASS [   0.014s] nexus-db-queries db::collection_detach_many::test::test_verify_query
        PASS [   1.705s] nexus-db-queries db::collection_detach::test::test_detach_missing_collection_fails
        PASS [   1.592s] nexus-db-queries db::collection_detach_many::test::test_detach_many
        PASS [   0.014s] nexus-db-queries db::collection_insert::test::test_verify_query
        PASS [   1.661s] nexus-db-queries db::collection_detach_many::test::test_detach_deleted_resource
        PASS [   0.013s] nexus-db-queries db::column_walker::test::test_all_columns_with_prefix
        PASS [   0.014s] nexus-db-queries db::column_walker::test::test_walk_columns
        PASS [   0.013s] nexus-db-queries db::column_walker::test::test_walk_table
        PASS [   1.740s] nexus-db-queries db::collection_detach::test::test_detach_without_update_filter
        PASS [   1.665s] nexus-db-queries db::collection_detach_many::test::test_detach_missing_collection_fails
        PASS [   1.070s] nexus-db-queries db::collection_detach_many::test::test_detach_missing_resource_succeeds
        PASS [   1.384s] nexus-db-queries db::collection_detach_many::test::test_detach_once_synchronous
        PASS [   1.365s] nexus-db-queries db::collection_detach_many::test::test_detach_while_already_detached
        PASS [   1.585s] nexus-db-queries db::collection_detach_many::test::test_detach_once
        PASS [   1.441s] nexus-db-queries db::collection_insert::test::test_collection_not_present
        PASS [   1.427s] nexus-db-queries db::collection_insert::test::test_collection_present
        PASS [   2.310s] nexus-db-queries db::datastore::bgp::tests::test_delete_bgp_config_and_announce_set_by_name
        PASS [   2.370s] nexus-db-queries db::datastore::allow_list::tests::test_allowed_source_ip_database_ops
        PASS [   2.448s] nexus-db-queries db::datastore::clickhouse_policy::tests::test_clickhouse_policy_basic
        PASS [   2.445s] nexus-db-queries db::datastore::cockroachdb_node_id::tests::test_cockroachdb_node_id
        PASS [   1.220s] nexus-db-queries db::datastore::db_metadata::test::ensure_schema_is_current_version
        PASS [   1.260s] nexus-db-queries db::datastore::db_metadata::test::schema_version_subcomponents_save_progress
        PASS [   2.103s] nexus-db-queries db::datastore::cockroachdb_settings::test::test_preserve_downgrade
        PASS [   1.852s] nexus-db-queries db::datastore::db_metadata::test::concurrent_nexus_instances_only_move_forward
        PASS [   2.367s] nexus-db-queries db::datastore::dataset::test::test_insert_if_not_exists
        PASS [   2.556s] nexus-db-queries db::datastore::dataset::test::test_upsert_and_delete_while_blueprint_changes
        PASS [   0.012s] nexus-db-queries db::datastore::dns::test::test_dns_builder_basic
        PASS [   2.340s] nexus-db-queries db::datastore::deployment::tests::test_empty_blueprint
        PASS [   2.519s] nexus-db-queries db::datastore::deployment::external_networking::tests::test_deallocate_external_networking
        PASS [   2.574s] nexus-db-queries db::datastore::deployment::external_networking::tests::test_allocate_external_networking
        PASS [   2.438s] nexus-db-queries db::datastore::deployment::tests::test_ensure_external_networking_works_with_good_target
        PASS [   2.360s] nexus-db-queries db::datastore::deployment::tests::test_set_target
        PASS [   2.644s] nexus-db-queries db::datastore::deployment::tests::test_ensure_external_networking_bails_on_bad_target
        PASS [   2.686s] nexus-db-queries db::datastore::deployment::tests::test_representative_blueprint
        PASS [   2.456s] nexus-db-queries db::datastore::deployment::tests::test_set_target_enabled
        PASS [   2.183s] nexus-db-queries db::datastore::disk::tests::test_undelete_disk_set_faulted_idempotent
        PASS [   2.090s] nexus-db-queries db::datastore::dns::test::test_dns_uniqueness
        PASS [   2.418s] nexus-db-queries db::datastore::dns::test::test_dns_update_from_version
        PASS [   2.334s] nexus-db-queries db::datastore::dns::test::test_read_dns_config_uninitialized
        PASS [   2.569s] nexus-db-queries db::datastore::dns::test::test_read_dns_config_basic
        PASS [   2.574s] nexus-db-queries db::datastore::dns::test::test_read_dns_config_complex
        PASS [   2.602s] nexus-db-queries db::datastore::external_ip::tests::test_service_ip_list
        PASS [   2.215s] nexus-db-queries db::datastore::instance::tests::test_instance_commit_update_is_idempotent
        PASS [   2.973s] nexus-db-queries db::datastore::dns::test::test_dns_update_incremental
        PASS [   2.256s] nexus-db-queries db::datastore::instance::tests::test_instance_fetch_all
        PASS [   2.382s] nexus-db-queries db::datastore::instance::tests::test_instance_set_migration_ids
        PASS [   3.466s] nexus-db-queries db::datastore::instance::tests::test_instance_and_vmm_list_by_sled_agent
        PASS [   2.157s] nexus-db-queries db::datastore::instance::tests::test_instance_update_invalidated_while_locked
        PASS [   2.184s] nexus-db-queries db::datastore::instance::tests::test_instance_updater_acquires_lock
        PASS [   2.208s] nexus-db-queries db::datastore::instance::tests::test_instance_updater_cant_unlock_someone_elses_instance_
        PASS [   2.281s] nexus-db-queries db::datastore::instance::tests::test_instance_updater_lock_is_idempotent
        PASS [   2.322s] nexus-db-queries db::datastore::inventory::test::test_find_hw_baseboard_id_missing_returns_not_found
        PASS [   2.584s] nexus-db-queries db::datastore::instance::tests::test_unlocking_a_deleted_instance_is_okay
        PASS [   2.621s] nexus-db-queries db::datastore::inventory::test::test_inventory_deletion
        PASS [   2.592s] nexus-db-queries db::datastore::inventory::test::test_representative_collection_populates_database
        PASS [   2.646s] nexus-db-queries db::datastore::ip_pool::test::test_default_ip_pools
        PASS [   2.067s] nexus-db-queries db::datastore::ip_pool::test::test_internal_ip_pool
        PASS [   3.795s] nexus-db-queries db::datastore::inventory::test::test_inventory_insert
        PASS [   2.163s] nexus-db-queries db::datastore::ipv4_nat_entry::test::ipv4_nat_sync_service_zones
        PASS [   2.000s] nexus-db-queries db::datastore::ipv4_nat_entry::test::nat_version_tracking
        PASS [   2.617s] nexus-db-queries db::datastore::ip_pool::test::test_ip_pool_utilization
        PASS [   2.167s] nexus-db-queries db::datastore::ipv4_nat_entry::test::table_allows_unique_active_multiple_deleted
        PASS [   2.164s] nexus-db-queries db::datastore::migration::tests::test_migration_query_by_instance
        PASS [   3.608s] nexus-db-queries db::datastore::ipv4_nat_entry::test::ipv4_nat_changeset
        PASS [   2.466s] nexus-db-queries db::datastore::network_interface::tests::test_service_network_interfaces_list
        PASS [   2.078s] nexus-db-queries db::datastore::oximeter::tests::test_oximeter_expunge
        PASS [   1.943s] nexus-db-queries db::datastore::oximeter::tests::test_producer_endpoint_reassigns_if_oximeter_expunged
        PASS [   2.023s] nexus-db-queries db::datastore::oximeter::tests::test_producers_list_expired
        PASS [   3.037s] nexus-db-queries db::datastore::oximeter::tests::test_oximeter_reassign_fails_if_no_collectors
        PASS [   2.281s] nexus-db-queries db::datastore::physical_disk::test::physical_disk_cannot_insert_to_expunged_sled
        PASS [   2.225s] nexus-db-queries db::datastore::physical_disk::test::physical_disk_deletion_idempotency
        PASS [   2.416s] nexus-db-queries db::datastore::physical_disk::test::physical_disk_insert_delete_reupsert_new_sled
        PASS [   2.422s] nexus-db-queries db::datastore::physical_disk::test::physical_disk_insert_different_disks
        PASS [   2.709s] nexus-db-queries db::datastore::physical_disk::test::physical_disk_insert_reupsert_new_sled
        PASS [   2.323s] nexus-db-queries db::datastore::physical_disk::test::physical_disk_insert_same_uuid_collides
        PASS [   2.764s] nexus-db-queries db::datastore::physical_disk::test::physical_disk_uninitialized_list
        PASS [   0.015s] nexus-db-queries db::datastore::region::test::test_extent_count
        PASS [   2.234s] nexus-db-queries db::datastore::rack::test::rack_set_initialized_missing_service_pool_ip_throws_error
        PASS [   2.622s] nexus-db-queries db::datastore::rack::test::rack_set_initialized_empty
        PASS [   2.280s] nexus-db-queries db::datastore::rack::test::rack_set_initialized_overlapping_ips_throws_error
        PASS [   3.736s] nexus-db-queries db::datastore::rack::test::allocate_sled_underlay_subnet_octets
        PASS [   2.587s] nexus-db-queries db::datastore::rack::test::rack_set_initialized_with_many_nexus_services
        PASS [   2.251s] nexus-db-queries db::datastore::rack::test::rack_sled_subnet_allocations
        PASS [   2.784s] nexus-db-queries db::datastore::rack::test::rack_set_initialized_with_services
        PASS [   2.026s] nexus-db-queries db::datastore::region_replacement::test::test_one_replacement_per_volume
        PASS [   2.168s] nexus-db-queries db::datastore::region_replacement::test::test_replacement_done_in_middle_of_drive_saga
        PASS [   7.778s] nexus-db-queries db::datastore::oximeter::tests::test_oximeter_reassigns_randomly
        PASS [   2.202s] nexus-db-queries db::datastore::region_replacement::test::test_replacement_done_in_middle_of_finish_saga
        PASS [   2.172s] nexus-db-queries db::datastore::region_snapshot_replacement::test::count_replacement_steps
        PASS [   2.218s] nexus-db-queries db::datastore::region_snapshot_replacement::test::region_snapshot_replacement_step_conflict
        PASS [   2.236s] nexus-db-queries db::datastore::region_snapshot_replacement::test::region_snapshot_replacement_step_conflict_with_region_replacement
        PASS [   2.221s] nexus-db-queries db::datastore::region_snapshot_replacement::test::region_snapshot_replacement_step_gc
        PASS [   2.189s] nexus-db-queries db::datastore::region_snapshot_replacement::test::test_one_replacement_per_volume
        PASS [   2.199s] nexus-db-queries db::datastore::region_snapshot_replacement::test::test_one_replacement_per_volume_conflict_with_region
        PASS [   1.868s] nexus-db-queries db::datastore::saga::test::test_create_event_idempotent
        PASS [   2.247s] nexus-db-queries db::datastore::region_snapshot_replacement::test::unique_region_snapshot_replacement_step_per_volume
        PASS [   2.226s] nexus-db-queries db::datastore::saga::test::test_list_candidate_sagas
        PASS [   2.022s] nexus-db-queries db::datastore::saga::test::test_list_no_unfinished_nodes
        PASS [   2.192s] nexus-db-queries db::datastore::saga::test::test_saga_reassignment
        PASS [   2.386s] nexus-db-queries db::datastore::saga::test::test_list_unfinished_nodes
        PASS [   2.245s] nexus-db-queries db::datastore::saga::test::test_update_state_idempotent
        PASS [   2.686s] nexus-db-queries db::datastore::sled::test::sled_list_batch
        PASS [   2.444s] nexus-db-queries db::datastore::sled::test::test_sled_expungement_also_expunges_disks
        PASS [   2.865s] nexus-db-queries db::datastore::sled::test::sled_reservation_create_non_provisionable
        PASS [   2.351s] nexus-db-queries db::datastore::sled::test::upsert_sled_doesnt_update_decommissioned
        PASS [   2.359s] nexus-db-queries db::datastore::sled::test::upsert_sled_updates_fails_with_stale_sled_agent_gen
        PASS [   2.133s] nexus-db-queries db::datastore::switch_port::test::test_bgp_boundary_switches
        PASS [   2.391s] nexus-db-queries db::datastore::sled::test::upsert_sled_updates_hardware
        PASS [   2.397s] nexus-db-queries db::datastore::test::test_deallocate_external_ip_by_instance_id_is_idempotent
        PASS [  13.792s] nexus-db-queries db::datastore::oximeter::tests::test_producer_endpoint_upsert_rejects_expunged_oximeters
        PASS [   1.948s] nexus-db-queries db::datastore::test::test_project_creation
        PASS [   2.192s] nexus-db-queries db::datastore::test::test_deallocate_external_ip_is_idempotent
        PASS [   1.980s] nexus-db-queries db::datastore::test::test_queries_do_not_require_full_table_scan
        PASS [   2.483s] nexus-db-queries db::datastore::test::test_external_ip_check_constraints
        PASS [   5.000s] nexus-db-queries db::datastore::sled::test::test_sled_transitions
        PASS [   2.144s] nexus-db-queries db::datastore::test::test_rack_initialize_is_idempotent
        PASS [   2.216s] nexus-db-queries db::datastore::test::test_region_allocation_not_enough_zpools
        PASS [   2.776s] nexus-db-queries db::datastore::test::test_region_allocation_only_considers_disks_in_service
        PASS [   2.706s] nexus-db-queries db::datastore::test::test_region_allocation_only_operates_on_zpools_in_inventory
        PASS [   2.226s] nexus-db-queries db::datastore::test::test_session_methods
        PASS [   2.166s] nexus-db-queries db::datastore::test::test_sled_ipv6_address_allocation
        PASS [   2.032s] nexus-db-queries db::datastore::test::test_ssh_keys
        PASS [   1.925s] nexus-db-queries db::datastore::test::test_table_scan
        PASS [   5.810s] nexus-db-queries db::datastore::test::test_region_allocation_is_idempotent
        PASS [   2.166s] nexus-db-queries db::datastore::virtual_provisioning_collection::test::test_instance_create_and_delete
        PASS [   1.978s] nexus-db-queries db::datastore::virtual_provisioning_collection::test::test_storage_create_and_delete
        PASS [   0.014s] nexus-db-queries db::datastore::volume::tests::test_read_only_target_in_vcr
        PASS [   0.013s] nexus-db-queries db::datastore::volume::tests::test_replace_read_only_target_in_vcr
        PASS [   2.260s] nexus-db-queries db::datastore::virtual_provisioning_collection::test::test_instance_create_and_delete_twice
        PASS [   1.995s] nexus-db-queries db::datastore::virtual_provisioning_collection::test::test_storage_create_and_delete_twice
        PASS [   5.460s] nexus-db-queries db::datastore::test::test_region_allocation_strat_random
        PASS [   5.852s] nexus-db-queries db::datastore::test::test_region_allocation_out_of_space_fails
        PASS [   5.607s] nexus-db-queries db::datastore::test::test_region_allocation_strat_random_with_distinct_sleds_fails
        PASS [   6.189s] nexus-db-queries db::datastore::test::test_region_allocation_strat_random_with_distinct_sleds
        PASS [   2.465s] nexus-db-queries db::datastore::vmm::tests::test_vmm_and_migration_update_runtime
        PASS [   2.114s] nexus-db-queries db::datastore::volume::tests::test_deserialize_old_crucible_resources
        PASS [   2.131s] nexus-db-queries db::datastore::volume::tests::test_find_volumes_referencing_socket_addr
        PASS [   2.234s] nexus-db-queries db::datastore::volume::tests::test_volume_replace_snapshot
        PASS [   2.490s] nexus-db-queries db::datastore::volume::tests::test_volume_replace_region
        PASS [   1.941s] nexus-db-queries db::datastore::volume_repair::test::volume_lock_conflict_error_returned
        PASS [   2.199s] nexus-db-queries db::datastore::vpc::tests::test_project_create_vpc_raw_returns_none_on_vni_exhaustion
        PASS [   0.930s] nexus-db-queries db::explain::test::test_explain_full_table_scan
        PASS [   0.988s] nexus-db-queries db::explain::test::test_explain_async
        PASS [   2.327s] nexus-db-queries db::datastore::vpc::tests::test_project_create_vpc_retries
        PASS [   0.014s] nexus-db-queries db::pagination::test::test_paginator
        PASS [   1.396s] nexus-db-queries db::pagination::test::test_paginated_multicolumn_descending
        PASS [   1.466s] nexus-db-queries db::pagination::test::test_paginated_multicolumn_ascending
        PASS [   1.254s] nexus-db-queries db::pagination::test::test_paginated_multicolumn_works_with_joins
        PASS [   2.841s] nexus-db-queries db::datastore::vpc::tests::test_vpc_router_rule_instance_resolve
        PASS [   1.410s] nexus-db-queries db::pagination::test::test_paginated_single_column_ascending
        PASS [   1.440s] nexus-db-queries db::pagination::test::test_paginated_single_column_descending
        PASS [   2.103s] nexus-db-queries db::lookup::test::test_lookup
        PASS [   3.402s] nexus-db-queries db::datastore::vpc::tests::test_vpc_resolve_to_sleds_uses_current_target_blueprint
        PASS [   3.151s] nexus-db-queries db::datastore::vpc::tests::test_vpc_system_router_sync_to_subnets
        PASS [   2.607s] nexus-db-queries db::queries::external_ip::tests::test_ensure_pool_exhaustion_does_not_use_other_pool
        PASS [   2.266s] nexus-db-queries db::queries::external_ip::tests::test_external_ip_allocate_omicron_zone_out_of_range
        PASS [   0.013s] nexus-db-queries db::queries::network_interface::tests::test_first_available_address
        PASS [   2.670s] nexus-db-queries db::queries::external_ip::tests::test_ephemeral_and_snat_ips_do_not_overlap
        PASS [   2.748s] nexus-db-queries db::queries::external_ip::tests::test_external_ip_allocate_omicron_zone_is_idempotent
        PASS [   2.322s] nexus-db-queries db::queries::external_ip::tests::test_insert_external_ip_is_idempotent
        PASS [   2.511s] nexus-db-queries db::queries::external_ip::tests::test_next_external_ip_allocation_and_exhaustion
        PASS [   2.444s] nexus-db-queries db::queries::external_ip::tests::test_next_external_ip_is_restricted_to_pools
        PASS [   2.335s] nexus-db-queries db::queries::external_ip::tests::test_next_external_ip_with_ephemeral_takes_whole_port_range
        PASS [   2.593s] nexus-db-queries db::queries::external_ip::tests::test_next_external_ip_out_of_order_allocation_ok
        PASS [   2.271s] nexus-db-queries db::queries::network_interface::tests::test_delete_service_is_idempotent
        PASS [   2.317s] nexus-db-queries db::queries::network_interface::tests::test_insert_multiple_vpc_subnets_succeeds
        PASS [   2.171s] nexus-db-queries db::queries::network_interface::tests::test_insert_no_instance_fails
        PASS [   2.868s] nexus-db-queries db::queries::network_interface::tests::test_detect_ip_exhaustion
        PASS [   2.311s] nexus-db-queries db::queries::network_interface::tests::test_insert_request_exact_ip
        PASS [   2.520s] nexus-db-queries db::queries::network_interface::tests::test_insert_multiple_vpcs_fails
        PASS [   2.268s] nexus-db-queries db::queries::network_interface::tests::test_insert_request_mac
        PASS [   0.014s] nexus-db-queries db::queries::network_interface::tests::test_last_available_address
        PASS [   2.416s] nexus-db-queries db::queries::network_interface::tests::test_insert_request_same_mac_fails
        PASS [   2.272s] nexus-db-queries db::queries::network_interface::tests::test_insert_request_same_slot_fails
        PASS [   2.487s] nexus-db-queries db::queries::network_interface::tests::test_insert_request_same_ip_fails
        PASS [   2.394s] nexus-db-queries db::queries::network_interface::tests::test_insert_request_slot
        PASS [   1.079s] nexus-db-queries db::queries::next_item::tests::test_explain_next_item_self_joined
        PASS [   0.012s] nexus-db-queries db::queries::next_item::tests::test_shift_indices
        PASS [   1.183s] nexus-db-queries db::queries::next_item::tests::test_next_item_query_is_ordered_by_indices
        PASS [   0.084s] nexus-db-queries db::queries::oximeter::test::expectorate_query_reassign_producers
        PASS [   1.262s] nexus-db-queries db::queries::next_item::tests::test_next_item_self_joined
        PASS [   1.158s] nexus-db-queries db::queries::next_item::tests::test_next_item_self_joined_with_gaps
        PASS [   0.103s] nexus-db-queries db::queries::oximeter::test::expectorate_query_upsert_producer
        PASS [   0.423s] nexus-db-queries db::queries::region_allocation::test::expectorate_query
        PASS [   2.429s] nexus-db-queries db::queries::network_interface::tests::test_insert_running_instance_fails
        PASS [   0.087s] nexus-db-queries db::queries::virtual_provisioning_collection_update::test::expectorate_query_delete_instance
        PASS [   2.470s] nexus-db-queries db::queries::network_interface::tests::test_insert_same_interface_fails
        PASS [   0.093s] nexus-db-queries db::queries::virtual_provisioning_collection_update::test::expectorate_query_delete_storage
        PASS [   2.508s] nexus-db-queries db::queries::network_interface::tests::test_insert_same_vpc_subnet_fails
        PASS [   0.092s] nexus-db-queries db::queries::virtual_provisioning_collection_update::test::expectorate_query_insert_instance
        PASS [   0.106s] nexus-db-queries db::queries::virtual_provisioning_collection_update::test::expectorate_query_insert_storage
        PASS [   2.402s] nexus-db-queries db::queries::network_interface::tests::test_insert_with_duplicate_name_fails
        PASS [   1.135s] nexus-db-queries db::queries::next_item::tests::test_wrapping_next_item_query
        PASS [   0.035s] nexus-db-queries db::queries::vpc::tests::test_vni_search_iter_full_count
        PASS [   0.017s] nexus-db-queries db::queries::vpc::tests::test_vni_search_iter_steps
        PASS [   0.015s] nexus-db-queries db::queries::vpc::tests::test_vni_search_iter_wrapping
        PASS [   0.015s] nexus-db-queries db::queries::vpc::tests::test_vni_shift_no_wrapping
        PASS [   2.624s] nexus-db-queries db::queries::network_interface::tests::test_insert_sequential_ip_allocation
        PASS [   0.012s] nexus-db-queries db::queries::vpc::tests::test_vni_shift_with_wrapping
        PASS [   2.544s] nexus-db-queries db::queries::network_interface::tests::test_limit_number_of_interfaces_per_instance_query
        PASS [   1.107s] nexus-db-queries db::queries::oximeter::test::explainable_reassign_producers
        PASS [   1.174s] nexus-db-queries db::queries::oximeter::test::explainable_upsert_producer
        PASS [   1.015s] nexus-db-queries db::queries::region_allocation::test::explainable
        PASS [   0.953s] nexus-db-queries db::queries::virtual_provisioning_collection_update::test::explain_delete_storage
        PASS [   1.070s] nexus-db-queries db::queries::virtual_provisioning_collection_update::test::explain_delete_instance
        PASS [   1.279s] nexus-db-queries db::queries::virtual_provisioning_collection_update::test::explain_insert_instance
        PASS [   1.238s] nexus-db-queries db::queries::virtual_provisioning_collection_update::test::explain_insert_storage
        PASS [   1.143s] nexus-db-queries db::queries::vpc_subnet::test::explain_insert_query
        PASS [   1.141s] nexus-db-queries db::queries::vpc_subnet::test::test_insert_vpc_subnet_query
        PASS [   1.118s] nexus-db-queries db::queries::vpc_subnet::test::test_insert_vpc_subnet_query_is_idempotent
        PASS [   1.699s] nexus-db-queries transaction_retry::test::test_transaction_retry_produces_samples
        PASS [   1.555s] nexus-db-queries transaction_retry::test::test_transaction_rollback_produces_no_samples
        PASS [   2.822s] nexus-db-queries policy_test::test_conferred_roles
        PASS [  19.285s] nexus-db-queries policy_test::test_iam_roles_behavior
------------
     Summary [  70.486s] 231 tests run: 231 passed, 1 skipped

So pretty good!