use the multi_thread executor to run the telemetry hang test (#922)

The test wasn't using the multi_thread executor, so eventually it would
trip over the bug in opentelemetry which we've hit in the past where the
runtime would hang for not fully understood reasons.

see: open-telemetry/opentelemetry-rust#536

Using the multi_thread executor is the recommended work-around.

I've also remove the redundant thread spawn from the telemetry Drop
implementation, since I don't think it's required as well as the
spawn_blocking() call.

I've run this test in a tight loop for a couple of hours (on my M1
laptop) with no issues.

fixes: #914

Author: garypen

apollo-router/src/plugins/telemetry/mod.rs

@@ -100,14 +100,13 @@ impl Drop for Telemetry {
             // Tracer providers must be flushed. This may happen as part of otel if the provider was set
             // as the global, but may also happen in the case of an failed config reload.
             // If the tracer prover is present then it was not handed over so we must flush it.
-            // The magic incantation seems to be that the flush MUST happen in a separate thread.
+            // We must make the call to force_flush() from spawn_blocking() (or spawn a thread) to
+            // ensure that the call to force_flush() is made from a separate thread.
             ::tracing::debug!("flushing telemetry");
-            std::thread::spawn(|| async {
-                let jh = tokio::task::spawn_blocking(move || {
-                    opentelemetry::trace::TracerProvider::force_flush(&tracer_provider);
-                });
-                futures::executor::block_on(jh).expect("failed to flush tracer provider");
+            let jh = tokio::task::spawn_blocking(move || {
+                opentelemetry::trace::TracerProvider::force_flush(&tracer_provider);
             });
+            futures::executor::block_on(jh).expect("failed to flush tracer provider");
         }
 
         if let Some(sender) = self.spaceport_shutdown.take() {

apollo-router/src/router_factory.rs

@@ -294,7 +294,9 @@ mod test {
         assert!(service.is_err())
     }
 
-    #[tokio::test]
+    // This test must use the multi_thread tokio executor or the opentelemetry hang bug will
+    // be encountered. (See https://github.com/open-telemetry/opentelemetry-rust/issues/536)
+    #[tokio::test(flavor = "multi_thread")]
     async fn test_telemetry_doesnt_hang_with_invalid_schema() {
         use crate::subscriber::{set_global_subscriber, RouterSubscriber};
         use tracing_subscriber::EnvFilter;