How to mix the traces from Datadog and OpenTelemetry?

[MacNale]

Team,

I am having some java services (SERVICE_A) being instrumented with Datadog agents. There are some services (SERVICE_B) which are not being instrumented by Datadog. These are legacy services written in a language which is not supported by any agent. The legacy services produces some traces in proprietary format. I am trying to convert these traces using opentelemetry SDK. This trace has Datadog parent TRACEID from SERVICE_A.

I am not sure, whether Datadog would be able to take traces from SERVICE_A instrumented using Datadog agent and SERVICE_B traces produced using OpenTelemetry SDK and present as a single transaction.

thanks,
Mac

[ericmustin]

👋 Hello , i'm glad you brought this up, I think I can help here. So, for services you're migrating to opentelemetry- sdks, but you still want them to connect to datadog-trace- sdk instrumented applications, as long as you're adding Propagation to both OpenTelemetry and Datadog using B3 Headers, you should be able to correctly pass trace context between the apps and they'll be connected in the datadog backend

Here's an example of how this would be enabled for opentelemetry-python <=> datadog-trace-rb, for example:

• https://github.com/open-telemetry/opentelemetry-python/blob/0cd037fdd396f68a8080f84b3a71ecea49c1a393/docs/getting-started.rst#configure-your-http-propagator-b3-baggage
• https://docs.datadoghq.com/tracing/setup_overview/custom_instrumentation/ruby/?tab=activespan#b3-headers-extraction-and-injection

Also worth noting that opentelemetry- sdk traces would have to go through the opentelemetry-collector+datadogexporter, and datadog-trace- sdk traces would have to go through the datadog-agent, but they'd both be connected in your backend/UI.

Does that help?

[MacNale]

@ericmustin Thank you for the links.

What happens if the existing application use proprietary context (e.g "Datadog: Datadog"). Will be I able to still fill in the trace from my legacy service and use opentelemetry collector to export to Datadog -

Sudo Java code in the service which builds the traces from legacy service -

void buildContext() {
  SpanContext remoteSpanContext = SpanContext.createFromRemoteParent( TRACEID, SPANID,
        TraceFlags.getSampled(), TraceState.getDefault());
  buildTraceAndSendToDataDogThroughCollector(remoteSpanContext);
}

void buildTraceAndSendToDataDogThroughCollector(SpanContext spanContext) {
  OpenTelemetry openTelemetryParent = ExampleConfiguration.initOpenTelemetry();
  Tracer trace = openTelemetryParent.getTracer("LegacyService");
   
   //Read the traces from the file which includes DataDog TraceID.

  Span parentSpan = trace.spanBuilder("Blah")
      .setStartTimestamp(getStartTime(), TimeUnit.MILLISECONDS)
      .setParent(Context.current().with(Span.wrap(spanContext)))
      .startSpan();  

  parentSpan.end(getEndTime(), TimeUnit.MILLISECONDS);
}

This is my converter service which uses opentelemetry sdk and emits the traces to OpenTelemetry Collector, so that I can ingest into Datadog. Do you see any problems with this?

Note: I will check with my application team, if they are okay to change the context to "B3" in service_A instrumented by datadog agent.

[ericmustin]

@MacNale apologies on the delay i had missed this msg. as far as I know, within an application process, traces generated from dd-trace-java are not interoperable with those from opentelemetry-java. Seperate services can be connected via distributed tracing.

[MacNale]

@ericmustin

These are two independent services. This is what I want to achieve.

[ericmustin]

@MacNale Ah, thank you for the clarification. My original answer should be what you're looking for here, apologies as this is not well documented in our docs but is possible, I'll work with the documentation team on updating for this use case.

A Datadog-instrumented application (service_a), emitting traces to a datadog-agent, can have it's traces connected to an opentelemetry-instrumented application (service_b), emitting traces to an opentelemetry-collector (which would have a datadogexporter configured). It is necessary to enable B3 trace context propagation (inject) on your datadog application, as well as on your opentelemetry-application. Here is an example for java:

• B3 datadog: https://docs.datadoghq.com/tracing/setup_overview/setup/java?tab=containers#b3-headers-extraction-and-injection
• B3 (multi) opentelemetry: https://github.com/open-telemetry/opentelemetry-java/blob/72db16e58e3e8cd83c213ae1292df698ecd9d553/sdk-extensions/autoconfigure/README.md#propagator

Does that help? If you are having issues with a specific lang i'm happy to try to help troubleshoot further

[ericmustin]

Are these services connected via HTTP? my answer here is specific to distributed tracing over HTTP

[MacNale]

@ericmustin These are connected with HTTP(SOAP/REST) and IBM MQ. How do you deal with MQ? The application is written in COBOL.

[matthiashaasesourceforge]

@MacNale Did you solve your problem last days/weeks/years ?
Same problem on my side. But not with COBOL. It's Fortran am mqtt.

[isaac621]

@ericmustin Do the datadog-trace- sdk instrumented applications have to enable the otel feature first, as instrumentation here https://docs.datadoghq.com/tracing/trace_collection/custom_instrumentation/php/otel/#setup? As I see tracecontext format ingestion and extaction should be enabled by default in dd-trace library.