Skip to content

Commit a444730

Browse files
committed
Simplified self-diagnostics example
1 parent 91f44ff commit a444730

File tree

6 files changed

+59
-243
lines changed

6 files changed

+59
-243
lines changed

examples/self-diagnostics/Cargo.toml

-4
Original file line numberDiff line numberDiff line change
@@ -9,11 +9,7 @@ publish = false
99
opentelemetry = { path = "../../opentelemetry" }
1010
opentelemetry_sdk = { path = "../../opentelemetry-sdk", features = ["rt-tokio"]}
1111
opentelemetry-stdout = { path = "../../opentelemetry-stdout"}
12-
opentelemetry-appender-tracing = { path = "../../opentelemetry-appender-tracing"}
1312
tokio = { workspace = true, features = ["full"] }
1413
tracing = { workspace = true, features = ["std"]}
1514
tracing-core = { workspace = true }
1615
tracing-subscriber = { version = "0.3.18", features = ["env-filter","registry", "std"]}
17-
opentelemetry-otlp = { path = "../../opentelemetry-otlp", features = ["http-proto", "reqwest-client", "logs"] }
18-
once_cell ={ version = "1.19.0"}
19-
ctrlc = "3.4"

examples/self-diagnostics/Dockerfile

-6
This file was deleted.

examples/self-diagnostics/README.md

+21-88
Original file line numberDiff line numberDiff line change
@@ -1,93 +1,26 @@
11
# Basic OpenTelemetry metrics example with custom error handler:
22

3-
This example shows how to setup the custom error handler for self-diagnostics.
4-
5-
## Custom Error Handling:
6-
7-
A custom error handler is set up to capture and record errors using the `tracing` crate's `error!` macro. These errors are then exported to a collector using the `opentelemetry-appender-tracing` crate, which utilizes the OTLP log exporter over `HTTP/protobuf`. As a result, any errors generated by the configured OTLP metrics pipeline are funneled through this custom error handler for proper recording and export.
3+
This example shows how to self-diagnose OpenTelemetry by enabling its internal
4+
logs. OpenTelemetry crates publish internal logs when "internal-logs" feature is
5+
enabled. This feature is enabled by default. Internal logs are published using
6+
`tracing` events, and hence, a `tracing` subscriber must be configured without
7+
which the logs are simply discarded.
88

99
## Filtering logs from external dependencies of OTLP Exporter:
1010

11-
The example configures a tracing `filter` to restrict logs from external crates (`hyper`, `tonic`, and `reqwest`) used by the OTLP Exporter to the `error` level. This helps prevent an infinite loop of log generation when these crates emit logs that are picked up by the tracing subscriber.
12-
13-
## Ensure that the internally generated errors are logged only once:
14-
15-
By using a hashset to track seen errors, the custom error handler ensures that the same error is not logged multiple times. This is particularly useful for handling scenarios where continuous error logging might occur, such as when the OpenTelemetry collector is not running.
16-
17-
18-
## Usage
19-
20-
### `docker-compose`
21-
22-
By default runs against the `otel/opentelemetry-collector:latest` image, and uses `reqwest-client`
23-
as the http client, using http as the transport.
24-
25-
```shell
26-
docker-compose up
27-
```
28-
29-
In another terminal run the application `cargo run`
30-
31-
The docker-compose terminal will display logs, traces, metrics.
32-
33-
Press Ctrl+C to stop the collector, and then tear it down:
34-
35-
```shell
36-
docker-compose down
37-
```
38-
39-
### Manual
40-
41-
If you don't want to use `docker-compose`, you can manually run the `otel/opentelemetry-collector` container
42-
and inspect the logs to see traces being transferred.
43-
44-
On Unix based systems use:
45-
46-
```shell
47-
# From the current directory, run `opentelemetry-collector`
48-
docker run --rm -it -p 4318:4318 -v $(pwd):/cfg otel/opentelemetry-collector:latest --config=/cfg/otel-collector-config.yaml
49-
```
50-
51-
On Windows use:
52-
53-
```shell
54-
# From the current directory, run `opentelemetry-collector`
55-
docker run --rm -it -p 4318:4318 -v "%cd%":/cfg otel/opentelemetry-collector:latest --config=/cfg/otel-collector-config.yaml
56-
```
57-
58-
Run the app which exports logs, metrics and traces via OTLP to the collector
59-
60-
```shell
61-
cargo run
62-
```
63-
64-
### Output:
65-
66-
- If the docker instance for collector is running, below error should be logged into the container. There won't be any logs from the `hyper`, `reqwest` and `tonic` crates.
67-
```
68-
otel-collector-1 | 2024-06-05T17:09:46.926Z info LogExporter {"kind": "exporter", "data_type": "logs", "name": "logging", "resource logs": 1, "log records": 1}
69-
otel-collector-1 | 2024-06-05T17:09:46.926Z info ResourceLog #0
70-
otel-collector-1 | Resource SchemaURL:
71-
otel-collector-1 | Resource attributes:
72-
otel-collector-1 | -> telemetry.sdk.name: Str(opentelemetry)
73-
otel-collector-1 | -> telemetry.sdk.version: Str(0.23.0)
74-
otel-collector-1 | -> telemetry.sdk.language: Str(rust)
75-
otel-collector-1 | -> service.name: Str(unknown_service)
76-
otel-collector-1 | ScopeLogs #0
77-
otel-collector-1 | ScopeLogs SchemaURL:
78-
otel-collector-1 | InstrumentationScope opentelemetry-appender-tracing 0.4.0
79-
otel-collector-1 | LogRecord #0
80-
otel-collector-1 | ObservedTimestamp: 2024-06-05 17:09:45.931951161 +0000 UTC
81-
otel-collector-1 | Timestamp: 1970-01-01 00:00:00 +0000 UTC
82-
otel-collector-1 | SeverityText: ERROR
83-
otel-collector-1 | SeverityNumber: Error(17)
84-
otel-collector-1 | Body: Str(OpenTelemetry metrics error occurred: Metrics error: Warning: Maximum data points for metric stream exceeded. Entry added to overflow. Subsequent overflows to same metric until next collect will not be logged.)
85-
otel-collector-1 | Attributes:
86-
otel-collector-1 | -> name: Str(event examples/self-diagnostics/src/main.rs:42)
87-
otel-collector-1 | Trace ID:
88-
otel-collector-1 | Span ID:
89-
otel-collector-1 | Flags: 0
90-
otel-collector-1 | {"kind": "exporter", "data_type": "logs", "name": "logging"}
91-
```
92-
93-
- The SDK will keep trying to upload metrics at regular intervals if the collector's Docker instance is down. To avoid a logging loop, internal errors like 'Connection refused' will be attempted to be logged only once.
11+
The example configures a tracing `filter` to restrict logs from external crates
12+
(`hyper`, `tonic`, and `reqwest`) used by the OTLP Exporter to the `error`
13+
level. This helps prevent an infinite loop of log generation when these crates
14+
emit logs that are picked up by the tracing subscriber. This is only a
15+
workaround until https://github.com/open-telemetry/opentelemetry-rust/issues/761
16+
is resolved.
17+
18+
## Filtering logs to be send to OpenTelemetry itself
19+
20+
If you use [OpenTelemetry Tracing
21+
Appender](../../opentelemetry-appender-tracing/README.md) to send `tracing` logs
22+
to OpenTelemetry, then enabling OpenTelemetry internal logs can also cause
23+
infinite, recursive logging. You can filter out all OpenTelemetry internal logs
24+
from being sent to [OpenTelemetry Tracing
25+
Appender](../../opentelemetry-appender-tracing/README.md) using a filter, like
26+
"add_directive("opentelemetry=off".parse().unwrap())".

examples/self-diagnostics/docker-compose.yaml

-11
This file was deleted.

examples/self-diagnostics/otel-collector-config.yaml

-29
This file was deleted.

examples/self-diagnostics/src/main.rs

+38-105
Original file line numberDiff line numberDiff line change
@@ -1,85 +1,16 @@
1-
use opentelemetry::global::{self, Error as OtelError};
1+
use opentelemetry::global;
22
use opentelemetry::KeyValue;
3-
use opentelemetry_appender_tracing::layer;
4-
use opentelemetry_otlp::{LogExporter, MetricExporter, WithExportConfig};
53
use opentelemetry_sdk::metrics::PeriodicReader;
6-
use tracing_subscriber::prelude::*;
7-
84
use std::error::Error;
9-
10-
use once_cell::sync::Lazy;
11-
use std::collections::HashSet;
12-
use std::sync::{Arc, Mutex};
13-
14-
use std::sync::mpsc::channel;
15-
16-
fn init_logger_provider() -> opentelemetry_sdk::logs::LoggerProvider {
17-
let exporter = LogExporter::builder()
18-
.with_http()
19-
.with_endpoint("http://localhost:4318/v1/logs")
20-
.build()
21-
.unwrap();
22-
23-
let provider = opentelemetry_sdk::logs::LoggerProvider::builder()
24-
.with_batch_exporter(exporter, opentelemetry_sdk::runtime::Tokio)
25-
.build();
26-
27-
let cloned_provider = provider.clone();
28-
29-
// Specialized filter to process
30-
// - ERROR logs from specific targets
31-
// - ERROR logs generated internally.
32-
let internal_and_dependency_filter = tracing_subscriber::filter::filter_fn(|metadata| {
33-
let target = metadata.target();
34-
35-
// Only allow ERROR logs from specific targets
36-
(target.starts_with("hyper")
37-
|| target.starts_with("hyper_util")
38-
|| target.starts_with("hyper")
39-
|| target.starts_with("tonic")
40-
|| target.starts_with("tower")
41-
|| target.starts_with("reqwest")
42-
|| target.starts_with("opentelemetry"))
43-
&& metadata.level() == &tracing::Level::ERROR
44-
});
45-
// Configure fmt::Layer to print detailed log information, including structured fields
46-
let fmt_internal_and_dependency_layer =
47-
tracing_subscriber::fmt::layer().with_filter(internal_and_dependency_filter.clone());
48-
49-
// Application filter to exclude specific targets entirely, regardless of level
50-
let application_filter = tracing_subscriber::filter::filter_fn(|metadata| {
51-
let target = metadata.target();
52-
53-
// Exclude logs from specific targets for the application layer
54-
!(target.starts_with("hyper")
55-
|| target.starts_with("hyper_util")
56-
|| target.starts_with("hyper")
57-
|| target.starts_with("tonic")
58-
|| target.starts_with("tower")
59-
|| target.starts_with("reqwest")
60-
|| target.starts_with("opentelemetry"))
61-
});
62-
63-
let application_layer = layer::OpenTelemetryTracingBridge::new(&cloned_provider)
64-
.with_filter(application_filter.clone());
65-
66-
tracing_subscriber::registry()
67-
.with(fmt_internal_and_dependency_layer)
68-
.with(application_layer)
69-
.init();
70-
provider
71-
}
5+
use tracing::info;
6+
use tracing_subscriber::fmt;
7+
use tracing_subscriber::prelude::*;
8+
use tracing_subscriber::EnvFilter;
729

7310
fn init_meter_provider() -> opentelemetry_sdk::metrics::SdkMeterProvider {
74-
let exporter = MetricExporter::builder()
75-
.with_http()
76-
.with_endpoint("http://localhost:4318/v1/metrics")
77-
.build()
78-
.unwrap();
11+
let exporter = opentelemetry_stdout::MetricExporterBuilder::default().build();
7912

80-
let reader = PeriodicReader::builder(exporter, opentelemetry_sdk::runtime::Tokio)
81-
.with_interval(std::time::Duration::from_secs(1))
82-
.build();
13+
let reader = PeriodicReader::builder(exporter, opentelemetry_sdk::runtime::Tokio).build();
8314

8415
let provider = opentelemetry_sdk::metrics::SdkMeterProvider::builder()
8516
.with_reader(reader)
@@ -92,41 +23,43 @@ fn init_meter_provider() -> opentelemetry_sdk::metrics::SdkMeterProvider {
9223

9324
#[tokio::main]
9425
async fn main() -> Result<(), Box<dyn Error + Send + Sync + 'static>> {
95-
let logger_provider = init_logger_provider();
26+
// OpenTelemetry uses `tracing` crate for its internal logging. Unless a
27+
// tracing subscriber is set, the logs will be discarded. In this example,
28+
// we configure a `tracing` subscriber to:
29+
// 1. Print logs of level INFO or higher to stdout.
30+
// 2. Filter logs from OpenTelemetry's dependencies (like tonic, hyper,
31+
// reqwest etc. which are commonly used by the OTLP exporter) to only print
32+
// ERROR-level logs. This filtering helps reduce repetitive log messages
33+
// that could otherwise create an infinite loop of log output. This is a
34+
// workaround until
35+
// https://github.com/open-telemetry/opentelemetry-rust/issues/761 is
36+
// resolved.
37+
38+
// Target name used by OpenTelemetry always start with "opentelemetry".
39+
// Hence, one may use "add_directive("opentelemetry=off".parse().unwrap())"
40+
// to turn off all logs from OpenTelemetry.
41+
42+
let filter = EnvFilter::new("info")
43+
.add_directive("hyper=error".parse().unwrap())
44+
.add_directive("tonic=error".parse().unwrap())
45+
.add_directive("h2=error".parse().unwrap())
46+
.add_directive("tower=error".parse().unwrap())
47+
.add_directive("reqwest=error".parse().unwrap());
48+
tracing_subscriber::registry()
49+
.with(fmt::layer().with_thread_names(true).with_filter(filter))
50+
.init();
9651

9752
// Initialize the MeterProvider with the stdout Exporter.
9853
let meter_provider = init_meter_provider();
54+
info!("Starting self-diagnostics example");
9955

100-
// Create a meter from the above MeterProvider.
10156
let meter = global::meter("example");
102-
// Create a Counter Instrument.
103-
let counter = meter.u64_counter("my_counter").build();
104-
105-
// Record measurements with unique key-value pairs to exceed the cardinality limit
106-
// of 2000 and trigger error message
107-
for i in 0..3000 {
108-
counter.add(
109-
10,
110-
&[KeyValue::new(
111-
format!("mykey{}", i),
112-
format!("myvalue{}", i),
113-
)],
114-
);
115-
}
116-
117-
let (tx, rx) = channel();
118-
119-
ctrlc::set_handler(move || tx.send(()).expect("Could not send signal on channel."))
120-
.expect("Error setting Ctrl-C handler");
121-
122-
println!("Press Ctrl-C to continue...");
123-
rx.recv().expect("Could not receive from channel.");
124-
println!("Got Ctrl-C, Doing shutdown and existing.");
57+
// Create a counter using an invalid name to trigger
58+
// internal log about the same.
59+
let counter = meter.u64_counter("my_counter with_space").build();
60+
counter.add(10, &[KeyValue::new("key", "value")]);
12561

126-
// MeterProvider is configured with an OTLP Exporter to export metrics every 1 second,
127-
// however shutting down the MeterProvider here instantly flushes
128-
// the metrics, instead of waiting for the 1 sec interval.
12962
meter_provider.shutdown()?;
130-
let _ = logger_provider.shutdown();
63+
info!("Shutdown complete. Bye!");
13164
Ok(())
13265
}

0 commit comments

Comments
 (0)