[BUG] OTEL traces/logs get constant 404s after otlp upgrade

[brooksmtownsend]

Affected project(s)

• documentation
• examples
• wasmCloud host
• wasmCloud CLI (wash)
• wasmCloud dashboard UI (washboard)
• capability providers
• provider bindgen
• control interface client
• other / not sure

Describe the bug

In the console when running the wasmCloud host from main, or any providers built from main, I get the following output constantly as direct console output (not formatted with our logs):

OpenTelemetry trace error occurred. HTTP status client error (404 Not Found) for url (http://localhost:4318/)
OpenTelemetry log error occurred. HTTP status client error (404 Not Found) for url (http://localhost:4318/)
OpenTelemetry trace error occurred. HTTP status client error (404 Not Found) for url (http://localhost:4318/)

Steps to reproduce

1. checkout main
2. Launch the OTEL docker compose under examples/docker
3. cargo run -- --enable-observability
4. See errors

Expected behavior

I expected for the default tracing/logging endpoints to keep working as before.

Environment

• OS: [e.g. Linux, MacOS] MacOS
• Shell [e.g. bash, zsh, powershell] zsh
• wasmcloud Version [e.g. 0.81.0] main

Screenshots / Logs / Additional context

I don't think this is technically a bug in the upstream otlp crate, it was just a breaking change that wasn't documented well enough for us to catch it from the changelog. open-telemetry/opentelemetry-rust#1866

Introduced in #2197 so this hasn't been released in any providers/the host

[brooksmtownsend]

As a workaround I've been able to test using our override endpoints.

./wasmcloud --override-traces-endpoint http://localhost:4318/v1/traces --override-logs-endpoint http://localhost:4318/v1/logs

[peter-jerry-ye]

I'm somehow still experiencing this using the docker-compose-full.yml in the repository. Shouldn't it been fixed?

[vados-cosmonic]

@peter-jerry-ye is this possibly a duplicate of #2839 ? How many of those messages are you getting versus how many providers do you have running?

[peter-jerry-ye]

@vados-cosmonic It's different in that I used the docker file directly or with endpoint http://otelcol:4318/ and they both gave me 404, which is exactly as described in this issue.

I'm not sure if we are expected to add /v1/entry? I thought it was just a hacking solution and the document (though might be outdated: https://wasmcloud.com/docs/reference/host-config) didn't mention it.

I got two messages and 0 providers. I just spin up the docker compose and deployed nothing else to it. It's kinda the first few messages that complained the error to me.