Can Filelog Receiver Support More Encoding Extensions?

[vaisu21]

Hi @djaglowski

I have developed a custom encoding extension that converts a non-OTLP format into OTLP, and I want to enable this encoding extension to be supported by the Filelog Receiver.

From the documentation, it is clear that Filelog Receiver supports encoding extensions:
Ref: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/filelogreceiver/README.md#supported-encodings

However, I noticed that the currently listed supported encoding extensions for Filelog Receiver do not include existing ones like otlpencodingextension or zipkinencodingextension.
Ref: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/filelogreceiver/README.md#supported-encodings

Is there any reason for not supporting existing encoding extensions (such as otlpencodingextension, zipkinencodingextension) in Filelog Receiver?

If support is possible, could you confirm whether the following implementation approach is appropriate?

Modified config.go and filelog.go to integrate the encoding extension.
Added a new file, unmarshaller.go, similar to Kafka Receiver, to register all encoding extensions that need to be used.
Looking forward to your insights on this!

[djaglowski]

The filelog receiver does not support encoding extensions. It supports encodings.

Encoding extensions decode bytes into pdata format (internal representation of OTLP data model). The encodings in the filelog receiver are character-level encodings, to decode bytes into a strings.

The use case you are describing sounds a lot like the otlpjson connector, which can be used to unmarshal the content of log bodies into pdata. Perhaps what is needed is to generalize that connector such that it unmarshals via encoding extensions.

[vaisu21]

@djaglowski Thanks for the information.

Based on the OpenTelemetry documentation I understand, connectors are primarily designed to process, route, or transform already structured pdata from one telemetry signal to another within the collector pipeline. However, my use case involves log producers generating logs in a non-OTLP format, which need to be transformed into pdata. This aligns better with the role of an encoding extension rather than a connector.

In the attached diagram, my use case includes multiple receivers such as filelogreceiver, kafkareceiver, and otlpreceiver, where encoding extensions should be applied for tranformation of bytes to pdata format. The kafkareceiver already supports encoding extensions, whereas otlpreceiver likely requires code modifications to enable support for encoding extensions.

A similar situation applies to exporters—fileexporter already supports encoding extensions, while other exporters may require updates in their respective modules to handle encoding extensions effectively.

Could you provide your view on the above?

[djaglowski]

I think it's valid to use a connector as the point of decoding, even if it is an encoding extension that does the actual work. As I said, this is essentially what the otlpjson connector is already doing, except it hasn't offloaded the work to an extension. (Though arguably it should.)

I would argue that the use case requires a connector because if you look at it more generally you may want to decode data into any type of telemetry (not just logs, but metrics, traces, or profiles). Therefore, you cannot solve it with a logs-only receiver.

The solution i'm recommending is to use filelog to read the file, but consider the body to be an encoded representation of pdata until is hits a connector that can decode it into whichever telemetry type is appropriate.

[vaisu21]

@djaglowski Thanks for the reply.

I noticed from the File Exporter’s README that it supports encoding extensions. If the exporter supports them, is there a reason why the File Receiver does not?

Ref: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/fileexporter/README.md

[djaglowski]

Yes, the filelog receiver is for logs only. The file exporter is for all telemetry types.