[Feature Request] Add routing to Translog

[kkewwei]

Is your feature request related to a problem? Please describe

In our product, we've developed a tool that replays the translog and write to multiple OpenSearch clusters. However, when performing indexing or deletion operations with routing, the target shards turn out to be incorrect. The root cause lies in the fact that the translog does not include the routing.

Describe the solution you'd like

Add routing to Translog

Related component

Indexing

Describe alternatives you've considered

No response

Additional context

No response

[andrross]

I assuming you mean this routing parameter, right? I can see why it's not included in the translog...the routing is used to determine the shard but the translog is per-shard so the routing decision has already been made by the time the translog is written to.

[kkewwei]

Yes, it's the parameter. When it's written in the shard, it's useless for the shard. In some scenarios, we will also reply this doc to other clusters/mysql, and we need this routing to continue makeing decision.

[andrross]

When it's written in the shard, it's useless for the shard

This is why I would be hesitant to include it in the translog, as it ends up being a small but unnecessary overhead for most use cases.

Would it be possible to implement an ingest pipeline that captures this parameter and writes it into some sort of specially defined field in the documents themselves? You could even add a search pipeline that would filter out this field from search responses if necessary.

[kkewwei]

Good idea, Thank you.