[FEATURE] [transport-netty4] Investigate support of the direct buffers by default

[reta]

Is your feature request related to a problem? Please describe

While working on #17408, we found out that some compression codecs (fe ZstdEncoder) use direct buffers unconditionally. The default NettyAllocator::ALLOCATOR is NoDirectBuffers and does not support direct buffers.

Describe the solution you'd like

• Understand why NoDirectBuffers had been chosen as the default
• Consider supporting direct buffers by default

Related component

Other

Describe alternatives you've considered

N/A

Additional context

• HTTP API calls hang with 'Accept-Encoding: zstd' #17408

[kkewwei]

NoDirectBuffers is mainly used in network, while Every thread has it's own DirectByteBuffert to use, there seems no need to create the directArena in PoolThreadCache
(https://github.com/opensearch-project/OpenSearch/blob/main/modules/transport-netty4/src/main/java/org/opensearch/transport/CopyBytesSocketChannel.java),

The purpose of this class is to allow the disabling of netty direct buffer pooling while allowing us to control how bytes end up being copied to direct memory.

[andrross]

Catch All Triage - 1