Support sending messages in the background

[hubcio]

At 25.02.2025 there was interesting discussion on discord.

The discussion revolved around performance optimizations and feature enhancements in iggy compared to Kafka. Participants debated strategies such as sticky partitioning, batching, and compression, noting that intelligent client implementations have a significant impact on overall throughput and latency. They compared Kafka’s round-robin partitioning with sticky partitioning, emphasizing that filling one partition’s batch before moving to the next can greatly improve performance and cost efficiency by reducing network load.

The conversation also touched on API for sending messages in background and error handling during background sends. Participants discussed the merits of different failure handling strategies, such as blocking until a send succeeds, blocking with a timeout, or failing immediately. To address these options, they proposed introducing a new enum, IggyBackgroundSendFailureMode, which would allow clients to choose the desired behavior when a background send fails.

As a summary of this discussion, I propose task to support this API:

Create an extension for the IggyProducerBuilder that introduces configurable parameters for background sending.

Changes in existing fields:

• batch_size - should be changed to batch_length.
• send_interval - should be changed to linger_time, determines how long messages should remain in the queue before sending

Add new fields (optional?):

• background_send - determines if send in background functionality should be enabled
• batch_size - value (in bytes) of how much data should be buffered before sending
• maximum_buffer_size - maximum value (in bytes) of buffered data, if exceeded error will be reported (see IggyBackgroundSendFailureMode)
• failure_mode - determines behavior of send_messages when buffer is full (or in general on failure)?

Ensure that the new API integrates seamlessly with the current send_messages method, providing optional background sending.
Handle all cases with conflicting parameters set in builder, e.g. maximum_buffer_size should not be set when background_send is set to false.
Is is not certain how errors on send should be handled when buffer is not full. Perhaps error channel to communicate errors would be sufficient.

Proposal of enum:

/// Determines how the `send_messages` API should behave when problem is encountered
#[derive(Debug, Clone)]
pub enum IggyBackgroundSendFailureMode {
    /// Block until the send succeeds
    Block,
    /// Block with a timeout, after which the send fails
    BlockWithTimeout(IggyDuration),
    /// Fail immediately without retrying
    FailImmediately,
}

@spetz @numinnex what's your opinion on this?

[numinnex]

We need to make sure to put "sensible" defaults in the newly added fields in order to make the change backward compatible and not mutually exclusive with our existing batching strategy. I am not sure what does the Block and BlockTimeout in the enum mean ? would it retry sending the batch ? but what if the retries are disabled, just some questions.

[hubcio]

Block - it will block the send until internal buffer_size is low enough so messages can be sent, BlockWithTimeout - it will keep trying to send the data for X amount of time, if timeout is exceeded it will return error to user. If retries are disabled and BlockTimeout is set, then builder should fail.

[MasteMind]

We can instead have a functionality called in-flight messages. This instead of Block and BlockWithTimeout, will give clients choice to limit further sending on a combination of pending_request_counts. Now each of these requests will just adhere to their own request-timeouts.

[hubcio]

great idea

[spetz]

I think that background_send flag is not needed - instead, we could add a new method e.g. send_in_background (or something like this), which would push the messages to the background sender task via channel. By doing so, we would let the user to explicitly decide, which method to use for the given scenario.

[hubcio]

not sure if I like this, shouldn't single producer be used one way or another?

[MasteMind]

One question: (ex-Colonel here :P)

Either by flag or method, the underlying producer implementation is same is it?

So now, if I have two methods, one calling default. another calling background_send:

• I would want to configure linger.ms near to 0 or 1-2 ms for default one as my response latencies might depend on it
• but a higher linger number for background send one like 15 or 25ms, as i want better batching there.

How will the underlying single producer take care of it? I will need an option to create multiple producers in this case.

[MasteMind]

background_send is an additional flag in send method or, producer initialization?

[spetz]

It's either having a single method, which works in the specific mode (background sender or not) or having 2 methods, sharing the same config for linger etc. IMHO having the additional configs per sender mode + 2 separate methods could be a bit too complex and confusing.

[hubcio]

How will the underlying single producer take care of it? I will need an option to create multiple producers in this case.
you are free to create as many producers as you like :)

background_send is an additional flag in send method or, producer initialization?
producer initialization

IMHO method send_messages should stay the same, only producer configuration should be changed. @spetz do you still insist on having multiple methods?

[MasteMind]

Why batch_length? batch_size seems intuitive enough.

[hubcio]

sometimes you might want to batch based on real size in bytes, sometimes you might want to batch based on length (numeric)

[MasteMind]

maximum_buffer_size signifies that there is a minimum_buffer_size somewhere too. Do we have plans to allow some range here?

If not, then just buffer_size seems intuitive enough again.

[hubcio]

what about pending_messages_buffer (bytes) and pending_messages_length (length)?

[MasteMind]

linger_time might require me to search for the time unit being followed by this configuration.

linger_ms itself speaks for me to use milliseconds only.

[hubcio]

in iggy, we use IggyDuration which supports constructs like:
IggyDuration::from("1ms").unwrap()
or
IggyDuration::from("100us").unwrap()
so this is much more convinient.

[MasteMind]

Even if background send is set to false, we might still need buffer_size. As there could be a back-pressure situation or some erroneous client implementation.

In these cases, client would need the capability to ensure these producers don't end up taking all the memory available. Buffer_size or buffer_memory should be set always with may be a default value of say 33 MB.

[hubcio]

I will think about it, good point.

[MasteMind]

"how errors on send should be handled when buffer is not full"

Errors for any sync send should be immediately communicated. These could be failures or timeouts.

Errors for any async send should have a callback method that can be triggered. Clients might need to log them, or send them to another dead-letter-queue, or may be do something custom with it.

[hubcio]

good point!

[hubcio]

@MasteMind @spetz @numinnex
Today/tomorrow I will create a github issue out of this discussion.