[DEVEX-222] Add built-in auto-serialization #329

oskardudycz · 2025-01-23T12:35:25Z

Motivation

KurrentDB users should be able to quickly feel efficient in using basic operations like appending new messages, reading them or subscribing to notifications about them. The current API is not fulfilling it as it doesn't provide safe defaults for the most common scenarios like:

building the state from events,
appending new events as an outcome of the business logic,
subscribing to stream etc.

Currently, they need to consider immediately in which order to read the stream, what the deadline is, and what max count to provide. And most importantly, how to serialize data and which serializer to use, as there's none. All of those customisations are needed in the end but shouldn't be thrown immediately on users if they just want to use the default recommended way. In most development environments, you can find the default, mature choices for serialization.

Such code may not be complex; once you get it right, you don't need to change it too often. But if it's stacked with multiple other things you need to keep in mind, then it's easy to miss something. Most importantly, we shouldn't require our users to build their own wrappers around KurrentDB, but we should provide an accessible API to benefit users from using KurrentDB from the get-go.

Solution

The code adds automatic serialization of message payloads. It uses JSON as a default but allows users to implement the binary format themselves. Thanks to that, user can:

append plain messages ,
use a simplified Message wrapper that takes data, and optionally metadata and message-id, this hides the unwanted complexity of EventData,
read messages deserialised automatically. Thanks to that, the user doesn't need to think about which resolved event is correct (event, original event, they can just use Message or DeserializedData fields).
subscribe to messages deserialised automatically.

To introduce those capabilities, it was necessary to introduce new methods not to make breaking changes. Having that, I benefited from it and reshaped their APIs.

Now, they just take mandatory data as parameters (e.g. stream name when appending or reading from stream) and pack the rest of the optional parameters into the options parameter. It also wraps safe defaults. That's aligned with how Java and Node.js clients are doing.

Thanks to that, users can use such simple APIs as:

var streamName = $"users-{Guid.NewGuid()}";

// Appending
var messages = [new UserRegistered(email, name)];
client.AppendToStreamAsync(streamName, messages);

// Appending with options
var messages = [new UserRegistered(email, name)];
await client.AppendToStreamAsync(
    streamName
    messages, 
    new AppendToStreamOptions{ ExpectedStreamState = StreamState.NoStream }
);

// Reading
List<UserRegistered> readMessages = 
    client.ReadStreamAsync(streamName)
    .Select(e => e.DeserializedData).ToListAsync();

// Reading with options
List<UserRegistered> readMessages = 
    client.ReadStreamAsync(streamName, new ReadStreamOptions { MaxCount = 3 })
    .Select(e => e.DeserializedData).ToListAsync();

// Subscribing
await using var subscription = client.SubscribeToStream(streamName);

// Subscribing with options
await using var subscription = client
    .SubscribeToStream(streamName, new SubscribeToStreamOptions{ Start = FromStream.End})

Next steps

After getting approval for changes in the .NET client, that should be applied accordingly to other clients (starting from Java and Node.js).

This change is foundational for enabling scenarios like getting state from events, running business logic, and appending new events as an outcome.

Details

0. No intentional breaking changes were made.

1. This PR introduces the default System.Text.Json serializer and uses it both for JSON and Binary serialization. Users can change the default serialization and also override it through each operation option.

2. Automatic serialization is only enabled for new methods. Thanks to that, people won't get additional overhead upon migration and will be motivated to migrate to the new methods. I didn't make old methods obsolete, but I'd suggest marking them as [Obsolete("Use the new method x. This method may be removed in future releases", false)]. Such registration won't be marked as a warning but as information, so we won't have friction and confusion. This would also give a hint for existing users that they can use new methods.

3. A basic client-side schema registry was introduced. Currently, it's a simple implementation that is responsible for finding the proper serializer based on the content type and mapping between the message type name and CLR type. Default convention is: {categoryName}-{CLR Type Full Name}.

The CLR name will be of course, specific to .NET, but if customers are also using other clients (e.g. Java, Node.js) then they can override the convention by injecting a custom implementation of the IMessageTypeNamingStrategy interface into settings. They can also define different, more universal conventions that are not based on C# or Java types, but then they'll also need to register those types up front.

Users can register message type mapping upfront or benefit from the automated registration. The automated registration will try to load type from assemblies available in AppDomain. It won't load assemblies. No assembly loading code is provided out of the box or registration by attribute. This can be added to the follow-up PR if needed. We could also use Assembly Qualified Name; then, we'd get assembly loading out of the box.

The Mappings are cached in the type registry to improve performance.

The automatic registration was added to allow automatic use of KurrentDB and diverge convention when needed.

4. General serialization settings can be set through client settings, which include:

Changing the default content type to octet stream,
Changing the default System.Text.Json options. That should help to adjust to settings that users are currently using (if they're different from the recommended ones),
Overriding serializer implementations. User can implement the new ISerializer interface for a specific content type. For now, I haven't provided a serializer for Protobuf and Avro, as the PR is already big. This can (and should) be added later.
Changing type name mapping convention,
Register type mappings manually. This can help register types manually upfront or those that are not following the default conventions.
Register CLR message types for the stream category. They will be mapped using the default (or registered) naming conventions.
Registering the custom metadata type. For now default serializer either takes TraceMetadata or same metadata type for all messages. Serialization will take data as it is. The more nuanced default handling can be provided by the user in the custom IMessageTypeNamingStrategy implementation. We can also expand it in the future.

All of that can also be customized for each operation.

5. The central piece of serialization is MessageSerializer class that orchestrates the serialization code. It does that by:

resolving message type using the registered in schema registry mapping strategy,
taking the serializer from the schema registry based on the intended content type,
(de)serializing message data using it,
(de)serializing message metadata as JSON

6. Old methods were made of wrappers for new ones. Thanks to that, the old test suite also covers new methods, and there was no need to repeat it. It'll also be easier to drop them in the future. The integration tests were added, covering additional scenarios.

TODO:

ensuring that the minimum set of types are expected, and the rest are made internally,
detailed unit tests after getting an agreement on the design.

…ate method This should enable easier customisation like changing serializer settings.

They currently use dummy serialisation

Currently it's implemented in a dummy way

…erialization It's not fully ready, as it has hardcoded schema serializer. Main question will be how much from schema registry I need to move here.

@RagingKore

…rom ResolvedEvent Per @RagingKore suggestion it'll be better not to keep the reference to serializer in ResolvedEvent to keep the clear intention behind it. Now, the serializer is resolved from schema registry based on the auto-serialization settings. Made possible to register more than one serializer per content type. Currently it doesn't support more schema definition types. To be discussed how to include them later on.

That will make resolution easier and give users ability to inject their own serializers implementation

…t Subscriptions

…rialization Context Previously we had DeserializationContext, but after consideration, it'll be better to merge those concepts to make easier customizations per operations (e.g. append, subscribe, etc.).

Refactored the code accordingly. It takes the full object instead of the limited number of parameters, as you may be using metadata to get parameters about clrtype.

…solution of CLR types

…tions

…ata header

…g resolved types

…rmation like category name

…pper to make clearer responsibilities Actually, it's just wrapping message serializer based on the client settings.

Removed also obsolete SerializationSettings

… IMessageSerializer There's no need to force all code to know about wrapper existence, it's okay to just create it during the setup.

…ied syntax Now they don't require all the fancy, but redundant in most cases Stream Positions, Directions, EventData etc.

…ad of parameters Thanks to that safe defaults are already provided

…n AppendToStreamOptions

alexeyzimarev · 2025-02-19T12:21:01Z

src/Kurrent.Client/Core/Serialization/SystemTextJsonSerializer.cs

+	readonly JsonSerializerOptions _options = options?.Options ?? SystemTextJsonSerializationSettings.DefaultJsonSerializerOptions;
+
+	public ReadOnlyMemory<byte> Serialize(object value) {
+		return Encoding.UTF8.GetBytes(JsonSerializer.Serialize(value, _options));


Why not use SerializeToUtf8Bytes?

Fixed in e8385b3

alexeyzimarev · 2025-02-19T12:23:23Z

src/Kurrent.Client/Core/Serialization/MessageSerializer.cs

+public interface IMessageSerializer {
+	public EventData Serialize(Message value, MessageSerializationContext context);
+
+#if NET48


This pragma is not needed if the project is compiled with latest SDK

alexeyzimarev · 2025-02-19T12:23:32Z

src/Kurrent.Client/Core/Serialization/MessageSerializer.cs

+		);
+	}
+
+#if NET48


This pragma is not needed if the project is compiled with latest SDK

alexeyzimarev · 2025-02-19T12:23:52Z

src/Kurrent.Client/Core/Serialization/MessageSerializer.cs

+			.TryResolveClrMetadataType(record.EventType, out clrMetadataType);
+}
+
+public class NulloMessageSerializer : IMessageSerializer {


What's Nullo?

Automatic serialization is only enabled for new methods. Thanks to that, people won't get additional overhead upon migration and will be motivated to migrate to the new methods.

To have that I need to pass IMessageSerializer to be able to (de)serialize data, but only for new methods. Having that, I could either make the serializer nullable or provide such nullo implementation that does not serialize data, just returns null. I took such a path, as then when we get rid at some point of the old methods, then we can just remove them without changing all code paths.

Also, someone may want to disable deserialization for specific methods, e.g. if they want to just restream bytes.

See how the serializer is created based on the operations settings:

EventStore-Client-Dotnet/src/Kurrent.Client/Core/Serialization/MessageSerializer.cs

Line 47 in 78d0f22

return NulloMessageSerializer.Instance;

Agreed, nonetheless Nullo is not an ideal name. I suggest Null or Nullable.

Renamed to NullMessageSerializer in f0a86fc

alexeyzimarev · 2025-02-19T12:23:58Z

src/Kurrent.Client/Core/Serialization/MessageSerializer.cs

+		throw new InvalidOperationException("Cannot serialize, automatic deserialization is disabled");
+	}
+
+#if NET48


This pragma is not needed if the project is compiled with latest SDK

I actually have the latest version, and its required indeed.

Yes, based on my findings, NotNullWhen is NetStandard 2.1 per: https://learn.microsoft.com/en-us/dotnet/api/system.diagnostics.codeanalysis.notnullwhenattribute?view=net-9.0#applies-to

and NetStandard 2.1 doesn't support .NET Framework, per: https://learn.microsoft.com/en-us/dotnet/standard/net-standard?tabs=net-standard-2-1.

alexeyzimarev · 2025-02-19T12:31:27Z

src/Kurrent.Client/PersistentSubscriptions/KurrentPersistentSubscriptionsClient.Read.cs

+
+	public class PersistentSubscriptionListener {
+#if NET48
+		/// <summary>


The docs comment should not be inside pragma

Fixed in e8385b3

alexeyzimarev · 2025-02-19T12:34:00Z

src/Kurrent.Client/PersistentSubscriptions/KurrentPersistentSubscriptionsClient.Read.cs

+						try {
+							await foreach (var message in _channel.Reader.ReadAllAsync(_cts.Token)) {
+								if (message is PersistentSubscriptionMessage.SubscriptionConfirmation(var subscriptionId
+								    ))


Can we merge this line to previous one?

Fixed in e8385b3

alexeyzimarev · 2025-02-19T12:35:13Z

src/Kurrent.Client/PersistentSubscriptions/KurrentPersistentSubscriptionsClient.Read.cs

-					_                                                         => null
-				}
-			);
+			public Task Nack(


These are formatting changes, can we avoid those? It seems that the formatting is not being applied correctly.

Yes, I tried to avoid them where possible, but when I applied automated formatting to the files, I changed them, and then those appeared. So, it seems that those files did not match the formatting rules. Should I revert those changes?

alexeyzimarev · 2025-02-19T12:38:16Z

src/Kurrent.Client/Streams/KurrentClient.Append.cs

+		/// <param name="cancellationToken">The optional <see cref="System.Threading.CancellationToken"/>.</param>
+		/// <returns></returns>
+		public Task<IWriteResult> AppendToStreamAsync(
+			string streamName,


Prepping for multi-stream transactions, can we already add a structure that represents an append to a stream? Where stream name, expected version for the whole operation, and new events are combined?

I'd prefer to have that as a follow-up PR, as I'd need to learn more about the multi-stream transactions. My gut feeling tells me that it can be a separate method or overload. Is it fine to move it to a separate discussion? I think that it'll still take some time for the rebranded client to be released.

alexeyzimarev · 2025-02-19T12:38:50Z

src/Kurrent.Client/Streams/KurrentClient.Append.cs

+
+			var eventsData = _messageSerializer.Serialize(messages, serializationContext);
+
+			return options.ExpectedStreamRevision.HasValue


I don't think the expected version belongs to Options tbh.

It already belongs to options in all the other clients.

The intention is to keep optional parameters with safe defaults there and not require to provide them.

Lets keep it in the options, but provide an extension so we dont have to create options just to pass the ExpectedStreamRevision.

I added such extension methods in 95f8546

Fixed also XML docs comments

…ision as a parameter

Used also two external assemlies - one that's loaded, - one that's never loaded. To double-check behaviour of loading types from different assemblies.

This reverts commit 44607ea.

[DEVEX-222] Added configuration options for KurrentClientSettings Cre…

f0e35bd

…ate method This should enable easier customisation like changing serializer settings.

oskardudycz force-pushed the DEVEX-222-AutoSerialization branch 3 times, most recently from 748955e to 64d583d Compare January 23, 2025 13:46

oskardudycz added 2 commits January 23, 2025 14:48

[DEVEX-222] Added overloads of Append methods that take regular events

f8b77d2

They currently use dummy serialisation

[DEVEX-222] Added TryDeserialize method to ResolvedEvent

8e27549

Currently it's implemented in a dummy way

oskardudycz force-pushed the DEVEX-222-AutoSerialization branch 2 times, most recently from 59666ee to d12c429 Compare January 23, 2025 15:15

[DEVEX-222] Added first working, but not yet complete JSON built-in s…

001f1a0

…erialization It's not fully ready, as it has hardcoded schema serializer. Main question will be how much from schema registry I need to move here.

oskardudycz force-pushed the DEVEX-222-AutoSerialization branch from d12c429 to 001f1a0 Compare January 23, 2025 15:24

oskardudycz added 7 commits January 27, 2025 12:33

[DEVEX-222] Made serialization settings to create specific serializers

8ce4f42

That will make resolution easier and give users ability to inject their own serializers implementation

[DEVEX-222] Added override of the serialization settings to Persisten…

cea88dc

…t Subscriptions

[DEVEX-222] Added serialization type and merged serialization into Se…

4992546

…rialization Context Previously we had DeserializationContext, but after consideration, it'll be better to merge those concepts to make easier customizations per operations (e.g. append, subscribe, etc.).

[DEVEX-222] Added message type name resolution strategy

1caba87

Refactored the code accordingly. It takes the full object instead of the limited number of parameters, as you may be using metadata to get parameters about clrtype.

[DEVEX-222] Refactored Event Type Mapper to not be responsible for re…

ef604ff

…solution of CLR types

[DEVEX-222] Added automatic message clr type resolution

64c79e7

oskardudycz force-pushed the DEVEX-222-AutoSerialization branch from 6677a46 to 64c79e7 Compare January 29, 2025 10:00

oskardudycz added 3 commits January 29, 2025 11:17

[DEVEX-222] Added metadata extensions to allow injecting type informa…

6db199d

…tions

[DEVEX-222] Made CLR type be resolved based on the message type metad…

e07b02a

…ata header

[DEVEX-222] Added the MessageTypeResolutionStrategyWrapper for cachin…

786caf3

…g resolved types

oskardudycz force-pushed the DEVEX-222-AutoSerialization branch from 4e88a61 to 786caf3 Compare January 29, 2025 11:04

oskardudycz added 8 commits January 29, 2025 13:39

[DEVEX-222] Added MessageSerializationContext to pass additional info…

6446ea2

…rmation like category name

[DEVEX-222] Refactored SerializationContext into MessageSerializerWra…

1e65b62

…pper to make clearer responsibilities Actually, it's just wrapping message serializer based on the client settings.

[DEVEX-222] Added message type maps registration

b395583

[DEVEX-222] Added helpers for custom type resolutions strategy

85bf401

Removed also obsolete SerializationSettings

[DEVEX-222] Replaced MessageSerializerWrapper usage with more generic…

edde73e

… IMessageSerializer There's no need to force all code to know about wrapper existence, it's okay to just create it during the setup.

[DEVEX-222] Made Message to be record instead of Struct

2bbb89b

[DEVEX-222] Refactored new AppendToStreamAsync method to have simplif…

c9f9305

…ied syntax Now they don't require all the fancy, but redundant in most cases Stream Positions, Directions, EventData etc.

[DEVEX-222] Refactored Reading events signatures to use options inste…

d555cd0

…ad of parameters Thanks to that safe defaults are already provided

oskardudycz added kind/enhancement .NET Pull requests that update .net code labels Feb 13, 2025

oskardudycz requested review from RagingKore and alexeyzimarev February 13, 2025 13:32

[DEVEX-222] Added Expected prefix to StreamState and StreamRevision i…

78d0f22

…n AppendToStreamOptions

oskardudycz marked this pull request as ready for review February 13, 2025 16:52

alexeyzimarev requested changes Feb 19, 2025

View reviewed changes

oskardudycz added 3 commits February 21, 2025 10:51

[DEVEX-222] Merged namespaces

e8385b3

Fixed also XML docs comments

[DEVEX-222] Renamed NulloMessageSerializer to NullMesageSerializer

f0a86fc

[DEVEX-222] Added AppendToStream methods that has expected stream rev…

95f8546

…ision as a parameter

oskardudycz force-pushed the DEVEX-222-AutoSerialization branch 3 times, most recently from 0cdb254 to 869e503 Compare February 25, 2025 13:11

[DEVEX-222] Added XML documentation for Serialization settings

62cebca

oskardudycz force-pushed the DEVEX-222-AutoSerialization branch 7 times, most recently from 8569d78 to 32461a5 Compare February 26, 2025 08:30

[DEVEX-222] Added Unit tests

e938afe

oskardudycz force-pushed the DEVEX-222-AutoSerialization branch from 32461a5 to e938afe Compare February 26, 2025 08:33

oskardudycz added 2 commits February 26, 2025 11:49

[DEVEX-222] Added tests for type resolution

9acba54

Used also two external assemlies - one that's loaded, - one that's never loaded. To double-check behaviour of loading types from different assemblies.

[DEVEX-222] Added SchemaRegistry tests

1af1def

oskardudycz force-pushed the DEVEX-222-AutoSerialization branch from a490f31 to 1af1def Compare February 26, 2025 11:27

w1am merged commit 44607ea into DEVEX-185-Rebranding Feb 27, 2025
45 checks passed

w1am added a commit that referenced this pull request Feb 27, 2025

Revert "[DEVEX-222] Add built-in auto-serialization (#329)"

4b3afeb

This reverts commit 44607ea.

w1am mentioned this pull request Feb 27, 2025

Revert "[DEVEX-222] Add built-in auto-serialization" #332

Merged

w1am added a commit that referenced this pull request Feb 27, 2025

Revert "[DEVEX-222] Add built-in auto-serialization (#329)" (#332)

07a8d87

This reverts commit 44607ea.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DEVEX-222] Add built-in auto-serialization #329

[DEVEX-222] Add built-in auto-serialization #329

oskardudycz commented Jan 23, 2025 •

edited

Loading

alexeyzimarev Feb 19, 2025

oskardudycz Feb 21, 2025

alexeyzimarev Feb 19, 2025

alexeyzimarev Feb 19, 2025

alexeyzimarev Feb 19, 2025

oskardudycz Feb 21, 2025

RagingKore Feb 21, 2025

oskardudycz Feb 21, 2025

alexeyzimarev Feb 19, 2025

RagingKore Feb 21, 2025

oskardudycz Feb 21, 2025

alexeyzimarev Feb 19, 2025

oskardudycz Feb 21, 2025

alexeyzimarev Feb 19, 2025

oskardudycz Feb 21, 2025

alexeyzimarev Feb 19, 2025

oskardudycz Feb 21, 2025

alexeyzimarev Feb 19, 2025

oskardudycz Feb 21, 2025

RagingKore Feb 21, 2025

alexeyzimarev Feb 19, 2025

oskardudycz Feb 21, 2025

RagingKore Feb 21, 2025

oskardudycz Feb 21, 2025


		var eventsData = _messageSerializer.Serialize(messages, serializationContext);

		return options.ExpectedStreamRevision.HasValue

[DEVEX-222] Add built-in auto-serialization #329

[DEVEX-222] Add built-in auto-serialization #329

Conversation

oskardudycz commented Jan 23, 2025 • edited Loading

Motivation

Solution

Next steps

Details

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oskardudycz commented Jan 23, 2025 •

edited

Loading