How to Store Serializer Definitions Alongside Your Data

Binary data is compact because it carries no description of itself. The reader has to construct serializers with the same type codes and member layout as the writer, and that configuration normally lives only in code. When the writer's registrations change, or the reader is a different program, older data can no longer be decoded reliably.

Sphere10 Framework can serialize the configuration itself. SerializerFactorySerializer writes a SerializerFactory to bytes and rebuilds an equivalent factory from them. This article shows how to save a factory next to the data it encoded, how to read the data back through the restored factory, and which registrations can and cannot be reconstructed.

A data container retaining serializer definitions beside its records.
A data container retaining serializer definitions beside its records.
💡 In short. Serialize a SerializerFactory with SerializerFactorySerializer so that stored binary data carries its own type codes and member layouts, and rebuild the reader factory from those bytes.

1. What a serializer factory holds

A SerializerFactory maps CLR types to serializers, and gives each registration a numeric type code. SerializerFactory.Default holds the built-in registrations for primitives, strings, collections and similar types, and is read-only. An application copies it and adds its own types.

C# 
private static SerializerFactory CreateFactory()
	=> new SerializerFactory(SerializerFactory.Default)
		.SetMinTypeCode(2100)
		.RegisterAutoBuild<StoredItem>();

RegisterAutoBuild<T>() assembles a composite serializer from the type's public read-write members. SetMinTypeCode sets the lowest code the factory generates for registrations that do not name one, which keeps application codes apart from the built-in range. GetSerializer<T>() then returns the serializer for a registered type.

C# 
var writerFactory = CreateFactory();
var payload = writerFactory.GetSerializer<StoredItem>().SerializeToBytes(item, Endianness.LittleEndian);
File.WriteAllBytes(payloadPath, payload);

A reader that calls CreateFactory() itself can decode this payload. The data then depends on that method staying unchanged in every program that reads it.

2. Save definitions alongside the export

SerializerFactorySerializer is an ordinary item serializer whose item type is SerializerFactory. It is also registered in SerializerFactory.Default.

C# 
var definitions = new SerializerFactorySerializer();
File.WriteAllBytes(definitionsPath, definitions.SerializeToBytes(writerFactory, Endianness.LittleEndian));
C# 
var restoredFactory = definitions.DeserializeBytes(File.ReadAllBytes(definitionsPath), Endianness.LittleEndian);
var decoded = restoredFactory.GetSerializer<StoredItem>().DeserializeBytes(File.ReadAllBytes(payloadPath), Endianness.LittleEndian);

The reader in the second block never calls CreateFactory(). It rebuilds the factory from the saved definitions and asks that factory for the serializer.

The byte order is the caller's choice, as with every Framework serializer. Use the same value for the definitions and for the payload, and keep it as part of the format.

3. What the definitions contain

The saved record holds the minimum generated type code, the registration count and one record for each registration. A registration record stores its type code, the CLR data type and serializer type, how the serializer is constructed and, for a composite, its ordered members.

The first row is the factory record's own two fields. The next three rows are one registration record, and the diagram shows the contents of the record and not the width of its fields.

Registration What is preserved
Built-ins copied from SerializerFactory.Default Their original type codes and standard construction.
Register<TItem, TSerializer>() Construction through the public parameterless constructor.
Register(dataType, serializerType) The factory's ordinary constructor injection.
Open generic registrations The generic serializer hierarchy and its dependent serializers.
RegisterEnum The enum, its nullable form and their codes.
RegisterAutoBuild composites Member order, declaring type, field or property identity, polymorphic wrappers and reference modes.

The record includes the built-in registrations as well as the application's, which is why the sample's definition file is about 44 KB for a 22-byte payload. Definitions are written once for a whole data set, not once for each record.

4. What cannot be reconstructed

⚠ī¸ No executable code is stored. The following registration forms are rejected when the factory is serialized, before any bytes are written.
  • Registrations made with a delegate or a captured closure.
  • A serializer instance that was configured independently and then registered.
  • Composites with a custom activator.
  • A registration owned by another factory.

The reader also needs the registered CLR types and serializer types to be loadable. The definitions describe how those types were serialized. They do not carry the types themselves, migrate a class whose members have changed, or authenticate the file.

ℹī¸ The sample writes two files to keep the two parts visible. Two file writes are not atomic, so a real format should place both in one container and record its version and byte order. A self-describing ObjectSpace does this for you by storing its serializer factory in the store's metadata.

5. Get the package and run the sample

Add the package to a .NET 10 project.

Bash 
dotnet add package Sphere10.Framework

SerializerFactorySerializer ships in the Sphere10.Framework NuGet package. Download the sample project. It encodes an item, saves the factory definitions, rebuilds the factory from them and checks that the restored factory decodes the same item. Use the .NET 10 SDK and Sphere10 Framework.

Bash 
dotnet run -c Release -- ./fresh-example-data
Plain text 
Decoded through restored factory: 2048, sample item
Definition bytes: 44202
Payload bytes: 22

Download the complete NuGet sample project. The archive contains the project, its model files and run instructions. Keep the pinned package version when reproducing the example.

sample.zip
Download the complete worked example and build instructions (ZIP).

The Sphere10 Framework overview describes the rest of the toolkit.

📚 Continue the series

Author: Herman Schoenfeld