How to Build a Self-Describing Object Store
An ObjectSpace file normally contains data but not its own schema, so every program that opens it must repeat the exact dimension and index declarations that created it. A viewer, a diagnostic tool or a second application then has to carry a copy of that definition and keep it in step.
Sphere10 Framework can store the definition inside the file instead. This article shows how StoreMetaData() records it, how ObjectSpace.TryOpenFromStream rebuilds a store from nothing but the bytes, and which parts of the configuration remain the caller's responsibility.
1. Create a store with metadata
Metadata is a single builder call. Everything else about the store is declared as usual.
using var raw = new MemoryStream();
var builder = new ObjectSpaceBuilder()
.UseMemoryStream(raw)
.WithEndianness(Endianness.LittleEndian)
.StoreMetaData();
using (var created = SampleSchema.Configure(builder).Build()) {
created.Save(new Customer { Id = 1, Email = "[email protected]", Region = 7, Name = "Ana" });
created.Save(new Purchase { Id = 100, Reference = "PO-0100", CustomerId = 1, Total = 125m });
created.Save(new Purchase { Id = 101, Reference = "PO-0101", CustomerId = 1, Total = 40m });
created.Flush();
}
File.WriteAllBytes(path, raw.ToArray());
SampleSchema.Configure adds two dimensions and their indexes, exactly as it would for a store without metadata. UseMemoryStream(raw) borrows the caller's stream, so raw stays usable after the space is disposed. The explicit Flush() settles the bytes before they are copied to a file.
2. What the metadata records
With metadata enabled, the ObjectSpace writes one record for the whole store and each dimension writes a record for itself. The records are ordinary serialized objects kept in the metadata region that follows the container header.
The magic, the traits and the metadata size are four bytes each and close the 256-byte header. The record begins with its own length, and padding fills any capacity that was reserved beyond it. Each dimension stream inside the clusters repeats this layout with its own header and record.
| Stored | Where | Contents |
| Traits | Header | Merkle, autosave, metadata and byte-order flags |
| ObjectSpace record | Store metadata | The hash function and one serializer factory shared by all dimensions |
| Dimension record | Each dimension's metadata | Item type, each index with its kind and B-tree specification, and change tracking |
A dimension record lists its fields in this order, with one index row for each declared index.
Together these are the same facts an ObjectSpaceBuilder is normally given in code.
StoreMetaData() allocates exactly the space the records need. StoreMetaData(capacity) reserves a fixed number of bytes instead, and a record larger than that budget is rejected when the store is created.
3. Reopen the transferred bytes without a definition
The reader needs only a stream and the byte order.
using var received = new MemoryStream();
using (var file = File.OpenRead(path))
file.CopyTo(received);
received.Position = 0;if (!ObjectSpace.TryOpenFromStream(received, Endianness.LittleEndian, out var reopened))
throw new InvalidOperationException("The self-describing store could not be opened.");
No schema code runs on this path. TryOpenFromStream works in four steps.
-
It reads the header at the stream's current position and checks the marker, the metadata flag and the byte order.
-
It reads the ObjectSpace record and every dimension record.
-
It rebuilds the equivalent builder configuration from those records.
-
It returns the stream to its starting position and loads the store through the normal path.
The result is an ordinary ObjectSpace. Lookups through identifiers and indexes behave as they do on a store opened with a supplied definition.
var customer = reopened.Get((Customer item) => item.Email, "[email protected]");
var purchases = IndexQueries.By(reopened, (Purchase item) => item.CustomerId, customer.Id);
The method borrows the stream, so dispose the returned space before closing or reusing it. Copying the file into an empty MemoryStream, as above, also gives the store room to grow. A MemoryStream constructed directly over a byte array has a fixed capacity.
4. Handle a store that cannot be opened
TryOpenFromStream separates stores it does not recognise from genuine failures.
| Condition | Result |
The store was created without StoreMetaData() |
false |
| The supplied byte order differs from the stored one | false |
| The metadata is missing or invalid | false |
| An I/O error or a deserialization error occurs | The exception propagates |
A store without metadata is still valid. It opens through a builder that supplies its definition, which is the only path it ever had. A metadata store can also be opened with a supplied definition, in which case the definition is checked against the stored records before any object loads.
5. What metadata does not cover
-
Byte order. The caller supplies it. Opening does not detect it.
-
Types. The records name the item types. The reading process must be able to load those types and a compatible Framework version.
-
Custom comparers and delegates. Custom comparer registrations and delegate-based change tracking cannot be recorded, so creating a metadata store with them is rejected.
-
Migration. The records describe the layout that was written. They do not convert old data to a changed class or add dimensions to an existing store.
-
Application versioning. Whether the contents are a version your application supports is a separate question. Keep your own version number in a stored record if you need to answer it.
-
Trust. Metadata describes a file. It does not authenticate who produced it.
6. Get the package and run the sample
Use .NET 10 and the Sphere10.Framework NuGet package.
dotnet new console --framework net10.0 --name ArticleSample
cd ArticleSample
dotnet add package Sphere10.FrameworkThe complete MIT-licensed sample contains the project and every file shown here. Extract the download over the generated project and run it against a new directory.
dotnet run -c Release -- ./fresh-data
The sample creates a store with metadata, reopens it from raw bytes, and exercises the first two false results in the table above.
Opened without a definition: 2 dimensions, 2 purchases for [email protected]
Wrong byte order refused: passed
Store without metadata refused: passedThe next article adds cryptographic integrity to a store. The self-describing-storage specification documents the header fields and the record formats byte by byte.
The Sphere10 Framework overview describes the rest of the toolkit.
Author: Herman Schoenfeld
