A binary store is only readable by code that knows how it was written. For an ObjectSpace that knowledge is the serializer layout of every type, the order of the dimensions and the specification of every index. Normally the application supplies all of it again each time it opens the file, and a tool that lacks the original definition cannot read the data.

Self-describing storage keeps that construction metadata inside the store. Sphere10 Framework can then rebuild the configuration from the bytes and open an ObjectSpace or ObjectStream without a supplied definition. This article explains what is recorded, where it lives, how reopening works and what the metadata does not cover.

1. A first look

One call on the builder, StoreMetaData(), turns the option on when a store is created.

using var raw = new MemoryStream();
var builder = new ObjectSpaceBuilder()
	.UseMemoryStream(raw)
	.WithEndianness(Endianness.LittleEndian)
	.StoreMetaData();
using (var created = SampleSchema.Configure(builder).Build()) {
	created.Save(new Customer { Id = 1, Email = "[email protected]", Region = 7, Name = "Ana" });
	created.Save(new Purchase { Id = 100, Reference = "PO-0100", CustomerId = 1, Total = 125m });
	created.Save(new Purchase { Id = 101, Reference = "PO-0101", CustomerId = 1, Total = 40m });
	created.Flush();
}

SampleSchema.Configure adds the sample's two dimensions and their indexes in the usual way. Reopening the bytes later takes a stream, the byte order and an out parameter.

received.Position = 0;
if (!ObjectSpace.TryOpenFromStream(received, Endianness.LittleEndian, out var reopened))
	throw new InvalidOperationException("The self-describing store could not be opened.");

The second call was given no builder and no dimension list, because the store carries its own definition. The sections that follow explain what is recorded, where it lives and how reopening works.

2. Supplied definition versus stored definition

There are two ways to open an existing store.

• Supplied definition. The store is created with an ordinary builder and reopened with the same builder configuration. The schema knowledge lives in application code. This suits one application that owns its files.
• Self-describing. The store is created with a builder that calls StoreMetaData() and reopened with ObjectSpace.TryOpenFromStream. The schema knowledge lives in the store, beside the data. This suits inspection tools, transfer between programs and long-lived files.

The supplied-definition path is the default and is what the embedded object database tutorial uses. It is a sound design when one application owns both the code and the data. Self-describing storage is an opt-in for cases where the reader should not need the writer's builder code.

3. Store the definition beside the records

Apart from the StoreMetaData() call shown in the first look, nothing about creating or using the store changes.

Two kinds of record are written. The ObjectSpace record is stored once and holds what all dimensions share. Each dimension's ObjectStream then stores its own record.

Because the serializer factory itself is saved, reopening does not inspect today's class members and guess the old layout. It restores the layout that wrote the data.

4. Where the metadata lives

---
config:
  packet:
    bitsPerRow: 16
    bitWidth: 22
    showBits: false
---
packet-beta
0-11: "Header fields, bytes 0-251"
12-15: "Capacity C"
16-19: "Length U"
20-27: "Rest of metadata record"
28-31: "Padding"
32-47: "Clusters, from byte 256 + C"

StoreMetaData() allocates exactly the size of the record. StoreMetaData(capacity) reserves a larger budget in bytes, and a negative value is rejected. A store created without the option keeps the plain layout, with clusters starting at byte 256.

5. Open a store without its definition

ObjectSpace.TryOpenFromStream, the call that reopened the store in the first look, needs no builder and no dimension list. The space it returns answers lookups through the stored indexes.

var customer = reopened.Get((Customer item) => item.Email, "[email protected]");
var purchases = IndexQueries.By(reopened, (Purchase item) => item.CustomerId, customer.Id);

---
config:
  sequence:
    actorMargin: 8
    width: 96
    mirrorActors: false
    messageMargin: 28
    noteMargin: 8
---
sequenceDiagram
    participant C as Caller
    participant O as ObjectSpace
    participant S as Stream
    C->>O: TryOpenFromStream
    O->>S: read header
    O->>S: read space record
    O->>S: read dimensions
    Note over O: rebuild the builder
    O->>S: seek to start
    O->>S: load as normal
    O-->>C: true, objectSpace

6. Capacity does not imply migration

7. Get the package and run the sample

Add the package to a .NET 10 project.

dotnet add package Sphere10.Framework

The complete MIT-licensed sample creates a self-describing store, writes it to a file, reopens the bytes without a definition and reads records through their indexes. It also confirms that the wrong byte order and a store without metadata are both refused. It targets .NET 10 and the Sphere10.Framework package.

dotnet run -c Release -- ./fresh-data

Opened without a definition: 2 dimensions, 2 purchases for [email protected]
Wrong byte order refused: passed
Store without metadata refused: passed

The metadata format reference gives the exact byte layout, and the self-describing store tutorial builds a store step by step.

For the broader Sphere 10 development toolkit, see the Sphere10 Framework overview .

Author: Herman Schoenfeld