A List<T> stops being practical when the data must survive a restart or is too large to hold in memory. The usual answer is a hand-written file format, which leaves the application responsible for record offsets, byte order and counts. Sphere10 Framework's stream-mapped collections keep the list, dictionary and set contracts and store the items in a stream instead.

This article creates a persistent list with StreamMappedFactory, appends to it, updates an item in place, and reopens the file to read a range. It also covers which other collections the factory builds, who owns the stream, and what must match when the file is reopened.

1. A first look

StreamMappedFactory.CreateList<T> takes the backing stream, a cluster size and an item serializer, and returns a list that stores its items in that stream.

using (var file = File.Open(path, FileMode.Create, FileAccess.ReadWrite)) {
	using var list = StreamMappedFactory.CreateList<int>(file, clusterSize: 4096, itemSerializer: PrimitiveSerializer<int>.Instance, autoLoad: true);
	list.AddRange(Enumerable.Range(0, count));
	list.Update(updatedIndex, -1);
}

AddRange appends 36,000 integers, and Update replaces one item by position. Reopening uses the same factory call over the existing file.

using var reopenedFile = File.Open(path, FileMode.Open, FileAccess.ReadWrite);
using var reopened = StreamMappedFactory.CreateList<int>(reopenedFile, clusterSize: 4096, itemSerializer: PrimitiveSerializer<int>.Instance, autoLoad: true);
var range = reopened.ReadRange(updatedIndex - 1L, 4).ToArray();

The second list reads its items from the file the first one wrote, and ReadRange deserializes only the requested items.

2. How a stream-mapped list is stored

block-beta
    columns 1
    L["IStreamMappedList: list contract"]
    S["IItemSerializer: item to bytes"]
    O["ObjectStream: item records"]
    C["ClusteredStreams: cluster allocation"]
    F[("File or any seekable stream")]

3. Create the list and write to it

The list that CreateList returned in the first look is an IStreamMappedList<int>, which is an IExtendedList<int>. That contract adds long-indexed and batch members to the familiar list operations.

The arguments to CreateList control the stored layout.

• clusterSize is the allocation unit of the container, in bytes.
• itemSerializer converts an item to and from bytes. PrimitiveSerializer<T> covers primitives, and any IItemSerializer<T> can be supplied for other types.
• endianness is the byte order of the stored data. It defaults to little-endian.
• autoLoad loads existing content during creation. Without it, call Load() when RequiresLoad is true.
• itemChecksummer and optionalItemChecksumIndexName add a checksum index so that IndexOf and Contains avoid reading every item.

4. Reopen the file and read a range

The first look reopened the file with the arguments it was created with, and that is a requirement. The cluster size, serializer and byte order are not negotiated at open time, so supply the values the file was created with. A different serializer would reinterpret the stored bytes.

5. Other stream-mapped collections

The same factory builds the other persistent collections, all over a stream or an existing ClusteredStreams container.

6. Ownership and limits

Disposing the list disposes the container the factory created for it. The backing stream stays with the caller, which is why the sample wraps both the file and the list in using. Dispose the list first.

The container adds headers and per-item records to the file. For fixed-width values that are only ever appended and read by position, a raw binary file is smaller. The stream-mapped list is the better fit when items vary in size, when they are inserted, updated or removed, or when the code should depend on a collection contract rather than a file layout.

7. Get the package and run the sample

Add the package to a .NET 10 project.

dotnet add package Sphere10.Framework

The collections ship in the Sphere10.Framework NuGet package. Download the sample project. It creates the list, appends the items, updates one, reopens the file and checks the range it reads back. Use the .NET 10 SDK and Sphere10 Framework.

dotnet run -c Release -- ./fresh-example-data

Items after reopen: 36000
Range from index 18008: 18008, -1, 18010, 18011

• Stream-mapped collection factories
• Collection selection guide

Download the complete NuGet sample project. The archive contains the project, its model files and run instructions. Keep the pinned package version when reproducing the example.

The Sphere10 Framework overview describes the rest of the toolkit.

Author: Herman Schoenfeld