A paged view over an in-memory dataset has to filter and sort the whole dataset before it can cut a page. Repeating that load, filter and sort for every Next click is wasted work when neither the data nor the query has changed.

TCustomDataSource<T> in Sphere 10 Pascal Framework implements that query path once, and can keep both the loaded dataset and the filtered, sorted view between page requests. This article shows the three methods a data source overrides, how a caller describes a filter and a sort, what the snapshot cache retains, and how to invalidate it.

1. Describe the query to the data source

A caller asks the data source for a page by describing a query. A query is an array of TColumnFilter records. Each one names a column and can carry a filter, a sort direction or both. The sample asks for records whose Value field is below 25, in ascending order of that same field.

    Source.SnapshotCacheEnabled := True;
    Provider := Source;
    SetLength(Filters, 1);
    Filters[0] := Default(TColumnFilter);
    Filters[0].ColumnName := 'Value';
    Filters[0].Filter := vgfNumericLT;
    SetLength(Filters[0].Values, 1);
    Filters[0].Values[0] := 25;
    Filters[0].Sort := sdAscending;

The first statement switches on the snapshot cache, which a later section covers. The page request combines the filters with a page index, a page size and the rule that joins several filters, which is foAnd or foOr.

      Params := TPageFetchParams.Create(PageIndex, 10, Filters, foAnd);
      Page := Provider.FetchPage(Params, Table);

Table.Rows holds the page, and each row exposes its fields by name, as in Table.Rows[I].Value. The returned TPageFetchResult reports PageIndex, PageCount and TotalDataCount for the whole filtered result. A page index beyond the last page is clipped to the last page, and a page size of zero or less returns everything as one page.

2. How a page is produced

---
config:
  sequence:
    actorMargin: 8
    width: 96
    mirrorActors: false
    messageMargin: 28
    noteMargin: 8
---
sequenceDiagram
    participant C as Consumer
    participant D as Base class
    participant S as Your overrides
    C->>D: FetchPage, page 0
    D->>S: FetchAll
    Note over D: filter, sort, keep view
    D-->>C: TDataTable
    C->>D: FetchPage, page 1
    Note over D: slice the kept view
    D-->>C: TDataTable
    C->>D: InvalidateSnapshot
    C->>D: FetchPage
    D->>S: FetchAll

3. Implement the data source

classDiagram
    direction TB
    class IDataSource {
      <<interface>>
      +FetchPage(params, table)
    }
    class TCustomDataSource~T~ {
      +SnapshotCacheEnabled
      +InvalidateSnapshot()
      #FetchAll(container)*
    }
    class TItemSource {
      #GetColumns()
      #FetchAll(container)
      #GetItemField(item, name)
    }
    IDataSource <|.. TCustomDataSource~T~
    TCustomDataSource~T~ <|-- TItemSource

  TItemSource = class(TCustomDataSource<TItem>)
  public
    FetchCount: Integer;
    ValueOffset: Integer;
  protected
    function GetColumns: TDataColumns; override;
    procedure FetchAll(const AContainer: TList<TItem>); override;
    function GetItemField({$IFDEF FPC}constref{$ELSE}const{$ENDIF} AItem: TItem; const ABindingName: String): Variant; override;
  end;

function TItemSource.GetColumns: TDataColumns;
begin
  Result := TDataColumns.Create(TDataColumn.From('Id', True), TDataColumn.From('Name'), TDataColumn.From('Value'));
end;

Because every comparison goes through GetItemField, the base class can filter and sort any record type without knowing its layout.

4. What the snapshot cache retains

Setting SnapshotCacheEnabled to True makes the data source keep two things, which are the raw dataset returned by FetchAll and one filtered and sorted view of it. Each later request is compared with the query that built the view.

The sample counts calls to FetchAll. It reads three pages of ten, ten and four rows from a single fetch, and then runs a different filter without fetching again.

5. Invalidate the snapshot when the data changes

The cache cannot know when the underlying data has changed, so the application tells it. InvalidateSnapshot marks both retained results as out of date, and the next FetchPage loads them again.

    Source.ValueOffset := 1;
    Source.InvalidateSnapshot;
    Page := Provider.FetchPage(Params, Table);

InvalidateSnapshot only records that a rebuild is due. It does not wait for a fetch that is running on another thread, which may finish with the older snapshot. Changing SnapshotCacheEnabled invalidates in the same way.

6. Limits

The data-source unit contains TCustomDataSource<T> and the filter types. The adaptive grid article binds the same kind of data source to TVisualGrid.

7. Get the Framework and run the example

Get Sphere 10 Pascal Framework and follow its package instructions . Use the units directly, or build the optional Lazarus or Delphi runtime package.

The sample was compiled and run on Windows with Free Pascal 3.2.0, against source revision 2cc6558 . Its Build.ps1 accepts the Framework, compiler, Lazarus and output paths. The program checks the order and size of each page and prints the number of FetchAll calls.

Filtered rows=24; pages=10,10,4; FetchAll calls=1
New filter: rows=49; FetchAll calls=1
After InvalidateSnapshot: rows=48; FetchAll calls=2

Download the complete runnable sample.

For the broader Sphere 10 development toolkit, see the Sphere10 Framework overview .

Author: Herman Schoenfeld