File format

One file, replayed in a single pass. Every record is [kind][collection-id][length][body], and the reasons for each exception are worth knowing before you touch the reader.

"FENECDB\x01"
  [6][u64 counter][u64 body length]        change counter (optional)
  [1][collection-id][length][schema]       create collection
  [7][collection-id][length][next-id]      id counter (optional)
  [5][collection-id][length][schema]       schema change (index)
  [2][collection-id][length]               drop collection (empty body)
  [3][collection-id][length][records]      data
  [4][collection-id][length][field][graph] HNSW graph (optional)

Records inside a data block are [op][doc-id][length][fields]. A half-written final record is truncated on open, so there is no separate recovery step after a crash.

The length is written even for an empty body — a drop record has one — and the reader must consume it. Otherwise the leftover byte is read as the next record kind and the file stops opening.

The graph record

It is derived data, not a cache. On open, the version, the dimension, the node count and the link bounds are validated; if any of them does not hold, the record is ignored and the index is rebuilt from scratch. A corrupt or stale graph therefore cannot cause data loss.

It is written only during snapshot, compact and checkpoint — never on the normal write path. What it stores is the links, not the vectors: the vectors are already in the document records, and the expensive part is the build.

The change counter, and why it is at the front

Record kind 6 sits at the front of the file and is fixed width. Both are deliberate.

At the front, because at the end a corrupt or half-written tail would break opening. The end of the file is exactly where the graph stops, and the graph is derived data — a record that errors out in that region would take that tolerance back.

Fixed width, because the body length is known only after the body is written. A placeholder can be filled in place, whereas a uvarint would mean shifting the whole image.

The body length means "this is where this image's own records end". Every record after it was written after the checkpoint and carries the counter forward. Old files without the record load fine — the counter is then counted from the records. The reverse does not hold: an old binary cannot open a new file and reports unknown record kind 6.

The id counter exists because of compact

A document id is normally derived from the records: a replay takes one more than the largest id it saw. Compaction throws tombstones away, so that derivation falls short — the highest deleted id disappears from the image entirely and would be handed out again on the next open.

An id coming back silently binds everything still holding it — a link handed out, a row a subscriber holds — to the wrong document. The record is per collection and comes right after the schema. Only snapshot writes it; the append path does not need it, because the tombstones are still there. Old files without the counter still load correctly.

Crates and layout

crates/
  fenec-core/   storage, HNSW, plan executor, plugin registry, JSON,
                calendar arithmetic, change stream          (no dependencies)
  fenec-ql/     FenecQL lexer and parser                    (core only)
  fenec-wasm/   the browser ABI                             (no wasm-bindgen)
  fenec-pg/     PostgreSQL v3 wire protocol, server and client
  fenec-http/   HTTP/JSON endpoint: REST, raw FenecQL, SSE  (core + ql only)
  fenec-import/ SQLite file reader, PostgreSQL COPY source
  fenec-cli/    the `fenec` shell, `fenec import`, `fenec types`
web/            fenec.js (builder + sync layer), fenec.d.ts,
                fenec.test.js and fenec.sync.test.js, the browser console

Maintenance operations

Operation	What it does	Peak memory
`snapshot`	Writes a fresh image including the graph and the id counter	≈ 3× the file
`checkpoint`	Lands the graph in the file so the next open does not rebuild it	≈ 3× the file
`compact`	Full rebuild: drops dead bytes and rebuilds every index	≈ 3× the file
open	Reads the file, then copies it into segments	≈ 2× the file

compact is not a garbage collection. Even with zero dead bytes every index is built from scratch, and writes block throughout: 20 000 × 32 takes 0.94 s, against 0.05 s on a vector-free text collection of the same size.

The rewrite goes to a side file and is renamed, so a half-written checkpoint cannot corrupt the database.