Import

One command turns an existing database into a fenecdb collection: it derives the schema, recognises vector columns and follows the bulk-load-then-index order by itself.

fenec import data.sqlite --table docs --into articles \
    --vector embed:384 --index "embed@hnsw(cosine)"

fenec import postgres://user@host:5432/database --table docs \
    --into articles --index "embed@hnsw(cosine)"

Look before you write

--dry-run prints the derived schema and any warnings, and writes nothing.

$ fenec import data.sqlite --table docs --into articles --vector embed:3 --dry-run
source   data.sqlite -> docs
target   articles.fenec -> articles

create collection articles (
  title     text           <- text
  category  text           <- text
  score     int            <- int
  published timestamp      <- datetime
  embed     vector<3>      <- blob
)
  id  <- id (INTEGER PRIMARY KEY)
create index on articles (embed) @hnsw(cosine, m=16, ef_construction=200, ef_search=100)

--count also prints how many rows will be read. It is a separate flag because counting means a full scan: SQLite keeps the row count nowhere and count(*) in PostgreSQL reads the table from the start. Silently reading everything twice would be a bad default.

Sources

SQLite

The file format is read directly — neither the sqlite3 library nor the binary is needed. INTEGER PRIMARY KEY becomes the document id, --vector field:N turns little-endian f32 BLOBs into vector<N>, and untyped columns are resolved from the first 1000 rows.

PostgreSQL

It connects to the live server and streams COPY ... TO STDOUT — no psql, no pg_dump. Authentication is SCRAM-SHA-256, using fenec-pg's own crypto in the client direction. pgvector's vector and halfvec columns are recognised along with their dimensions.

Type mapping

Source	fenecdb
`integer` `bigint` `int2/4/8`	`int`
`real` `double` `float4/8`	`float`
`text` `varchar` `char` `clob`	`text`
`blob` `bytea`	`bytes`
`boolean`	`bool`
`date` `datetime` `timestamp(tz)`	`timestamp`
`int[]` `float[]` `text[]`	`[int]` `[float]` `[text]`
`vector(N)` `halfvec(N)`	`vector<N>` `vector<N, f16>`
`uuid`	`text`
`numeric` `decimal`	no equivalent — `--cast` required
`json` `jsonb`	no equivalent — `--cast` required

Types without an equivalent are not silently rounded: say what you want with --cast <field>=<type>. On PostgreSQL, --cast price=text preserves a decimal exactly. In SQLite a decimal is already stored as an int or a float at the source, so the exact value is gone before the import starts.

Filtering

--where is a FenecQL expression and behaves identically on both sources: it is applied after the row becomes a document and before it is written. The field names are the target schema's — the same expression you would later write with get.

fenec import data.sqlite --table docs --into articles \
    --where 'category = "book" and score >= 10 and tags has "rust"'

On PostgreSQL there is also --source-where, which goes into the SELECT inside the COPY so non-matching rows never travel over the wire. That one is SQL, not FenecQL. They are separate flags on purpose: one is a portable expression, the other a win pushed down to the source. Both can be used together.

fenec import postgres://... --table docs --into articles \
    --source-where "year >= 2024" --where 'tags has "rust"'

With a local --where in place, --limit is not pushed to the source: cutting on the server first and filtering locally afterwards would yield fewer rows than asked for. --limit always counts rows that pass the filter.

Speed and testing

100 000 rows × 3 dimensions, index build included: about 3.1 s from either source. Every option is in fenec import --help.

make import-test        # the PostgreSQL arm — needs Docker
cargo test -p fenec-import --test pg -- --ignored

Import is a default feature of the fenec binary and accounts for 146 KB of it. --no-default-features removes it; with the cli profile as well, the binary drops from 863 KB to 636 KB (make small).