Backups & cold tier

Two related mechanisms use collection snapshots and an object store: periodic backups (durability) and the cold tier (evict idle collections from RAM, restore lazily on access). Backups work in both single-node and -cluster mode — in a cluster each node backs up the shards it owns, plus the cluster catalog. The cold tier is single-node only (-cold-tier-after is rejected in cluster mode: vectors are partitioned across shards).

Filesystem backups

rostam-server -http 127.0.0.1:8080 -data ./data \
  -backup-dir /var/backups/rostam \
  -backup-interval 30m \
  -backup-retention 24

Every -backup-interval, each collection is snapshotted into -backup-dir (keyed under -backup-prefix, default default); only the newest -backup-retention snapshots per collection are kept (default 24).

S3-compatible backups

Point the same machinery at S3, MinIO, Cloudflare R2, or any SigV4-compatible store — the client is stdlib-only, no AWS SDK:

rostam-server ... \
  -backup-bucket rostam-backups \
  -backup-region us-east-1 \
  [-backup-endpoint https://minio.internal:9000] \
  [-backup-tenant default] \
  [-s3-path-style=true]        # default true; false for AWS virtual-host style

Credentials come from the standard AWS environment variables. Object layout:

<tenant>/<escaped-collection-name>/<RFC3339-timestamp>.snap
<tenant>/<escaped-collection-name>/<RFC3339-timestamp>.cfg.json

On-demand backup & restore

With an object store configured (these return 412 Precondition Failed otherwise):

POST /v1/admin/backup                       # snapshot all collections now
GET  /v1/admin/backups                      # list available snapshots
POST /v1/collections/{name}/restore         # restore a collection

Cluster backups & restore

The same flags in -cluster mode take periodic point-in-time backups of each node's owned shards (cache + vectors, via the shard FSM snapshot) and the MetaRaft catalog. Restoring is a one-shot disaster-recovery flag:

rostam-server -cluster ... -restore -backup-dir /var/backups/rostam

Run once on every node of a fresh cluster: after bring-up, each node restores the shards it owns and the catalog from -backup-dir (or -backup-bucket), then continues serving. The cluster must have the same topology (shard count and node IDs) as the backup — a mismatch fails loud. A shard with no backup artifact also fails loud by default, so an incomplete backup cannot silently lose a shard's keys; -allow-missing-shards is the explicit operator override that brings such shards up empty (logged per shard).

Cold tier

For long-tail workloads (many collections, few hot), the cold tier evicts collections idle longer than a threshold to the object store, leaving a stub that restores lazily on the next access:

rostam-server ... -backup-bucket rostam-cold -backup-region us-east-1 \
  -cold-tier-after 1h

Manual control:

POST /v1/collections/{name}/evict           # evict to object store now
POST /v1/collections/{name}/restore         # bring it back

The first request against an evicted collection pays the restore latency; subsequent requests are served from RAM as usual. Library-level APIs (EvictCollection, SweepCold) are described in Persistence.