Production readiness¶

Rostam is self-hosted — you own running it. This is a pre-flight checklist for putting it in front of production traffic: the operational subset that matters most, in one place. Nothing here is new mechanism; each item links to the page with the full detail. Walk it once before go-live, and again when you change topology.

Security & access¶

[ ] Authentication is on. Never run open on a network-reachable bind — the server refuses to start on a non-loopback address without auth unless you pass -insecure. Prefer a -keys-file registry (per-key RBAC) over a single -api-key, and pass the secret via ROSTAM_API_KEY (env), not a flag, so it isn't visible in /proc. Authentication modes
[ ] Keys are least-privilege. Scope each key read:/write:/admin: with a collection pattern; reserve *:* superuser for setup, not for services. RBAC keys and scopes
[ ] Multi-tenant isolation is enforced (if applicable). Bind keys to a tenant and run -tenant-isolation; the static -api-key/ROSTAM_API_KEY is a superuser that sees every tenant. Tenant isolation
[ ] TLS on client listeners. -tls-cert/-tls-key cover HTTP, gRPC, and TCP (≥ 1.2); misconfiguration fails at startup rather than falling back to plaintext. For mTLS, add -tls-ca — and to make a client certificate the required identity (not merely an accepted one: -tls-ca alone still lets a certless client fall back to token auth), also set -tls-require-client-cert. TLS
[ ] Inter-node auth set (cluster). -internal-token (prefer ROSTAM_INTERNAL_TOKEN) must be the same on every node — an authenticated cluster cannot function without it. Consider inter-node TLS (-tls-node-cert, -node-cn-allowlist). Inter-node auth
[ ] Audit trail shipped. -audit-log emits a JSON record per authorization decision to stderr — only under -keys-file RBAC (it is a no-op with a single -api-key, so don't tick this box while running the static key); route stderr to your log pipeline. Audit log

Durability & replication¶

[ ] Durability rung chosen deliberately. The default (fsync every Raft log write) is the strongest; -nosync and -volatile-log step down for throughput. Pick by what you can afford to lose, not by the benchmark number. The durability ladder
[ ] -volatile-log nodes rejoin fresh. A crashed rung-3 node must rejoin as a fresh member and catch up from a snapshot — never resume in place (a correctness requirement). Rungs 1–2 tolerate in-place restart. Durability ladder
[ ] Replication factor ≥ 2 for HA. With a majority intact, a follower failure is invisible; a leader election briefly fails that shard's writes with retryable errors. Failure behavior
[ ] Replication mode chosen deliberately. raft (default) is the safe, proven choice. -replication-mode=pb (primary-backup / ISR) is also supported for production — its correctness hazards are closed. The no-acked-loss-across- failover guarantee holds only under all of: -min-isr ≥ 2, the default full-ISR commit (-pb-commit-primary=false), and a bounded cross-node clock rate. It is recommended at RF=2, where it beat raft RF=3 (both -nosync -volatile-log) ~1.7× on throughput (and p50 latency) in the 2026-07 real-network gate — a cross-RF comparison (PB RF=2 holds one fewer copy), whose raft baseline also ran under the old epoll default, so re-verify the margin on your hardware; at RF=3 its full-ISR commit is slower than raft's majority. If you run PB: set -min-isr ≥ 2 (keeps every acked write on ≥2 nodes — =1 can lose acked writes across failover — and requires at least that many replicas; the default 0 is rejected in pb mode), set PBAutoFailover: true if you construct cluster.Config or rostam.EmbeddedConfig directly (the -pb-auto-failover server flag defaults on, the Go struct field does not — otherwise a failed primary stays DOWN), and leave -pb-commit-primary at its default (setting it is a durability downgrade). Note PB is nosync (no per-shard WAL/fsync): durability is "acked on ≥ min-ISR nodes in memory," so the guarantee covers losing individual nodes, not the simultaneous loss of every in-sync node for a shard. It's newer than raft — validate on your workload. Replication engine
[ ] Shard count has headroom. -shards is fixed for the life of the cluster; choose shards ≫ nodes if you expect to grow (membership/RF changes redistribute the fixed shards, they don't add more). Reconfigure

Backups & disaster recovery¶

[ ] Periodic backups configured. -backup-dir (filesystem) or -backup-bucket (S3/MinIO/R2, stdlib SigV4 — no AWS SDK), with -backup-interval and -backup-retention. Backups
[ ] Restore rehearsed, not assumed. Cluster restore requires the same topology (shard count + node IDs) as the backup, and a shard with no artifact fails loud unless you pass -allow-missing-shards. Practise a restore before you need one. Cluster backups & restore
[ ] Backup destination is in-boundary if data residency matters — point the S3 client at an in-network store (e.g. MinIO). Rostam initiates no egress otherwise. S3-compatible backups

Observability & alerting¶

[ ] Metrics scraped. Prometheus against /metrics with a read-scoped token (the endpoint is scope-gated when auth is on). Monitoring
[ ] Probes wired to the right question. /v1/health is liveness (a restart decision — it stays green without quorum, so it's wrong for routing); /v1/ready is readiness — use it for load-balancer membership and as the Kubernetes readinessProbe. Liveness and readiness
[ ] Alerts on the signals that matter: quota/rate-limit rejections rising, the search-latency histogram shifting right, degraded/missing fields in search responses (partitions unreachable), and 503s on writes (transient elections). Operational signals
[ ] Memory alerts on RSS, not VIRT. Vector collections reserve address space far beyond what they use, so a virtual-size threshold fires spuriously. Alert on resident memory

Capacity & sizing¶

[ ] Per-collection quotas set. MaxVectors / MaxBytes / MaxInsertsPerSecond so a runaway workload fails cleanly (ErrCollectionFull / ErrCollectionRateLimited) instead of exhausting the host — and MaxVectors also sizes the memory reservation. Collection limits
[ ] KV at-capacity policy chosen. PolicyRingbufEvict (bounded cache, overwrites oldest) vs PolicyRejectWrites (returns cache.ErrFull) — pick per workload. Cache configuration · Sizing guidance
[ ] Persistence on disk you control. -data (and -persistent-vectors to mmap-back vectors off-heap); review RAM-vs-disk sizing before load. Persistence & warm restart

Rollout & day-2 operations¶

[ ] Rolling restarts respect the durability rung — rungs 1–2 tolerate an in-place restart of a node; rung 3 (-volatile-log) does not (see above).
[ ] Membership / RF changes run online. -reconfigure with the desired end-state -peers; size the context deadline to your data volume. A departing leader can cause a transient retryable write error mid-rebalance. Reconfigure
[ ] Partition-count changes use online reshard (vector collections): a dual-write + atomic cutover that is resumable. Resharding online
[ ] Clients retry transient errors. The Go client retries not-leader/503s automatically; ensure any other client retries with backoff. Failure behavior

Before you publish an incident channel¶

[ ] The team knows the private disclosure path for security issues — security@rostamlabs.com or the advisory form, never a public issue. Reporting vulnerabilities