Changelog

Notable user-visible changes. Entries that alter existing behaviour are marked Breaking and say what to do about it.

Unreleased

v0.7.0 — 2026-09-27

• go install …/cmd/rostam-server@v0.7.0 and go get github.com/rostamlabs/rostam work outside the repo. Earlier tags could not be installed with the Go toolchain: the root go.mod carried replace directives, which go install rejects, and required an sdk version that predated packages the engine imports. The root module now requires tagged sdk/v0.2.0 and client/v0.2.0 and has no replace directives.

• The container image carries the official MCP Registry label for io.github.rostamlabs/rostam, and the repo ships the matching server.json, so the image can be published to the registry. docker run -i --rm ghcr.io/rostamlabs/rostam mcp is the stdio server.

• The TCP server answers a pipelined batch with one write instead of one per frame. A client that sends several requests in a single write now gets its replies back in a single write, which is a large share of the server's CPU at the point where the cache itself is cheap. The wire is byte for byte what it was and replies keep request order, so a client cannot tell except by counting reads. Nothing changes for a client that sends one request at a time.

• PipelineDepth now applies to CallFunc, not just Call. Setting it previously did nothing at all for a CallFunc caller: the client opened the usual pooled connections, kept one request in flight on each, and reported no error, so the knob looked enabled and was not. If you set PipelineDepth and measured no change, this is why, and you will see a change now. Note what comes with it: pipelined requests are concurrently in flight, so two writes to the same key on one connection may apply in either order. On the pipelined path the payload CallFunc hands to fn is that response's own allocation rather than a slice of the connection's read buffer; the contract is unchanged (valid only for the duration of fn), but the reason differs.

• Concurrent pipelined calls now share a write. A pipelined connection flushed after every frame, so pipelining removed the wait for a response but not the syscall per request. Callers whose writes overlap now leave in one write. The saving needs concurrency ON ONE connection: spreading the same rate over more pipelined connections drops each below one concurrent caller and there is nothing left to combine, so PipelineConns wants to be small.

• A persistent cache now fails closed if it cannot make a reused or recovered page's cleared header durable. When the cache hands a page back to the write path (after emptying, discarding a corrupt page, or reclaiming dead space) it first makes the page's cleared header durable on disk; if that step fails, a write that would require the unsafe page reuse now returns an error instead of proceeding. Likewise, at startup a shard whose recovered page bounds cannot be made durable now refuses to open rather than serving them. Previously these cases logged a durability warning and continued, which could let a stale page be resurrected after a later crash. The failure is transient — retry the write, or restart the shard once the underlying disk fault clears.

• The persistent cache on-disk format is now v5, with keyed per-entry frame integrity, so recovery and eviction lose only a damaged entry instead of the rest of the page. Each entry is now protected by a keyed message authentication code, bound to the entry's position, rather than a plain checksum. When a persistent (mmap) shard meets a corrupted or torn entry — at warm restart or during eviction — it now skips just that entry and continues with the entries after it, where before a single damaged entry discarded everything following it on the page. Because the integrity code is keyed by a per-file secret and tied to position, a value crafted to look like a stored entry cannot be mistaken for one during this recovery. Upgrade/downgrade: a v4 data file is upgraded to v5 automatically the first time this build opens it (the upgrade is crash-safe and leaves the original in place if it cannot complete); a v5 file cannot be opened by a build older than v0.7.0-beta5, which refuses it rather than rewriting it. Because a v5 page reserves slightly more per-page overhead than v4, a persistent shard that was packed essentially full under v4 may not have room for the upgraded copy; the upgrade then fails closed — it leaves the original v4 file untouched and refuses to open the shard until capacity is raised — rather than losing any data.

• Security (GHSA-m63m-rp87-w4rf): a persistent cache no longer trusts a page's stored bounds after an unclean host shutdown. On a power loss or kernel panic (not an ordinary process crash), a persistent (mmap) shard could recover a page whose stored start/end offsets named byte ranges whose contents had not reached disk, or a reused page whose old contents were still present, and index records from the page's previous life. Depending on timing this could resurrect a deleted or overwritten record, or — for a client that could choose stored values — let a crafted record be recovered ahead of the genuine one. The page bounds are now written to disk only after the entry bytes they describe, and a reused page's cleared bounds are made durable before the page is written again, so recovery can only index data that was durably written in the page's current life. No on-disk format change; existing data files are read as-is. A file written by a newer build is now refused rather than silently rotated aside and served empty.
• Dropping a multi-vector collection through the generic collection drop now removes it, so it no longer reports success without doing anything and no longer comes back on restart. The generic drop (DELETE /v1/collections/{name}, the gRPC DropCollection call and CollectionStore.DropCollection) did not recognise a multi-vector collection at all: it returned success while leaving the index live in memory — so the drop was a no-op in every mode, heap included. Where the collection had files (a persistent or WAL single-node collection, or any collection on a -cluster node, which always writes its .mvcfg marker and may write generation mmap files), those files were left on disk too, so the collection reloaded on the next start. The generic drop now performs the same cleanup as the multi-vector drop (DELETE /v1/multivector/{name}). Heap-only multi-vector collections write no files, so for them only the in-memory no-op is fixed.
• Dropping a named-vector collection through the generic collection drop now removes its files, so it no longer comes back on restart. The generic drop (DELETE /v1/collections/{name}, the gRPC DropCollection call and CollectionStore.DropCollection) accepted a named-vector collection and removed it from memory, but left its .ncfg marker, snapshot and write-ahead log on disk. A single-node collection with a write-ahead log therefore reloaded on the next start — and if the name had been reused for a dense collection in the meantime, the name loaded as both. The generic drop now deletes the same files as the named-vector drop. Heap-only named collections, and named collections on a -cluster node, write no files and were not affected. A name already loaded in two families by this bug still needs the stale collection dropped by hand: DELETE /v1/named/{name} removes the named-vector collection and its files and leaves the dense one in place.
• The cache eviction knobs can now be set on rostam-server and through rostam.CacheConfig. Relocating eviction, its background reserve cadence, in-place same-size updates, lock-free (seqlock) reads for them, and the SIEVE visited bit existed only as cache.Config fields, so nothing short of constructing the cache directly could turn them on. They are now the flags -relocating-eviction, -relocate-reserve-interval, -in-place-same-size-update, -in-place-seqlock-reads and -sieve-visited-bit (each with its ROSTAM_* variable), and CacheConfig fields of the matching names. None has any effect by default: the four switches are off, and the reserve interval's default does nothing without relocating eviction, so nothing changes unless you set them. They are flags and environment variables only; the -config file does not accept them.

Most of them have no effect on most topologies. None acts on a -cluster node, whose shards are file-backed and refuse writes at capacity instead of evicting; in-place updates, their seqlock reads and the background reserve act only on a single-node server without -data. A knob set where it cannot act — or paired without the knob it depends on — is accepted and logged as a startup warning naming the flag, the deployment and the reason, rather than silently ignored. The matrix is in Running the server.

-in-place-seqlock-reads is an opt-in performance trade whose read protocol is a deliberate data race validated after the fact: it is not covered by race detection, and its concurrent tests are skipped under -race. - Breaking: objstore.ObjectStore gains PutIfAbsent, and two single-node backups at one timestamp can no longer both publish. A backup refused an existing snapshot key by listing it first and then writing, and nothing made those steps atomic: two runs starting in the same second — an on-demand backup landing in the periodic run's second, or two servers backing up the same prefix — could both see the key absent and both write, the second replacing the first's snapshot and config.

PutIfAbsent(ctx, key, r, size) stores an object only if the key does not exist, as one atomic step, and returns objstore.ErrExists otherwise. A backup now claims its timestamp by creating the config object with it, then creates the snapshot the same way, so exactly one concurrent run publishes and the rest report backup.ErrSnapshotExists without writing anything.

This covers single-node collection backups only. Cluster backups (shard snapshots, their sidecars and the catalog) still write with Put, so two cluster backups at one timestamp can still overwrite each other; making that path write-once is not done yet.

• S3 sends If-None-Match: *; the service answers 412 when the key exists. Some S3-compatible stores — older MinIO releases among them — ignore that header and overwrite. A store cannot be told apart by its response, so the first conditional write checks once, with a throwaway object next to the key, that the service really refuses a second create. A service that ignores or rejects the header makes every PutIfAbsent fail with objstore.ErrConditionalWriteUnsupported, and single-node backups to it fail loudly rather than silently losing the guarantee. Upgrade the store to a release that supports conditional writes.
• Filesystem (-backup-dir) stages as before and publishes with a hard link, which refuses an existing name where a rename replaces it. The backup directory must be on a filesystem with hard links (not FAT32 or exFAT); on one without, backups fail with the link error.
• An orphaned config — left by a backup that died between its two writes — now holds its timestamp: a retry at that exact timestamp is refused, since it cannot be told apart from a run still writing. Periodic and on-demand backups use the current time and are unaffected.

The filesystem store no longer fails every write on Windows by trying to fsync a directory, which Windows does not support. The separate reason filesystem backups on Windows still failed — the : in backup keys — is fixed in the next entry.

What to do: code outside this repository that implements objstore.ObjectStore no longer compiles until it adds PutIfAbsent. Implement it atomically against the backing store; if the store cannot guarantee that, return objstore.ErrConditionalWriteUnsupported rather than calling Put, or concurrent backups will silently overwrite each other again. A wrapper that embeds another store and overrides Put must override PutIfAbsent as well, or the embedded one is used and bypasses the override. - Filesystem backups (-backup-dir) now work on Windows. Every backup key embeds an RFC 3339 timestamp, and the filesystem store used the key as the file name, so on Windows every snapshot and config write failed: : is not allowed in a Windows file name. The store now writes each : as %3A in file and directory names, on every platform, and maps names back to keys when listing. Object keys are unchanged on every backend, so retention still ranks snapshots by time and S3 backups are untouched.

Backup directories written by earlier builds, whose file names contain a literal :, stay fully usable on Linux and macOS: restore, LatestKey and retention find them, a key present under both spellings is listed once, a delete removes both, and a write-once create is refused when the key exists under either spelling. New backups are written under the escaped names. Such a directory cannot be carried to Windows as-is, since its names cannot exist there.

The filesystem store now refuses a key that itself contains %3A (in any case), since such a key could not be told apart from an escaped :. The collection segment of a backup key never contains it (it is percent-escaped, which turns % into %25), so this only affects a backup prefix (-backup-prefix) that contains %3A itself. - A corrupt entry met during eviction on a persistent (mmap) cache left the evicted page's keys behind in the index. When the in-place drain could not frame an entry it reset the page, but dropped none of the index slots still pointing into it. A reset keeps the page's generation, so those slots stayed valid to the read path: until the page was rewritten they served records the drain had evicted, and once it refilled, the bytes at their offsets belonged to other records — which Iterate then reported twice. The slots were never reclaimed, so Entries stayed inflated for the life of the process. The drain now drops every slot addressing the page before resetting it, counting and notifying the removals as an ordinary drain would. A key is reported to the removal hook only when its bytes still hash to the slot, so the damaged record itself notifies nothing rather than a key read out of garbage.

• On 32-bit builds, one corrupt length in a persistent cache's page file crashed every restart. The entry decoders added a length read off disk into a total before bounding it; a value length near the 32-bit maximum wrapped that total negative, slipped past the length check, and panicked on the slice — during the warm-restart walk, so again on the next start. Eviction's framing had the same sum and stored the wrapped value as the page head without reporting an error. Both now compare the length with the room left instead. 64-bit builds are unaffected.

• New counter CorruptionBytesDiscarded (rostam_kv_corruption_bytes_discarded_total) reports how many page bytes the warm restart and the mmap eviction drain discarded because they could not read them. A warm restart that meets an entry failing its checksum still discards that page from the entry to its tail, and this is where that loss now shows; a page whose persisted head/tail are out of range counts as its whole capacity. It does not resynchronise past the bad entry: a value can legitimately contain bytes that form a complete, checksummed entry, so a resync could index a key nobody wrote. Recovering the rest of the page safely needs framing a client cannot forge, which is a format change.

It is not a per-incident size for CorruptionErrors, whose description was also wrong: that counter never counted checksum mismatches on read (reads verify none). It counts once per read that reaches an entry which will not frame, so a damaged entry read repeatedly counts repeatedly, and once per region the warm restart or the drain gives up on.

• An mmap cache page whose persisted head/tail were corrupted while the cache was running could panic the process at its next eviction, or, on 32-bit builds, at the next write into it. The drain now checks a page's bounds before reading through them and resets a page whose bounds are out of range, with its index slots dropped.
• Breaking (rolling upgrades): a node now HALTS on cluster metadata it does not recognise, instead of silently skipping it. When the meta-Raft log delivered an entry whose op this binary did not know, the node returned an error that Raft ignores, recorded the entry as applied, and carried on. Its peers on the newer binary that proposed the entry applied it, so the node's cluster metadata diverged from theirs permanently — and because the entry was recorded as applied, not even a restart would revisit it. Shard groups have long failed closed on the equivalent condition; the meta log now does too.

The node exits with an error log naming the entry's op and log index, the applied index it stopped at, and the remedy. Nothing about the entry is recorded, so on every restart the node replays the log, meets the same entry and exits again, until its binary is upgraded to one that knows the op. A process supervisor will show this as a crash loop repeating that one line.

The same gap existed through snapshots: once the leader compacts its log past such an entry it sends a snapshot instead, and a snapshot carrying state from a newer binary had that state silently dropped on restore. Snapshots now record which fields carry data, and a node refuses one that carries data in a field its binary does not have, logging which field. Raft retries a refused snapshot install, so the node stays behind rather than wrong. Snapshots written by earlier binaries carry no such record and restore as before, so this protection starts with the binaries that write it.

What to do about it: upgrade every node's binary before using a feature that introduces new cluster metadata. That was always required for correct behaviour; it is now enforced by a visible stop rather than risked as silent divergence. If a node is halting with this message, upgrade that node — no data needs repairing, since nothing was applied.

• A restore from a bad backup no longer destroys the collection it was meant to replace. backup.Restore and CollectionStore.RestoreCollection / RestoreCollectionWithConfig are create-or-replace, and they dropped the existing collection — removing its files — before reading the snapshot. A truncated or corrupt snapshot then failed with an empty collection left where the original had been, and a config the index rejected left no collection at all.

The snapshot is now restored into a collection built outside the catalog while the existing one keeps serving, and the existing one is dropped only after the restore has succeeded. A failed restore leaves it untouched and nothing of the attempt behind. (#112)

What to expect:

• The swap is not atomic. Between dropping the existing collection and registering the restored one, the name resolves to no collection, and an operation on it fails as it would on an absent name. That window covers draining the existing collection's in-flight operations, closing it and removing its files, and writing the new config marker (and opening the log of a WAL collection) — not the restore itself. A filesystem failure inside it still leaves neither collection, and the error says so.
• A restore now holds both the existing collection and the restored copy in memory until the swap, where it used to need room for one. No store-wide quota counts this; running out fails the restore with the existing collection intact.
• Operations that change what a collection name refers to now wait for one another instead of interleaving: creating, dropping, restoring, promoting a cold collection, and committing an eviction. So while a restore of a name runs, a create or drop of that name, or a second restore of it, waits rather than failing, then acts on the result — a create reports ErrCollectionExists if the restore succeeded, and a drop removes the restored collection. Reads and writes on an existing hot collection are not held up while the snapshot is restored.
• Reading a cold (evicted) collection that is being restored blocks until the restore finishes, because resolving it promotes it and promotion takes part in that ordering. It then sees the restored collection. In the other direction, a drop, create or restore of a cold name waits for a promotion already in progress, including its read of the snapshot from object storage, and then acts on the promoted collection.
• A whole-store RestoreAll swap that arrives while a restore is registering its replacement waits for that step, at most the window described above, rather than landing in the middle of it.
• A dense restore over a named-vector or multi-vector collection is refused with ErrCollectionExists. Over a named-vector collection it used to succeed while leaving that collection's files on disk, so after a restart the name loaded as both kinds. A restore with a named-vector config is also refused up front; it could never succeed, and used to drop the existing collection before failing.

Cluster disaster recovery (-restore) installs whole shard snapshots through Raft and does not use this path, so it is unaffected.

• Catalog operations on one collection name no longer undo each other. These races existed independent of restore, and the ordering above closes them:

• A drop removes the collection from the catalog, then waits for in-flight operations before deleting its files by name. A create (or restore) of the same name in that gap wrote its config marker and log, which the drop's cleanup then deleted — the call succeeded and the collection vanished on restart.

• A cold promotion that lost a race with a drop discards what it built by deleting the name's files, including files a create or restore of the name had just written.

• An eviction snapshots a collection, releases it, and then replaced whatever was registered under the name with a cold entry for that snapshot. If the name had been dropped and recreated, or restored, in between, the new collection was replaced by the old data. It now commits only if the collection it snapshotted is still the one registered.

• Server.Close could panic the process, or return while a connection handler was still running. The accept loop published a connection into the tracked set and only then counted its handler goroutine, while Close walked that set and then waited on the counter. Nothing ordered those two steps, so a connection could be visible to Close but not yet counted.

That produced two failures. Close could observe a zero counter and return while a handler was only starting, making its documented promise to wait for handler goroutines false — and since shutdown closes the TCP server before the store, a handler could then still be dispatching against a closing store. Or, when other handlers drained the counter to zero at that moment, sync.WaitGroup detected the misuse itself and panicked inside Close with "WaitGroup is reused before previous Wait has returned". A connection accepted after Close's walk was also never force-closed, so nothing ended it but the idle timeout.

Admission now counts and publishes the connection under the same lock Close holds before it waits, and refuses once Close has begun. A connection is therefore either fully admitted before that critical section — so it is force-closed and its handler is waited for — or refused, and the accept loop drops it rather than starting a handler Close cannot wait for. Ordering the two steps differently, or checking a shutdown flag in the accept path, does not close the window: Close can run to completion between the accept and the count either way.

• Breaking: operate now bounds the total BYTES its returns may produce, not just how many of them there are. maxRet capped the return-spec list at 4096, but a spec is a few bytes on the wire and a record-kind VALUE copies the whole record — so an 8 KiB request asking for 4096 record values against a 64 KiB record returned 256 MiB and allocated 288 MiB of heap, a 32,701x amplification, and against the 16 MiB record cap the worst case was ~64 GiB from one small call. None of it could ever have reached the caller: a reply that large is far past the 16 MiB frame limit, so the memory was spent only to be thrown away.

A new cap, maxRetBytes, is charged as each return is evaluated, so a call that would cross it is refused with wire.ErrOperateCap before the full return list is materialised — the peak a refused call reaches is the budget plus the one value that crossed it, not the whole list. Its value is 16,711,680 B: the 16 MiB frame limit less a flat 64 KiB, which is what the reply's own framing and the result envelope (a length per value, plus the status, count and failedOp fields) need alongside the values. A budget of exactly 16 MiB would have made the largest ACCEPTED call fail at the transport instead, which is the same refusal arriving later and naming the wrong cause.

Every operate cap refusal also reaches the client as the refusal it is, on all three transports: wire.ErrOperateCap was unclassified at the TCP, REST and gRPC edges alike, so a caller that had simply asked for too much was answered "internal error" (500 / Internal) instead of a 400 / InvalidArgument naming the cap. It is now matched both by sentinel and by exact message shape, which is what makes it work on a cluster: an operate APPLIES inside the Raft FSM, so the engine's own caps come back rebuilt and lose errors.Is identity on the way out.

What to do about it: a call whose returns come to more than 16,711,680 B in total now fails with the record unchanged, like every other cap, where it previously applied the ops and then failed to frame the reply. Ask for fewer returns, or for narrower paths than the whole record — the budget is per CALL, not per key, so splitting an oversized read across several operate calls is the remedy.

• rostam_kv_entries and rostam_kv_tombstones: how many keys a node is actually holding. The KV metrics covered reads, writes, evictions and bytes, but never the count of the thing being stored — so "how many keys fit in this budget" had no answer. It could be inferred from misses only until the first eviction, after which an evicted key that comes back misses again and the inference breaks; that is precisely when the question matters.

entries is the denominator the eviction counters previously lacked, and what to divide a memory budget by. Note what the ratio is and is not: bytes_used / entries is occupancy per indexed key INCLUDING every byte no live key owns — superseded copies, TTL-expired entries not yet swept, and the ghost bytes of deleted slots. It is the right figure for sizing a budget, since all of that occupies the budget, but it is not an average record size: overwriting, expiry and deletion each raise it while the records themselves are unchanged. tombstones counts deleted slots not yet reclaimed — a large share of entries means the index wants compacting.

Both come from counters the index already maintains, read under the same lock bytes_used uses. No walk.

• Three allocation leaks on the KV path. The append path landed in v0.7.0-beta4, but profiling a write-heavy workload showed operations still allocating in a minority of calls. All three causes are fixed:

• A connection never kept a reply buffer it had to grow. The epoll loop adopted the payload only when it still aliased the buffer it handed down — but a payload that outgrew that buffer is a NEW array, and is exactly the one worth keeping. A connection whose records exceeded its buffer therefore reallocated on every request, forever. dispatchInto now reports whether the append path ran, and that is the signal to keep the buffer.

• copyRecord reserved a flat 64 bytes of slack. One row insert routinely exceeds that on a record with growing tables, so insertGap reallocated — the largest single allocation site in the profile. Slack now scales with the record.
• The create path ignored the caller's scratch, allocating a fresh record for every first write to a key, while the handler's pooled buffer sat idle.

Before the fix, insertGap and newSchemaRecord each allocated on roughly a quarter of the calls that reach them, copyRecord on a sixth, and getIntoCore on a seventh.

• get and operate can be served without allocating a reply at all. The reply payload was the last per-call allocation on a KV node: tx.Get copies the stored value into a fresh slice for every read, and operate builds its result frame the same way. AppendHandler is a new OPTIONAL per-op twin of Handler that appends the payload to a caller-owned buffer instead, and server.AppendDispatcher lets a transport supply one. Only those two ops opt in; every other op, and every dispatcher without CallAppend, is served exactly as before.

	before	after
get	256 B/op, 1 alloc, 118 ns	0 B/op, 0 allocs, 60 ns
operate (with the pooled record buffer)	1 alloc	0 allocs

Store.Call deliberately keeps the allocating path: it is public API handing bytes to in-process callers that may retain them. The append form's payload aliases the transport's buffer and is valid only until that buffer is reused, which the epoll server satisfies by copying the payload into its response frame before reading the next request.

• The operate reply frame is built in one allocation, whatever it returns. EncodeOperateResult reserved room for each value's 4-byte length prefix but not for the value BYTES, so any non-empty return made append grow the array — on exactly the calls asking for the most data back. It also reserved a failedOp field that only a failed CHECK ever writes. The size is now computed exactly, and a test pins it (cap == len on the finished frame).

BenchmarkEncodeOperateResult, 32-byte values:

values	before	after
1	64 B, 2 allocs	48 B, 1 alloc
4	360 B, 4 allocs	160 B, 1 alloc
16	2112 B, 5 allocs	640 B, 1 alloc

Roughly 2.6x faster at 16 values. A call that returns nothing was already one tiny allocation and still is.

• A warm operate write allocates nothing for the record itself. The apply path worked on a private copy of the stored record (copyRecord), one allocation per call and — after the result frame moved to the stack — the last one left in applyRecordBytes. handleOperate now supplies that buffer from a pool and keeps it across calls, so a steady-state record is patched in a recycled array. A record that grew keeps the larger array, which also stops insertGap reallocating on every write once a record reaches its working size — up to 64 KiB. Past that the buffer is dropped rather than pooled (the same bound the read buffer uses), so records larger than that keep reallocating and only lose the result-frame allocation.

Measured through the handler (BenchmarkOperateHandlerExistingRow): 2 allocs/op to 1, 169 B/op to 71. The call is ~3% slower in ns/op — about 14 ns for the pool round trip — which is the trade. The remaining allocation is the reply frame EncodeOperateResult builds.

The recycled buffer is the one out points at, not the one passed in: insertGap REPLACES the array when a record outgrows it. out aliases the scratch whenever the record fit, so a caller holding those bytes past its next call holds a buffer someone else is writing — tx.Put copies into the page arena, so the store itself never does.

• The operate APPLY path costs one allocation per call instead of two. The result frame was built as &wire.OperateResult{...} and returned up the apply path, one 32-byte heap object per call. It never outlives the handler — the op encodes it and drops it — so it is now returned by value, removing that per-call heap allocation. Measured on BenchmarkOperateSchemaExistingRow and BenchmarkOperateDynamicExistingRow: 2 allocs/op to 1, with bytes down by exactly the struct. The one remaining allocation is the private record copy the design doc's §2.6 budget allows, and a test now pins that budget.

Scope: this is applyRecordBytes, not a whole operate call. Encoding the reply still allocates its response buffer in EncodeOperateResult, and vector_operate still heap-allocates one result, because the mutation callback captures its address — unchanged in count there, just moved. Small in bytes, since the struct is 32 of them, but in profiling this was the largest single allocation site by object COUNT, and object count is what allocator and sweep work follow. It is not what paces GC — that is heap bytes against GOGC, and 32 bytes a call barely moves it — nor what mark cost follows, which is live pointer-bearing memory.

• KV cache metrics are now scrapeable, and capacity loss is distinguishable from TTL turnover. A KV-only node previously reported nothing about itself: /metrics renders dense-collection stats and returns an empty body without a vector store, and cache.Stats() was only surfaced through the Raft path a single-node deployment never runs. A new shardless read-only op __kv_metrics__, served at /kv-metrics and /v1/kv-metrics, renders the node's cache counters as Prometheus text. In cluster mode it is dispatched node-locally (like __ready__/__repl_metrics__) and aggregates every shard the receiving node hosts, rather than routing to one shard's cache.

Alongside it, rostam_kv_evictions_live_total separates records displaced because the cache was full from records that simply reached their TTL. rostam_kv_evictions_total counts every entry a reclaimed page held — including versions a newer write had already superseded and entries that had expired but not yet been swept — so it overstates what a cache actually lost. The _live_ counter rising while rostam_kv_expirations_total stays low is the signal that the memory budget, not the TTL, is deciding how long entries survive.

• kv_query: find KV keys by what their record holds. A new read op answers "which keys hold a record matching this filter" over the KV keyspace, using the same JSON filter grammar the vector side uses with bare record paths as fields (rc, b#count, b/42/hi). Named, cluster-wide KV index definitions — one top-level field or a table's #count, scoped to a key prefix — are committed through the meta log and backfilled per shard group; an eq/in/range leaf on an index's own path (at the filter root or in a top-level and) drives the candidate set, and everything else, every negation included, is re-checked against the live value, so a stale posting can never produce a wrong row. An indexed query answers "the keys under this index's prefix that match", never "the keys that match"; a filter no index can drive needs explicit scan: true consent and is bounded by a per-node scan budget. Pages are bounded by a row limit (≤1000) and an 8 MiB page budget, resume through an opaque composite cursor with one continuation per shard group, and come back as keys, values or records. Read consistency is leader by default, with any and linearizable available. On the Go client: CreateKVIndex / ListKVIndexes / DropKVIndex / KVQuery; over REST: POST /v1/kv/query plus POST/GET/DELETE /v1/kv/indexes (index creation and drops are admin-scoped). New Stats.KVIndex counters make the index observable. See querying records in the KV store.

• Filter and index operate records stored in vector payloads. A payload value of kind record (the bytes operate writes) can now be addressed directly by filters: a path like session/rc, session/b/42/hi, or session/b#count resolves into the record, and two new leaf ops, row_exists and row_absent, test table-row presence. A collection auto-indexes every top-level scalar field and table row count for eq/ in/range acceleration; row/column/positional paths and contains/match/regex/row_exists/row_absent are always evaluated against the live record. A point whose stored record is malformed fails the whole payload key closed (evaluated live, still correct) until it is repaired or reclaimed. row_exists and row_absent follow the same exact-payload-key-first rule as every other op — a literal key such as "session/b/42" wins over splitting, and makes both ops false — and row_absent is true only when the named field really is a table on that point, so a record with no such field at all answers false rather than true. See record paths in filters.

• operate: server-side atomic multi-field updates on a record. A new built-in op updates several fields of one record atomically, in one round trip, against a hot key — the way you'd reach for a Redis hash command or an Aerospike Operate call, without writing a custom Go op or a client-side get-modify-put CAS loop. The record/table model is record → table of keyed rows → columns, addressed by path and applied under the same shard lock as every other read-write op. Two storage modes share one apply engine: schema mode packs a client-defined, versioned layout with nothing per value (fastest, smallest, for a known shape); dynamic mode is schema-free, Redis-hash-ergonomic, and lets any op add a new name/type at any time. A dynamic record can later be frozen into schema mode with MIGRATE. Available on the Go client via client.NewOperate's typed op-list builder. See the KV overview for the full path/type/op vocabulary, caps, and worked examples.

• vector_operate: mutate a record stored in a vector payload in place. New built-in ops vector_operate / vector_named_operate / vector_mv_operate apply the same operate op-list to the record held under one point's payload key, atomically, under the collection's write lock, in one apply — the read-back paths, types and OperateResult are exactly operate's. A point's TTL and the op-list's inner key don't apply here and are rejected rather than silently ignored; create can still create the record, and create = NONE against a missing one is an error, matching KV operate. The mutated record's top-level fields are reindexed in the same apply, so a filtered search sees the change immediately. The call is refused while its collection is resharding (retryable after cutover) and is not replayable. Available on the Go client via the typed Collection.Operate (dense collections; named/MV have engine support with a typed client handle as a follow-up). See updating a record in place.

• Malformed record payload values are now rejected at ingest. Every entry that stores a payload value of kind record — insert, insert-if-absent, upsert, set_payload, overwrite_payload, and the bulk paths — now validates the bytes decode as a well-formed operate record, not just that they fit the size cap. A caller that hands in bytes no operate engine can open now gets a 400 at write time instead of a payload that later poisons that key's filter acceleration for the whole collection. Only a record written by a version before this change can still be malformed; the fail-closed indexing behaviour for that case is unchanged.

• New -online-compaction flag. Opts every replicated mmap reject-writes shard into online relocating compaction, so a write/overwrite-heavy replicated shard reclaims dead ("ghost") bytes while the process runs instead of only at restart (cold compaction). Off by default — recycling retired page bytes is memory-safe only when every read is released within the alias fence (2×WriteTimeout, default 60s), which holds for the standalone server since all reads go through the transport. No-op on heap/single-node/ring-buffer shards. Validated under a 9-hour nosync write-endurance soak (memory held flat, zero errors).

• Primary-backup / ISR replication (-replication-mode=pb) is no longer experimental. Its blocking correctness hazards are closed — a lease-fenced primary plus full-ISR commit, backed by partition and crash-stop failover tests. The no acked-write loss across an automatic failover guarantee holds under all of: -min-isr ≥ 2 (=1 can lose acked writes when a post-failover ISR resets to the lone new primary), the default full-ISR commit (-pb-commit-primary=false — the opt-in -pb-commit-primary is a durability downgrade), and a bounded cross-node clock rate. It is recommended at replication-factor 2, where it beat raft RF=3 (both -nosync -volatile-log) ~1.7× on throughput (and p50 latency) in the 2026-07 real-network gate — a cross-RF comparison (PB RF=2 holds one fewer copy), whose raft baseline also ran under the old epoll default, so the margin is unverified under the current server; re-verify on your hardware. At RF=3 its full-ISR commit (wait for the slowest of every replica) is slower than raft's majority, so raft stays the default. Automatic failover is on via the -pb-auto-failover server flag; cluster.Config / rostam.EmbeddedConfig users must set PBAutoFailover: true. Durability scope: PB is nosync (no per-shard WAL / fsync) — durability is "acked on ≥ min-ISR nodes in memory," so the guarantee covers losing individual nodes, not the simultaneous loss of every in-sync node for a shard. The mode is newer than raft — validate on your workload.

• Hardening: 32-bit builds no longer panic on an oversized request or response frame length. The client-facing TCP protocol decoders (DecodeRequest, DecodeResponse) bounds-checked a declared frame length after converting it to int; on a 32-bit build (GOARCH=386/arm) a declared length of 2 GiB or more wrapped negative during that conversion, defeating the bounds check and panicking an unauthenticated server on a 7-byte frame. Found by the new protocol fuzz targets. Both decoders now reject the oversize length while it is still an unsigned 32-bit value, so the int conversion can never go negative. They also now reject a per-field-valid length whose reconstructed total frame size exceeds MaxFrameSize — the bound the corresponding encoder already panics on — closing a gap where an accepted frame could panic on re-encode.

v0.6.0 — 2026-09-02

• Online compaction for replicated shards (opt-in, off by default). A persistent (mmap) cluster-replicated shard is append-only under reject-writes: an overwrite writes a new copy and leaves the old one as dead "ghost" bytes, so under write/overwrite churn a shard could climb to capacity and reject writes with "at capacity" until the next restart reclaimed the space (cold compaction). With ServerConfig.EnableOnlineCompaction set, the background sweeper now reclaims that dead space while the process runs: it relocates the live entries out of fragmented pages, retires the emptied pages, and — once an alias-drain fence has elapsed — resets those pages back into writable space, so a shard recovers write capacity online instead of only at restart. It leaves the lock-free zero-copy read path untouched and is a no-op on heap, single-node, and ring-buffer shards. It is off by default and must be enabled explicitly because of its safety contract: recycling overwrites retired page bytes, which is memory-safe only when every read is released within the fence — i.e. when all reads flow through the server transport, whose write timeout bounds the zero-copy response. Do not enable it if an in-process embedder retains a raw Store.Get / Node.Call result (a cache alias) past the fence; those readers must copy the value out. The fence is derived from the new WriteTimeout and is enforced fail-closed at startup, so it can never fall below the real write deadline. Stats.ReclaimableBytes (a gauge) exposes the current ghost-byte pressure whether or not compaction is enabled.

• New ServerConfig.WriteTimeout. Bounds how long a single TCP response write may block on a stalled client before the connection is aborted (default 30s). Besides slow-client hygiene it is the source of truth for the online- compaction alias fence above, so raising it raises the fence in lockstep. A consumer slower than this is disconnected rather than allowed to pin a response buffer indefinitely.

• New key-value op: flush. Wipes the entire KV keyspace in one call, in O(shards) rather than O(keys) — it swaps each shard's in-memory index for an empty one and records a small per-shard durability watermark, so pages are reclaimed lazily by later compaction instead of scanned and deleted up front. It replicates like any other write (applied through each shard group's log, so every replica converges on an empty keyspace) and, in a cluster, one call fans out to every shard group. Requires a global write scope (write:*); a read-only or single-collection key cannot flush. Available on the Go client as Flush(ctx) and over REST as POST /v1/kv/flush. This removes the need for the generation-counter workaround a cache driver otherwise needs for Cache::flush()-style clears (which is not correct on the client side, because the counter is itself an evictable cache entry). Caveat: flush empties the keyspace but leaves the page allocation intact, so under the non-default PolicyRejectWrites a shard that was full can still reject the next write with "at capacity" until its space is reclaimed by a later cold compaction (at restart) — under the default ring-buffer eviction policy this is invisible.

• Embedded web dashboard. The server now ships a browser UI at /dashboard/ for inspecting and operating on a running instance — vector collections, cluster topology, and KV keys — with no separate install. It reads over new REST endpoints: GET /v1/topology, GET /v1/collections, and GET /v1/collections/{name}. The KV store also gains a REST data plane alongside the existing binary protocol: GET, PUT, and DELETE /v1/kv/{key}.

• Slabs grow in place on Windows too (heap-backed). The address-space reservation that removed the multi-hundred-millisecond reader stall at a slab growth boundary now exists on 64-bit Windows, built on VirtualAlloc's native reserve/commit split. It covers the heap-backed slabs — arena.vecs and the level-0 graph in heap mode, and the quantization codes always — which is where the larger stall was measured (1.502 s to 10.2 ms on Linux, against 188.9 ms to 17.1 ms for the mmap-backed ones). An mmap-backed slab on Windows still grows the old copy/remap way: committing a file view into the middle of an existing reservation needs the Windows 10 1803 placeholder API, which is not wired up. Nothing changes for Linux, and a reservation remains an optimization — where it cannot be made, growth simply falls back.

• Cluster metadata stability fixes. Three correctness fixes in the Raft-replicated metadata plane: Node.Close() now waits for its background cluster-management goroutines (the PB control-plane seeder and the shard-formation seeder/driver) to exit before tearing down shared resources, removing a shutdown use-after-free; MetaFSM.State() returns a deep copy of ShardFormer so a caller can no longer mutate the internal map or race a concurrent Apply; and the ApplySetMembersIfLeader idempotency check now accounts for ReplicationFactor, so an RF change is no longer silently skipped when members, shard count, and MinISR are unchanged.

v0.5.0 — 2026-08-30

• Eight new key-value ops: exists, getdel, getset, persist, ttl, incr_ex, compare-and-expire, and mget. The KV store gains the everyday Redis-style verbs: exists reports whether a key is present, getdel atomically reads a key and deletes it, getset atomically replaces a value and returns the old one, persist removes a key's TTL, and ttl reports the remaining time-to-live (-2 absent, -1 no expiry, else remaining ms). Two are primitives for correct distributed coordination: incr_ex increments a counter but sets the TTL only when the key is first created — so an existing counter's window never slides, giving you a correct fixed-window rate limiter in one round trip — and compare-and-expire refreshes a key's TTL only if it still holds an expected token, the safe lock-renewal counterpart to compare-and-delete. mget fetches many keys at once, returning a value (or a distinct "missing") per key in request order; the Go client groups keys by owning shard and fans out one batched read per shard. Available on the Go client as Exists / GetDel / GetSet / Persist / TTL / IncrEx / CompareAndExpire / MGet and on the Python client (rostam-client) as exists / getdel / getset / persist / ttl / incr_ex / compare_and_expire / mget.
• Persistence works on Windows. cache.Config.DataDir and mmap-backed (Persistent) vector collections are now supported on Windows, so an embedded store — and the MCP memory server on its default -data auto — keeps its data across restarts there instead of refusing to open with cache.Config: DataDir set but mmap not supported on windows. Windows previously had no file mapping wired up at all, only stubs that returned that error, which made -data "" (heap, no persistence) the one working mode. Durability means the same thing on both platforms: the flush pairs FlushViewOfFile with FlushFileBuffers, so -durable is not claiming a guarantee Windows was never asked for. One difference to plan for: NTFS does not hand out sparse files, so a shard file occupies its full configured size on disk the moment it is created. A fresh embedded store therefore reserves its whole cache budget up front (1 GiB by default off Linux, where system memory is not probed), while on Linux the same file starts as a hole and consumes space only as it fills.
• Atomic conditional KV writes: set_nx, cas, and compare-and-delete. Three new key-value ops give you an atomic conditional write, the primitive needed for correct distributed locks, once-only writes, and idempotency keys: set_nx stores a value only if the key is absent (or expired); cas (compare-and-swap) stores only if the current value equals an expected value; and compare-and-delete removes a key only if it still holds an expected value (a safe lock release). Each is a single atomic operation — two writers racing on the same key can no longer both win — and an expired key reads as absent, so a lock re-acquires cleanly once its TTL lapses. Available on the Go client as SetNX / CAS / CompareAndDelete and on the Python client (rostam-client) as set_nx / cas / compare_and_delete.
• Python client: Mem0, Semantic Router, CrewAI, and DSPy adapters. (First shipped in rostam-client 0.3.0; this release cuts the client at 0.4.0 for the new KV methods listed above.) Framework integrations join the existing LangChain, LlamaIndex, and Haystack adapters, each an optional in-package submodule behind its own extra so the base import rostam stays standard-library only: rostam.mem0 (RostamVectorStore, a Mem0 vector-store provider), rostam.semantic_router (RostamIndex), rostam.crewai (RostamStorage memory backend), and rostam.dspy (RostamRetriever). Install with the matching extra, e.g. pip install rostam-client[mem0].
• In-process local embeddings, pure Go (no cloud API, no ONNX Runtime). Rostam can generate semantic embeddings in-process via rembed — pure Go, no cgo, no ONNX Runtime, and no build tag: it is compiled into every binary and image. Select a model with ROSTAM_EMBED_LOCAL=<name> from a catalog of six models — the 384-dim minilm-l6-v2 (default), bge-small-en-v1.5, gte-small, and the 768-dim bge-base-en-v1.5, gte-base, all-mpnet-base-v2 — or pass any Hugging Face org/model id directly. Weights download from the Hugging Face Hub on first use (cache dir REMBED_CACHE, or the legacy ROSTAM_EMBED_MODELS_DIR). A configured local embedder also powers the generic vector-DB tools (upsert auto-embeds content; search embeds query_text). See docs/server/mcp.md. Note: the default weight-cache location is now rembed's (REMBED_CACHE, default the OS user cache dir) rather than ~/.rostam/models; set REMBED_CACHE/ROSTAM_EMBED_MODELS_DIR to pin it. Selecting a model by its full Hugging Face id instead of its catalog short name uses a distinct cache scope key, so pick one form and keep it.
• Reliability: a failed vector-slab grow no longer bricks a collection. When a persistent collection must grow its on-disk vector slab and the OS grow fails (disk full, address-space exhaustion), the index re-maps its existing vectors and stays usable — only the write that triggered the grow errors, and it can be retried once space frees (previously the whole collection went to a terminal error state). A failure while growing the smaller graph-edge slab, which happens after the point is already committed, still takes the collection terminal, since there is no safe rollback from that half-committed point.
• Hardening: the wire decoders reject hostile length fields on 32-bit builds. A crafted request could previously panic a decoder on a 32-bit target via an integer-width overflow; every decoder now bounds-checks in an overflow-safe form. No effect on the shipped 64-bit binaries; closes the class the 32-bit CI lane exists to catch.

v0.4.1 — 2026-08-24

• Fix the Docker image build after the module split. The server image's go mod download layer copied only the root go.mod/go.sum, but the root module now has replace directives to the in-repo sdk and client modules, so the download step failed reading sdk/go.mod/client/go.mod. The Dockerfiles now copy the sub-module manifests and go.work before download. (v0.4.0 shipped binaries + PyPI but no Docker images; v0.4.1 completes it.)

v0.4.0 — 2026-08-24

• Security: bump google.golang.org/grpc to v1.82.1. Addresses GHSA-hrxh-6v49-42gf (a DoS in the xDS RBAC / HTTP/2 server paths) present in v1.81.1. The gRPC API surface is unchanged.

• Configurable TTL sweep interval, with a calmer default (-ttl-sweep-interval). The per-shard background reaper that reclaims memory from expired TTL keys is now operator-tunable, and the server default relaxes from 1s to 30s — 1s meant every shard rescanned its index every second for no correctness benefit (lazy-on-read expiry already returns an expired key as not-found immediately). It is a memory-reclaim-latency vs CPU-churn tradeoff: 0 disables active reaping entirely (lazy expiry, plus cold compaction on persistent shards, still apply); lower it if a write-heavy replicated node climbs toward the cache cap between sweeps. Embedded (rostam.CacheConfig.TTLSweepIntervalMs) is unchanged by default — 0 still means the 1s library default; set it to opt in. See docs/server/running.md.

• The Go client is now its own module — go get github.com/rostamlabs/rostam/client pulls no engine. The client-facing leaf packages moved into a shared github.com/rostamlabs/rostam/sdk module (sdk/vtypes, sdk/wire — was ops/wire, sdk/pb — was grpcapi/pb), and client became its own module requiring only sdk. So an external go get of the client resolves a small graph (sdk + protobuf/xxhash/puddle/x/sync) and never downloads the engine module or its requires. The client import path is unchanged. Breaking (direct importers of the moved packages only): if you imported .../ops/wire, .../grpcapi/pb, or .../vtypes directly, update to .../sdk/wire, .../sdk/pb, .../sdk/vtypes. No wire or behaviour change. See docs/RELEASING_MODULES.md for the tag order.

• New vtypes package: the engine-free vector data types. The pure data types of the vector API (Config, Filter, Value, Metadata, QuerySpec, Result, GroupOpts, OrderBy, …) now live in github.com/rostamlabs/rostam/sdk/vtypes; the vector package re-exports every one via an alias, so vector.Config etc. keep working unchanged. This lets the wire codec and network client depend on the types without pulling in the engine — client and ops/wire now transitively import zero engine packages (a prerequisite for a lightweight go get .../client). Breaking (minor, direct vector-package API only): two exported engine-coupled methods became free functions — cfg.Validate() → vector.ValidateConfig(cfg), filter.Compile() → vector.CompileFilter(filter); if you call these on a vector.Config/vector.Filter directly, switch to the function form. No wire or behaviour change.

• Python client: collection handle. r.collection(name) returns a handle bound to one collection so its name stops being repeated on every call — r.collection("posts").search(vec, k=10) instead of r.search("posts", vec, k=10). Mirrors the Go client's client.Collection; construction does no I/O, and transport rules are preserved (e.g. query / delete_by_filter still raise TransportError on a TCP client). Exported as rostam.Collection. Additive — the flat API is unchanged.

• The Python client (rostam-client on PyPI) unifies on a single Rostam class — v0.2.0 (Breaking). The Python client is versioned and released independently of the server/project (this changelog's v0.3.0 entry below) — 0.2.0 is the client's own version, not this project's. RostamClient (HTTP) and the native-TCP Rostam/RostamKV pair used to be two classes with two different vocabularies for the same server — one flat, one nesting vector ops under .vector. Both collapse into one Rostam(target): the transport is chosen from the target string — http:///https:// for REST, tcp://host:port or a bare host:port for the native binary protocol — and the vector API is flat (r.search, r.upsert, r.hybrid_text, ...) on both transports. Key-value operations move to r.kv.* and stay TCP-only: on an HTTP-connected client, any r.kv operation (e.g. r.kv.get(...)) raises TransportError — reading r.kv itself does not raise — as does any other op with no equivalent on the connected transport (the general query() is HTTP-only; TCP callers use recommend() instead). RostamClient and RostamKV are removed, not deprecated — importing either now raises ImportError. Note r.get is REPURPOSED, not preserved: on the old native Rostam class it was a KV read; on the unified client it means vector point-get (r.get(collection, id, ...)). KV reads move to r.kv.get(...).

Before	After
RostamClient("http://host:8080")	Rostam("http://host:8080")
Rostam("host", 7000) (native)	Rostam("tcp://host:7000") or Rostam("host:7000")
r.vector.hybrid_text(...)	r.hybrid_text(...) (flat)
r.get("k") (KV read) / RostamKV	r.kv.get("k") — r.get now means r.get(collection, id, ...) (vector point-get)
query(prefetch=...) on the native client	HTTP-only — raises TransportError on TCP; TCP callers use recommend()

v0.3.0 — 2026-08-16

The Python native client reaches vector parity over TCP

Rostam(...).vector now speaks the full set of point, search, hybrid, and recommend ops over the native TCP protocol, not just the basics. Added: get_batch, scroll, search_docs, search_groups, hybrid_search, hybrid_text, recommend, query, and upsert_batch — so hybrid (BM25 + dense) search and recommendations no longer require the HTTP client. Every request is byte-identical to the Go encoder (verified by a cross-stack golden oracle) and every method is exercised against a live server. recommend/query currently encode the recommend-shaped query spec; a general multi-leaf query encoder is future work.

A RAG CLI with no pipeline to assemble

rostam-server rag turns the engine into a retrieval-augmented-generation tool you can point at a directory:

rostam-server rag ingest ./docs
rostam-server rag query "How does the LLM proxy decide what's cacheable?"

ingest chunks and indexes every recognized file into a local corpus (./.rostam-rag by default) — recognized meaning a known text extension whose contents are valid UTF-8; anything else is skipped and reported, and a source that becomes invalid has its stale chunks purged rather than left to be returned by searches. query returns the matching chunks with a source#chunk-index and score, and ask sends them to an LLM and prints an answer that cites the chunks it used. There is no separate ingestion service and no vector store to stand up first.

As with the MCP server, it does something useful before you configure anything: retrieval runs on BM25 with no embedder, no API key and no network. Pointing it at an OpenAI-compatible embedding endpoint upgrades retrieval in place, without changing the commands. Full flag and file-type reference: RAG CLI.

Retrieval fuses dense and BM25 by default

With an embedder configured, rag query and rag ask now run both a dense KNN search and a BM25 search and fuse the two ranked lists, rather than searching dense alone. Fusion is reciprocal-rank by default, with a weighted rank-blend available; chunks are deduplicated on source#index, and ties break in a fixed first-seen order so the same corpus and query always produce the same output.

This matters because the two lanes fail differently: dense retrieval misses exact identifiers, error strings and flag names that a user copies verbatim into a question, and BM25 misses paraphrase. Fusing them covers both without asking the user to know which kind of question they are asking.

-no-hybrid restores pure dense KNN. Note the returned Score is the fusion score, not the original per-lane score — they are not comparable across modes.

A native-protocol Python client for KV and the vector database

Two things reachable from Go's remote client but not Python have both landed: the key-value store (never on REST — it lives only on the binary TCP protocol, built for sub-microsecond ops an HTTP round trip would defeat) and the same transport for vector operations. The new Rostam client speaks that protocol directly, standard library only:

from rostam import Rostam, filters

r = Rostam("127.0.0.1", 7000)          # the server's -tcp port

# key-value
r.put("user:42", b'{"coins":100}', ttl_ms=300_000)
r.incr("views:42", 1)                  # atomic; missing key counts as 0

# vector database, same connection
r.vector.create_collection("docs", dim=768, metric="cosine")
r.vector.upsert("docs", 1, embedding, content="hello", metadata={"tenant": "acme"})
r.vector.search("docs", query, k=10, filter=filters.eq("tenant", "acme"))

KV covers get/put/delete/incr/expire (plus a ping heartbeat); .vector covers create_collection, upsert, insert, search, get, delete and exists. auth_token= rides the protocol-v2 frame when the server requires one. RostamKV remains as an alias for the KV-focused name.

The vector arg layouts — including create_collection's fragile config trailer — are differential-tested byte-for-byte against the Go encoders (a Go oracle emits golden hex the Python encoders must reproduce); the JSON-carrying parts (metadata, filter, content) round-trip through a real server. Custom ops and WASM procedures stay Go-only (per-op encoders). Pairs with the container serving the TCP port by default.

The container serves the Go client out of the box

The published image bound only REST (:8080), but the Go remote client (rostam.NewClient) speaks the binary TCP protocol and nothing else — so a Go service could not reach the container without overriding its command. The default now binds REST and TCP (:7000) and documents both with EXPOSE.

Nothing new is published to the host by default: a bound port is not a published one, so docker run -p 8080:8080 still maps only REST to the host, and a Go deployment adds -p 7000:7000. The token guards every transport identically — the startup gate refuses an unauthenticated non-loopback bind on any of them — so publishing 7000 is no less safe than 8080. gRPC stays opt-in (-grpc 0.0.0.0:9090).

The Python client reaches the Query API

rostam-client gains query(), recommend() and discover(). Recommend and discover have no standalone route on any transport — they are leaves of the composable Query API — so they were previously unreachable from Python without hand-building the request. recommend(collection, positive=[...], k=...) scores toward example ids (and away from negative ones); discover(...) guides a search with (positive, negative) context pairs and an optional anchor; both take an optional metadata filter. query() is the general form for multi-lane fusion or rerank.

The Go library defaults HNSW parameters

vector.NewCollection used to reject a config that left M, EfConstruction or EfSearch at zero, even though Config.M documents "default 16" and the HTTP and Python layers fill exactly those. The same create therefore succeeded over the wire and failed from the Go library — including the quickstart's embedded example. The engine now fills the standard 16 / 200 / 64 when they are left zero (or any non-positive value), matching the other entry points. Only the HNSW (default) index type is affected; IVF still requires and validates its parameters, and Config.Validate() is unchanged.

The MCP server explains itself to the agent

Tool schemas say what a tool does; they do not say when to reach for it. So an agent with memory available would still re-read a file it had already summarised. The server now returns MCP's top-level instructions from initialize — clients inject it into the model's context — and the remember and recall descriptions carry the same guidance at the point of use.

The doctrine is short and deliberate: recall at the start of a task before re-exploring a codebase, namespace per project rather than default, one self-contained durable fact per remember, and never store secrets.

Memory keeps one entry per key for live state

remember now takes an optional key. Without one, nothing changes — a memory is still identified by its (namespace, content), so re-remembering an edited fact leaves the old version behind. With a key, the memory is identified by (namespace, key) instead, so re-remembering the same key replaces the prior entry in place. That makes it the right shape for live, in-flight state — a PR's status, what you're mid-task on — which otherwise piles up as stale snapshots that a later agent can't tell apart from the current one.

recall and list_memories now surface each memory's key and its created and updated times, so a reader can spot a keyed live-state entry and judge its freshness at a glance; a keyed memory keeps its original created across updates while updated moves to now. forget can delete by key as well as by id (pass namespace + keys). The MCP instructions and the remember description now teach the pattern: use a key for live state, omit it for durable facts.

Fixed: the MCP server reported the wrong version

initialize answered with a hardcoded 0.1.0, so a v0.2.0 binary introduced itself to Claude Code, Cursor and every other client as 0.1.0 — and every bug report filed through an MCP client named the wrong release. It now reports the binary's real version, derived from the same source -version uses.

v0.2.0 — 2026-08-15

The MCP server: agent memory with nothing to set up

rostam-server mcp runs Rostam as a Model Context Protocol server over stdio, giving Claude Code, Claude Desktop, Cursor or any MCP client persistent agent memory and generic vector-DB tools:

claude mcp add rostam -- rostam-server mcp

There is no daemon and no account. The process embeds the engine directly and persists to ~/.rostam/memory, so the agent has durable memory seconds after the command above.

It works with no embedder at all. remember and recall run on the built-in BM25 index, so there is no API key and no external service in the default path — the usual reason an agent-memory tool cannot be tried in one command. Pointing it at any OpenAI-compatible /embeddings endpoint (OpenAI, Azure, Ollama, LM Studio, TEI, LiteLLM) upgrades recall to hybrid dense+BM25: set ROSTAM_EMBED_ENDPOINT, ROSTAM_EMBED_MODEL and ROSTAM_EMBED_DIM together — the endpoint alone is a configuration error and refuses to start, naming the variable it is missing. The tools and their call shapes do not change; only how well results rank.

Five memory tools (remember, recall, forget, list_memories, list_namespaces) and four collection tools (create_collection, upsert, search, get) are always registered. delete and delete_by_filter are absent from tools/list unless -enable-destructive is passed, rather than present and refusing — a model cannot call a tool it cannot see.

-connect host:port runs the tools against a remote rostam-server over the binary TCP protocol instead of embedding the engine, with the usual token/mTLS options. Full tool reference and client config: MCP server.

The LLM caching proxy

rostam-server llm-proxy is an OpenAI-compatible caching reverse proxy: point an existing OpenAI SDK client at it instead of api.openai.com, and a chat completion that hits the cache is answered locally, at no generation cost and without the round trip. A cacheable request that misses is still forwarded and still costs what it always did — eligibility decides whether the cache is consulted, not whether you pay.

Anything the cache cannot answer is forwarded upstream verbatim: every other /v1/* route, and any chat request whose shape or scope makes it uncacheable. The two exceptions are handled locally rather than passed on — a chat body over the size limit gets a 413, and one that will not parse gets a 400.

Like the MCP server it is useful before you configure anything: with no embedder, byte-identical prompts are served from cache. With one, it upgrades to semantic matching, so a reworded or differently-whitespaced prompt may hit as well — subject to the same eligibility and cache-identity rules, and to the configured similarity threshold.

Chat responses carry an x-rostam-cache header (hit, miss, uncacheable, bypass) so it is visible which requests are still costing generation tokens. Generic passthrough routes such as /v1/models deliberately omit it — they were never cache candidates, and a verdict there would be noise. See LLM caching proxy.

Breaking: NaN no longer matches range filters

A point whose numeric payload field is NaN previously matched gte and lte against every bound, because the comparison classified "neither less nor greater" as equal. It now follows IEEE 754: a NaN operand makes the comparison unordered, so gt, gte, lt and lte are all false — for a NaN field value and for a NaN bound alike.

eq, ne and in are unchanged (they compare with ==, under which NaN was never equal to anything), and is_null/is_empty are unchanged (a NaN is a present, non-null value).

Who is affected: anyone whose payloads contain NaN, which in practice comes from a division by zero or a failed numeric parse upstream. Those points will stop appearing in gte/lte results.

What to do: write a real sentinel value rather than NaN if such points should remain matchable.

Why: the payload index never agreed with the old behaviour — it excluded NaN from every range posting list — so a filtered search could already return different rows depending on which query path the planner chose. Making NaN unordered is what lets the index and the predicate answer the same question, and it matches Go, Rust, Milvus and Qdrant. See Metadata filtering.

Breaking: three misconfigurations now refuse to start instead of serving

All three used to come up and serve traffic (one printed a warning):

• An open non-loopback bind with no authenticator. With neither -keys-file nor -api-key set, binding a non-loopback address is refused at startup, because every request would be served unauthenticated. Loopback-only binds — the dev default — are unaffected. Configure auth, bind 127.0.0.1, or pass the new -insecure flag to run open deliberately (dev/trusted networks only).
• Inter-node TLS without a CA. In cluster mode, -tls-cert without -tls-ca encrypted the replication listener while authenticating nobody: any peer that completed a TLS handshake could replicate writes. -tls-ca is now required whenever -tls-cert is set in cluster mode.
• Tenant-scoped keys with -tenant-isolation off. A key carrying a real tenant while isolation is off silently crosses tenant boundaries through glob scopes; the old startup warning is now a refusal. Enable -tenant-isolation, or mark deliberately cross-tenant keys with Tenant: "*".

See Security for the full model.

Breaking: WASM registration sends the module out of band

Store.RegisterWASM now takes the module bytes as a separate argument and returns a push report:

pushReport, err := store.RegisterWASM(ctx, reg, moduleBytes)

WASMRegistration no longer carries the bytes inline — the Bytes field is replaced by a content address computed for you from the module argument — and KeyExtractorHandle is gone: every WASM op uses the standard extractor, and the field's other value could make replicas silently diverge, so it cannot exist. Pass the bytes as the new argument and delete any KeyExtractorHandle assignment; the compiler will point at both sites.

Two behaviour changes ride along. Updating a live module's bytes is now supported on replicated stores — the version a shard group executes is bound to that group's own log, which is what made updates unsafe before — while changing a registered op's Kind is still refused; use a new op name for that. And the push report is part of the result, not diagnostics: when non-empty it names the cluster members that did not acknowledge the module bytes and will have to fetch them on demand.

Range filters are no longer a throughput cliff

A range predicate over a high-cardinality field (id >= N and friends) used to cost a large multiple of the same search unfiltered, most of it spent building a candidate set the planner then discarded. Filtered throughput is now close to unfiltered for high-pass-rate ranges, with no configuration change and no API change. Two new counters expose which acceleration a filtered search used: filter_column_gates_total and filter_complement_gates_total, alongside the existing filter_gates_total.

Numeric range filters may now build a column sidecar — one float64 per slot per range-queried field, at most eight fields at a time, least-recently-used evicted beyond that. It is built lazily on the first range query over a field.

The sidecar counts against MaxBytes, and writes always win: an insert that would otherwise be refused reclaims the sidecar and proceeds, so query traffic can never make a collection permanently reject writes. ErrCollectionFull continues to mean what it always meant — the collection's own durable data has filled its budget. The filter_column_drops_total counter reports how often a write had to reclaim columns; a value that climbs steadily means reads and writes are fighting over the same bytes and MaxBytes wants raising. In the other direction, a search on a collection with no byte headroom simply skips the sidecar and uses the slower path.

On low-dimensional collections the sidecar is a meaningful fraction of memory — a full set of columns is a quarter of the vector data at 64 dimensions and twice it at 8 — so size MaxBytes with that in mind.

Filtered searches spend less time re-checking the filter

When a filtered search cannot brute-force the candidate set the payload index narrowed, it falls back to graph traversal — and that traversal used to re-derive each candidate's filter membership from scratch, a fact the index had already established. It now folds the same narrowing plan into a membership bitset and consults one bit per candidate instead. No API change, no configuration, and no change to which points a filter matches: it is the same plan, read a cheaper way.

Binary query wire

/points/search and /points/search/docs now accept a dense binary body, selected by Content-Type: application/octet-stream — the ingest wire's counterpart on the read side. The query vector travels as raw f32 instead of base-10 text, which it turns out is a large share of a small request: at dim=768, k=10 on a kept-alive connection, building the JSON body measured 0.258 ms of a 0.845 ms search, against 0.011 ms to write the same vector as bytes. The server's matching decode of those literals goes with it.

That is the mechanism; the effect on a real corpus was measured separately, on a dedicated 12-core EPYC Genoa server against Cohere-1M at k=100, m=16, ef_construction=200, ef_search=300. Three same-session pairs, each loading the corpus once and then running both wires against that one index, gave 2,718 / 2,739 / 2,777 QPS over JSON against 3,163 / 3,178 / 3,277 over the binary wire — +16.8% — with single-client p99 falling from 7.8–7.9 ms to 6.2–6.5 ms. One of the three pairs reverses which arm loads the corpus and which inherits the warm index, and the advantage does not move, which is what distinguishes a transport effect from a page-cache one. Recall is identical within every pair to four decimals: the binary path returns the same answers, sooner.

Adds no semantics, on the same terms as the bulk wire: a binary body decodes into exactly the request its JSON body produces, then runs the identical validation and dispatch, and any other content type takes the JSON path unchanged. The framing, its limits (dim ≤ 65,536, filter ≤ 1 MiB, NaN/Inf refused) and its error behaviour are in Binary query body.

Clients need no flag day. A server that predates this answers a binary body with 400 invalid JSON body: ..., which is specific enough to fall back on; the Python client does so automatically and permanently.

The Python client reuses connections

rostam-client opened a new TCP connection per request. It now keeps a small pool, and is still safe to share between threads. Combined with the binary query wire, repeated searches measured 724–990/s before and 1911–3179/s after at dim=768 against the same server — non-overlapping ranges, on a laptop.

That figure is the two changes together and cannot be divided between them; it is also a laptop measuring a small index, where per-request overhead dominates in a way it does not at scale. For the query wire on its own, on server hardware and a 1M corpus, see Binary query wire above.

Reads are retried once when a pooled connection turns out to have been closed while idle; writes are not, because that failure surfaces after the request is on the wire and a replayed insert is worse than an error. close() releases the pool. Note that http.client does not consult HTTP_PROXY/HTTPS_PROXY the way the previous urllib call did.

Binary bulk-ingest wire

/points/bulk and /points/batch now accept a dense binary body, selected by Content-Type: application/octet-stream, carrying vectors as raw f32 rather than as base-10 JSON text. On a large initial load the JSON encode/decode — not the index build — was the larger cost: a 1M × 768d load measured locally at roughly 1043 s dropped to roughly 508 s.

This adds no semantics. Each route decodes a binary body into exactly the request its JSON body produces and then runs the identical code, so results are unaffected, and any other content type takes the JSON path unchanged. The framing, its limits, and its error behaviour are in Binary bulk ingest.

Bulk ingest carries payloads, so filtered workloads get the fast path

A load that needed metadata on each point previously had exactly one route: /points/batch, one indexed insert per point. The staging path carried ids and vectors only, so every filtered workload was pushed off the multi-core bulk build and onto the slow inline route. /points/bulk now accepts per-point payloads and stages them for the same concurrent build. Measured locally at 100k × 768d with a single scalar payload per point, ingest went from 760 to 4915 vectors/s (6.46×); carrying the payloads costs nothing measurable against a vectors-only staged load, because they ride a pass that was already visiting every slot.

Content and sparse vectors still have no bulk representation. The staging route now rejects a point carrying content, sparse, ttl_ms, key_ttl_ms or expected_version with a 400 rather than staging it with those fields quietly dropped.

Queries no longer stall while a collection grows

A collection's per-slot arrays used to be reallocated and copied as it grew, and a query arriving at a growth boundary waited for the copy. They now grow in place on a reserved address range, so nothing moves and nothing waits. The effect on worst-case latency, the virtual-versus-resident memory consequence, and the platforms that fall back to the old behaviour are covered in Growing a collection doesn't stall queries.

Cluster backups and one-shot restore

-backup-dir/-backup-interval now cover -cluster deployments: each node streams per-shard artifacts plus the meta catalog, and a fresh same-topology cluster is restored by starting every node once with -restore. A restore fails loud on a topology mismatch or a missing shard artifact rather than bringing a shard up empty. See Backups.

A readiness probe that reflects shard leadership

GET /v1/ready now answers from cluster state — shard leadership backed by per-shard replication-lag and ISR-health metrics — rather than from mere process liveness, so a load balancer stops routing to a node that cannot actually serve. /v1/health remains the liveness check. Both endpoints are auth-exempt, since an infrastructure probe carries no token.

Structured logs, request ids, and an opt-in access log

-log-format json switches server logs to one JSON object per line (text, the default, keeps the historical stderr format); -log-level sets the floor. -access-log emits one structured line per request on every transport — HTTP, gRPC and TCP — with a request id (an inbound X-Request-Id is reused), the op, status, latency, bytes, and a redacted principal; raw tokens are never logged. Both new logs are off by default and cost nothing on the hot path when off.

Experimental: primary-backup replication mode

-replication-mode=pb selects a primary-backup/ISR replication engine for every shard in place of per-shard Raft; the default (raft) is byte-identical to previous behaviour. PB mode requires -min-isr and a per-node PB address, promotes a verified ISR survivor automatically when a primary dies (-pb-auto-failover, default on within pb mode), and offers -pb-commit-primary to trade acked-write durability for lower write latency. Explicitly experimental — see shard/pbisr/BENCHMARK.md for the measured comparison it must clear.

Fixed: a point stored under id 0 was never returned by search

A point inserted with id 0 was stored, live, returned by Get, and counted by the live-count metric — but no search of any kind returned it, on any index type. A collection holding exactly one point, id 0, answered its own vector with an empty result set. Search now returns it like any other point.

The same absent-versus-zero confusion reached two /query discover cases, both also fixed. A discover query with "target": 0 silently lost its anchor and was answered from the mean of its context positives instead of erroring; it now anchors on point 0 as written. And a context pair specifying one side as a vector and the other as an id — for example {"positive_vec": [...], "negative": 5} — used to be accepted and searched against an anchor the caller never named. A pair must now be wholly the id form or wholly the vector form; a mixed pair is rejected with a 400. If you were sending a half-specified pair, it was not doing what it appeared to do, and it now fails loudly.

On CUDA builds (-tags cuda), the same slot guard ran in the GPU exact-scan lanes with a second consequence: a slot freed by a delete keeps a stale id, which the old check waved through, so a deleted point could resurface in GPU search results. The same fix covers both, and is verified on real CUDA hardware.

Fixed: a filter on $content returned no rows

Any filter naming the $content field returned zero results, even when every point matched — for example And(Eq("tag", "a"), Match("$content", "quick")) over a set in which all points satisfy both clauses. $content is deliberately not indexed, and the query planner read "no index entries" as "no matching points" and returned early without ever evaluating the predicate. The predicate is now consulted, and such filters return the rows they always should have. This affected every filter operator on $content, not just text match.

Fixed: points could become permanently unsearchable under several workload shapes

Under each of these shapes, a handful of points ended up in the collection — byte-correct, live, returned by Get and by scans — yet reachable by no search at any ef_search:

• The default multi-worker bulk build, and upserts overlapping a reshard, could link a point against a neighbour that had no usable edges yet, leaving it with no path back from the entry point.
• An insert arriving when everything reachable was dead — every candidate the graph traversal could see being deleted or expired, a reshard's ordinary lazy-delete band and a natural state of any delete-heavy collection — wrote an empty edge set for the new point. Worse than orphaning that one point: if it drew a taller level it promoted itself to entry point, severing the whole live graph behind it. Linking now falls back to structural neighbours (still-traversable tombstoned slots) rather than accepting emptiness.
• Repeated upsert sweeps at low graph degree (m of about 4–6) fed each reused slot self-loops and one-way upper edges and discarded the edges the slot inherited, thinning the graph below its configured degree with every sweep until points fell off it.

All are fixed, and a standing invariant test now requires every live point to be findable by its own vector.

If you loaded a large collection through the bulk build, ran upsert churn, or deleted heavily before these fixes, a small number of points may still be missing from your search results. The stored data is intact — Get and scans return those points — but only a re-load on a fixed build puts them back in the graph.

Fixed: a malformed bulk-ingest body could reserve memory before being rejected

The bulk-ingest routes sized their allocations from counts the request itself declared, before the bytes backing them arrived — and on the JSON path the encoder ran ahead of the authorization check, so an anonymous caller could trigger the allocation and only then be told 401. Worse, the JSON path sized its buffer from the point count and the first vector's length, two numbers the body chooses independently: a measured 289 KB anonymous request allocated 1.61 GB, and the same shape scaled within the body cap reached an allocation no process survives. Requests are now authorized before anything is sized by the body, every declared count and length is bounded before it sizes anything, and each section of a binary body is read in windows that grow only as bytes actually arrive. A body that over-declares its size is refused having consumed only what it really sent.

Two related input errors are now rejected at the edge rather than deeper in: a bulk-stage batch whose vectors are not all the same length, and a bulk body whose payload section ends early. Neither was merely an error before. A ragged batch whose short vector sat mid-batch shifted every following row during encoding, so ids were reconstructed out of vector bytes — points stored under ids nobody sent, with no error anywhere. And the early-ending payload section mattered most on a cluster, where such a body could be replicated before it was decoded and then bring down every node that applied it.

Fixed: an upsert into an over-budget collection could resurrect the point it replaced

An upsert reclaims the dead slot of the id it replaces — free it, then immediately reuse it — and both quota checks sat between those two steps. An upsert refused for being over MaxVectors/MaxBytes had therefore already freed the slot and then abandoned it, with the old id, vector and payload still in place and still reachable through the graph's in-edges: a search could return the deleted point, scored against its stale vector and carrying its old payload. A collection over its budget is reachable in the first place because the bulk load does not consult the quotas.

The quota verdict now precedes the reclaim, judged against the accounting the reclaim is about to produce — so an upsert into a merely full collection is a replace and still succeeds, while a rejected upsert mutates nothing at all.

Fixed: loading a snapshot silently dropped most of the collection's config

The snapshot header carries six config fields (dimension, metric, m, ef_construction, ef_search, seed). Reading a snapshot rebuilt the live config from those six and zeroed everything else, while Config() kept reporting the original values — so nothing looked wrong. Every path that loads a snapshot was affected: a restart with persistence enabled, a backup restore, and a Raft snapshot install. After any of them, MaxVectors/MaxBytes quotas were silently lifted, every subsequent insert wrote half the intended level-0 graph degree, a quantized build silently switched to exact-float navigation, and the max_ef_search request ceiling vanished. The stored data was never harmed; the config now survives the round trip, and only the six header-carried fields are taken from the stream.

Fixed: TTL deadlines could differ between replicas

In a cluster, each replica applying a write stamped TTL deadlines from its own clock, so replicas at skewed clocks stored different absolute deadlines for the same committed write — and could then disagree about whether a point was live: different search results, and a different insert-if-absent outcome, depending on the replica asked. Every clock-dependent decision in the apply path — point TTLs, per-key payload TTLs, expiry-aware liveness for CAS and reclaim — now uses a single leader-assigned stamp carried with the entry, across dense, multi-vector and named collections. Single-node behaviour is unchanged.

Fixed: a deleted key could come back after a warm restart

In the KV cache, Del tombstoned only the in-memory index; nothing was recorded on the page, so a warm restart rebuilt the index straight off the bytes and the deleted key returned — and in a cluster the restarted node then disagreed with its peers forever, because nothing re-applies a delete below the applied index. Deletes now append a tombstone entry that outranks every earlier copy of the key in the rebuild.

Fixed: a handler panic no longer takes down the whole process

There was no recover() anywhere in the op path, so a panic in any request handler — reachable from a malformed frame through the argument decoders — crashed the entire node: every shard, every connection. Every transport (HTTP, gRPC, TCP) now recovers handler panics into an internal error response plus a server-side log with the stack; the panic detail never reaches the client. The same hardening pass made the -api-key comparison constant-time and put body-size, idle and write timeouts on the HTTP server (the long-running bulk-build route is exempted from the write deadline, which otherwise cut it off mid-build on any large corpus).