Skip to content

Migrating from Pinecone to Rostam

This guide maps Pinecone concepts and API calls to Rostam so you can move an existing workload over. Rostam is open source (Apache-2.0) and self-hosted — one Go binary you run in your own environment — so the main reasons teams make this move are cost control and keeping vector data inside a boundary you own (data residency, on-prem, air-gapped). There is no per-project minimum, and Rostam makes no outbound calls on its own — data leaves only through egress you explicitly configure, which you control.

The two places data can leave — both under your control

  1. Backups / cold tier. If you configure S3 backups or a cold tier (Backups), Rostam writes snapshots to that object store. Point it at an in-boundary store (e.g. MinIO) to keep everything inside your network, or leave it unconfigured.
  2. Embedding. Embedding is a separate step from storage. If you generate embeddings with a hosted API (OpenAI, Cohere, …), your text leaves your network there regardless of where the vectors live. Embed with a local/self-hosted model for a fully in-boundary pipeline.

Neither happens unless you set it up — Rostam initiates no egress by default.

If you just want the API side by side, jump to Code, side by side.

Concept mapping

Pinecone Rostam Notes
Account / project Your Rostam server (or cluster) You run it; no hosted control plane.
Index Collection create_collection(name, dim, metric).
Metric cosine / euclidean / dotproduct metric="cosine" / "l2" / "dot" Set per collection at creation.
Vector (id, values, metadata) Point (id, embedding, metadata, optional content) Rostam ids are integers; see IDs.
Namespace Tenant (<tenant>/<collection>) A real primitive — and an optional security boundary. See Namespaces.
Metadata filter ($eq, $in, …) filters helpers (f.eq, f.in_, …) Full translation table below.
Serverless / pods The single binary, or a Raft cluster Scale by running more shards/nodes; see Clustering.
Managed, multi-tenant SaaS You operate it See Security and Backups.

Run Rostam

# one static binary; data stays on disk you control.
# pass the token via the environment, NOT -api-key (a flag secret leaks via /proc and shell history):
export ROSTAM_API_KEY="a-strong-token"
rostam-server -http :8080 -data ./data

The server refuses to bind a reachable address without auth. See Running the server for TLS, clustering, and Docker.

Install the Python client:

pip install rostam-client

Code, side by side

Connect

# Pinecone
from pinecone import Pinecone
pc = Pinecone(api_key="...")
index = pc.Index("docs")

# Rostam
from rostam import Rostam, filters as f
c = Rostam("http://localhost:8080", api_key="a-strong-token")

Create the index / collection

# Pinecone
pc.create_index(name="docs", dimension=384, metric="cosine", spec=...)

# Rostam
c.create_collection("docs", dim=384, metric="cosine")   # metric: cosine | l2 | dot

Upsert

# Pinecone
index.upsert(vectors=[
    {"id": "1", "values": embedding, "metadata": {"doc_id": 7, "lang": "en"}},
])

# Rostam  (content is an optional stored text payload returned with hits)
c.upsert("docs", 1, embedding, content="the chunk text",
         metadata={"doc_id": 7, "lang": "en"})

Query

# Pinecone
res = index.query(vector=embedding, top_k=5,
                  filter={"doc_id": {"$eq": 7}}, include_metadata=True)
for m in res["matches"]:
    print(m["id"], m["score"], m["metadata"])

# Rostam  (hits carry .id / .content / .metadata / .distance — smaller distance = closer)
hits = c.search_docs("docs", embedding, k=5, filter=f.eq("doc_id", 7))
for h in hits:
    print(h.id, h.distance, h.metadata)

Fetch and delete

# Pinecone
index.fetch(ids=["1"])
index.delete(ids=["1"])

# Rostam
c.get("docs", 1)      # -> Point | None
c.delete("docs", 1)

Translating metadata filters

Pinecone filters are dicts of operators; Rostam uses the filters helpers (from rostam import filters as f). They compose the same way.

Pinecone Rostam
{"g": {"$eq": "x"}} f.eq("g", "x")
{"g": {"$ne": "x"}} f.ne("g", "x")
{"n": {"$gt": 3}} f.gt("n", 3)
{"n": {"$gte": 3}} f.gte("n", 3)
{"n": {"$lt": 3}} f.lt("n", 3)
{"n": {"$lte": 3}} f.lte("n", 3)
{"g": {"$in": ["a","b"]}} f.in_("g", ["a", "b"])
{"g": {"$nin": ["a","b"]}} f.not_(f.in_("g", ["a", "b"]))
{"$and": [A, B]} f.and_(A, B)
{"$or": [A, B]} f.or_(A, B)
{"g": {"$exists": false}} {"op": "is_empty", "field": "g"} (raw dict)
{"g": {"$exists": true}} f.not_({"op": "is_empty", "field": "g"})
implicit {"a": 1, "b": 2} (AND) f.and_(f.eq("a", 1), f.eq("b", 2))

Rostam has is_empty and is_null predicates for this — the Python filters helpers just don't expose builders for them, so pass the raw predicate dict (a filter is a plain dict). is_empty matches a field that is absent, null, "", or an empty array; is_null matches a field that is present and explicitly null. Note the semantic gap from Pinecone's $exists, which tests key presence alone: is_empty treats a present-but-empty value as "empty" too. If you need exact key-presence semantics, a boolean sentinel field (has_x) written at upsert time is still the precise option.

Rostam runs filters through an exact, filter-first path — a selective filter does not degrade recall. See Filtering.

Namespaces → tenants

Pinecone namespaces partition one index. Rostam's equivalent is a tenant: collection names can be written <tenant>/<collection> (a bare name lands in the default tenant). Unlike a Pinecone namespace — which is only a partition — a Rostam tenant can also be an authoritative security boundary: bind an API key to a tenant and run the server with -tenant-isolation, and that key can only see its own tenant's collections.

# Pinecone: index.query(vector=v, top_k=5, namespace="user-42")
# Rostam:   the namespace becomes the tenant prefix
c.search_docs("user-42/docs", v, k=5)

Alternatives when you do not need isolation: store the namespace as a metadata field and add f.eq("ns", "user-42") to every query, or use a separate collection per namespace. See Collections, tenants & aliases.

IDs: string vs. integer

Pinecone ids are strings; Rostam point ids are integers (uint64). Map strings deterministically with the client's helper:

from rostam._ids import to_uint64   # stable str -> uint64; used by all the framework adapters
c.upsert("docs", to_uint64("abc-123"), embedding, metadata={"pinecone_id": "abc-123"})

to_uint64 is one-way. Neither the raw client nor TextStore preserves the original string for you — if you need it back on read, store it in metadata yourself (as above) and read it from hit.metadata["pinecone_id"]. Pick a key your own metadata doesn't already use.

Migrating your data

There is no import tool — you re-upsert. If you still have the source embeddings, upsert them straight into Rostam. If not, read them out of Pinecone in pages and write them across, keeping the original id in metadata:

from rostam._ids import to_uint64

for ids in paginate_pinecone_ids(index):          # your paging over index.list()
    fetched = index.fetch(ids=ids)["vectors"]
    for pid, v in fetched.items():
        meta = dict(v.get("metadata") or {})
        meta["pinecone_id"] = pid                  # preserve the original id for lookup
        c.upsert("docs", to_uint64(pid), v["values"], metadata=meta)

Re-embedding from source documents is often cleaner than exporting vectors, and it lets you switch embedding models at the same time. Rostam can also embed for you — see TextStore and the built-in embedders in the Python client (note that a hosted embedder sends text out at the embedding step; see the egress caveat above).

What is different (read before you commit)

  • You operate it. No managed control plane — you run the server, back it up (Backups), and secure it (Security). That is the point (your data, your boundary), but it is real work.
  • Integer ids — map strings with to_uint64, keep the original in metadata (see above).
  • Distance, not score — hits carry distance (smaller = closer), not a Pinecone-style similarity score. Convert if your app expects scores.
  • No server-side embedding — generate embeddings before you upsert (see above). ($exists is covered — it maps to the is_empty/is_null predicates in Translating metadata filters.)

Next steps