Migrating from Pinecone to turbopuffer: mapping namespaces, metadata and cutover
How to move a Pinecone vector index to turbopuffer: map each Pinecone namespace to a turbopuffer namespace, convert ids, metadata and score thresholds, backfill with batched retried writes, then cut over per namespace.
A Pinecone to turbopuffer migration maps each Pinecone index and namespace pair to one turbopuffer namespace, and copies records by listing ids and fetching them in batches of 1,000, then writing those batches as columns. Do it in this order: turn on dual writes first, backfill with insert-only conditional writes, reconcile ids, compare results with shadow reads, and only then move reads over, one namespace at a time. Keep Pinecone current until the rollback window closes.
Neither vendor ships a tool for this. The turbopuffer pricing FAQ says "We don't currently offer migration tools", and offers Solutions Engineering help to Enterprise customers. Pinecone's docs describe backups that restore into another Pinecone index, and we found no documented export to files, so the way out is the data plane API. Everything below was checked against the docs of both vendors as of October 2026. We had no account on either, so the script is written against pinecone 10.2.0 and turbopuffer 3.1.0 from PyPI and labelled illustrative. We installed both SDKs to check method signatures, and ran the pure-Python parts (mapping, batching, retries, threshold and filter conversion) locally with tests. If you are still deciding between the two engines, the turbopuffer vs Pinecone comparison covers that. This post covers the move.
Pinecone to turbopuffer concept map
Pinecone has two levels, an index that fixes dimension and metric, and namespaces inside it. On the turbopuffer side there is one level: a namespace owns its own schema and distance metric, and names must match [A-Za-z0-9-_.]{1,128} per the write docs. Encode the Pinecone index in the namespace name, as docs.tenant-a.
| Pinecone | turbopuffer | What to do |
|---|---|---|
| Index (one dimension, one metric) | Namespace prefix | Name namespaces <index>.<namespace> |
Namespace, up to 512 ASCII chars, default __default__ | Namespace, [A-Za-z0-9-_.]{1,128} | Replace other characters, hash names that are too long |
| Record id, up to 512 chars | u64, UUID, or string up to 64 bytes | Hash longer ids, keep the original as an attribute |
Dense values, up to 20,000 dims | [N]f32 or [N]f16 with ann: true, up to 10,752 dims | Check dimensions before you start |
sparse_values, up to 2,048 non-zero values | {}f16 with string keys and sparse_knn, up to 1,024 dims per vector | Lossy above 1,024, see below |
Metric cosine, euclidean or dotproduct | cosine_distance or euclidean_squared | Dot product needs unit vectors |
| Metadata: string, number (float64), boolean, list of strings | string, int, float, bool, []string, plus uuid and datetime | Declare types in the schema |
list (serverless only) plus fetch, 1,000 ids | Query paged by id, limit 10,000 | Used for export and reconciliation |
| Upsert, 1,000 records or 2 MB | Write, up to 512 MB per request | Bigger batches cost less |
The Pinecone figures come from its operation limits and identifier limits pages, and the turbopuffer ones from its limits page, all as of October 2026.
Most of the risk sits in a few rows. Ids first: Pinecone allows 512 characters and turbopuffer allows 64 bytes, so structured ids like s3://bucket/path/report.pdf#chunk12 can overflow. The script hashes those with SHA-256, whose hex digest is exactly 64 bytes, and stores the original in a non-filterable pinecone_id attribute. If you list chunks by id prefix in Pinecone, store document_id as a filterable attribute and filter on it instead.
Sparse vectors come next. Pinecone indices are integers and turbopuffer keys are strings, which is a cheap conversion. The limit is not. A Pinecone sparse vector can hold 2,048 non-zero values and a turbopuffer one holds 1,024, stored as f16. The script keeps the 1,024 largest weights by magnitude and counts what it drops, so you know whether this affects you before anyone queries. If your sparse vectors were BM25-style term weights, consider replacing them with native BM25, which our post on turbopuffer hybrid search with BM25 and vectors sets up.
Then metadata typing. The turbopuffer write API infers a type from the first value it sees and rejects later values of another type, and Pinecone stores every number as a float64. Scan a sample, decide int or float per field, and send an explicit schema with the first write. Layout questions such as one namespace per tenant are covered in turbopuffer namespace design, and the same rules apply to the namespaces you create here.
The migration order
flowchart TD
A["Map schema<br/>ids, types, metric"] --> B["Dual writes on<br/>app writes both engines"]
B --> C["Backfill<br/>list + fetch, insert-only writes"]
C --> D["Reconcile<br/>id sets and deletes"]
D --> E["Shadow reads<br/>overlap at 10, labelled queries"]
E --> F["Move reads<br/>one namespace at a time"]
F --> G["Rollback window<br/>dual writes still on"]
G --> H["Stop Pinecone writes"]
G -.->|regression| R["Flip reads back<br/>Pinecone is current"]
R --> EDual writes go first because of a race. A backfill reads a record at one moment and writes it a little later, and if the application updated that record in between, a plain upsert writes the stale copy over the new one. Conditional writes in turbopuffer close that gap. With upsert_condition: ["id", "Eq", null], the write docs say a write applies when the document does not exist and is skipped when it does, so anything the live dual write has already stored wins.
sequenceDiagram
participant B as Backfill
participant P as Pinecone
participant A as App dual write
participant T as turbopuffer
B->>P: fetch doc-7, gets v1
A->>P: upsert doc-7 v2
A->>T: upsert doc-7 v2
B->>T: upsert doc-7 v1 if id is null
Note over T: doc-7 exists, v1 skipped, v2 keptThere are two costs. The write docs say a conditional write "is billed as both a query and a write operation", with one query per batch, and that skipped documents are not billed as writes. Conditional writes also cannot use disable_backpressure, so once a namespace has more than 2 GB or 1 million rows waiting to be indexed, writes get HTTP 429 until indexing catches up, and the backoff below handles that. If you can pause application writes for the whole backfill instead, plain upserts with disable_backpressure: true avoid the query charge. You then lose the rollback safety of a Pinecone index that keeps receiving writes.
Deletes remain a gap. A record deleted in both engines after the backfill fetched it gets re-inserted. The reconcile step catches that.
Exporting from Pinecone with list and fetch
Pinecone's list returns up to 100 ids per page and works only on serverless indexes, according to its list ids guide. fetch takes up to 1,000 ids and returns each record's vectors and metadata. As of October 2026 the rate limits page caps list at 200 requests per second per index and fetch at 100 per second per index, returns 429 - TOO_MANY_REQUESTS above them, and says both consume read units. Starter is capped at 1,000,000 read units per month per organization, so check your plan before you export a large index. On a pod-based index, list is not available and the ids have to come from your source system. Indexes that use Pinecone's document schema use a separate Documents API, which this post does not cover.
The helpers below are the parts we ran locally.
# Illustrative: written to pinecone 10.2.0 and turbopuffer 3.1.0, not run against live accounts.
# The pure-Python helpers (mapping, batching, backoff, thresholds) are tested locally.
import hashlib
import math
import random
import re
import time
TP_ID_MAX_BYTES = 64 # turbopuffer: string ids up to 64 bytes
TP_SPARSE_MAX_DIMS = 1024 # turbopuffer: max dimensions per sparse vector
RETRYABLE = {408, 409, 429, 500, 502, 503, 504}
def tp_namespace(index_name: str, pc_namespace: str) -> str:
"""Pinecone index + namespace -> one turbopuffer namespace name, [A-Za-z0-9-_.]{1,128}."""
ns = pc_namespace if pc_namespace not in ("", "__default__") else "default"
name = re.sub(r"[^A-Za-z0-9\-_.]", "-", f"{index_name}.{ns}")
if len(name) > 128: # keep it unique and stable
name = name[:111] + "." + hashlib.sha256(name.encode()).hexdigest()[:16]
return name
def tp_id(pc_id: str) -> str:
"""Pinecone ids go up to 512 chars; hash the ones turbopuffer cannot take."""
if len(pc_id.encode()) <= TP_ID_MAX_BYTES:
return pc_id
return hashlib.sha256(pc_id.encode()).hexdigest() # 64 hex chars = 64 bytes
def infer_schema(sample_metadata: list[dict]) -> dict:
"""Pinecone metadata (string, float64 number, bool, list of strings) -> turbopuffer types."""
seen: dict[str, set] = {}
for meta in sample_metadata:
for key, v in (meta or {}).items():
if isinstance(v, bool):
t = "bool"
elif isinstance(v, (int, float)):
t = "int" if float(v).is_integer() and abs(v) < 2**53 else "float"
elif isinstance(v, list):
if not v:
continue # empty list says nothing about the type
t = "[]string"
else:
t = "string"
seen.setdefault(key, set()).add(t)
schema = {}
for key, types in seen.items():
if types == {"int", "float"}:
types = {"float"}
if len(types) > 1:
raise TypeError(f"metadata field {key!r} has mixed types {sorted(types)}")
schema[key] = types.pop()
return schema
def map_metadata(meta: dict, schema: dict) -> dict:
out = {}
for key, v in (meta or {}).items():
t = schema.get(key)
if t == "int":
if not float(v).is_integer():
raise ValueError(f"{key!r}={v!r} is not integral; set it to float in the schema")
v = int(v)
elif t == "float":
v = float(v)
out[key] = v
return out
def map_sparse(indices: list[int], values: list[float]) -> tuple[dict, int]:
"""Pinecone {indices, values} -> turbopuffer {}f16 with string keys.
Keeps the TP_SPARSE_MAX_DIMS largest weights and returns how many it dropped."""
pairs = sorted(zip(indices, values), key=lambda p: -abs(p[1]))
kept = pairs[:TP_SPARSE_MAX_DIMS]
return {str(i): float(w) for i, w in kept}, len(pairs) - len(kept)
def batches(items, size: int):
batch = []
for item in items:
batch.append(item)
if len(batch) == size:
yield batch
batch = []
if batch:
yield batch
def with_backoff(fn, *, attempts=6, base=0.5, cap=30.0, sleep=time.sleep):
"""Retry fn() on throttling, timeouts and 5xx with full-jitter exponential backoff."""
for attempt in range(attempts):
try:
return fn()
except Exception as e: # both SDKs expose status_code on HTTP errors
status = getattr(e, "status_code", None)
transient = status in RETRYABLE or (
status is None and any(w in type(e).__name__ for w in ("Connection", "Timeout")))
if not transient or attempt == attempts - 1:
raise
sleep(random.uniform(0, min(cap, base * 2**attempt)))
Retry status 409 deliberately. The turbopuffer Python SDK retries 408, 409, 429 and 5xx responses four times by default, and the script turns that off with max_retries=0 so one policy governs both vendors. Both SDKs' HTTP errors carry status_code, which we confirmed in the installed packages. Retrying any batch is safe because writes are keyed by id.
Batched writes into turbopuffer
The write path accumulates ten list pages, fetches the 1,000 ids, converts them to columns and writes them as one request. The write docs recommend the column layout for bulk loads, and the performance guide suggests several processes writing in parallel, because Python serialization becomes the bottleneck. Run one process per Pinecone namespace.
def to_columns(vectors: dict, schema: dict) -> tuple[dict, int]:
cols = {"id": [], "vector": [], "pinecone_id": []}
sparse, dropped = [], 0
for pc_id, v in vectors.items():
cols["id"].append(tp_id(pc_id))
cols["pinecone_id"].append(pc_id)
cols["vector"].append(list(v.values))
if v.sparse_values is not None:
s, d = map_sparse(v.sparse_values.indices, v.sparse_values.values)
sparse.append(s)
dropped += d
else:
sparse.append(None)
meta = map_metadata(v.metadata, schema)
for key in schema:
cols.setdefault(key, []).append(meta.get(key))
if any(s is not None for s in sparse):
cols["sparse"] = sparse
return cols, dropped
def migrate_namespace(index, tpuf, index_name, pc_ns, dims, metric, schema, *,
start_token=None, fts_field=None, has_sparse=False):
"""Copy one Pinecone namespace into one turbopuffer namespace. Re-runnable."""
tp_schema = {"vector": {"type": f"[{dims}]f32", "ann": True},
"pinecone_id": {"type": "string", "filterable": False},
**{k: {"type": t} for k, t in schema.items()}}
if has_sparse:
tp_schema["sparse"] = {"type": "{}f16", "sparse_knn": {"distance_metric": "dot_product"}}
if fts_field: # e.g. chunk_text: index it for BM25 now
tp_schema[fts_field] = {"type": "string", "full_text_search": True}
ns = tpuf.namespace(tp_namespace(index_name, pc_ns))
token, ids = start_token, []
while True:
page = with_backoff(lambda: index.list_paginated(
namespace=pc_ns, limit=100, pagination_token=token))
ids += [item.id for item in page.vectors]
token = page.pagination.next if page.pagination else None
while len(ids) >= 1000 or (ids and token is None):
chunk, ids = ids[:1000], ids[1000:]
fetched = with_backoff(lambda: index.fetch(ids=chunk, namespace=pc_ns))
cols, dropped = to_columns(fetched.vectors, schema)
if not cols["id"]:
continue # every id in the chunk was deleted after listing
with_backoff(lambda: ns.write(
upsert_columns=cols, distance_metric=metric, schema=tp_schema,
upsert_condition=["id", "Eq", None], # insert only: dual writes win
))
print(f"{pc_ns}: wrote {len(chunk)}, sparse dims dropped {dropped}, resume={token if not ids else None}")
if token is None:
return
if __name__ == "__main__":
import os
import turbopuffer
from pinecone import Pinecone
pc = Pinecone(api_key=os.environ["PINECONE_API_KEY"])
index = pc.index(host=os.environ["PINECONE_INDEX_HOST"])
tpuf = turbopuffer.Turbopuffer(region=os.environ["TURBOPUFFER_REGION"], max_retries=0)
sample = index.fetch(ids=[i.id for i in index.list_paginated(namespace="tenant-a", limit=100).vectors],
namespace="tenant-a")
schema = infer_schema([v.metadata for v in sample.vectors.values()])
migrate_namespace(index, tpuf, "docs", "tenant-a", dims=768, metric="cosine_distance",
schema=schema, fts_field="chunk_text")
Infer the schema from a larger sample than 100 records in practice, because a field that is integral in the sample and fractional later raises ValueError mid-run. Set fts_field now if you want BM25 later. Enabling full_text_search on an existing attribute is allowed in place, but the write docs say queries that need the index get HTTP 409 until it is built. Declaring it at creation avoids that wait.
We ran the helpers and the loop on Python 3.13.6. The loop ran against stub Pinecone and turbopuffer objects that page through 2,350 ids. The test file asserts the id hashing, type inference, the 1,024 sparse cut, batch sizes, retry on 429 and connection errors with no retry on 400, threshold conversion and equal-length columns:
$ python3 test_migrate.py
all checks passed
$ python3 run_with_stubs.py
tenant-a: wrote 1000, sparse dims dropped 0, resume=1000
tenant-a: wrote 1000, sparse dims dropped 0, resume=2000
tenant-a: wrote 350, sparse dims dropped 0, resume=None
docs.tenant-a [1000, 1000, 350] 2350
What the backfill costs to write
The pricing page lists writes at $2 per GB on Launch and Scale ($2.70 on Enterprise), with a 10 KB minimum per request and a batch discount of min(0.5, max(0, 0.2 * (log10(KB) - 1))) applied to the whole request, as of October 2026. Vectors are billed at 4 bytes per dimension for writes even when stored as f16. Take an illustrative 10M records with 768 dimensions (3,072 bytes) plus 1,024 bytes of attributes, so 4,096 bytes per row. The figures come from this function, run for each batch size:
def write_billed_bytes(request_bytes: int) -> float:
"""turbopuffer write billing per request: 10 KB floor, batch discount up to 50%."""
kb = max(request_bytes, 10_000) / 1000
discount = min(0.5, max(0.0, 0.2 * (math.log10(kb) - 1)))
return kb * 1000 * (1 - discount)
Billed GB for 10M rows is write_billed_bytes(4096 * n) * (10_000_000 / n) / 1e9 for n rows per request:
| Rows per request | Request size | Discount | Billed GB for 10M rows | Cost at $2/GB |
|---|---|---|---|---|
| 1,000 | 4,096 KB | 50% | 20.48 | $40.96 |
| 100 | 409.6 KB | 32.25% | 27.75 | $55.50 |
| 1 | 4.1 KB, billed as 10 KB | 0% | 100.00 | $200.00 |
The conditional-write query charge is small next to that. Bill every one of the 10,000 batches at the final namespace size of 40.96 GB. With the 80% marginal discount between 32 and 128 GB, that is 32 + 0.2 × 8.96 = 33.79 GB per query, or 0.338 PB in total, which comes to $0.34 at $1 per PB. Attributes that have FTS and filtering both enabled are billed at 200% of their size, so the attribute share grows if you index text for BM25.
Scores, thresholds and filters
Pinecone reports similarity, turbopuffer reports distance in $dist, and lower is better there for both metrics. Any min_score in your application changes meaning.
| Pinecone metric | Pinecone score | turbopuffer metric | $dist | Converted cut-off |
|---|---|---|---|---|
cosine | Cosine similarity, higher is better | cosine_distance | 1 - cosine similarity, 0 to 2 | keep $dist <= 1 - min_score |
euclidean | Squared Euclidean distance, lower is better | euclidean_squared | Squared Euclidean distance | Same number, same direction |
dotproduct | Dot product, higher is better | none for dense vectors | Only for unit vectors: use cosine, 1 - min_score |
The metric definitions are from Pinecone's index configuration docs and turbopuffer's write docs. The write docs list only cosine_distance and euclidean_squared for dense vectors, while dot_product exists only for sparse attributes. If your Pinecone index uses dotproduct on vectors that are not normalized, rankings will change after the move, and you will need to normalize the vectors or evaluate the new order. Treat converted thresholds as a starting point. ANN results differ between engines, and turbopuffer's limits page documents recall@10 of 90 to 100%, so re-tune the cut-off on labelled queries.
def tp_threshold(pc_metric: str, pc_min_score: float, vectors_unit_norm: bool) -> tuple[str, float]:
"""Pinecone score cut-off -> (turbopuffer distance_metric, max $dist to keep)."""
if pc_metric == "cosine": # Pinecone: cos sim, higher is better
return "cosine_distance", 1.0 - pc_min_score # turbopuffer: 1 - cos sim, lower is better
if pc_metric == "euclidean": # Pinecone already reports squared euclidean, lower is better
return "euclidean_squared", pc_min_score
if pc_metric == "dotproduct":
if not vectors_unit_norm:
raise ValueError("no dense dot product metric in turbopuffer: normalize or re-tune")
return "cosine_distance", 1.0 - pc_min_score # dot == cos on unit vectors
raise ValueError(pc_metric)
Filters need translating as well, and two semantic differences are easy to miss. Pinecone matches a list-of-strings field when any element satisfies the filter, which in turbopuffer is Contains or ContainsAny, so a plain Eq on a []string attribute will not behave like Pinecone's $eq. The turbopuffer query docs also say Lt and Lte match null values, so a record with no price passes price < 100 unless you exclude nulls.
OPS = {"$eq": "Eq", "$ne": "NotEq", "$gt": "Gt", "$gte": "Gte",
"$lt": "Lt", "$lte": "Lte", "$in": "In", "$nin": "NotIn"}
ARRAY_OPS = {"$eq": "Contains", "$ne": "NotContains", "$in": "ContainsAny", "$nin": "NotContainsAny"}
def tp_filter(f: dict, schema: dict):
"""Pinecone metadata filter -> turbopuffer filter."""
out = []
for key, cond in f.items():
if key in ("$and", "$or"):
out.append(("And" if key == "$and" else "Or", [tp_filter(c, schema) for c in cond]))
continue
for op, v in (cond if isinstance(cond, dict) else {"$eq": cond}).items():
if op == "$exists":
out.append((key, "NotEq" if v else "Eq", None))
elif op == "$not":
out.append(("Not", tp_filter({key: v}, schema)))
elif schema.get(key) == "[]string": # Pinecone matches any element
out.append((key, ARRAY_OPS[op], v))
elif op in ("$lt", "$lte"): # turbopuffer Lt/Lte also match null
out.append(("And", [(key, OPS[op], v), (key, "NotEq", None)]))
else:
out.append((key, OPS[op], v))
return out[0] if len(out) == 1 else ("And", out)
Output from a real run:
{'tags': {'$in': ['a', 'b']}} -> ('tags', 'ContainsAny', ['a', 'b'])
{'tags': {'$ne': 'x'}} -> ('tags', 'NotContains', 'x')
{'year': {'$lt': 2020}} -> ('And', [('year', 'Lt', 2020), ('year', 'NotEq', None)])
{'$or': [{'genre': {'$eq': 'drama'}}, {'year': {'$gte': 2020}}]}
-> ('Or', [('genre', 'Eq', 'drama'), ('year', 'Gte', 2020)])
The turbopuffer docs define NotEq as the inverse of Eq, so a record without the field should match it, and Pinecone's filter guide does not say what $ne does there. Test it on a handful of records in both engines before you trust the translation for access-control filters.
Dual-run, cutover and rollback
Reconcile before anyone reads from the new side. The turbopuffer export docs page through a namespace by sorting on id and filtering id > last_id with limit 10,000. Collect pinecone_id from those pages, diff it against Pinecone's list, delete in turbopuffer whatever Pinecone no longer has, and backfill whatever is missing. Compare counts as well: list_namespaces reports record_count per namespace, and the turbopuffer metadata endpoint reports approx_row_count.
For shadow reads, send a sample of production queries to both engines and log the share of Pinecone's top 10 that turbopuffer also returns:
def overlap_at_k(pc_ids: list[str], tp_ids: list[str], k: int = 10) -> float:
"""Share of Pinecone's top k that turbopuffer also returns in its top k."""
top = pc_ids[:k]
return len(set(top) & set(tp_ids[:k])) / len(top) if top else 1.0
Do not expect 1.0 from two different ANN indexes. A low overlap tells you where to look. It does not tell you which engine is right, so judge the disagreements with labelled queries. Our post on why an agent is only as good as its retriever explains why that measurement matters downstream.
Move reads behind a per-namespace flag, starting with a low-traffic namespace. Each newly migrated namespace starts cold, and the first queries read from object storage, so warm it before you flip the flag, as our post on turbopuffer cold start latency describes. Rolling back means flipping the flag, which works only while dual writes keep Pinecone current.
| Check | How | Done when |
|---|---|---|
| Schema matches | Metadata endpoint vs your declared schema | Every field has the intended type |
| Ids reconciled | Diff pinecone_id export against Pinecone list | No ids missing or extra in either |
| Counts agree | record_count vs approx_row_count | Equal once indexing has caught up |
| Sparse loss known | Sum of sparse dims dropped in the log | Zero, or accepted |
| Filters verified | Translated filters on records with and without each field | Same ids from both engines |
| Thresholds re-tuned | Labelled queries on turbopuffer | Cut-off set from turbopuffer distances |
| Shadow reads reviewed | overlap_at_k plus judged disagreements | Disagreements explained |
| Namespace warm | Warm-up query before the flag flips | First real query is not cold |
| Rollback tested | Flip one namespace back and forth | Pinecone answers with current data |
| Pinecone writes stopped | After the rollback window | Dual-write code removed |
The same order applies to moving from Pinecone to Qdrant, where collection and payload limits replace the ones covered here.
Where we can help
We run Pinecone migrations end to end, from schema mapping and the backfill to the labelled-query evaluation that decides when reads can move. See our turbopuffer consulting service.
Abhilash B is an advisor at Query Quotient and an information retrieval and AI expert who holds patents in the field. Abhilash brings that retrieval depth to the search, RAG and agent systems Query Quotient builds for clients.
