turbopuffer cold start latency: hint_cache_warm and designing agents around cold namespaces

A turbopuffer cold start is the first query to a namespace that is not yet cached on a query node: it reads object storage directly, so its p50 is hundreds of milliseconds where a cached query takes about 14ms. The fix is the hint_cache_warm endpoint (GET /v1/namespaces/:namespace/hint_cache_warm), which tells the service to start loading the namespace before your latency-sensitive query arrives. Send it when a user session starts, for every namespace that user can touch, and the first real query in an agent run no longer pays the cold price.

Everything here was checked against turbopuffer's documentation as of October 2026. We did not run benchmarks for this post, so every latency figure below is either quoted from the docs or computed from those quoted figures, and labelled as such.

Why a turbopuffer cold start is slow

The storage of record is object storage. Each namespace has its own prefix there, holding a write-ahead log and the indexes built from it. Query nodes keep a memory and NVMe SSD cache in front of that bucket. turbopuffer's architecture docs describe the routing: after the first query, a namespace's documents are cached on NVMe, and subsequent queries are routed to the same query node for cache locality, though any node can serve any namespace.

A cold query cannot use any of that. It has to fetch the storage engine metadata, then the first level of the indexes (filter index, centroid index and any unindexed WAL entries), then the vector clusters it needs. The architecture page puts each object-storage round trip at roughly 100ms and says the 3-4 round trips of a cold query "often take as little as ~400ms". The vector index is centroid-based (SPFresh), which the docs chose because it needs fewer round trips on object storage than graph indexes like HNSW.

flowchart LR
  C([Client query]) --> LB[Load balancer]
  LB --> Q{"Namespace cached<br/>on this query node?"}
  Q -- "yes: warm" --> M[Memory / NVMe cache]
  M --> R([Results])
  Q -- "no: cold" --> RT1["Round trip 1<br/>metadata"]
  RT1 --> RT2["Round trip 2<br/>filter + centroid index, WAL"]
  RT2 --> RT3["Round trip 3<br/>clusters"]
  RT3 --> FILL[Fill NVMe cache]
  FILL --> R

Even a warm query is not free of object storage by default. Strongly consistent queries (the default) make one round trip to check for writes the cache has not seen. That is the source of the gap between the ~14ms strong-consistency warm number and the "sub 10ms" eventually consistent figure the architecture page mentions.

The documented numbers

These are the figures turbopuffer publishes, as of October 2026. They come from the vendor, on its own hardware and data, so treat them as a starting point for your own measurements.

Sources: the benchmark header on the query docs and the performance guide, plus the warm cache page. Two inconsistencies are worth knowing about. The architecture page quotes the same 874ms / 14ms p50s but labels them "for 1M documents", while the query page labels them 10M docs; we use the query page's label. If you cite a number, cite it with the page and the date, because these figures change between doc revisions.

User reports line up with the docs: a G2 reviewer describes the cold start as noticeable.

What one cold call does to an agent run

A single search box can absorb an 874ms first query. An agent usually can't, because its tool calls are sequential: the model has to read the result of call 1 before it writes call 2. The cold call sits on the critical path, and the remaining calls cannot hide it.

The table below is arithmetic on the documented vector p50s (874ms cold, 14ms warm), assuming every tool call in the run hits the same namespace and only the first is cold. Sums of medians are not medians of sums, so read these as orders of magnitude, not predictions. LLM time is excluded.

xychart-beta
  title "Retrieval time per agent run, computed from documented p50s"
  x-axis "Sequential tool calls" [1, 3, 5, 10]
  y-axis "Milliseconds" 0 --> 1100
  bar [874, 902, 930, 1000]
  bar [14, 42, 70, 140]

Blue bars are runs where the first call is cold, grey bars are runs where every call is warm. A ten-call run with a cold first call spends 87% of its retrieval time on one query. If the agent fans out across three cold namespaces (say a user's docs, tickets and code), that cold cost appears three times, once per namespace, each on whichever step first touches it. The reliability math of AI agents makes the same point about errors that this table makes about latency: per-step costs compound across a run, and the agent reliability calculator lets you play with the step count.

hint_cache_warm: what it does and when it is billed

The endpoint takes no body. It returns immediately with {"status": "ACCEPTED", "message": "cache warm hint accepted"} and the warming happens in the background. The docs show it in every official SDK: hint_cache_warm() in Python, Ruby and Rust, hintCacheWarm() in TypeScript and Java, HintCacheWarm() in Go and C#.

curl https://gcp-us-central1.turbopuffer.com/v1/namespaces/acme-docs/hint_cache_warm \
  -X GET --fail-with-body \
  -H "Authorization: Bearer $TURBOPUFFER_API_KEY"

import turbopuffer

tpuf = turbopuffer.Turbopuffer(region="gcp-us-central1")  # reads TURBOPUFFER_API_KEY
print(tpuf.namespace("acme-docs").hint_cache_warm())
# NamespaceHintCacheWarmResponse(status='ACCEPTED', message='cache warm hint accepted')

The billing rule, quoted from the warm cache docs: if the namespace is already ready for low-latency queries, or already being warmed, the request is free. Otherwise it is "billed as a query that returns zero rows". A query is billed as data queried plus data returned, and data queried is the namespace size or 1.28 GB, whichever is greater, per turbopuffer pricing as of October 2026.

That table is our arithmetic from the published rates, assuming zero bytes returned and no large-namespace discount (those start at 32 GB). Enterprise's data-queried rate is $1.35 per PB. The practical upshot: a billed hint costs the same as one cold query, and a hint to a namespace that is already warm costs nothing. So you can hint liberally. The worst case is that you pay for a query you were about to run anyway, a few hundred milliseconds earlier.

The docs do not say how long warming takes after the hint, or how long an idle namespace stays cached. Both depend on namespace size and node load. Measure them for your data with the script at the end.

Where to send the hint

At session start, for every namespace the user can reach

This is the pattern the warm cache page itself recommends, and the first one we would add. When a user logs in or opens the chat panel, hint every namespace tied to that user, concurrently, and do not await anything on the hot path.

import asyncio
import turbopuffer

tpuf = turbopuffer.AsyncTurbopuffer(region="gcp-us-central1")

async def warm_user(namespaces: list[str]) -> None:
    results = await asyncio.gather(
        *(tpuf.namespace(n).hint_cache_warm() for n in namespaces),
        return_exceptions=True,  # a failed hint must never fail the session
    )
    for name, r in zip(namespaces, results):
        if isinstance(r, Exception):
            print(f"warm hint failed for {name}: {r}")

# in your session handler:
# asyncio.create_task(warm_user([f"{tenant}-docs", f"{tenant}-tickets"]))

How many namespaces belong to a user is a namespace design question. One namespace per tenant means one hint per session; one namespace per document collection means several. The turbopuffer vs Pinecone comparison covers how the two engines model tenancy.

Before the agent's first tool call

If the session-start hook is not available (API clients, background jobs, a Slack bot), send the hint the moment the user's message arrives, in parallel with the first LLM call. The model's planning step usually takes longer than the cold-to-warm transition, so by the time it emits a search tool call the namespace is warming or warm.

sequenceDiagram
  participant U as User
  participant A as Agent runtime
  participant L as LLM
  participant T as turbopuffer
  U->>A: message
  par warm in background
    A->>T: GET hint_cache_warm
    T-->>A: ACCEPTED
    Note over T: loads namespace into<br/>NVMe and memory
  and plan
    A->>L: system prompt + message
    L-->>A: tool call search
  end
  A->>T: POST query
  T-->>A: rows, cache_temperature warm
  A->>L: tool result
  L-->>U: answer with citations

Whether the planning step really covers the warming time is something to measure. If it doesn't, the first query is partly warm and you still save part of the cold cost.

Keeping hot tenants warm

For a short list of tenants where the first query must be fast (a large customer with spiky traffic, a demo account), a periodic hint is cheap: it is free whenever the namespace is still warm, and billed as one query only when it has gone cold. Because the cache lifetime is undocumented, pick the interval from measurement, not from a guess.

Past a certain size and traffic level, stop hinting and pin. Namespace pinning reserves compute and NVMe cache for one namespace, switches billing from per-query to GB-hours, and keeps the cache warm. The pinning docs suggest evaluating it around >16 GB and >10 QPS, with break-even typically around 10 queries per second, and note that pinning capacity in gcp-us-central1 is currently limited.

Eventual consistency for lower warm latency

Warming fixes the first query. Eventual consistency trims the rest. With consistency={"level": "eventual"}, a query skips the per-query object-storage check and reuses the node's cached view of the namespace for up to 60 seconds, refreshing it in the background. The first eventually consistent query after a namespace has been idle on a node for more than 60 seconds still waits for one round trip. max_staleness_seconds raises that window to at most one hour.

The cost is freshness, and the query docs are specific about it. Eventual queries search at most 128MiB of unindexed writes. Over 99.8% of them return consistent data, but staleness of about 100ms can show up during scaling or failover, and up to about an hour after heavy writes.

Set it per query, at the request root. For a multi-query, all subqueries share one consistency level.

Measuring your own cold and warm p50/p95

The script below runs one query against each namespace you pass in, records the client wall time alongside the server's own performance.server_total_ms and performance.cache_temperature, then repeats the query to get warm numbers. Pass namespaces that have been idle for a while. The script does not assume the first query was cold: it prints the temperature the server reports, so you can drop rows that came back warm or hot. Run it from the same cloud region as your application, or network time will dominate the warm numbers.

# measure_cold_start.py
# pip install turbopuffer ; export TURBOPUFFER_API_KEY=...
import argparse
import random
import time

import turbopuffer


def pct(xs: list[float], p: float) -> float:
    xs = sorted(xs)
    return xs[min(len(xs) - 1, round(p / 100 * (len(xs) - 1)))]


def run_query(ns, dims: int, consistency: str):
    vec = [random.uniform(-1, 1) for _ in range(dims)]
    t0 = time.perf_counter()
    r = ns.query(
        rank_by=("vector", "ANN", vec),
        limit=10,
        consistency={"level": consistency},
    )
    wall_ms = (time.perf_counter() - t0) * 1000
    return wall_ms, r.performance.server_total_ms, r.performance.cache_temperature


def main() -> None:
    ap = argparse.ArgumentParser()
    ap.add_argument("namespaces", nargs="+", help="namespaces idle long enough to be cold")
    ap.add_argument("--region", default="gcp-us-central1")
    ap.add_argument("--dims", type=int, required=True, help="vector dimensions of the namespaces")
    ap.add_argument("--repeats", type=int, default=20)
    ap.add_argument("--consistency", choices=["strong", "eventual"], default="strong")
    ap.add_argument("--hint-wait", type=float, default=None,
                    help="send hint_cache_warm, wait this many seconds, then query")
    a = ap.parse_args()

    tpuf = turbopuffer.Turbopuffer(region=a.region)
    first, repeat = [], []
    for name in a.namespaces:
        ns = tpuf.namespace(name)
        if a.hint_wait is not None:
            ns.hint_cache_warm()
            time.sleep(a.hint_wait)
        wall, server, temp = run_query(ns, a.dims, a.consistency)
        first.append(wall)
        print(f"{name}: first query {wall:.0f}ms wall, {server}ms server, temperature={temp}")
        repeat += [run_query(ns, a.dims, a.consistency)[0] for _ in range(a.repeats)]

    for label, xs in (("first query", first), ("repeat queries", repeat)):
        print(f"{label}: n={len(xs)} p50={pct(xs, 50):.0f}ms p95={pct(xs, 95):.0f}ms")


if __name__ == "__main__":
    main()

Run it three ways on three different sets of idle namespaces: with no flags for the cold baseline, with --hint-wait 1 (and then 5, 30) to find how long warming takes for your namespace size, and with --consistency eventual to see what the consistency round trip costs you. A cold namespace queried once is warm afterwards, so each set of namespaces gives you one cold sample apiece. Collect at least 20 before you trust a p95. If your namespaces use a vector column other than vector, change the rank_by tuple.

For BM25-only namespaces, swap the rank_by for a full-text query such as ("content", "BM25", "refund policy"). Cold behaviour on disk-based vector indexes in other engines follows the same physics: see how Elasticsearch BBQ HNSW and BBQ disk trade memory for latency, and how Qdrant's TurboQuant quantization changes what fits in RAM.

Our recommendation, in order: hint at session start for every namespace the user can reach, hint again in parallel with the agent's first LLM call, use eventual consistency on read paths that never read their own writes, and pin only the few namespaces that meet the pinning thresholds. Warm latency matters less than retrieval quality in the end, which is the argument of your agent is only as good as its retriever, but a cold first call is the easiest latency problem in the stack to remove.

Where we can help

We design turbopuffer namespaces, warming and consistency settings around how your agents actually call them, and measure cold and warm latency on your data before anything ships. See our turbopuffer consulting service.