Back to blog
Performance

Cutting OpenSearch vector memory: 1-bit quantization, disk mode and rescoring

On OpenSearch 3.9.0, compression_level 32x is 1-bit scalar quantization with rescoring on by default. We indexed FiQA under twelve settings and measured native memory and recall@10 for each.

Abhilash B12 min read
#opensearch#vector search#quantization#k-nn#faiss

OpenSearch vector quantization cuts k-NN memory by storing a compressed copy of each vector in the Faiss or Lucene index and keeping the full-precision vectors on disk for rescoring. On OpenSearch 3.9.0, "compression_level": "32x" gives you 1-bit scalar quantization with rescoring switched on by default. In our test it used 12% of the memory of a float32 index, and recall@10 fell from 0.970 to 0.951. Raising oversample_factor to 30 took it to 0.991, above the unquantized index. Our recommendation for most float32 fields is 32x with a measured oversample_factor, plus 8x where recall has to stay closest to float.

Everything below ran on OpenSearch 3.9.0 (Lucene 10.5.1, k-NN plugin 3.9.0.0), the current release as of 31 October 2026, in a single-node Docker container with security off. The corpus is the BEIR FiQA set: 57,638 passages embedded with BAAI/bge-small-en-v1.5 (384 dimensions, normalized, innerproduct space). The queries are the first 500 FiQA queries with the model's retrieval prefix. Ground truth is exact inner product over all 57,638 vectors, computed in NumPy. Each index has one shard, is force-merged to one segment, refreshed and warmed up. HNSW uses the defaults (m 16, ef_construction 100, ef_search 100) and every query uses k: 10.

Where k-NN memory goes

The Faiss engine loads each segment's native index into off-heap memory, outside the JVM heap. The methods and engines page estimates HNSW at 1.1 × (4 × dimension + 8 × m) bytes per vector. The quantization pages swap the 4 × dimension term for the compressed size: 2 × d for FP16, d for byte vectors, and d × bits / 8 for 1, 2 and 4-bit quantization. The 8 × m term is the graph, and quantization does not touch it.

That second term matters more than it looks. At 768 dimensions and m 16, the graph is 128 bytes per vector against 3,072 bytes of float32. At 1 bit the vector shrinks to 96 bytes, so the graph becomes the larger half. At 32x, lowering m saves more memory than any further compression could.

Our float32 index reported 95,029 KB in _plugins/_knn/stats, which is exactly the size of its .faiss file (97.3 MB). The formula predicts 105.5 MB: 1.1 × (4 × 384 + 8 × 16) × 57,638. For the 1-bit index the formula gives 11.2 MB and the files came to 11.5 MB, because the real graph took 152 bytes per vector instead of 128. So the 1.1 headroom shrinks as the graph share grows.

The OpenSearch vector quantization options on 3.9

The memory-optimized vectors page and the Faiss scalar quantization page list these, with the version each arrived in:

OptionMappingBits per dimensionRescore by defaultNotes
float32default32NoBaseline
half_float"data_type": "half_float"16No3.9+. Stored natively as FP16
Faiss SQ fp16encoder: sq, bits: 16 or 2x16NoInput stays float32
byte"data_type": "byte"8NoYou quantize before ingest, values in [-128, 127]
compression_level: 8x4-bit SQ4Yes, factor 1.04-bit SQ is 3.9+
compression_level: 16x2-bit SQ2Yes, factor 1.02-bit SQ is 3.9+
compression_level: 32x1-bit SQ1Yes, factor 2.01-bit SQ is 3.6+. Same as mode: on_disk
Binary quantizationencoder: binary, bits: 11, 2 or 4NoFaiss only. ADC and random rotation are 3.2+
Product quantizationtrained model, encoder: pqm × code_size per vectorNoFaiss only, needs the Train API

The mode, compression_level and data_type parameters themselves, and the errors you get when you combine them wrongly, are in our knn_vector mapping guide. Here are the field definitions we ran, inside a knn: true index:

// 1-bit SQ: these three gave identical recall and identical files on 3.9.0
{ "type": "knn_vector", "dimension": 384, "space_type": "innerproduct", "compression_level": "32x" }
{ "type": "knn_vector", "dimension": 384, "space_type": "innerproduct", "mode": "on_disk" }
{ "type": "knn_vector", "dimension": 384, "space_type": "innerproduct",
  "method": { "name": "hnsw", "engine": "faiss",
    "parameters": { "encoder": { "name": "sq", "parameters": { "bits": 1 } } } } }

// Older binary quantization encoder, 1 bit
{ "type": "knn_vector", "dimension": 384, "space_type": "innerproduct",
  "method": { "name": "hnsw", "engine": "faiss",
    "parameters": { "encoder": { "name": "binary", "parameters": { "bits": 1 } } } } }

// FP16 storage, 3.9+
{ "type": "knn_vector", "dimension": 384, "space_type": "innerproduct", "data_type": "half_float",
  "method": { "name": "hnsw", "engine": "faiss" } }

The mode: on_disk field logged The 'mode' parameter is deprecated starting with version 3.8.0. Mode is now derived from 'compression_level'. That derivation explains a behavior the docs do not spell out. The docs say in_memory fields get no rescoring by default, but a field with only "compression_level": "8x" and no mode rescored by default on 3.9.0: omitting rescore gave the same recall as "rescore": true, and "rescore": false dropped recall from 0.981 to 0.884.

Memory for 10 million 768-dimension vectors

These are the documented formulas applied to an illustrative corpus of 10M vectors, 768 dimensions, m 16, one copy. Add one full copy per replica. GB here is 10^9 bytes.

OptionVector bytesPer vector: 1.1 × (vector + 8 × 16)10M vectors
float324 × 768 = 3,0723,520 B35.2 GB
FP16 (half_float or sq 16-bit)2 × 768 = 1,5361,830.4 B18.3 GB
byte768985.6 B9.86 GB
4-bit (8x)768 × 4 / 8 = 384563.2 B5.63 GB
2-bit (16x)192352 B3.52 GB
1-bit (32x, on_disk, binary 1-bit)96246.4 B2.46 GB
HNSW + PQ, pq_m 96, code_size 896 + 24 overheadsee below2.81 GB

The PQ row uses the product quantization formula with 100 segments: 1.1 × ((8 / 8 × 96 + 24 + 8 × 16) × 10,000,000 + 100 × (2^8 × 4 × 768)) = 2.81 GB. PQ needs a training step and is not supported by memory-optimized search, so it only earns its place when you can't run 1-bit quantization. We did not test it.

Rescoring needs the full-precision vectors, and quantized fields keep them on disk: 30.72 GB of float32 for 10M × 768 (10,000,000 × 3,072 bytes), read only for the rescored candidates. The vector sizing calculator runs these formulas for your own dimension, m and replica count, and the cost calculator turns the result into instance cost.

Measured memory and recall@10 on FiQA

The quantized fields never appeared in indices_in_cache in _plugins/_knn/stats, even after warmup and 3,500 queries each. The float32 index with index.knn.memory_optimized_search: true didn't either. The memory-optimized search page explains why: the engine memory-maps the index file and relies on the operating system's page cache, so nothing is charged to the k-NN cache. The Faiss docs state this for 1-bit quantization, and on 3.9.0 we saw the same for 2-bit and 4-bit. For those rows we report the working set from the segment files: the 8,777,838-byte graph file plus the quantized codes (57,638 × 384 × bits / 8). The codes sit in Lucene's compound file, and its size moved by exactly that amount between 8x and 16x.

IndexNative memoryRecall, rescore: falseRecall, defaultoversample_factor 30oversample_factor 100
float3297.3 MB (stats)0.9700.9700.9700.970
float32, memory-optimized97.3 MB file, mmap0.9720.9720.9930.999
half_float53.0 MB (stats)0.9710.9710.9710.971
sq 16-bit53.0 MB (stats)0.9690.9690.9710.971
byte, our own int8 scaling30.9 MB (stats)0.9330.9330.9330.933
8x (4-bit SQ)19.8 MB, mmap0.8840.9810.9960.999
16x (2-bit SQ)14.3 MB, mmap0.7490.9770.9950.999
32x / on_disk / sq 1-bit11.5 MB, mmap0.5970.9510.9910.999
half_float + 16x (1-bit)11.5 MB, mmap0.5970.9500.9890.998
Lucene engine, 32xnot in k-NN stats0.6020.9590.9910.999
Binary encoder, 1-bit11.5 MB (stats)0.4440.4440.9620.986
Binary 1-bit, ADC + rotation11.5 MB (stats)0.4400.4400.9570.984

For the binary rows the default does not rescore, and "rescore": true gave 0.876 and 0.873. The byte row is illustrative: we scaled every value by one global 127 / max|x| factor, which is the crudest way to make byte vectors, so treat it as a lower bound for byte fields.

xychart-beta
  title "Recall@10 on FiQA, OpenSearch 3.9.0, k = 10"
  x-axis ["4-bit SQ 8x", "2-bit SQ 16x", "1-bit SQ 32x", "binary 1-bit"]
  y-axis "Recall@10" 0 --> 1
  bar [0.884, 0.749, 0.597, 0.444]
  bar [0.981, 0.977, 0.951, 0.876]
  line [0.996, 0.995, 0.991, 0.962]

Blue bars are rescore: false, grey bars are rescoring with factor 1.0 or 2.0 (100 candidates), and the yellow line is oversample_factor 30, from the table above.

xychart-beta
  title "Vector index memory for 57,638 x 384 vectors, MB"
  x-axis ["float32", "FP16", "byte", "8x", "16x", "32x"]
  y-axis "MB" 0 --> 100
  bar [97.3, 53.0, 30.9, 19.8, 14.3, 11.5]

Bars are the measured native memory from the table: k-NN stats for the first three, segment file sizes for the memory-mapped rows.

The 1-bit scalar quantizer from 3.6 beats the older binary encoder by a wide margin on this data, 0.951 against 0.876 with the same 100-candidate rescore. OpenSearch's 3.6 release post reports 24% better recall and 15% lower latency for 1-bit SQ over "existing binary methods"; our recall gap points the same way, and we did not measure the latency claim. Turning on ADC and random rotation for the binary encoder did not help here. And the in-memory float32 index ignored oversample_factor completely, while the memory-optimized one used it to search a larger first pass and reached 0.999.

Disk shrank as well. The float32 Faiss index stored its vectors twice, once in Lucene's .vec file and again inside the .faiss file, for 186.5 MB in total. The 32x index was 101.7 MB, almost all of it the float32 vectors kept for rescoring.

How rescoring and oversample_factor work

Rescoring is a two-phase search. The quantized index returns oversample_factor × k candidates with approximate scores, then OpenSearch reads those candidates' full-precision vectors from disk, rescores them against the float query and keeps the top k.

flowchart LR
  Q["knn query, k = 10"] --> A["Phase 1: HNSW over<br/>quantized codes"]
  A --> C["Candidates = oversample_factor x k,<br/>clamped to 100 .. 10,000"]
  C --> R["Phase 2: load float32 vectors<br/>for candidates from disk"]
  R --> S["Exact score vs float query"]
  S --> T["Top k results"]

The clamp is the detail that trips people up. The docs say the first-pass count "is guaranteed to be between 100 and 10,000", so at k: 10 any factor from 1 to 10 asks for the same 100 candidates. Our runs agree: rescore: true, factor 2.0 and factor 10.0 returned identical recall on every quantized index. Factor 30 (300 candidates) is where recall moved, and 100 (1,000 candidates) pushed every scalar-quantized index to 0.998 or higher. At k: 100 the same factors mean real work, so tune the factor for the k you send. By default the k-NN plugin first combines results from all segments of a shard and rescores only that shard's top results; index.knn.disk.vector.shard_level_rescoring_disabled: true rescores per segment instead.

The factor accepts 1.0 to 100.0. A query that sets it looks like this:

GET fiqa-32x/_search
{
  "size": 10,
  "query": { "knn": { "v": {
    "vector": [0.018, -0.044, ...],
    "k": 10,
    "rescore": { "oversample_factor": 30.0 }
  } } }
}

We timed every variant, and all of them answered in 1 to 4 ms on a laptop, with run-to-run noise larger than the differences between settings. A corpus of 57,638 vectors fits in page cache, so it says nothing about disk reads at 10M vectors. The 3.6 release post also describes a prefetch feature that loads vectors before distance computation, with up to 2x lower latency in memory-constrained setups; the docs pages we read describe no setting for it, and we did not test it.

Measure recall on your own queries

Recall depends on your embedding model and data, so run this before you choose a level. It compares each index against exact search over the same vectors (D is your corpus matrix, Q a sample of real query embeddings, both normalized for innerproduct):

import json, numpy as np, requests

def recall_at_10(index, Q, D, rescore=None):
    truth = np.argsort(-(Q @ D.T), axis=1)[:, :10]       # exact top 10 per query
    total = 0.0
    for q, t in zip(Q, truth):
        knn = {"vector": q.tolist(), "k": 10}
        if rescore is not None:
            knn["rescore"] = rescore
        r = requests.post(f"http://localhost:9200/{index}/_search",
                          headers={"Content-Type": "application/json"},
                          data=json.dumps({"size": 10, "_source": False,
                                           "query": {"knn": {"v": knn}}})).json()
        ids = {int(h["_id"]) for h in r["hits"]["hits"]}   # docs indexed with _id = row number
        total += len(ids & set(t.tolist())) / 10
    return total / len(Q)

for factor in [None, False, {"oversample_factor": 30.0}]:
    print(factor, recall_at_10("my-index-32x", Q, D, factor))

For corpora too large for a NumPy matrix, compute the truth with an exact script_score query instead, as in the recall check from our nmslib to Faiss migration guide. Run the check with your production filters too, because a filtered query on a quantized field still follows the exact-or-approximate rules covered in why k-NN filters return fewer than k results. If you also run Elasticsearch or Qdrant, the same trade-off appears in BBQ HNSW vs BBQ disk and the Qdrant quantization guide.

Which compression level to pick

Start with "compression_level": "32x" for float32 fields of 384 dimensions or more. Leave mode out, since 3.9 derives it and logs a deprecation if you set it. Then measure recall at the k you serve and raise oversample_factor until it stops improving. On FiQA that point was 30 for k: 10, and it beat the float32 index at default ef_search while using 12% of its memory. If recall must sit within a point of float32 without a large factor, use 8x, which gave 0.981 at the default rescore. Skip the binary encoder for new fields; the 1-bit sq encoder replaced it as the 32x default in 3.6 and scored better here. Keep half_float for fields where you want FP16 storage with no rescoring path at all. For memory-mapped indexes, budget the operating system's page cache for the graph and codes, because _plugins/_knn/stats will not show their footprint.

Where we can help

We size and tune OpenSearch vector indexes for teams moving to quantized storage: picking the compression level and oversample_factor from recall measured on their own queries, then checking memory under real traffic. Our OpenSearch work for GrabJobs delivered 5ms queries and 40% lower cost across 10M+ listings. If your k-NN memory bill is the problem, start with our OpenSearch consulting service.

Advisor, Query Quotient

Abhilash B is an advisor at Query Quotient and an information retrieval and AI expert who holds patents in the field. Abhilash brings that retrieval depth to the search, RAG and agent systems Query Quotient builds for clients.

Keep reading

Building something similar?

Tell us what you are working on. In a free 30-minute call we will give you an honest view of the approach, the effort and how to measure success.