Migrating from Elasticsearch to Vespa: mappings to schemas, Query DSL to YQL, scoring to rank profiles
An Elasticsearch to Vespa migration, run on both engines: one mapping, one bool plus function_score query, and the Vespa schema, YQL and rank profile that reproduce its ranking on Vespa 8.778, top four scores included.
An Elasticsearch to Vespa migration maps each index to a Vespa schema, each mapping field to a schema field with an indexing: statement, each bool query to a YQL where clause, and each function_score to a rank profile expression. The data moves as JSON put operations through vespa feed. Port the existing relevance exactly first, prove parity on real queries, cut over, and only then start using Vespa's phased ranking. Moving the engine and changing the ranking in the same step leaves you unable to tell a migration bug from a ranking change.
Everything below was run on 31 October 2026 with Elasticsearch 9.5.5 (Lucene 10.5.1, single node, security off) and Vespa 8.778.45 (single content node, Vespa CLI 8.778.45), both in Docker. The corpus is 12 outdoor products written for this post. It is illustrative, small enough to read every score, and the results are a parity demonstration only. They are not a benchmark.
The Elasticsearch to Vespa migration concept map
Most Elasticsearch concepts have a direct Vespa counterpart. A few have none, and those are where migrations stall.
| Elasticsearch | Vespa | Notes from our run |
|---|---|---|
| Index | Document type inside a schema, deployed in an application package | One .sd file per document type, plus services.xml |
| Shards and replicas | Content cluster nodes and redundancy / min-redundancy in services.xml | No shard count to pick; Vespa distributes buckets across nodes |
text field | indexing: index plus index: enable-bm25 | bm25(field) only works on fields with enable-bm25 |
keyword field | indexing: attribute, attribute: fast-search for filtering | Case-insensitive by default; ES keyword is case-sensitive |
integer, float, boolean | int, float, bool attributes | scaled_float has no equivalent; we used double |
date | long attribute holding epoch seconds | Convert during export |
Multi-field (title.raw) | Synthetic field outside the document block, indexing: input title | attribute | Recomputed by Vespa on every feed |
| Analyzer | Linguistics: OpenNLP by default, or Lucene linguistics | Stemmers differ; see the stemming section |
bool filter clauses | YQL and with attribute predicates | Attribute matches add no text score |
multi_match with ^ boosts | userInput() per field, boosts in the rank profile | YQL has no per-field query boost |
function_score | first-phase expression in a rank-profile | log1p becomes log10(1 + x) |
_explain | match-features on each hit | Declared per rank profile, returned on every hit |
_bulk | vespa feed or vespa-feed-client | JSONL of put operations |
_reindex | Reindexing from stored documents, or a refeed from source | See the cutover section |
| Index aliases | No direct equivalent | Schema changes apply in place; route at the application layer |
Vespa has an official Migrating from Elasticsearch guide built around ElasticDump and a parser script that converts a mapping into a schema. It describes that script as "a basic script with minimal error checking". It gives you a first schema to edit. It does not carry over analyzers or scoring, which is the part this post covers.
The Elasticsearch side
The mapping uses the built-in english analyzer on two text fields, keyword for facets, and a title.raw multi-field:
{
"mappings": {
"properties": {
"sku": { "type": "keyword" },
"title": { "type": "text", "analyzer": "english",
"fields": { "raw": { "type": "keyword" } } },
"description": { "type": "text", "analyzer": "english" },
"brand": { "type": "keyword" },
"category": { "type": "keyword" },
"price": { "type": "scaled_float", "scaling_factor": 100 },
"rating": { "type": "float" },
"sales": { "type": "integer" },
"in_stock": { "type": "boolean" }
}
}
}
The query is a common product search shape: a multi_match with a title boost, three filters, and a popularity multiplier from function_score:
{
"size": 10,
"_source": ["title", "brand", "price", "sales"],
"query": {
"function_score": {
"query": {
"bool": {
"must": {
"multi_match": {
"query": "waterproof hiking boots",
"fields": ["title^3", "description"]
}
},
"filter": [
{ "terms": { "category": ["boots", "trail-shoes"] } },
{ "range": { "price": { "lte": 200 } } },
{ "term": { "in_stock": true } }
]
}
},
"field_value_factor": { "field": "sales", "modifier": "log1p", "missing": 0 },
"boost_mode": "multiply"
}
}
}
Read the scoring as a formula before porting it. multi_match defaults to best_fields, which takes the single best field score, so the text part is max(3 × BM25(title), BM25(description)). The filters add nothing to the score. The Elasticsearch docs define log1p as adding 1 and taking the common logarithm, and boost_mode: multiply multiplies the two. The full score is:
max(3 × BM25(title), BM25(description)) × log10(1 + sales)
Six of the 12 products pass the filters. The top hit came back as "_id": "p01", "_score": 21.543879.
The Vespa schema
Each mapping row becomes a field. Text fields get index and enable-bm25. Facets and numerics become attributes. The title.raw multi-field becomes a synthetic field declared outside the document block, which Vespa fills from title at feed time:
schema product {
document product {
field sku type string {
indexing: attribute | summary
}
field title type string {
indexing: index | summary
index: enable-bm25
}
field description type string {
indexing: index | summary
index: enable-bm25
}
field brand type string {
indexing: attribute | summary
attribute: fast-search
}
field category type string {
indexing: attribute | summary
attribute: fast-search
match: cased
}
field price type double {
indexing: attribute | summary
}
field rating type float {
indexing: attribute | summary
}
field sales type int {
indexing: attribute | summary
}
field in_stock type bool {
indexing: attribute | summary
}
}
field title_raw type string {
indexing: input title | attribute
match: exact
}
fieldset default {
fields: title, description
}
rank-profile es_parity {
function text() {
expression: max(3 * bm25(title), bm25(description))
}
first-phase {
expression: text * log10(1 + attribute(sales))
}
match-features: text bm25(title) bm25(description) attribute(sales)
}
}
The rank profile is the function_score formula written out. The match-features line returns every input on each hit, which replaces _explain during parity work. Vespa's BM25 reference uses the same IDF formula as Lucene, log(1 + (N - n + 0.5) / (n + 0.5)), with k1 = 1.2 and b = 0.75 by default, so identical inputs give identical scores.
Case sensitivity needs a decision per field. Without match: cased, category in ('BOOTS') matched 6 documents in Vespa while the Elasticsearch terms filter matched 0. With match: cased both engines returned 0 for BOOTS and 6 for boots. Adding it to a live field made the deploy report a required restart of the search node ("change property 'dictionary: cased/uncased' from 'UNCASED' to 'CASED'"), so settle it before the first production deploy.
services.xml is minimal: one container with the document and search APIs, and one content cluster. The config element is the stemming fix explained below.
<services version="1.0">
<container id="default" version="1.0">
<config name="ai.vespa.opennlp.open-nlp">
<snowballStemmingForEnglish>true</snowballStemmingForEnglish>
</config>
<document-api/>
<search/>
<nodes><node hostalias="node1"/></nodes>
</container>
<content id="shop" version="1.0">
<min-redundancy>1</min-redundancy>
<documents>
<document type="product" mode="index"/>
</documents>
<nodes><node hostalias="node1" distribution-key="0"/></nodes>
</content>
</services>
Feeding: export from Elasticsearch, vespa feed into Vespa
A Vespa document id has the form id:<namespace>:<document-type>::<user-id>. We exported with a point-in-time and search_after on _shard_doc, which pages through an index without a scroll context, and wrote one put per hit:
pit = call("POST", "/products/_pit?keep_alive=2m")["id"]
after = None
with open("feed.jsonl", "w") as out:
while True:
body = {"size": 500, "pit": {"id": pit, "keep_alive": "2m"},
"sort": [{"_shard_doc": "asc"}]}
if after:
body["search_after"] = after
hits = call("POST", "/_search", body)["hits"]["hits"]
if not hits:
break
for h in hits:
out.write(json.dumps({"put": f"id:shop:product::{h['_id']}",
"fields": h["_source"]}) + "\n")
after = hits[-1]["sort"]
vespa feed feed.jsonl
{
"feeder.operation.count": 12,
"feeder.ok.count": 12,
"feeder.error.count": 0,
"http.response.code.counts": { "200": 12 }
}
Type conversion belongs in this export step. Dates become epoch long values and scaled_float becomes double or integer cents. Drop any _source field the schema doesn't declare, because the document API rejects a put that names an unknown field.
sequenceDiagram
participant App as Application
participant ES as Elasticsearch
participant X as Export job
participant V as Vespa
X->>ES: open point-in-time, search_after pages
ES-->>X: _source hits
X->>X: convert types, build put operations
X->>V: vespa feed feed.jsonl
App->>ES: live writes
App->>V: same writes, dual-written
App->>V: shadow queries, results logged onlyQuery DSL to YQL, and the fieldset trap
The direct translation of the query uses userQuery(), which searches the default fieldset:
{
"yql": "select sku, title, brand, price, sales from product where userQuery() and category in ('boots', 'trail-shoes') and price <= 200 and in_stock = true",
"query": "waterproof hiking boots",
"ranking": "es_parity",
"hits": 10
}
It returned the same 6 documents, with scores about 20 times lower than Elasticsearch: 1.1002 for p01 against 21.5439. The queryTermDocumentFrequency(title) rank feature showed why. Through the fieldset, every query term reported a document frequency of 12, the whole corpus, so every IDF collapsed to the same small value. Querying title contains "waterproof" directly reported 4, which matches the n in Elasticsearch's _explain. A probe with laces, which appears in the title and the description of one product, reported 2, so in our run the fieldset value was the sum of the per-field document frequencies. We found no statement of this in the docs, so check it on your version with the same feature.
The fix is to query each field separately, which is also closer to what multi_match does. userInput() takes a defaultIndex and a grammar annotation (see the YQL reference), and grammar: 'any' gives Elasticsearch's default or operator:
{
"yql": "select sku, title, price, sales from product where (({defaultIndex: 'title', grammar: 'any'}userInput(@q)) or ({defaultIndex: 'description', grammar: 'any'}userInput(@q))) and category in ('boots', 'trail-shoes') and price <= 200 and in_stock = true",
"q": "waterproof hiking boots",
"ranking": "es_parity",
"hits": 10
}
{
"relevance": 21.543878632632797,
"fields": {
"matchfeatures": {
"attribute(sales)": 1840.0,
"bm25(description)": 0.5985169322638356,
"bm25(title)": 2.199440910538444,
"text": 6.5983227316153314
},
"sku": "p01",
"title": "Ridgeline Waterproof Hiking Boot"
}
}
bm25(title) of 2.1994 is the exact sum of the three per-term weights in Elasticsearch's _explain for the same document. The default model.type for userQuery() is weakAnd, an approximate OR that passes a limited number of hits to ranking, 100 by default. On a large corpus that changes recall against Elasticsearch's exhaustive OR, so pick the grammar on purpose.
Analyzers to linguistics: the stemming difference
Before the services.xml fix, the ranking was wrong as well as the scores: "Summit Mid Hiking Boots" came first. The query trace (trace.level: 3) showed the stemmed query as WEAKAND waterproof hike boots. Vespa's default OpenNLP linguistics uses kStem for English, which the OpenNLP linguistics page describes as a simpler stemmer with lower recall. kStem reduced hiking to hike and left boots alone, so among the filtered products it only matched the one whose text contains "boots". Elasticsearch's english analyzer turns the same text into waterproof, hike, boot.
Setting snowballStemmingForEnglish changed the query stems to waterproof hike boot. Two things caught us here. The deploy listed no refeed or reindex action, yet the documents already indexed kept their kStem tokens, so p02 lost its "boots" match until we fed the documents again. Make every linguistics change before the bulk load, or plan a refeed. Also, Elasticsearch's english analyzer drops stop words and Vespa's default linguistics keeps them. For p04 the description length was 8 terms in Elasticsearch (average 8.08) and 13 in Vespa (average 10.83). For custom analyzer chains, Lucene linguistics configures tokenizers and token filters per language in services.xml. We did not test it in this run.
Relevance parity, side by side
With per-field userInput() and Snowball stemming, the two engines returned the same order:
| Rank | Product | Elasticsearch _score | Vespa relevance |
|---|---|---|---|
| 1 | p01 Ridgeline Waterproof Hiking Boot | 21.543879 | 21.543879 |
| 2 | p12 Highland Waterproof Walking Boot | 14.442671 | 14.442671 |
| 3 | p02 Summit Mid Hiking Boots | 10.553432 | 10.553432 |
| 4 | p09 Moorland Wellington Boot | 5.444240 | 5.444240 |
| 5 | p04 Drizzle Trail Runner | 0.827849 | 0.762011 |
| 6 | p11 Scree Approach Shoe | 0.310981 | 0.335981 |
The top four match to six decimals because their best field is the title, where stop words never occur. Ranks 5 and 6 score on description only, where the field lengths differ by the stop words. The order still holds here. On real queries it will not always hold, and that is the measurement that matters.
For a real migration, take a few thousand logged queries, run each against both engines with the parity profile, and record top-10 overlap and the rank of each Elasticsearch top-3 hit in Vespa. Sort the queries by overlap and read the worst 50 with match-features open. Check the causes this run surfaced first: stemming, stop words, fieldset statistics and case handling. The judgement process in our relevancy techniques post applies when you need labelled queries for a decision.
A cutover plan, and how to sequence it
flowchart TD
A["Port mapping to schema<br/>and function_score to es_parity profile"] --> B["Bulk export with PIT,<br/>vespa feed"]
B --> C["Dual-write new changes<br/>to both engines"]
C --> D["Shadow queries:<br/>log top-10 overlap per query"]
D --> E{"Overlap acceptable<br/>on logged queries?"}
E -- no --> F["Fix linguistics, fieldsets,<br/>case, field lengths"]
F --> D
E -- yes --> G["Shift read traffic<br/>by percentage in the application"]
G --> H["Keep Elasticsearch on dual-write<br/>as rollback"]
H --> I["Then change ranking:<br/>second-phase, global-phase"]Index aliases have no Vespa equivalent, so the traffic switch belongs in your application or gateway, with a flag that sends a share of queries to each engine. On the Vespa side you rarely need an alias swap afterwards. Schema changes deploy in place, and the deploy output names any change that needs a restart, a refeed or a reindex. The reindexing guide covers re-running the indexing pipeline over stored documents, and for an atomic change to a field it suggests adding a synthetic field, reindexing, then switching queries over, which is the alias pattern inside one schema. Our zero-downtime Elasticsearch deployments post covers the alias side you are leaving.
Our recommendation is to keep the es_parity profile as the production profile until read traffic is fully on Vespa and Elasticsearch is only a rollback target. Only then write new profiles that inherit from it: priors in second-phase, a cross-encoder in global-phase, as in our phased ranking walkthrough. If the migration adds vector retrieval, the reciprocal rank fusion post shows the hybrid profile and its rerank-count trap. Whether the move is worth making at all is a separate question, which the Elasticsearch vs Vespa comparison covers.
Where we can help
We run Elasticsearch to Vespa migrations parity-first. That means porting the schema and rank profile, then proving parity on your query logs before any traffic moves. On the Elasticsearch side, our Elasticsearch consulting work includes making AlphaSense's Elasticsearch search 90% faster. For the Vespa side, see our Vespa consulting service.
Abhilash B is an advisor at Query Quotient and an information retrieval and AI expert who holds patents in the field. Abhilash brings that retrieval depth to the search, RAG and agent systems Query Quotient builds for clients.
