Back to blog
Architecture

Migrating from Elasticsearch to Vespa: mappings to schemas, Query DSL to YQL, scoring to rank profiles

An Elasticsearch to Vespa migration, run on both engines: one mapping, one bool plus function_score query, and the Vespa schema, YQL and rank profile that reproduce its ranking on Vespa 8.778, top four scores included.

Abhilash B11 min read
#vespa#elasticsearch#migration#yql#ranking

An Elasticsearch to Vespa migration maps each index to a Vespa schema, each mapping field to a schema field with an indexing: statement, each bool query to a YQL where clause, and each function_score to a rank profile expression. The data moves as JSON put operations through vespa feed. Port the existing relevance exactly first, prove parity on real queries, cut over, and only then start using Vespa's phased ranking. Moving the engine and changing the ranking in the same step leaves you unable to tell a migration bug from a ranking change.

Everything below was run on 31 October 2026 with Elasticsearch 9.5.5 (Lucene 10.5.1, single node, security off) and Vespa 8.778.45 (single content node, Vespa CLI 8.778.45), both in Docker. The corpus is 12 outdoor products written for this post. It is illustrative, small enough to read every score, and the results are a parity demonstration only. They are not a benchmark.

The Elasticsearch to Vespa migration concept map

Most Elasticsearch concepts have a direct Vespa counterpart. A few have none, and those are where migrations stall.

ElasticsearchVespaNotes from our run
IndexDocument type inside a schema, deployed in an application packageOne .sd file per document type, plus services.xml
Shards and replicasContent cluster nodes and redundancy / min-redundancy in services.xmlNo shard count to pick; Vespa distributes buckets across nodes
text fieldindexing: index plus index: enable-bm25bm25(field) only works on fields with enable-bm25
keyword fieldindexing: attribute, attribute: fast-search for filteringCase-insensitive by default; ES keyword is case-sensitive
integer, float, booleanint, float, bool attributesscaled_float has no equivalent; we used double
datelong attribute holding epoch secondsConvert during export
Multi-field (title.raw)Synthetic field outside the document block, indexing: input title | attributeRecomputed by Vespa on every feed
AnalyzerLinguistics: OpenNLP by default, or Lucene linguisticsStemmers differ; see the stemming section
bool filter clausesYQL and with attribute predicatesAttribute matches add no text score
multi_match with ^ boostsuserInput() per field, boosts in the rank profileYQL has no per-field query boost
function_scorefirst-phase expression in a rank-profilelog1p becomes log10(1 + x)
_explainmatch-features on each hitDeclared per rank profile, returned on every hit
_bulkvespa feed or vespa-feed-clientJSONL of put operations
_reindexReindexing from stored documents, or a refeed from sourceSee the cutover section
Index aliasesNo direct equivalentSchema changes apply in place; route at the application layer

Vespa has an official Migrating from Elasticsearch guide built around ElasticDump and a parser script that converts a mapping into a schema. It describes that script as "a basic script with minimal error checking". It gives you a first schema to edit. It does not carry over analyzers or scoring, which is the part this post covers.

The Elasticsearch side

The mapping uses the built-in english analyzer on two text fields, keyword for facets, and a title.raw multi-field:

{
  "mappings": {
    "properties": {
      "sku":         { "type": "keyword" },
      "title":       { "type": "text", "analyzer": "english",
                       "fields": { "raw": { "type": "keyword" } } },
      "description": { "type": "text", "analyzer": "english" },
      "brand":       { "type": "keyword" },
      "category":    { "type": "keyword" },
      "price":       { "type": "scaled_float", "scaling_factor": 100 },
      "rating":      { "type": "float" },
      "sales":       { "type": "integer" },
      "in_stock":    { "type": "boolean" }
    }
  }
}

The query is a common product search shape: a multi_match with a title boost, three filters, and a popularity multiplier from function_score:

{
  "size": 10,
  "_source": ["title", "brand", "price", "sales"],
  "query": {
    "function_score": {
      "query": {
        "bool": {
          "must": {
            "multi_match": {
              "query": "waterproof hiking boots",
              "fields": ["title^3", "description"]
            }
          },
          "filter": [
            { "terms": { "category": ["boots", "trail-shoes"] } },
            { "range": { "price": { "lte": 200 } } },
            { "term":  { "in_stock": true } }
          ]
        }
      },
      "field_value_factor": { "field": "sales", "modifier": "log1p", "missing": 0 },
      "boost_mode": "multiply"
    }
  }
}

Read the scoring as a formula before porting it. multi_match defaults to best_fields, which takes the single best field score, so the text part is max(3 × BM25(title), BM25(description)). The filters add nothing to the score. The Elasticsearch docs define log1p as adding 1 and taking the common logarithm, and boost_mode: multiply multiplies the two. The full score is:

max(3 × BM25(title), BM25(description)) × log10(1 + sales)

Six of the 12 products pass the filters. The top hit came back as "_id": "p01", "_score": 21.543879.

The Vespa schema

Each mapping row becomes a field. Text fields get index and enable-bm25. Facets and numerics become attributes. The title.raw multi-field becomes a synthetic field declared outside the document block, which Vespa fills from title at feed time:

schema product {
    document product {
        field sku type string {
            indexing: attribute | summary
        }
        field title type string {
            indexing: index | summary
            index: enable-bm25
        }
        field description type string {
            indexing: index | summary
            index: enable-bm25
        }
        field brand type string {
            indexing: attribute | summary
            attribute: fast-search
        }
        field category type string {
            indexing: attribute | summary
            attribute: fast-search
            match: cased
        }
        field price type double {
            indexing: attribute | summary
        }
        field rating type float {
            indexing: attribute | summary
        }
        field sales type int {
            indexing: attribute | summary
        }
        field in_stock type bool {
            indexing: attribute | summary
        }
    }

    field title_raw type string {
        indexing: input title | attribute
        match: exact
    }

    fieldset default {
        fields: title, description
    }

    rank-profile es_parity {
        function text() {
            expression: max(3 * bm25(title), bm25(description))
        }
        first-phase {
            expression: text * log10(1 + attribute(sales))
        }
        match-features: text bm25(title) bm25(description) attribute(sales)
    }
}

The rank profile is the function_score formula written out. The match-features line returns every input on each hit, which replaces _explain during parity work. Vespa's BM25 reference uses the same IDF formula as Lucene, log(1 + (N - n + 0.5) / (n + 0.5)), with k1 = 1.2 and b = 0.75 by default, so identical inputs give identical scores.

Case sensitivity needs a decision per field. Without match: cased, category in ('BOOTS') matched 6 documents in Vespa while the Elasticsearch terms filter matched 0. With match: cased both engines returned 0 for BOOTS and 6 for boots. Adding it to a live field made the deploy report a required restart of the search node ("change property 'dictionary: cased/uncased' from 'UNCASED' to 'CASED'"), so settle it before the first production deploy.

services.xml is minimal: one container with the document and search APIs, and one content cluster. The config element is the stemming fix explained below.

<services version="1.0">
  <container id="default" version="1.0">
    <config name="ai.vespa.opennlp.open-nlp">
      <snowballStemmingForEnglish>true</snowballStemmingForEnglish>
    </config>
    <document-api/>
    <search/>
    <nodes><node hostalias="node1"/></nodes>
  </container>
  <content id="shop" version="1.0">
    <min-redundancy>1</min-redundancy>
    <documents>
      <document type="product" mode="index"/>
    </documents>
    <nodes><node hostalias="node1" distribution-key="0"/></nodes>
  </content>
</services>

Feeding: export from Elasticsearch, vespa feed into Vespa

A Vespa document id has the form id:<namespace>:<document-type>::<user-id>. We exported with a point-in-time and search_after on _shard_doc, which pages through an index without a scroll context, and wrote one put per hit:

pit = call("POST", "/products/_pit?keep_alive=2m")["id"]
after = None
with open("feed.jsonl", "w") as out:
    while True:
        body = {"size": 500, "pit": {"id": pit, "keep_alive": "2m"},
                "sort": [{"_shard_doc": "asc"}]}
        if after:
            body["search_after"] = after
        hits = call("POST", "/_search", body)["hits"]["hits"]
        if not hits:
            break
        for h in hits:
            out.write(json.dumps({"put": f"id:shop:product::{h['_id']}",
                                  "fields": h["_source"]}) + "\n")
        after = hits[-1]["sort"]
vespa feed feed.jsonl
{
  "feeder.operation.count": 12,
  "feeder.ok.count": 12,
  "feeder.error.count": 0,
  "http.response.code.counts": { "200": 12 }
}

Type conversion belongs in this export step. Dates become epoch long values and scaled_float becomes double or integer cents. Drop any _source field the schema doesn't declare, because the document API rejects a put that names an unknown field.

sequenceDiagram
  participant App as Application
  participant ES as Elasticsearch
  participant X as Export job
  participant V as Vespa
  X->>ES: open point-in-time, search_after pages
  ES-->>X: _source hits
  X->>X: convert types, build put operations
  X->>V: vespa feed feed.jsonl
  App->>ES: live writes
  App->>V: same writes, dual-written
  App->>V: shadow queries, results logged only

Query DSL to YQL, and the fieldset trap

The direct translation of the query uses userQuery(), which searches the default fieldset:

{
  "yql": "select sku, title, brand, price, sales from product where userQuery() and category in ('boots', 'trail-shoes') and price <= 200 and in_stock = true",
  "query": "waterproof hiking boots",
  "ranking": "es_parity",
  "hits": 10
}

It returned the same 6 documents, with scores about 20 times lower than Elasticsearch: 1.1002 for p01 against 21.5439. The queryTermDocumentFrequency(title) rank feature showed why. Through the fieldset, every query term reported a document frequency of 12, the whole corpus, so every IDF collapsed to the same small value. Querying title contains "waterproof" directly reported 4, which matches the n in Elasticsearch's _explain. A probe with laces, which appears in the title and the description of one product, reported 2, so in our run the fieldset value was the sum of the per-field document frequencies. We found no statement of this in the docs, so check it on your version with the same feature.

The fix is to query each field separately, which is also closer to what multi_match does. userInput() takes a defaultIndex and a grammar annotation (see the YQL reference), and grammar: 'any' gives Elasticsearch's default or operator:

{
  "yql": "select sku, title, price, sales from product where (({defaultIndex: 'title', grammar: 'any'}userInput(@q)) or ({defaultIndex: 'description', grammar: 'any'}userInput(@q))) and category in ('boots', 'trail-shoes') and price <= 200 and in_stock = true",
  "q": "waterproof hiking boots",
  "ranking": "es_parity",
  "hits": 10
}
{
  "relevance": 21.543878632632797,
  "fields": {
    "matchfeatures": {
      "attribute(sales)": 1840.0,
      "bm25(description)": 0.5985169322638356,
      "bm25(title)": 2.199440910538444,
      "text": 6.5983227316153314
    },
    "sku": "p01",
    "title": "Ridgeline Waterproof Hiking Boot"
  }
}

bm25(title) of 2.1994 is the exact sum of the three per-term weights in Elasticsearch's _explain for the same document. The default model.type for userQuery() is weakAnd, an approximate OR that passes a limited number of hits to ranking, 100 by default. On a large corpus that changes recall against Elasticsearch's exhaustive OR, so pick the grammar on purpose.

Analyzers to linguistics: the stemming difference

Before the services.xml fix, the ranking was wrong as well as the scores: "Summit Mid Hiking Boots" came first. The query trace (trace.level: 3) showed the stemmed query as WEAKAND waterproof hike boots. Vespa's default OpenNLP linguistics uses kStem for English, which the OpenNLP linguistics page describes as a simpler stemmer with lower recall. kStem reduced hiking to hike and left boots alone, so among the filtered products it only matched the one whose text contains "boots". Elasticsearch's english analyzer turns the same text into waterproof, hike, boot.

Setting snowballStemmingForEnglish changed the query stems to waterproof hike boot. Two things caught us here. The deploy listed no refeed or reindex action, yet the documents already indexed kept their kStem tokens, so p02 lost its "boots" match until we fed the documents again. Make every linguistics change before the bulk load, or plan a refeed. Also, Elasticsearch's english analyzer drops stop words and Vespa's default linguistics keeps them. For p04 the description length was 8 terms in Elasticsearch (average 8.08) and 13 in Vespa (average 10.83). For custom analyzer chains, Lucene linguistics configures tokenizers and token filters per language in services.xml. We did not test it in this run.

Relevance parity, side by side

With per-field userInput() and Snowball stemming, the two engines returned the same order:

RankProductElasticsearch _scoreVespa relevance
1p01 Ridgeline Waterproof Hiking Boot21.54387921.543879
2p12 Highland Waterproof Walking Boot14.44267114.442671
3p02 Summit Mid Hiking Boots10.55343210.553432
4p09 Moorland Wellington Boot5.4442405.444240
5p04 Drizzle Trail Runner0.8278490.762011
6p11 Scree Approach Shoe0.3109810.335981

The top four match to six decimals because their best field is the title, where stop words never occur. Ranks 5 and 6 score on description only, where the field lengths differ by the stop words. The order still holds here. On real queries it will not always hold, and that is the measurement that matters.

For a real migration, take a few thousand logged queries, run each against both engines with the parity profile, and record top-10 overlap and the rank of each Elasticsearch top-3 hit in Vespa. Sort the queries by overlap and read the worst 50 with match-features open. Check the causes this run surfaced first: stemming, stop words, fieldset statistics and case handling. The judgement process in our relevancy techniques post applies when you need labelled queries for a decision.

A cutover plan, and how to sequence it

flowchart TD
  A["Port mapping to schema<br/>and function_score to es_parity profile"] --> B["Bulk export with PIT,<br/>vespa feed"]
  B --> C["Dual-write new changes<br/>to both engines"]
  C --> D["Shadow queries:<br/>log top-10 overlap per query"]
  D --> E{"Overlap acceptable<br/>on logged queries?"}
  E -- no --> F["Fix linguistics, fieldsets,<br/>case, field lengths"]
  F --> D
  E -- yes --> G["Shift read traffic<br/>by percentage in the application"]
  G --> H["Keep Elasticsearch on dual-write<br/>as rollback"]
  H --> I["Then change ranking:<br/>second-phase, global-phase"]

Index aliases have no Vespa equivalent, so the traffic switch belongs in your application or gateway, with a flag that sends a share of queries to each engine. On the Vespa side you rarely need an alias swap afterwards. Schema changes deploy in place, and the deploy output names any change that needs a restart, a refeed or a reindex. The reindexing guide covers re-running the indexing pipeline over stored documents, and for an atomic change to a field it suggests adding a synthetic field, reindexing, then switching queries over, which is the alias pattern inside one schema. Our zero-downtime Elasticsearch deployments post covers the alias side you are leaving.

Our recommendation is to keep the es_parity profile as the production profile until read traffic is fully on Vespa and Elasticsearch is only a rollback target. Only then write new profiles that inherit from it: priors in second-phase, a cross-encoder in global-phase, as in our phased ranking walkthrough. If the migration adds vector retrieval, the reciprocal rank fusion post shows the hybrid profile and its rerank-count trap. Whether the move is worth making at all is a separate question, which the Elasticsearch vs Vespa comparison covers.

Where we can help

We run Elasticsearch to Vespa migrations parity-first. That means porting the schema and rank profile, then proving parity on your query logs before any traffic moves. On the Elasticsearch side, our Elasticsearch consulting work includes making AlphaSense's Elasticsearch search 90% faster. For the Vespa side, see our Vespa consulting service.

Advisor, Query Quotient

Abhilash B is an advisor at Query Quotient and an information retrieval and AI expert who holds patents in the field. Abhilash brings that retrieval depth to the search, RAG and agent systems Query Quotient builds for clients.

Keep reading

Building something similar?

Tell us what you are working on. In a free 30-minute call we will give you an honest view of the approach, the effort and how to measure success.