How neural sparse search works

Neural sparse search first expands text, whether a query or a document, into a larger set of terms, each weighted by its semantic relevance. It then uses Lucene's term vector computation to identify the highest-scoring results. Like dense semantic matching, it interprets queries using semantic techniques, so it can handle terms that traditional lexical search might not understand, while sparse representations also capture specific terms and exact matches. OpenSearch states that the approach generalises well in out-of-domain scenarios.

  • Encodes text into (token, weight) entries
  • Builds and searches indexes via Lucene's inverted index
  • Captures both semantic similarity and specific terms
  • Introduced in OpenSearch 2.11

Sources: Introducing the neural sparse two-phase algorithm - OpenSearch, Improving search efficiency and accuracy with the newest v2 neural sparse models - OpenSearch

Index and memory characteristics

OpenSearch reports that neural sparse search uses a native Lucene index and therefore avoids any increase in RAM cost at search time, whereas dense encoding using k-NN retrieval increases RAM costs by 7.9% at search time. Index size is also smaller than with dense encoding. A document-only model generates an index that is 10.4% the size of a dense encoding index, and a bi-encoder index is 7.2% of a dense encoding index.

  • No RAM increase at search time with the native Lucene index
  • Doc-only index: 10.4% of a dense encoding index
  • Bi-encoder index: 7.2% of a dense encoding index

Source: Introducing the neural sparse two-phase algorithm - OpenSearch

Operating modes

Neural sparse search runs in doc-only mode and bi-encoder mode. In doc-only mode a sparse encoding model is registered for ingestion and retrieval requires no model inference; in bi-encoder mode a sparse encoding model is used at both ingestion and query time, so model inference speed affects client-side search latency. The sparse encoding model influences document scoring and ranking, and so directly affects search relevance.

  • doc-only mode — no inference required for retrieval
  • bi-encoder mode — model inference applied at query time

Source: Improving search efficiency and accuracy with the newest v2 neural sparse models - OpenSearch

The v2 series models

The v1 series models are based on the BERT base model, a 12-layer transformer with 133 million parameters, giving an inference cost 2x to 4x higher than dense embedding models such as tas-b and all-MiniLM-L6-v2. The v2 series reduces parameters using knowledge distillation from an ensemble of heterogeneous teacher models. The v2-distill model reduces parameters by 50% to 67M and supports both doc-only and bi-encoder modes, while v2-mini reduces parameters by 75% to 23M and supports doc-only mode. All v2 models are available in both OpenSearch and Hugging Face.

  • v2-distill — 67M parameters, AVG NDCG@10 of 0.528 in bi-encoder form
  • v2-distill doc — AVG NDCG@10 of 0.504, no inference for retrieval
  • v2-mini doc — 23M parameters, AVG NDCG@10 of 0.497
  • Ingestion throughput: v2-distill 1.39x on GPU, 1.74x on CPU
  • Ingestion throughput: v2-mini 1.74x on GPU, 4.18x on CPU
  • Available in OpenSearch and on Hugging Face

Source: Improving search efficiency and accuracy with the newest v2 neural sparse models - OpenSearch

The two-phase search pipeline

OpenSearch 2.15 introduced a two-phase search pipeline that splits neural sparse query terms into high-scoring tokens and low-scoring tokens. In the initial phase the algorithm selects candidate documents using the high-scoring tokens; in the recalculation phase it rescores those candidates using both high- and low-scoring tokens. OpenSearch reports p99 latency falling from 198 ms to 124 ms in doc-only mode and from 617 ms to 122 ms in bi-encoder mode. Use of the pipeline involves setting up a neural_sparse_two_phase_processor and setting it as the default search pipeline.

  • Introduced in OpenSearch 2.15
  • Initial phase selects candidates from high-scoring tokens
  • Recalculation phase rescores using all query tokens
  • Configured via neural_sparse_two_phase_processor

Source: Introducing the neural sparse two-phase algorithm - OpenSearch

Deployment options

Models can be deployed remotely on a GPU or locally on a CPU, and the published throughput figures cover both. The OpenSearch blog documents registering and deploying v2 models separately for doc-only mode and for bi-encoder mode. Model inference speed affects ingestion throughput in both modes and client-side search latency in bi-encoder mode.

  • Remote deployment on a GPU
  • Local deployment on a CPU
  • Separate registration steps for doc-only and bi-encoder modes

Source: Improving search efficiency and accuracy with the newest v2 neural sparse models - OpenSearch

Sources

  1. Improving search efficiency and accuracy with the newest v2 neural sparse models - OpenSearch https://opensearch.org/blog/neural-sparse-v2-models/ Verified 01 Nov 2026
  2. Introducing the neural sparse two-phase algorithm - OpenSearch https://opensearch.org/blog/introducing-a-neural-sparse-two-phase-algorithm/ Verified 01 Nov 2026

Last verified 01 Nov 2026. This entry is compiled from the public web pages listed above. Nothing here is stated that those pages do not, and each of them was read on the date shown.