📚 Collection | 📝 Blogpost | 🛠️ Training and Evaluation
🎯 TL;DR: A 174M-parameter English sparse retrieval model built from DenseOn through sparse autoencoder (SAE) pretraining and supervised retrieval fine-tuning. SlideOn reaches 56.68 nDCG@10 on BEIR-15, surpassing DenseOn's 56.20, while producing sparse representations compatible with inverted-index search. Joint query/document Matryoshka training preserves retrieval quality under pruning, allowing search and storage costs to be adjusted at inference without retraining.
With DenseOn and LateOn, then mDenseOn and mLateOn, we explored dense and late-interaction retrieval using our open data recipe. SlideOn and mSlideOn reuse these models' retrieval knowledge to learn sparse representations through SAE pretraining followed by supervised fine-tuning.
SlideOn starts from DenseOn for English retrieval. mSlideOn starts from mLateOn for multilingual, long-context, and code retrieval. We release both the pretrained SAE initializations and the final retrieval models.
See our blog post for the full method, results, and ablations.
SlideOn encodes queries and documents into 32,768-dimensional nonnegative sparse vectors. A learned linear encoder maps each normalized DenseOn token representation to its top-32 latent activations, followed by ReLU. Max pooling across tokens produces one vector per query or document, and retrieval uses the dot product between these vectors.
The dimensions are learned features, independent of the tokenizer vocabulary. Active latents can be stored in an inverted index, where query latents select posting lists and their weights contribute to document scores.
SlideOn notably:
BEIR-15 scores are average nDCG@10 across 15 datasets, reported on a 0–100 scale. Sparse models use dot-product scoring and exact search over each full evaluation corpus. All rows use unpruned query and document vectors. Dense and late-interaction results are taken from our DenseOn and LateOn blog post.
SlideOn achieves the highest BEIR-15 score among the sparse models below 3B parameters in this comparison. It improves on SPARSEUP by 2.22 points and on DenseOn by 0.48 points, while remaining 0.54 points behind LateOn. The figures below show how pruning changes the comparison when search and storage costs are taken into account.
Query pruning reduces the number of posting lists consulted during search. Document pruning shortens those lists and reduces the number of stored document weights. Matryoshka training concentrates retrieval information in the strongest latents so that both budgets can be adjusted after training.
Retrieval FLOPS estimates the expected number of shared active features per query–document pair. It is a search-cost proxy, not measured latency or encoder FLOPs. Document NNZ is the average number of nonzero document features and serves as a storage-cost proxy, rather than a measurement of index size in bytes. Efficiency measurements use the fixed sample of 6,000 documents and 1,271 queries from SciFact, NFCorpus, and FiQA.
| Model | Description |
|---|---|
| SlideOn (this model) | English sparse retrieval model, after SAE pretraining and retrieval fine-tuning |
| SlideOn-SAE | DenseOn-based SAE initialization before retrieval fine-tuning |
| mSlideOn | Multilingual sparse retrieval model for natural language, long documents, and code |
| mSlideOn-SAE | mLateOn-based SAE initialization before retrieval fine-tuning |
| DenseOn | English dense retriever used to initialize SlideOn's backbone |
The SAE checkpoints are starting points for training. Use SlideOn or mSlideOn for the retrieval results reported in the blog.
See our blog post for training data, SAE pretraining, and supervised retrieval fine-tuning.
| Property | Value |
|---|---|
| Model type | Learned sparse retriever; Sentence Transformers SparseEncoder |
| Backbone | DenseOn / ModernBERT-base |
| Total parameters | 174M: 149M backbone + 25M sparse head |
| Maximum sequence length | 8,192 tokens, including prompts and special tokens |
| Output dimensionality | 32,768 learned latent features |
| Token representation | 768 dimensions, L2-normalized before sparse encoding |
| Token sparsity | Top-32 latent selection followed by ReLU |
| Sequence pooling | Maximum over token activations |
| Similarity function | Dot product |
| Prompts | query: for queries; document: for documents |
| Language | English |
Install the supported library versions:
pip install -U "sentence-transformers>=6.1,<6.2" "transformers>=5.3,<6"
Load the model and encode queries and documents. This example uses the q-64 / d-512 configuration:
from sentence_transformers import SparseEncoder
model = SparseEncoder("lightonai/SlideOn", trust_remote_code=True)
queries = ["What causes a solar eclipse?"]
documents = [
"A solar eclipse occurs when the Moon passes between the Sun and Earth.",
"Wind turbines convert the kinetic energy of wind into electricity.",
]
query_embeddings = model.encode_query(queries, top_k=64)
document_embeddings = model.encode_document(documents, top_k=512)
print(query_embeddings.shape, document_embeddings.shape)
# torch.Size([1, 32768]) torch.Size([2, 32768])
scores = model.similarity(query_embeddings, document_embeddings)
print(scores)
encode_query() and encode_document() apply the appropriate prompts and preprocessing automatically. Pass the raw text without manually adding prefixes. The returned embeddings are PyTorch sparse COO tensors. The model and custom inference code are included in this repository; loading does not require the training package or PyLate.
Use top_k independently on each encoding call to cap the number of nonzero features after pooling. This is separate from the per-token top-32 operation and does not change the vector dimensionality.
To reproduce the unpruned setting:
query_embeddings = model.encode_query(queries, top_k=0)
document_embeddings = model.encode_document(documents, top_k=0)
Omitting top_k also uses the saved unpruned defaults. To use the q-full / d-512 setting, keep the query call at top_k=0 and set the document call to top_k=512.
Dimensions represent learned latents, so they do not map directly to tokenizer vocabulary entries. Inspect the sparse indices and weights; SparseEncoder.decode() cannot label them as vocabulary tokens. Retrieval uses raw dot products, so leave final vector normalization disabled.
If you find this work useful for your research, please consider citing us:
@misc{moura2026slideon,
author = {Moura Junior, Paulo Roberto and Sourty, Raphaël},
title = {{SlideOn} \& {mSlideOn}: Sparsified late-interaction and dense retrievers},
year = {2026},
howpublished = {Hugging Face Blog},
url = {https://huggingface.co/blog/lightonai/slideon-mslideon}
}