LFM2.5-Encoder-350M

LFM2.5-Encoder is a family of multilingual bidirectional encoders built on the LFM2 architecture, available in two sizes:

• LFM2.5-Encoder-230M — a lightweight encoder for tight latency and memory budgets, punching above its size class.
• LFM2.5-Encoder-350M (this model) — a larger sibling for maximum downstream quality.

Both are masked language models with full bidirectional attention, designed to be fine-tuned into task-specific models (classification, token classification, retrieval, reranking, and semantic similarity) across 15 languages, and to run efficiently on-device.

Find more details about our encoders in our blog post.

Key highlights:

• Top quality for its size. Ahead of every model its size or smaller, and ~5 points above our own retrieval siblings.
• General-purpose. 8k context, strong across NLI, paraphrase, sentiment, and multilingual tasks.
• Fast and on-device. Matches or beats ModernBERT throughput, with a long-context edge on CPU.

[!NOTE] 💻 Demos: We built the demos below from fine-tuned LFM2.5-Encoders. Each one runs in a CPU-only Hugging Face space:

• Zero-shot prompt routing — define your own routing lanes as free text. The model scores the whole prompt against every lane in one pass.
• Zero-shot policy linting — check text against your company's rules, written as free text. It scores every token against every rule in one pass.
• Spell checking — correct misspellings token by token.
• PII detection — spot and remove 40 kinds of personal information across 16 languages.
• Masked-diffusion text generation — bonus: run the encoder as a chatbot that generates text by iteratively unmasking instead of left to right.

📄 Model details

Property	LFM2.5-Encoder-230M	LFM2.5-Encoder-350M
Type	Bidirectional encoder (masked language model)	Bidirectional encoder (masked language model)
Backbone	LFM2	LFM2
Total parameters	~229.7M	~354.5M
Hidden size	1024	1024
Vocabulary size	65,536	65,536
Context length	8,192 tokens	8,192 tokens
License	LFM Open License v1.0	LFM Open License v1.0

Supported languages: English, German, Spanish, French, Italian, Dutch, Polish, Portuguese, Arabic, Hindi, Japanese, Russian, Turkish, Vietnamese, Chinese (15).

Architecture. LFM2.5-Encoder is built on the LFM2 hybrid backbone, which interleaves gated short-convolution blocks with grouped-query attention. For encoder use, the causal mask is replaced with full bidirectional (non-causal) attention and the model is trained with a masked language modeling head. The encoder body is exposed as Lfm2BidirectionalModel; masked-LM loading uses Lfm2BidirectionalForMaskedLM. Both are wired through auto_map and require trust_remote_code=True.

Lfm2BidirectionalForMaskedLM(
  (lfm2): Lfm2BidirectionalModel
  (lm_head): Linear(in_features=1024, out_features=65536, bias=False)
)

Training. LFM2.5-Encoder-350M is adapted from the LFM2 base and trained with a masked language modeling objective on a large multilingual corpus. Pre-training uses a two-stage schedule that extends the context window to up to 8,192 tokens.

We recommend fine-tuning LFM2.5-Encoder-350M for a range of downstream tasks, such as:

• Text classification: sentiment, topic, intent/routing, moderation, and business-text linting.
• Token classification: named-entity recognition, span extraction, and sequence labeling.
• Retrieval and reranking: a backbone for dense embedding or late-interaction (ColBERT-style) retrievers.
• Semantic similarity: STS, paraphrase, and duplicate detection.
• Natural language inference and extractive QA: sentence-pair reasoning and answer-span extraction.

🏃 How to run

Install the latest version of transformers:

pip install -U transformers

Run masked-token prediction:

from transformers import AutoModelForMaskedLM, AutoTokenizer
import torch

tok = AutoTokenizer.from_pretrained("LiquidAI/LFM2.5-Encoder-350M", trust_remote_code=True)
mlm = AutoModelForMaskedLM.from_pretrained("LiquidAI/LFM2.5-Encoder-350M", trust_remote_code=True)

text = f"The capital of France is {tok.mask_token}."
enc = tok(text, return_tensors="pt")
with torch.no_grad():
    logits = mlm(**enc).logits
pos = (enc["input_ids"][0] == tok.mask_token_id).nonzero()[0].item()
print([tok.decode([t]).strip() for t in logits[0, pos].topk(5).indices.tolist()])
# -> ['Paris', 'Strasbourg', 'Paris', 'Lyon', 'Versailles']

For downstream tasks, load the encoder body and attach your own head (classification, token classification, regression, retrieval):

from transformers import AutoModel
body = AutoModel.from_pretrained("LiquidAI/LFM2.5-Encoder-350M", trust_remote_code=True)

If your GPU supports it, we recommend using LFM2.5-Encoder-350M with Flash Attention 2 to reach the highest efficiency. To do so, install Flash Attention as follows, then use the model as normal:

pip install flash-attn

📊 Performance

For each benchmark task, we run a full supervised fine-tune and report that fine-tuned model's score. The results below span 14 models across 17 tasks from GLUE, SuperGLUE, and multilingual classification tasks. The full evaluation harness is open-sourced in the eurobert-repro repository.

17-task results (avg@5 fresh seeds ± std)

Rank	Model	Params	17-task mean	± std
1	XLM-R XL (3.5B)	3.5B	83.06	±1.16
2	ModernBERT-large (395M)	395M	81.68	±2.49
3	XLM-R large (560M)	560M	81.34	±1.66
4	LFM2.5-Encoder-350M (ours)	350M	81.02	±1.00
5	mDeBERTa-v3 (280M)	280M	80.37	±1.06
6	LFM2.5-Encoder-230M (ours)	230M	79.29	±1.02
7	ModernBERT-base (149M)	149M	78.19	±1.39
8	XLM-R base (280M)	280M	77.46	±1.63
9	EuroBERT-210M	210M	76.87	±2.00
10	mGTE-MLM (305M)	305M	76.53	±1.85
11	LFM2.5-ColBERT-350M	350M	76.18	±1.25
12	EuroBERT-610M	610M	75.87	±2.03
13	LFM2.5-Embedding-350M	350M

Model	XNLI	PAWS-X	Amazon	MASSIVE	SeaHorse	CoLA*	SST-2*	MRPC*	STS-B*	QQP*	MNLI*	QNLI*	RTE*	BoolQ*	CB*	WiC*	WSC*	ALL
XLM-R XL (3.5B)	87.12±0.45	93.30±0.41	62.29±0.05	88.21±0.32	59.51±3.52	84.58±1.18	95.69±0.30	87.65±1.69	90.20±0.93	91.72±0.07	90.09±0.11	94.47±0.28	82.38±3.51	83.70±0.24	89.88±3.72	66.55±1.37	64.62±1.58	83.06
ModernBERT-large (395M)	81.76±0.38	92.46±0.18	60.42±0.15	85.65±0.94	40.20±17.18	83.37±0.20	96.10±0.53	88.14±1.79	92.16±0.24	91.81±0.13	90.65±0.18	94.36±0.10	81.59±4.92	81.68±2.34	88.21±3.24	70.16±2.57	69.81±7.18	81.68
XLM-R large (560M)	84.69±0.59	93.23±0.78	61.58±0.13	88.50±0.17	56.12±2.31	83.34±1.75	93.83±1.04	88.77±2.15	91.35±0.23	90.48±0.23	88.29±0.07	93.08±0.23	80.79±3.14	80.54±0.79	78.21±8.69	66.24±5.41	63.65±0.43	81.34
LFM2.5-Encoder-350M (ours)	79.82±0.29	91.53±0.73	60.57±0.09	85.70±0.13	54.96±0.41	84.43±0.81	95.11±0.26	87.21±1.86	91.59±0.05	92.08±0.10	89.03±0.17	93.97±0.23	75.23±3.84	81.52±0.69	83.21±2.04	69.66±2.23	61.73±3.15	81.02
mDeBERTa-v3 (280M)	83.01±0.47	92.59±0.39	60.62±0.31

* = dev split (GLUE/SuperGLUE test labels hidden). The 5 multilingual columns are labeled test. SeaHorse & STS-B are Spearman×100. All other tasks are accuracy.

Inference speed

The LFM2 backbone was built for fast inference, and the encoders inherit it. While ModernBERT-base is faster at short sequences in Apple GPU inputs, LFM2.5-Encoders overtake it as inputs grow. At long input sequences of 8k on CPU, the encoders run 3.3× faster than ModernBERT-base.

🔧 Fine-tuning

LFM2.5-Encoder-350M follows standard BERT-style fine-tuning. Attach a task head to the encoder body and train end-to-end. Suggested starting points (tune per task):

Hyperparameter	Suggested range
Learning rate	1e-5 – 5e-5
Warmup ratio	0.1
Weight decay	0.1
Epochs	3 – 20 (early stopping, patience 3)
Precision	bf16 autocast (fp32 master weights)

📬 Contact

• Got questions or want to connect? Join our Discord community
• If you are interested in custom solutions with edge deployment, please contact our sales team.

Citation

@article{liquidAI2026Encoders,
  author = {Liquid AI},
  title = {LFM2.5-Encoders: Fast at Long Context, Even on CPU},
  journal = {Liquid AI Blog},
  year = {2026},
  note = {www.liquid.ai/blog/lfm2-5-encoders},
}