Mythos2.0-2B

A Frontier Sparse Mixture-of-Experts (SMoE) Foundation Translation Model for 500+ Global Languages

---

Welcome to Mythos AI

Mythos is an open-source foundation model family built by Adithyan AI. In this organization, we develop and open-source state-of-the-art Sparse Mixture-of-Experts (SMoE) language models, universal translation engines, parallel multilingual datasets, and ultra-efficient inference runtimes targeting 500+ languages.

• Mission: Bridge the digital divide for underserved languages worldwide through efficient, open-weights AI architectures.
• 100% Free and Open-Source: Released under the permissive Apache 2.0 license with zero paywalls, metered tokens, or subscription fees.
• Community: Explore our models and datasets on Hugging Face and view development on GitHub.

---

🌐 Supported Languages Directory (500+ Languages)

Mythos2.0-2B was trained on a massive 16.5B sentence-pair parallel corpus covering 500+ languages and regional dialects across Africa, the Americas, Asia, Europe, and Oceania.

To translate any text into a desired language, simply prepend the target language tag <2code> (e.g. <2es> for Spanish, <2ml> for Malayalam, <2hi> for Hindi, <2fr> for French, <2de> for German, <2ta> for Tamil).

Major Language Hubs Supported:

• Global Commercial Languages: English (eng), Spanish (spa), French (fra), German (deu), Italian (ita), Portuguese (por), Russian (rus), Mandarin Chinese (cmn), Japanese (jpn), Korean (kor), Arabic (ara), Turkish (tur), Vietnamese (vie), Indonesian (ind), Dutch (nld), Polish (pol).

• South Asian & Indian Languages: Hindi (hin), Malayalam (mal), Tamil (tam), Telugu (tel), Bengali (ben), Marathi (mar), Gujarati (guj), Kannada (kan), Punjabi (pan), Urdu (urd), Odia (ori), Assamese (asm), Sanskrit (san), Nepali (nep), Sinhala (sin), Maithili (mai), Bhojpuri (bho), Sindhi (snd), Kashmiri (kas), Konkani (kok).

• African Languages: Swahili (swa), Amharic (amh), Yoruba (yor), Igbo (ibo), Hausa (hau), Somali (som), Oromo (orm), Zulu (zul), Xhosa (xho), Shona (sna), Tigrinya (tir), Malagasy (mlg), Kinyarwanda (kin), Lingala (lin), Bambara (bam), Wolof (wol).

• European & Slavic Languages: Ukrainian (ukr), Czech (ces), Romanian (ron), Greek (ell), Hungarian (hun), Danish (dan), Finnish (fin), Norwegian (nob), Swedish (swe), Bulgarian (bul), Croatian (hrv), Serbian (srp), Slovak (slk), Catalan (cat), Basque (eus), Galician (glg), Irish (gle), Welsh (), Scottish Gaelic ().

[!TIP] Universal Language Prompting:
To translate into any supported language, prepend <2{iso_code}> to your source text (e.g., <2es> for Spanish, <2hi> for Hindi, <2fr> for French, <2de> for German, <2ml> for Malayalam). All 500+ ISO-639 language codes are mapped directly into the model vocabulary and registered in the metadata above for automatic Hugging Face search filtering.

---

Mythos2.0 Model Collections

Following the modular design of frontier foundation families like Qwen, the Mythos2.0 Series spans foundation models, specialized context engines, and quantized edge runtimes:

Collection / Model	Architecture	Parameters (Total / Active)	Context Window	Target Capability	Status
Mythos2.0-2B	Sparse MoE (8E, Top-2)	2.04B / 678.7M	8,192	Flagship Universal 500+ Language Translation	🟢 Active Run
Mythos2.0-4B	Sparse MoE (16E, Top-2)	4.10B / 1.10B	16,384	Long Document and Legal/Technical Translation	🔄 Pipeline
Mythos2.0-Edge-2B	2-bit / 4-bit SMoE	2.04B (~1.2 GB RAM)	4,096	Sub-2-bit Edge and Mobile Phone Deployment	📋 In Dev
Mythos-Tokenizer	Byte-Level BPE	128,000 Vocab	-	Balanced Compression for 552 Global Languages	🟢 Available
Mythos-16B-Corpus	Parallel Bilingual Corpus	16 Billion Pairs	-	Bicleaner & LASER Curated Parallel Training Data	🟢 Open Data

Explore all models in the official Hugging Face Collection:
👉 https://huggingface.co/collections/AdithyanAI

---

Key Features of Mythos2.0-2B

1. 500+ Global Languages Supported: Native, high-fidelity translation across major world languages plus 250+ underserved African, Indigenous American, and Regional South/Central Asian languages with zero coverage in commercial translation APIs.
2. Sparse Mixture-of-Experts Efficiency: Employs 8 SwiGLU experts with Top-2 routing. With 2.04B total parameters, only 678.7M parameters are activated per token, delivering the translation capacity of a 7B-class model with the inference speed and memory footprint of a sub-1B model.
3. 8k Native Context with Document Packing: Features a native 8,192-token context window (4,096 encoder + 4,096 decoder) with Block-Diagonal Attention Packing. Translates whole articles, SRT/VTT subtitles, and markdown documents without chunking or losing discourse context.
4. FP32 Master Precision Embeddings: Maintains a 131M-parameter shared 3-way tied embedding table (src_embed, tgt_embed, proj.weight) in full 32-bit FP32 master weights, ensuring stable representation across rare scripts.

---

Comparison with Frontier Models

Specification	Mythos2.0-2B	TranslateGemma-7B	NLLB-200 (3.3B)	Google Cloud API
Architecture	Sparse MoE (8E, Top-2)	Dense Transformer	Dense Enc-Dec	Proprietary LLM
Total Parameters	2.04B	7.0B	3.3B	Closed
Active Parameters / Token	678.7M	7.0B	3.3B	Closed
Context Window	8,192 tokens	2,048 tokens	1,024 tokens	Dynamic
Supported Languages	500+	55	200	189
Min Inference VRAM	~4 GB	16 GB	8 GB	Cloud API
License	Apache 2.0 (100% Free)	Community License	CC-BY-NC 4.0	Paid Metered API

---

Model Architecture Overview

• Model Family: Mythos2.0
• Model ID: AdithyanAI/Mythos2.0-2B
• Architecture: Encoder-Decoder Sparse Mixture-of-Experts (SMoE)
• Total Parameters: 2.04B (2,037,643,264)
• Active Parameters / Token: 678.7M (678,688,768)
• Layers: 24 Transformer Blocks (12 Encoder Layers + 12 Decoder Layers)
• Hidden Dimension ($d_{\text{model}}$): 1,024
• Attention Mechanism: Grouped Query Attention (GQA)
  • 16 Query Heads, 4 Key-Value Head Groups (4x KV compression)
  • Head Dimension: 64
  • RoPE Base Frequency: $\theta = 100,000.0$
• Feed-Forward Network (Sparse MoE):
  • 8 SwiGLU Experts per layer (Intermediate Dim: 3,072)
  • Top-2 Routing with Switch-Transformer Capacity Factor (1.35)
  • Calibrated Load Balancing Loss (aux_loss_weight = 0.01)
• Context Capacity: 8,192 tokens (4k Source + 4k Target Document-Packed)
• Vocabulary: 128,000 Byte-Level BPE Tokens (552 languages)

---

Quickstart: Free Offline Inference

1. Installation

pip install torch transformers tokenizers sacrebleu

2. Python Inference Code

import torch
from tokenizers import Tokenizer

# Load Tokenizer
tokenizer = Tokenizer.from_file("multilingual_tokenizer.json")
sos_id = tokenizer.token_to_id("[SOS]")
eos_id = tokenizer.token_to_id("[EOS]")
pad_id = tokenizer.token_to_id("[PAD]")

def translate(model, text: str, tgt_lang: str = "fra", max_len: int = 128, device: str = "cuda:0"):
    clean_tgt = tgt_lang.split("_")[0]
    prompt = f"<2{clean_tgt}> {text}"
    tokens = tokenizer.encode(prompt).ids
    src_tensor = torch.tensor([tokens], dtype=torch.long, device=device)
    src_mask = (src_tensor != pad_id).unsqueeze(1).unsqueeze(2)

    with torch.no_grad():
        enc_out = model.encode(src_tensor, src_mask)
        gen_tokens = torch.tensor([[sos_id, tokenizer.token_to_id(f"<2{clean_tgt}>")]], device=device)
        
        for _ in range(max_len):
            cur_len = gen_tokens.size(1)
            causal_mask = torch.tril(torch.ones((cur_len, cur_len), dtype=torch.bool, device=device)).unsqueeze(0).unsqueeze(0)
            dec_out = model.decode(gen_tokens, enc_out, src_mask, causal_mask)
            logits = model.project(dec_out[:, -1:])
            next_token = logits.argmax(dim=-1).item()
            if next_token == eos_id:
                break
            gen_tokens = torch.cat([gen_tokens, torch.tensor([[next_token]], device=device)], dim=1)

    return tokenizer.decode(gen_tokens[0].tolist()[2:])

---

Training Infrastructure and Engineering

• Distributed Engine: PyTorch Fully Sharded Data Parallel (FSDP) and DDP.
• Precision Policy: Native 32-bit FP32 Master Weights with FP16 compute and unscaled FP32 logits projection.
• Zero-Host RAM Footprint: Streaming disk-spooler architecture keeping host CPU memory strictly below < 1.0 GB throughout training.
• Router Stabilization: Switch-Transformer dynamic capacity factor capping (capacity_factor = 1.35) with calibrated auxiliary loss (0.01) preventing expert collapse.

---

Community and Collaboration

We welcome researchers, linguists, and compute sponsors to join the Mythos AI initiative:

• Compute Sponsors: Pooling idle GPU hours (RTX 3090/4090, A100, H100) to scale Mythos2.0-4B and Mythos2.0-7B pre-training.
• Researchers and Engineers: Optimizing routing loss, sparse kernels, and sub-2-bit quantization.
• Native Linguists: Auditing translation quality and expanding low-resource parallel corpora.

---

Citation

If you use Mythos2.0-2B in your research or applications, please cite:

@misc{mythos2026multilingual,
  author       = {Adithyan AI and Community Contributors},
  title        = {Mythos2.0-2B: A Free and Open-Source Sparse Mixture-of-Experts Translation Foundation Model for 500+ Languages},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/AdithyanAI/Mythos2.0-2B}}
}