skilld
All skills
huggingface avatar

/train-sentence-transformers

@0201949 official
by Hugging Facehuggingface/skills11k stars
753

Train or fine-tune sentence-transformers models across `SentenceTransformer` (bi-encoder, dense or static embedding model for retrieval, similarity, clustering, classification, paraphrase mining, dedup, multimodal), `CrossEncoder` (reranker, pair scoring for two-stage retrieval / pair classification), `SparseEncoder` (SPLADE, sparse embedding model for learned-sparse retrieval), and `MultiVectorEncoder` (ColBERT / late-interaction, per-token embeddings scored with MaxSim). Covers loss selection, hard-negative mining, evaluators, distillation, LoRA, Matryoshka, and Hugging Face Hub publishing. Use for any sentence-transformers training task.

Ask your Agent

Use this Skill: https://skilld.dev/gh/huggingface/skills/train-sentence-transformers

This session only. Nothing lands on disk.

referencesbase_model_selection.md

≈1.9k tokens on demand. Your agent reads this file only when SKILL.md points to it.

Base Model Selection

Leaderboards rotate every few months. Don't trust any hardcoded "best" pick. Discover current options live. Run both sort orders since most-downloaded surfaces proven options and trending surfaces recent SOTA that may not have download volume yet.

Discovery commands

[BI]:

hf models list --filter sentence-transformers --sort downloads --limit 20
hf models list --filter sentence-transformers --sort trending  --limit 20

[CE]:

hf models list --filter sentence-transformers --filter text-ranking --sort downloads --limit 20
hf models list --filter sentence-transformers --filter text-ranking --sort trending  --limit 20

[SPARSE]:

hf models list --filter sentence-transformers --filter sparse-encoder --sort downloads --limit 20
hf models list --filter sentence-transformers --filter sparse-encoder --sort trending  --limit 20

Optional language narrowing (any type): add --filter <language-code>. Not all multilingual models tag each language, so missing matches doesn't mean the model can't handle that language. Re-run without the filter to compare.

hf models card <id> --text                        # confirm dimensions, max_seq_length, license, languages

Cross-check the MTEB leaderboard (pick the relevant tab) before committing to a multi-hour run.

[BI] Bi-Encoder

Continue from an existing retriever beats fresh-start + 100k-500k pairs. Common namespaces as of 2026-Q2 (verify against discovery commands, since the field rotates):

  • English encoder retrievers: sentence-transformers/all-* (MiniLM-L6-v2, mpnet-base-v2 still the most-downloaded models on the Hub), BAAI/bge-*-en-v1.5, nomic-ai/nomic-embed-text-v1.5, mixedbread-ai/mxbai-embed-large-v1, Alibaba-NLP/gte-*, Snowflake/snowflake-arctic-embed-*, jinaai/jina-embeddings-v5-text-small / -nano, microsoft/harrier-oss-v1-270m / -0.6b.
  • Multilingual encoder retrievers: sentence-transformers/paraphrase-multilingual-*, intfloat/multilingual-e5-*, ibm-granite/granite-embedding-*-multilingual-r2, google/embeddinggemma-300m, voyageai/voyage-4-nano.
  • Long documents (8k+): nomic-ai/modernbert-embed-*, answerdotai/ModernBERT-large.
  • Decoder LLM retrievers (multilingual, need last-token pooling): Qwen/Qwen3-Embedding-* (0.6B / 4B / 8B), Qwen/Qwen3-VL-Embedding-* (multimodal), codefuse-ai/F2LLM-v2-*.
  • Fresh-start English (≥500k pairs + domain-fit reason): microsoft/mpnet-base, answerdotai/ModernBERT-base, google-bert/bert-base-uncased, jhu-clsp/ettin-encoder-* (17m / 32m / 68m / 150m / 400m / 1b, paired ModernBERT encoder family).
  • Fresh-start multilingual: FacebookAI/xlm-roberta-base (MLM-only, needs contrastive training), microsoft/mdeberta-v3-base, jhu-clsp/mmBERT-base / -small.
  • CPU / small footprint (StaticEmbedding): StaticEmbedding(tokenizer, embedding_dim=...). Model size = vocab_size × dim × 4 bytes. Pick a small-vocab tokenizer or you get a giant model: 30k-vocab bert-base-uncased × 128 dim ≈ 15 MB, 250k-vocab paraphrase-multilingual-MiniLM-L12-v2 × 256 dim ≈ 256 MB. Random init needs 1M+ pairs. Warm-start (StaticEmbedding.from_distillation(...)) helps under ~100k pairs.

Architecture variants (encoder / decoder / static / Router), pooling rules, and decoder-vs-encoder setup paths: model_architectures.md.

ModernBERT-family bases default to max_seq_length=8192. That allocates activation memory for 8192-token sequences regardless of your data length and silently drives Windows VRAM into "shared memory" spillover. After loading any ModernBERT / mmBERT / Ettin / gte-modernbert / nomic-modernbert base, explicitly set model.max_seq_length = 256 (or 512 for documents) unless you actually need long context.

[CE] Cross-Encoder

Continue from an existing reranker beats fresh-start + 100k-500k pairs in most domains. Default to this unless you have a strong reason otherwise. Common namespaces as of 2026-Q2:

  • English encoder rerankers: cross-encoder/ms-marco-*, BAAI/bge-reranker-*, mixedbread-ai/mxbai-rerank-*-v1 / -v2, Alibaba-NLP/gte-reranker-modernbert-*, ibm-granite/granite-embedding-reranker-english-*.
  • Multilingual encoder rerankers: cross-encoder/mmarco-*, BAAI/bge-reranker-v2-m3, Alibaba-NLP/gte-multilingual-reranker-*, ibm-granite/granite-embedding-reranker-multilingual-*.
  • Decoder LLM rerankers (multilingual, num_labels=1 last-token-style scoring): Qwen/Qwen3-Reranker-* (0.6B / 4B / 8B), Qwen/Qwen3-VL-Reranker-* (multimodal).
  • Fresh-start: microsoft/MiniLM-L12-H384-uncased, answerdotai/ModernBERT-base / -large, jhu-clsp/ettin-encoder-*, FacebookAI/xlm-roberta-base (multilingual), microsoft/mdeberta-v3-base (multilingual), jhu-clsp/mmBERT-base / -small (multilingual). Pass num_labels >= 2 for classification cross-encoders.

Encoder-only bases are still the latency-efficient default (bidirectional attention is well-suited to the reranking use case at small parameter counts), but decoder LLM rerankers are now competitive at the top of MTEB Reranking when latency / memory budget allows.

Minimum dataset: 500k+ labeled (query, passage, label) tuples for production. 10k-100k labeled pairs for continue-training on domain data. Low-resource languages may have less than 10k labeled pairs. In that case, lean on a multilingual base's pretraining and accept a noisier signal.

"Small" multilingual is ~100M+ params, not 17M-50M like the English small models. mMiniLMv2-L12-H384 (~117M) is roughly the small-end for usable multilingual rerankers.

[SPARSE] Sparse Encoder (SPLADE)

SPLADE requires a fill-mask / AutoModelForMaskedLM-compatible checkpoint. Encoder-only MLM models work out of the box. Decoder LLMs do not.

  • Continue from existing SPLADE (English): naver/splade-* (the canonical family), opensearch-project/opensearch-neural-sparse-encoding-* (incl. -doc-v2-distill, -doc-v3-distill / -doc-v3-gte), prithivida/Splade_PP_en_v*, ibm-granite/granite-embedding-30m-sparse.
  • Continue from existing SPLADE (multilingual): opensearch-project/opensearch-neural-sparse-encoding-multilingual-v1.
  • Fresh-start English (≥500k pairs): any encoder with an MLM head (distilbert/distilbert-base-uncased, google-bert/bert-base-uncased). Pure AutoModel checkpoints without MLM won't work. Discover MLM bases: hf models list --filter fill-mask --sort downloads --limit 20.
  • Fresh-start multilingual: FacebookAI/xlm-roberta-base (has MLM head). For other multilingual MLM bases: add --filter <language-code>.

Minimum dataset: 500k+ triplets (with mined hard negatives) for a competitive SPLADE. 50k+ triplets for domain adaptation on existing SPLADE.

Cross-cutting tips

  • Non-English retrieval starting points (when language tag returns 0 results): check intfloat/multilingual_e5_train_data for parallel pair data, MIRACL via the sentence-transformers/miracl mirror for multilingual retrieval, mMARCO via unicamp-dl/mmarco (14 languages, parquet-backed).
  • Avoid script-based dataset loaders. datasets >= 4 rejects them with RuntimeError: Dataset scripts are no longer supported. Look for parquet-backed mirrors (e.g. sentence-transformers/miracl instead of miracl/miracl).
  • hf datasets sql requires DuckDB (pip install duckdb). Without it, fall back to python -c "from datasets import load_dataset; ds = load_dataset('<id>', ...); print(ds.column_names, ds[0])".

Source: SKILL.md on GitHub

Third-party checks

1 warning16d3 checks · Risk SAFE
  • Gen Agent Trust Hub16d

    This skill provides a comprehensive environment for training sentence-transformers models, including production-ready scripts and detailed documentation. It involves standard practices such as downloading packages from official registries and processing datasets from remote sources.

  • Socket16d

    No alerts

  • Snyk16d

    Risk: MEDIUM · 1 issue

Provenance

Signed by skilld at 0201949. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub last week.

Activeupdated last month

Topics

  • sentence-transformers
  • training
  • fine-tuning
  • embeddings
  • retrieval
  • cross-encoder
  • sparse-encoder
  • ner
  • classification

README badge

README badge for huggingface/skills/train-sentence-transformers

What it does

Trains or fine-tunes sentence-transformers models across SentenceTransformer (bi-encoder for dense embeddings), CrossEncoder (reranker for pair scoring), and SparseEncoder (SPLADE for sparse vectors). Covers loss selection, hard-negative mining, evaluators, distillation, LoRA, and Hub publishing. Use this skill for any sentence-transformers training task.

Generated from the current SKILL.md.

Frequently asked

Does this skill cover all three model types (SentenceTransformer, CrossEncoder, SparseEncoder)?
Yes. The skill routes you to type-specific references and production templates. Use section 1 to identify which model type matches your task, then load the corresponding references and example script.
Can I use this skill to fine-tune models with LoRA, distillation, or Matryoshka?
Yes. The skill includes variant scripts for `train_sentence_transformer_with_lora_example.py`, `train_sentence_transformer_distillation_example.py`, and `train_sentence_transformer_matryoshka_example.py`, plus distillation variants for CrossEncoder and SparseEncoder.
Do I need to write my own training script or can I copy from the templates?
Copy from the production templates (`scripts/train_<type>_example.py`). The skill explicitly states not to synthesize from the routing file alone; templates contain load-bearing scaffolding (autocast helpers, seed handling, version-compatible imports, required callbacks) that prior runs have missed when rolling their own.
What if my task involves hard-negative mining or training on multiple datasets?
The skill includes `scripts/mine_hard_negatives.py` for hard-negative mining and a `train_sentence_transformer_multi_dataset_example.py` variant. Check section 2 (Variant scripts) and `references/dataset_formats.md` for reshaping recipes.
Does this work with multimodal models or non-English languages?
For multimodal: install `sentence-transformers[train,image]` or add audio/video extras. For non-English: the skill references `references/base_model_selection.md` (non-English shortcuts) and `references/prompts_and_instructions.md` for prompt-tuned bases (E5, BGE, Qwen3-Embedding, etc.).

Generated from the current SKILL.md. These answers refresh after source changes.