Embedding Finetune RAG
Fine-tune the retriever so RAG actually finds the right law.
The problem
A RAG pipeline is only as good as its retriever. A generic multilingual embedding model doesn't know that, in Swiss tenancy law, 'Kaution' and 'Sicherheitsleistung' mean the same thing under the same article — so the right passage lands far down the ranking and the LLM never sees it.
The solution
A few minutes of contrastive fine-tuning on domain query→article pairs, measured honestly on real, verbatim statute text (Swiss Code of Obligations, Art. 253-274g). recall@1 jumped from 37.5% to 89.6% and MRR from 0.55 to 0.95 — all really measured on an Apple Silicon Mac.
What it does
- ▹Contrastive fine-tuning (MultipleNegativesRankingLoss) on a small multilingual model
- ▹Real, verbatim corpus — 79 Swiss tenancy-law articles (OR, SR 220)
- ▹Measured before/after: recall@1 0.375 → 0.896, MRR 0.55 → 0.95
- ▹Group-level split avoids paraphrase leakage — gains are real generalisation
- ▹Injected encoder interface: 33 tests, 97% core coverage, no model download
How it works
- 1
Build pairs
The real statute corpus (Art. 253-274g) is turned into 260 train / 48 test query→article pairs, split at the article level.
- 2
Fine-tune on MPS
A ~120 MB multilingual base model is fine-tuned with in-batch negatives (MNRL) in ~28 s on Apple Silicon.
- 3
Measure before/after
Both models encode the corpus; cosine top-k search is scored with recall@k and MRR, then a chart reports the gain.
Architecture
Click a node to see what it does.
How it looks
Tech stack
- Python 3.11
- sentence-transformers
- PyTorch
- Apple MPS
- pytest