RAG Experiment Playground
The regression harnesses that gate changes to this sandbox — small golden sets run before/after any retrieval, chunking, or backend swap, not large-scale benchmarks. Numbers below are from live runs, not hand-picked or simulated.
Retrieval quality
eval_retrieval.py — real BAAI/bge-m3
embeddings + TopicBoostedRetriever against a 7-query golden corpus tagged
with the production topic taxonomy. Gate: fail CI below 70% overall hit rate.
Golden query examples
Two of the 7 golden queries, from the same live run above — the actual top-3 retrieved passage IDs, not simulated. One is a Malay-language query, exercising the multilingual embedding path the ADIME nutrition persona relies on.
"Saya ada masalah darah tinggi, makanan apa yang sesuai?"
expected: blood pressure, hypertension → htn-1
retrieved: htn-1, cholesterol-1, general-1
"I have kidney disease — should I avoid bananas and nuts?"
expected: CKD, potassium restriction, renal nutrition → ckd-1
retrieved: ckd-1, htn-1, cholesterol-1
Structured-lookup backend comparison
market_llm_results.json — the same 5 golden structured queries (spreadsheet
lookups: nutrient values, reference ranges) run against two swappable LLM backends. Both are kept swappable
for cost/latency, not accuracy — this golden set shows no meaningful gap between them.
| Query | Claude | Gemini |
|---|---|---|
| Protein content of Glucerna TC | ✓ | ✓ |
| Sodium reference range for adults | ✓ | ✓ |
| Energy kcal/scoop for Ensure RTD | ✓ | ✓ |
| Diabetasol sachet nutrition info | ✓ | ✓ |
| Haemoglobin reference range | ✓ | ✓ |
Vision experiment: I-JEPA food scanning
JEPA/ is a separate playground inside this repo: Meta's
facebook/ijepa_vith14_1k self-supervised vision encoder, used for few-shot
nearest-neighbor food-photo ID instead of training anything from scratch. I-JEPA has no text tower, so unlike
CLIP there's no zero-shot "match this image against text labels" path — the appeal was a bounded output space
(always one of N known labels, no free-text hallucination to parse) at a lighter 2.4GB footprint than a
vision-LLM. A blind 24-image bake-off against a cloud LLM and a local vision-LLM checked whether that
trade-off was actually paying for itself.
| Approach | Accuracy | Model size | Latency | Offline |
|---|---|---|---|---|
| Claude (cloud, via subagent) | 96% (23/24) | N/A (API) | ~seconds | No |
| llava:7b (local, Ollama) | 92% (22/24) | 4.7 GB | 0.33s/img | Yes |
| I-JEPA + nearest-neighbor | 87% (leave-one-out*) | 2.4 GB | 0.45s/img | Yes |
* Not apples-to-apples — see hover tooltip. Kept in the table anyway rather than dropped, since it's the only number nearest-neighbor classification can produce.
Finding: this isn't currently JEPA's selling point.
A generic, off-the-shelf local vision-LLM beat the custom JEPA pipeline on accuracy while running fully offline at comparable latency, with none of the scrape/curate/rebuild cycle the JEPA reference set required. The experiment is paused, not shipped into /scan — it stays parked for the day one of these becomes a real constraint:
- A resource envelope where even a 4.7GB vision-LLM is too heavy (e.g. an actual phone, not a dev machine)
- A hard requirement for a strictly bounded output space — always one of N known labels, zero hallucination risk — rather than parsing an LLM's prose
- Reusing the same embeddings for something beyond classification (similarity search, clustering, anomaly detection) where an LLM call is the wrong tool regardless of accuracy