Self-LLM Eval Report

Real output of the field-level scoring harness (scripts/run_eval.py) against 41 golden clinical notes with hand-labeled expected ADIME fields, run on qwen2.5:14b. Nothing here is hand-picked — the full 41-case run is charted below, warts included.

80.1%

core field accuracy

4.1%

hallucination rate (9/221)

0.0%

NCPT code fill rate

14.4s

avg. generation / note

Hallucination target is <2% per the project plan — 4.1% (plus 10 fabricated list items not captured in that rate) means this isn't gated for production yet. NCPT fill rate is expected to be 0%: the model isn't given the NCPT vocabulary in-context in this phase, so it's scored separately rather than folded into "accuracy" and hidden.

Per-case core field accuracy

All 41 golden cases, hover a bar for the exact match count. Dashed lines mark the 85% / 70% bands used above.

85%

70%

≥85% 70–84% <70%

Golden case examples

Two of the 41 golden cases, re-run live against qwen2.5:14b for this page (not pulled from the saved report) — a strong case and the weakest one, showing what "80.1% core accuracy" actually looks like on real input.

case_001 — post-op reduced intake 90% (18/20 fields)

Input note (excerpt)

Ht 165cm, Wt 58kg (usual wt 63kg, ~8% loss over 1 month). BMI 21.3. Albumin 3.2 g/dL (L). Reduced oral intake x5 days post-op, tolerating <50% of meals offered.

Expected

Diagnosis: Inadequate oral intake — problem_ncpt_code: "NI-2.1"
Intervention: High-protein ONS TDS, 1800 kcal/day

Actual (live re-run)

Diagnosis: Inadequate oral intake — problem_ncpt_code: null
Intervention: High-protein ONS TDS, 1800 kcal/day ✓

Every clinical field matches. The only miss is the NCPT code — expected, since the model isn't given the NCPT vocabulary in-context yet (the reported 0% fill rate isn't a fluke, it shows up on the very first case).

case_039 — diabetic foot ulcer (Bahasa Malaysia note) 48% (19/40 fields) — weakest case

Input note (excerpt, original language)

"Diabetes Mellitus Jenis 2 (DMT2) x 12 tahun, ulser kaki diabetik... DIAGNOSIS PEMAKANAN: Ambilan protein tidak mencukupi RT selera makan berkurangan... INTERVENSI PEMAKANAN: Diet tinggi protein (1.25-1.5g/kg/hari)..."

Expected

Diagnosis: Inadequate protein intake (1 entry)
Intervention: 3 entries (diet, ONS, education)
M&E: 3 entries (HbA1c, glucose, ulcer status)

Actual (live re-run)

Assessment: correctly extracted & translated (anthropometrics, labs, clinical findings all match)
Diagnosis: [ ] — empty
Intervention: [ ] — empty
M&E: [ ] — empty

The concrete failure mode behind the 48% score: on this Bahasa Malaysia note, the model translates and extracts the assessment section correctly, then returns empty arrays for diagnosis, intervention, and monitoring & evaluation entirely — it doesn't hallucinate or mistranslate them, it just drops them. This is a real gap the separate translate-then-extract path (see extract.py's non-English handling) hasn't fully closed yet.

Roadmap