Report-Generation Model Eval

The deterministic risk math (Framingham/REDISCOVER) needs no evaluation — it's a pure function checked by unit tests. What does need evaluating is the LLM that drafts the narrative report: does it stay grounded in the computed scores, keep the disclaimer, and avoid inventing findings. Below is the local-model bake-off (eval_report_models.py) run over 4 fixed clinical scenarios, judged for grounding, format, and hallucination.

Model comparison — latest run (2026-08-18)

Pass rate across the 4 scenarios. All 4 models scored "no hallucination" from the LLM judge — the gap here is grounding & format compliance, not fabrication.

qwen2.5:32b production
100%
llama3.1:8b
87.5%
meditron:7b
59.4%
malaysian-llama-3.2-3b
12.5%

qwen2.5:32b — pass rate across iterations

Same 4 scenarios, re-run as the prompt/report generator changed. Chosen model went from "usually right" to "always right" on this scenario set.

96.9%
Aug 17, 05:02 UTC
96.9%
Aug 17, 14:04 UTC
100%
Aug 18, 02:42 UTC

Scenario examples

Two of the 4 fixed eval scenarios, re-run live for this page via report_generator.generate_report() on qwen2.5:14b (the size available on this dev machine — the production baseline in the model-comparison chart above is qwen2.5:32b), with real pgvector retrieval, not mocked. Patient data is a synthetic test fixture; grounding/format checks are the harness's own, run against this real output.

high_risk_agreeing 5/5 grounding · 3/3 format ✓
Patient: Male, 60, Malay ethnicity — Diabetes, Hypertension
Framingham 10-yr risk: 18.2%   REDISCOVER: 9.1%   Category: HIGH
TSN echo model: MI risk score = 0.834 — positive at both default (0.5) and validation-optimal (Youden's J, 0.7685) thresholds
Recommended Next Step: Referral to cardiologist based on high cardiovascular risk scores and positive AI echo results; intensify treatment for all CV risk factors in line with [Primary & Secondary Prevention of Cardiovascular Disease.pdf] guidelines — cited from the retrieved corpus, not invented.
echo_threshold_disagreement 5/5 grounding · 3/3 format ✓
Patient: Male, 60, Malay ethnicity — Diabetes, Hypertension
Framingham 10-yr risk: 18.2%   REDISCOVER: 9.1%   Category: MODERATE
TSN echo model: MI risk score = 0.620 — MI at default threshold (0.5), Normal at validation-optimal threshold — the two-threshold disagreement this project deliberately reports rather than hides
Clinical Significance (verbatim): "...this highlights the need for careful interpretation, as the model balances the risk of missing an MI against the risk of false alarms."
Recommended Next Step: Comprehensive risk assessment, individualized glucose control and statin therapy, close monitoring for cardiovascular events — cites [CPG Management of Ischaemic Stroke (3rd Edition)] from the retrieved corpus.

This is the scenario the two-threshold design exists for: the model doesn't collapse the disagreement into a single confident verdict, it surfaces it and reasons about the tradeoff — exactly what report_generator.py's prompt asks for.

Repository: private