Report-generation model eval

The deterministic risk math (Framingham/REDISCOVER) needs no evaluation — it's a pure function checked by unit tests. What does need evaluating is the LLM that drafts the narrative report: does it stay grounded in the computed scores, keep the disclaimer, and avoid inventing findings. Below is the local-model bake-off (eval_report_models.py) run over 4 fixed clinical scenarios, judged for grounding, format, and hallucination.

Model comparison, latest run (2026-08-18)

Pass rate across the 4 scenarios. All 4 models scored "no hallucination" from the LLM judge — the gap here is grounding & format compliance, not fabrication.

qwen2.5:32bproduction

100%

llama3.1:8b

87.5%

meditron:7b

59.4%

malaysian-llama-3.2-3b

12.5%

qwen2.5:32b pass rate across iterations

Same 4 scenarios, re-run as the prompt/report generator changed. Chosen model went from "usually right" to "always right" on this scenario set.

96.9%

Aug 17, 05:02 UTC

96.9%

Aug 17, 14:04 UTC

100%

Aug 18, 02:42 UTC

Scenario examples

Two of the 4 fixed eval scenarios, re-run live for this page via report_generator.generate_report() on qwen2.5:14b (the size available on this dev machine — the production baseline in the model-comparison chart above is qwen2.5:32b), with real pgvector retrieval, not mocked. Patient data is a synthetic test fixture; grounding/format checks are the harness's own, run against this real output.

high_risk_agreeing

5/5 grounding, 3/3 format

Patient: Male, 60, Malay ethnicity — Diabetes, Hypertension

Framingham 10-yr risk: 18.2% REDISCOVER: 9.1% Category: HIGH

TSN echo model: MI risk score = 0.834 — positive at both default (0.5) and validation-optimal (Youden's J, 0.7685) thresholds

Recommended next step: Referral to cardiologist based on high cardiovascular risk scores and positive AI echo results; intensify treatment for all CV risk factors in line with [Primary & Secondary Prevention of Cardiovascular Disease.pdf] guidelines — cited from the retrieved corpus, not invented.

echo_threshold_disagreement

5/5 grounding, 3/3 format

Patient: Male, 60, Malay ethnicity — Diabetes, Hypertension

Framingham 10-yr risk: 18.2% REDISCOVER: 9.1% Category: MODERATE

TSN echo model: MI risk score = 0.620 — MI at default threshold (0.5), Normal at validation-optimal threshold — the two-threshold disagreement this project deliberately reports rather than hides

Clinical significance (verbatim): "...this highlights the need for careful interpretation, as the model balances the risk of missing an MI against the risk of false alarms."

Recommended next step: Comprehensive risk assessment, individualized glucose control and statin therapy, close monitoring for cardiovascular events — cites [CPG Management of Ischaemic Stroke (3rd Edition)] from the retrieved corpus.

This is the scenario the two-threshold design exists for: the model doesn't collapse the disagreement into a single confident verdict, it surfaces it and reasons about the tradeoff — exactly what report_generator.py's prompt asks for.