Skip to content

Conference abstract · AHS 2026 · Poster CP 17

Comparative evaluation of domain-specific and general-purpose large language models for ICHD-3 headache diagnosis: A controlled simulation study

Samneet Singh, BS; Tarvinder Singh, MS, MD

FIRY AI, USA

Headache. 2026;66(Suppl 1). Abstracts of the American Headache Society 68th Annual Scientific Meeting, Orlando, June 4 to 7, 2026.

DOI(opens in a new tab)PubMed(opens in a new tab)

Study summary

Objective

To compare the FIRY AI engine, a domain-specific system, with general-purpose foundation models on ICHD-3 headache diagnosis.

Methods

Controlled simulation using synthetic patient cases with pre-assigned ICHD-3 ground truth, compared with general-purpose foundation models.

Results

  • 91.7%

    top-1 ICHD-3 diagnostic accuracy, 95% CI 78.8 to 94.0

  • 29 to 62

    percentage points above general-purpose models, odds ratios 5.2 to 21.6

Results for the FIRY AI engine in a controlled simulation: 70 synthetic patient cases with pre-assigned ICHD-3 ground truth, compared with general-purpose foundation models. Real-world clinical validation is the stated next step.

Conclusion

In this controlled simulation, the FIRY AI engine was more accurate at ICHD-3 headache diagnosis than general-purpose foundation models. Real-world clinical validation is the next step.