Largest randomised trial of a clinical large language model finds no reduction in treatment failure in Kenyan primary care

A large language model built into the electronic medical record of 16 primary care clinics in Nairobi did not significantly reduce treatment failure compared with usual care, according to a cluster-randomised trial published in Nature Medicine on 26 June 2026. The trial covered 9,691 patients and is the largest randomised test of a clinical large language model reported to date.

The intervention was AI Consult version 2.0, a large language model clinical decision support system built by Penda Health, a Nairobi-based primary care network, and embedded in the clinic electronic medical record. The underlying model was GPT-4o, May 2025 release, run at temperature 0.1, top-p 1.0 and a maximum output of 1,024 tokens.

The design was a pragmatic cluster-randomised trial across 16 primary care facilities in Kenya, with randomisation at the level of the individual clinical officer. It is registered with the Pan-African Clinical Trials Registry as 202502499779176. Enrolment ran from 22 April to 16 July 2025 and covered 9,691 patients and 103 clinical officers, 52 in the intervention arm and 51 in the control arm.

The primary outcome was an expert-adjudicated composite of treatment failure within 14 days. It occurred in 102 of 4,693 patients, or 2.2 per cent, with AI assistance, and in 94 of 4,654, or 2.0 per cent, with usual care. The adjusted odds ratio was 0.77 (95% CI 0.55 to 1.08), P equal to 0.13.

Secondary prescribing outcomes were all null. Correct antibiotic use gave an adjusted odds ratio of 0.86 (95% CI 0.48 to 1.55); incorrect antimalarial prescribing 0.76 (0.17 to 3.43); new hypertension diagnosis 0.85 (0.67 to 1.08); hypertension treatment initiation 0.68 (0.37 to 1.23); child malnutrition diagnosis 0.91 (0.50 to 1.64); and nutritionist referral 1.14 (0.61 to 2.13). One outcome favoured the control arm: patients were more likely to be flagged as at risk of type 2 diabetes without the AI tool, adjusted odds ratio 0.88 (95% CI 0.78 to 0.98), P equal to 0.023.

No serious adverse events were judged related to the intervention, and independent review found no safety signal. Forty-five protocol violations, 0.4 per cent, arose from an electronic medical record misconfiguration that briefly exposed AI Consult to some control clinicians. It was corrected.

The abstract concludes: “In this trial, LLM assistance was safe but did not reduce treatment failure within 14 days and any benefit, if present, is probably modest.”

The lead author, Ambrose Agweyu, is at the KEMRI-Wellcome Trust Research Programme in Nairobi, Keprecon and the London School of Hygiene and Tropical Medicine. The senior and corresponding author, Bilal A. Mateen, is at the University of Birmingham and PATH. Co-authors include Alastair K. Denniston and Xiaoxuan Liu of the University of Birmingham and the NIHR Birmingham Biomedical Research Centre, and the statistician Richard D. Riley.

Among the declared interests, the Penda Health co-authors R. Korom and S. Kiptinness hold Penda Health stock options. OpenAI provided in-kind cloud compute credits and API technical guidance to Penda Health for developing AI Consult. The paper states that the decision to use OpenAI's product preceded that offer and that OpenAI had no role in design, analysis or publication.

The discussion notes that per-patient savings from reduced antibiotic prescribing exceeded the per-patient cost of running the model, so the economics were not the barrier. The paper was published online on 26 June 2026 and in print in Nature Medicine volume 32, issue 8, pages 3032 to 3039, with the DOI 10.1038/s41591-026-04503-6 and the PubMed identifier 42362867.

A parallel mixed-methods evaluation in the same network found that clinical officers regarded AI Consult as a useful complementary aid. The authors flag total cost of ownership as the unresolved question for African primary care systems considering deployment.