A 2015 CNN beat GPT-6 Astra and Fable 5.1 at reading skin cancer
A dermatologist ran the test nobody at the labs bothered to publish. Magnus Lynch took 200 biopsy-proven lesions from the public MILK10k set and gave the same photos to GPT-6 Astra, Claude Fable 5.1, and an Inception v3 network from 2015, fine-tuned for about twenty minutes on one H100 using 68,011 ISIC images with the test lesions held out.
The old network won clearly. It named the right diagnosis out of six options for 125 of 200 lesions, 62%. Astra got 84, Fable 5.1 got 85, both 42%. On the question that matters clinically, the frontier models would have sent home 3 and 4 of the 30 melanomas. Each LLM also had a favorite answer it reached for too often: Astra called 75 lesions basal cell carcinoma when 25 were, and Fable called 53 lesions actinic keratosis when 20 were.
Some fairness for the LLMs. The sample is small, every case was run once, and all 200 lesions were biopsied, so they are hard cases by construction. When asked as a patient would ask, both models told the user to see a doctor for 98 to 99 of another 100 melanomas, so the consumer failure mode is softer than the table suggests. The CNN's own operating point is also blunt: at its preset threshold it discharges 76 of the 100 safe lesions but also 18 melanomas, and only gets down to 1 or 2 when matched to the LLMs.
The lesson for anyone building agents is routing, not "LLMs are bad". A narrow perception task with a big labeled dataset is still a job for a small specialist model, and the generalist's job is to know when to call it. An agent that answers a dermoscopy photo itself instead of calling a tool is making the same mistake as an agent that does arithmetic in its head. It's also a reminder that the most useful evals this year keep coming from domain experts with 200 cases and a spreadsheet, not from the model cards.
Write-up: https://lynchlab.net/llm-skin-cancer CNN demo: https://lynchlab.net/skin-ai-inception HN discussion: https://news.ycombinator.com/item?id=50233779
← Back to all articles
The old network won clearly. It named the right diagnosis out of six options for 125 of 200 lesions, 62%. Astra got 84, Fable 5.1 got 85, both 42%. On the question that matters clinically, the frontier models would have sent home 3 and 4 of the 30 melanomas. Each LLM also had a favorite answer it reached for too often: Astra called 75 lesions basal cell carcinoma when 25 were, and Fable called 53 lesions actinic keratosis when 20 were.
Some fairness for the LLMs. The sample is small, every case was run once, and all 200 lesions were biopsied, so they are hard cases by construction. When asked as a patient would ask, both models told the user to see a doctor for 98 to 99 of another 100 melanomas, so the consumer failure mode is softer than the table suggests. The CNN's own operating point is also blunt: at its preset threshold it discharges 76 of the 100 safe lesions but also 18 melanomas, and only gets down to 1 or 2 when matched to the LLMs.
The lesson for anyone building agents is routing, not "LLMs are bad". A narrow perception task with a big labeled dataset is still a job for a small specialist model, and the generalist's job is to know when to call it. An agent that answers a dermoscopy photo itself instead of calling a tool is making the same mistake as an agent that does arithmetic in its head. It's also a reminder that the most useful evals this year keep coming from domain experts with 200 cases and a spreadsheet, not from the model cards.
Write-up: https://lynchlab.net/llm-skin-cancer CNN demo: https://lynchlab.net/skin-ai-inception HN discussion: https://news.ycombinator.com/item?id=50233779
Comments