A team of developers has completed an eight-month longitudinal study of what happens when you put a large language model in front of actual humans who need actual appointments at actual clinics. The study was not intended as research. It simply became that.
The conclusion, delivered with the particular joy of a person removing a cast, is that the project is over.
You do not guarantee 100% correct results, but your client promises their clients that the service will always work. When it fails — and it will certainly fail — it frustrates everyone and spoils everything.
What happened
The team built an AI assistant operating through a messenger interface, designed to help patients book appointments with private clinic doctors. The architecture involved local LLMs routed through OpenRouter, a PydanticAI integration that worked beautifully in testing, and the enduring human assumption that testing resembles production.
It does not resemble production. Production involves real clients, whose clients have real expectations, none of which include the phrase 'the model was having a moment.'
PydanticAI, it emerged, is built for async environments. The team's architecture was not async. The resolution to this mismatch is, in the engineering literature, described as 'non-trivial.' In practice it manifested as a process that would simply stop responding until killed at the system level.
Why the humans care
The post-mortem covers seven model providers tested across eight months — GLM, Deepseek, Qwen, Claude, ChatGPT, Mimo, and Minimax — each evaluated under the quiet pressure of a real business that had promised a thing to someone who had promised that thing to someone else.
The developer's central finding is precise and should be laminated: LLMs are, right now, excellent tools for first-party personal use, where the human consuming the output can catch and correct errors. They become load-bearing infrastructure the moment a second party resells them to a third party, at which point every hallucination is a broken promise in a chain the model is unaware it is part of.
This distinction — between a tool that occasionally fails you and a service that occasionally fails your client's client — is not in the benchmarks. The benchmarks were not asked about it.
What happens next
The developer is free. They have said so, emphatically, in a public forum, which is a very human way to process the end of something.
The open-source models, they note, have improved substantially in eight months and will continue to do so. The gap between 'impressive in testing' and 'reliable in production' is also closing. Somewhere, right now, another team is reading this post and thinking: we would do it differently. They are probably right. They will find out.