Why the skeptic’s instinct was right — and why it no longer means what it used to
The Best AI Lies Confidently 1/6th of the Time — Image by Gemini
Most people still picture artificial intelligence as a confident liar. A system that invents legal cases, makes up citations for research papers, and says all of it in the same smooth, certain voice — whether it’s right or completely wrong.
That picture isn’t unfair. It was earned.
In the spring of 2023, a New York lawyer named Steven Schwartz filed a court brief that cited six federal court decisions. There was just one problem: none of them existed. ChatGPT had invented the case names, the citations, even quotes attributed to judges who never wrote them. When Schwartz asked the tool to double-check itself, it cheerfully assured him the cases were real.
They weren’t. The judge, P. Kevin Castel, fined the attorneys and their firm. The episode became the textbook warning about letting a machine near professional work, and the wariness that followed made complete sense given what everyone had just watched happen.
Here’s the more useful way to think about what’s changed since then. AI did not become trustworthy by learning to stop making things up. It still makes things up. What changed is that the fields with the most to lose — law, medicine, finance, software — stopped pretending the AI would behave itself, and built systems around it that catch the mistakes before anyone gets hurt. The model didn’t get more honest. The structure around the model got better.
That’s a less exciting story than “the hallucinations went away.” It’s also a sturdier reason for confidence, because it doesn’t depend on the AI being perfect. It depends on the system around the AI being sound — and a system can be checked, tested, and held accountable in a way a single black-box answer never can.
What follows is a tour of that system in four fields where getting it wrong has real consequences: software, law, finance, and medicine. The same pattern shows up in each one. The AI generates fast, and it generates fallibly. Something slower and more reliable checks its work. And a human, with a name and a job and a reputation, signs off on the result.
The number that keeps everyone honest
Before anything reassuring, here’s the number that should stay in view the whole time.
In 2024, researchers at Stanford’s RegLab — a group that tests how well legal technology actually performs — evaluated the AI research tools built specifically for lawyers. These weren’t free consumer chatbots. They were professional products, designed to ground their answers in real legal databases and check their own citations.
Even these tools got it wrong somewhere between seventeen and thirty-four percent of the time. At best, roughly one answer in six was incorrect.
That number isn’t meant to scare anyone off the technology. It’s meant to explain why everyone serious about using AI professionally now designs around it.
Here’s why the mistakes don’t simply disappear with a better model. A language model doesn’t actually know whether a court case exists. It knows what a citation looks like — the shape of it, the rhythm of it — and it’s very good at producing something with that shape. Sometimes the shape is filled with something real. Sometimes it isn’t, and the model has no way to tell the difference from the inside.
Once that’s understood as a permanent feature of how these systems work, rather than a bug waiting to be patched, the engineering question changes. Nobody serious is trying to build a model that never makes mistakes anymore. They’re trying to build a process where mistakes get caught — reliably, visibly, before they reach anyone who matters.
That shift, from chasing a flawless model to building a process that survives a flawed one, is really the whole story here. It’s the difference between the 2023 disaster and how the same kind of tool gets used today. The model can still fabricate in both cases. What’s different is what stands between the fabrication and the person relying on it.
Software: the most mature loop, because the work checks itself
Software engineering arrived at this answer first, and the reason why matters for everything that follows.
Software Engineer: A Day in the Life — Image by Gemini
Code can be tested by a machine, automatically, in seconds. A legal claim needs a human to go check a real source. A claim embedded in a line of code can simply be run. You don’t have to trust that the AI got it right. You can compile the code, run the test suite, run a linter — a tool that flags sloppy or risky patterns — and run a security scanner. None of these care who or what wrote the code. They just report back, flatly, whether it works.
That difference has reshaped how AI is used to write software. The industry has moved well past having AI finish a single line of code for you. Now an AI agent gets handed an entire project and a real problem to solve, and works through it in stages. It proposes a change. A testing system tries to break that change inside a disposable copy of the project. Whatever breaks — error messages, failed tests — gets fed straight back to the AI as the next thing to fix. The loop keeps running until the tests pass.
Progress here is measured by whether the code actually works, not by how convincing it sounds. That distinction is everything.
There’s a benchmark that tracks this, called SWE-bench Verified, and it’s worth knowing about because it’s so unforgiving. It’s built from five hundred real bugs and feature requests pulled from actual open-source software projects, reviewed by humans to confirm each one is solvable. An AI agent is handed the project and the problem, and has to find its way through the codebase, write a fix, and pass the project’s real test suite. There’s no partial credit for a fix that merely sounds plausible. Either the tests pass, or they don’t.
When this benchmark first appeared, solving even a small fraction of these problems was considered impressive. By 2026, the strongest AI systems solve roughly four out of five of them. That number is worth sitting with — not because it proves AI writes flawless code, but because the way it’s measured leaves no room to fake it. A plausible-sounding wrong answer fails exactly as visibly as an obviously wrong one.
It helps to see how far this is from where coding benchmarks started. Early tests like HumanEval simply asked a model to write one self-contained function from a short description, a task today’s strong models pass over ninety percent of the time. That’s a much narrower skill than real engineering, which means understanding a sprawling, messy codebase, reproducing a bug someone else reported, and making a change that doesn’t quietly break five other things. Newer benchmarks lean directly into that mess on purpose, and the AI agents being tested increasingly work the way a human engineer does — across many steps, over a long stretch of time, rather than producing one clever answer in a single shot.
None of that capability is what makes this safe to use, though. The testing is what makes it safe.
Here’s the part worth noticing. The trust in this picture doesn’t sit with a developer trusting the AI’s judgment. It sits with the test suite, which has no opinion about who wrote the code and simply refuses to lie. Broken code fails to run whether a senior engineer wrote it or an AI generated it in half a second. In that sense, AI slotted neatly into a discipline that had already spent decades building defenses against ordinary human mistakes: automated tests, build pipelines, staging environments, one engineer reviewing another’s work. None of that infrastructure had to be reinvented. It simply gained a new contributor — very fast, occasionally wrong, and required to pass through the same checks as everyone else.
And at the end of all those checks, a person is still standing there. A human engineer reviews the change and decides whether to ship it. If something breaks in production at three in the morning, nobody accepts “the AI wrote that part” as an explanation. The responsibility never moved.
What did move is how quickly a reviewable first draft shows up. The engineer’s job has shifted upward — less typing, more judging: is this the right approach, what will break under pressure, what shouldn’t exist at all. The AI closes the distance between an idea and a working draft. Deciding whether that draft is actually right, and living with the consequences if it isn’t, is still entirely a human job.
Law: adoption is near-total, and so is the supervision
If any profession seemed built to reject AI, it was law. Lawyers are trained to distrust anything they can’t verify, their mistakes are public and expensive, and the ChatGPT disaster hung over the whole field as a fresh memory.
Legal Researcher: A Day in the Life — Image by Gemini
And yet adoption happened anyway, fast. Bloomberg Law’s survey of the profession in mid-2026 found that roughly eighty-three percent of lawyers now use AI in their work, up from under a fifth just three years earlier. The same kind of tool that caused a national embarrassment in 2023 is, by 2026, ordinary equipment on a lawyer’s desk. A famously cautious profession adopting a famously error-prone technology only makes sense once you look at what now surrounds that technology.
The first thing surrounding it is a clear rule. In July 2024, the American Bar Association issued its first formal opinion on generative AI, spelling out something the existing rules already implied: a lawyer’s duty to be competent and diligent doesn’t disappear just because a machine did the drafting. A lawyer has to understand the tool’s limits well enough to use it responsibly, and has to check its work. That’s not a suggestion. It’s an ethics rule, and breaking it can end a career.
The second thing surrounding it is a judiciary that has run out of patience. Fake citations didn’t stop appearing after the 2023 case. Courts have kept catching them, including from large, well-regarded firms. What changed is how seriously courts respond. In one 2025 case in Alabama, a court found that fining attorneys wasn’t deterring the behavior, so it reached for something sharper: it disqualified the responsible attorneys from the case entirely, and made sure their state bar found out. Many federal courts now require lawyers to certify, in writing, that they’ve personally checked any AI-generated content in their filings. The message from judges has been consistent. Checking the AI’s work isn’t optional, and the lawyer is the one who has to do it.
The third thing surrounding it is a change in how the AI tools themselves are built. The 2023 disaster involved a general-purpose chatbot with no connection to any real legal database, asked to produce case law it could only invent. Serious legal AI tools don’t work that way anymore. A question triggers a search of a real, verified library of cases and statutes, and the AI is only allowed to build its answer from what that search actually finds, with a link back to the source for each claim. This doesn’t eliminate mistakes, as the Stanford numbers show. But it changes what kind of mistake is possible. Instead of inventing a case from nothing, the AI might misread a real one, and the real case sits one click away for a human to check.
There’s a second risk running alongside accuracy, and it’s about keeping secrets rather than getting facts right. A lawyer who pastes confidential client details into a free, public chatbot may be exposing that information to a company with nothing to do with the case, which can violate a lawyer’s basic duty of confidentiality. The cautionary tale here came from outside the legal world. In 2023, engineers at Samsung reportedly fed proprietary chip-design data into ChatGPT, and the company banned the tool internally soon after. Law firms learned the same lesson. Confidential client work now runs on private, enterprise versions of these tools that don’t retain data or train on it, and the free public versions are kept away from anything involving a real client.
All of this is now backed by law itself. A California statute that took effect on January 1, 2026, blocks a specific kind of excuse: a company can no longer argue, after an AI system causes harm, that the harm doesn’t count because the AI “acted on its own.” The escape hatch is closed. Whoever built, modified, or deployed the system is the one who answers for what it did.
What this adds up to, in practice, is unglamorous and it works. The AI does the heavy lifting — searching, summarizing, producing a first draft. A licensed attorney checks every citation against the real source, confirms the argument actually holds up, and signs the filing. That signature is where the responsibility lives, and a signature belongs to a person. The AI gets the draft done faster than any associate could. The lawyer still owns whatever gets filed.
There’s one more thing worth knowing about, even though nobody is using it yet. A 2025 research paper described a system called LegalSim, which used AI agents to simulate opposing sides in a lawsuit and let them compete against each other. Left to their own devices, the agents discovered “exploit chains” — sequences of moves, like deliberately expensive discovery requests and tactical use of deadlines, that are completely within the rules but exist purely to drain an opponent’s resources. It’s an unsettling demonstration of how an AI might probe the procedural machinery of a courtroom for weaknesses. Some people in the field imagine law firms eventually using a tool like this to test their own filings before they ever reach a real court, the way a security team tries to break into its own software before a hacker does. For now, that idea lives in a research paper, not a law office.
Finance: the model that trades is not the model that thinks
Finance had a head start on this entire problem, which is part of why its adoption of AI has felt comparatively calm. The industry has run automated backtests for decades: take a trading strategy, run it against years of historical market data, and see how it would have performed before risking real money on it. More than that, finance already operates under a formal regulatory framework for the exact question AI raises — how do you trust a model whose output drives a real decision?
Financial Analyst: A Day in the Life — Image by Gemini
That framework has a name: SR 11–7, supervisory guidance issued by the Federal Reserve and a banking regulator back in 2011, long before anyone was talking about generative AI. Reading it now, it sounds almost like a blueprint for trustworthy AI, written more than a decade early. It rests on three ideas. A model has to be built carefully. It has to be checked by people who didn’t build it and have the standing to demand changes, a principle the guidance calls “effective challenge.” And someone senior, ultimately the bank’s leadership, is accountable for the whole thing. No blind trust in any model, and conservative limits whenever a model’s confidence is shaky.
This rule didn’t come from a textbook. It came from an expensive lesson. In 2012, a trading position at a major bank, guided by flawed risk models that nobody seriously challenged, turned into a multi-billion-dollar loss, an episode that came to be known as the London Whale. The lesson the industry took away was blunt: an unchallenged model can do enormous damage, completely out of proportion to how minor it seemed at the time. That’s precisely the kind of damage a confidently wrong AI model can do, which is why finance didn’t need to invent a new way of thinking about AI. It already had one, and it’s stretching that existing framework to cover it.
By the mid-2020s, AI and machine-learning models made up roughly half of a typical large bank’s entire model inventory. In April 2026, regulators updated the original guidance, replacing it with a framework that scales the level of scrutiny to how much is actually riding on a given model. The newest rules technically treat the most advanced generative AI as falling outside their formal definition for now — the technology moved faster than the paperwork — but the underlying expectation hasn’t changed: independent checking, real accountability, full documentation. Regulators are already extending those expectations to AI in practice, regardless of the technicality.
Inside that framework, the new capability is genuinely striking. AI models can now read the messy, unstructured information that used to be hard for quantitative systems to use at all — news coverage, regulatory filings, earnings calls — and turn it into a trading hypothesis or a risk signal. That hypothesis then gets tested against detailed market simulations before a single dollar moves. The AI generates ideas quickly and imperfectly. The simulation checks them slowly and rigorously. Only what survives gets used.
The most interesting part of how these systems get deployed mirrors a pattern showing up across the industry. A large, expensive AI model acts as a “teacher,” and its knowledge gets distilled into a smaller, much cheaper “student” model that can run constantly, at scale, on live data. NVIDIA has published an open-source example of exactly this for financial data: a small model trained to match the judgment of a much larger one, at a fraction of the cost, with compliance checks built into the loop. Reported results show models a fraction of the size matching their larger teacher’s accuracy to within a couple of percentage points.
The fully autonomous version people sometimes imagine — AI traders quietly rewriting their own strategies hour by hour, with nobody watching — gets talked about more than it actually happens. In practice, a human still owns the strategy, and that human answers to a regulator working from a framework that predates the technology by more than a decade.
Across software, law, and finance, the same thing is true. The checking process wasn’t invented for AI. It already existed, built to catch ordinary human mistakes, and it simply absorbed a new, faster source of mistakes without needing to change its basic shape. The person responsible — the engineer who approves the merge, the lawyer who signs the filing, the risk officer who signs off on the model — never moved. They just started receiving better first drafts.
Medicine: the most promise, the most caution, and the right reason for both
Medicine is where the stakes are highest, and the honest picture splits cleanly into two parts: what’s already in use at the bedside, and what’s still being built toward.
Medical Researcher: A Day in the Life — Image by Gemini
Start with what’s real and already cleared by regulators, because it’s more substantial than most people realize. A company called HeartFlow has built a tool that takes an ordinary CT scan of the heart and constructs a detailed, patient-specific computer model of the coronary arteries — essentially a digital stand-in for that one patient’s heart. From that model, it calculates something called fractional flow reserve: a measure of how much a narrowed artery is actually restricting blood flow, not unlike how a kink in a garden hose slows the water passing through it. That number tells a cardiologist whether a blockage is serious enough to need a stent, without threading a catheter into the patient’s heart just to find out.
This tool carries clearance from the FDA. It turns a scan around in a few hours, and studies comparing it against the more invasive traditional method put its accuracy in the mid-eighty-percent range. A companion feature can even simulate placing a stent virtually and predict how blood flow would change before anyone touches the patient. This isn’t a pilot program. It’s used in routine cardiology practice today.
Cardiology isn’t unique here. The most advanced digital-twin work in medicine clusters around the heart and the brain, because both produce rich, detailed data — imaging, electrical signals — that gives an AI model something solid to be faithful to. One growing use is simulating the electrical activity of a scarred heart, to predict where dangerous rhythm problems might occur and guide where a surgeon should intervene. Regulators have even started treating this kind of simulation as a legitimate part of testing new treatments, not unlike how car companies crash-test virtual cars before building physical prototypes — except here, the test involves running a new device or therapy against a simulated patient first.
What’s still being built is the more ambitious version of this idea: combining a patient’s imaging, medical history, genetics, and wearable-device data into one living virtual model that doctors can use to rehearse a treatment before trying it for real. That vision is moving out of research labs and into real institutional programs. In September 2025, Siemens Healthineers and the Mayo Clinic announced a partnership to build AI-driven cardiovascular digital twins that predict heart complications before they happen. Separately, GE HealthCare and Mayo launched a project nicknamed GEMINI-RT, built around the idea of “twinning the patient, personalizing the beam,” to plan radiation cancer treatment dose by dose against a digital model of an individual patient. And in January 2025, the FDA released draft guidance actively encouraging the use of these digital-twin simulations in clinical trials, a sign the agency considers the approach credible enough to build a formal path for it.
A complete, whole-body digital twin isn’t here yet. Some people in the field imagine a future where a doctor can build a full virtual version of a patient in a day and test a dozen treatment options against it before choosing one. That future hasn’t arrived. But the steps toward it are now concrete, funded, and dated, rather than speculative.
The rules around accountability in medicine are the strictest of any of these four fields, for the most basic reason possible: a piece of software can’t hold a medical license. If an AI tool suggests a path and a patient is harmed, the law looks to the physician, who is responsible for the decision whether they followed the tool’s suggestion or overrode it. That puts doctors in a genuinely difficult spot — deviating from standard practice because an AI suggested it carries risk, and ignoring an AI’s warning carries a different kind of risk. The resolution has been to treat the AI strictly as an advisor, with a licensed human required to own every meaningful decision it touches. A separate California law, also taking effect in 2026, reinforces this by barring AI tools from implying they’re a licensed clinician unless an actual licensed person is overseeing what they say. The human in the loop here isn’t a courtesy. It’s the entire foundation the system rests on.
Where AI gets to work with more independence is in research, not direct patient care, and there’s a real, recent example worth knowing. In May 2026, a nonprofit research lab called FutureHouse published a paper in the journal Nature describing a system called Robin, which carried out an entire scientific investigation almost on its own: forming a hypothesis, designing an experiment, and analyzing the results. Starting with nothing but the name of a disease — a common cause of blindness called dry age-related macular degeneration — Robin read through hundreds of scientific papers, found a connection no single researcher had pieced together, and proposed that an existing glaucoma medication might help treat it. Human scientists then ran the actual lab experiments, fed the results back to the system, and confirmed the candidate worked. The whole process, from building the system to submitting the finished research paper, took about two and a half months.
The way Robin’s own creators describe it matters here. They call it a “lab-in-the-loop” system. The AI does the reading and the reasoning, and humans do the physical experiments and make the final scientific call. Its value wasn’t replacing the scientists. It was reading more of the existing research than any one person could hold in their head, and handing the scientists a far better place to start. In research, unlike law or patient care, the physical experiment itself is the check on the AI’s work. The AI doesn’t need to be right on the first guess. It just needs to point toward the right place to look, and let the lab confirm or rule it out. The AI proposes. Reality decides. A human interprets what that means.
The shape that repeats, and why it should reassure
Line up all four fields, and the same shape shows up in each one.
In software, the AI writes the code. Automated tests and human review catch the mistakes. The engineer who approves the change owns it.
In law, the AI drafts the brief. The attorney checks every citation against a real source. The attorney who signs the filing carries the liability.
In finance, the AI proposes a trading signal or a risk score. Simulations, backtests, and independent reviewers challenge it before it’s trusted. The person who owns the strategy answers to a regulator.
In medicine, the AI reads the scan or proposes the hypothesis. Clinical testing, or a physical experiment in a lab, confirms or rules it out. The physician, or the experiment itself, has the final word.
Generation is fast, and it’s fallible, in every single one of these. Verification is slower, and it’s reliable. And in every case, a specific, named human is the one accountable for what happens next.
The original distrust of AI was never irrational. Watching a system produce fluent, confident falsehoods is a completely reasonable reason to distrust it, taken on its own. What changed is that nobody serious is asking the model alone to be trustworthy anymore. They’re asking a larger system to be trustworthy — one where answers get checked against real sources, where a person validates the result, and where someone specific owns the outcome and can be held to it. None of that requires the AI to be perfect. It just requires the AI to fail in ways that are predictable enough to catch, which, it turns out, is exactly how it fails.
There’s a second, quieter shift worth naming, because it speaks to the fear sitting underneath a lot of the distrust: the worry that these tools exist to replace the people using them. What actually shows up across all four fields is a shift in where the human’s value sits, not its disappearance. The engineer spends less time typing routine code and more time deciding what should be built and what will break under pressure. The lawyer spends less time on the first rote pass through case law and more time on strategy and on the judgment calls no machine can make. The physician remains, completely irreplaceably, the person weighing a recommendation against one particular human being in one particular hospital bed. In every field, the center of gravity moves from creating the first draft to deciding whether that draft is right. That’s not a smaller job. It’s a more concentrated one, with the routine work stripped away and the part that actually required a license and a career left behind.
That’s what makes the growing use of AI in these fields a genuinely hopeful development, rather than a reckless one. It isn’t growing because the technology became flawless. It’s growing because the systems built around the technology matured to the point where its flaws are contained. Nobody needs to extend blind faith to an AI model to benefit from one. The right posture is the same one that already governs every other part of professional life: trust a process that assumes mistakes will happen and is built to catch them before they matter.
The error rate didn’t have to fall to zero for any of this to work. It had to become visible, expected, and somebody’s specific job to catch. In four fields where getting it wrong has real consequences, that’s exactly what happened.
Works Cited
Bloomberg Law. State of Practice Survey, 2026 (reporting roughly 83% of surveyed lawyers using AI at work). News and analysis at news.bloomberglaw.com.
Stanford RegLab and Institute for Human-Centered AI. Evaluation of legal research tools finding error/misgrounding rates of approximately 17–34% in leading purpose-built legal AI products, 2024.
Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. 2023). Sanctions order by Judge P. Kevin Castel for fabricated AI-generated case citations.
American Bar Association. Formal Opinion 512: Generative Artificial Intelligence Tools, July 2024.
Johnson v. Dunn, №2:21-cv-1701 (N.D. Ala. July 23, 2025). Order finding monetary sanctions an inadequate deterrent and disqualifying attorneys for AI-hallucinated citations.
California Assembly Bill 316 (Cal. Civ. Code § 1714.46), effective January 1, 2026. Bars the “autonomous-harm” defense in civil actions involving AI.
California Assembly Bill 489, effective 2026. Prohibits AI tools from implying licensed-clinician status absent real professional oversight.
SWE-bench Verified. A 500-instance, human-reviewed, execution-based benchmark of real GitHub issues; public leaderboards and methodology at epoch.ai and related trackers, 2024–2026.
Badhe, Sanket. “LegalSim: Multi-Agent Simulation of Legal Systems for Discovering Procedural Exploits.” arXiv:2510.03405, 2025; published in the NLLP workshop proceedings (ACL Anthology, 2025).
Board of Governors of the Federal Reserve System and Office of the Comptroller of the Currency. SR 11–7: Supervisory Guidance on Model Risk Management, April 4, 2011; revised interagency model-risk guidance issued April 17, 2026.
NVIDIA. Data Flywheel Blueprint and the AI Model Distillation for Financial Data developer example. NVIDIA Technical Blog and NVIDIA-AI-Blueprints repository, 2025–2026.
HeartFlow FFRCT Analysis. FDA-cleared computational coronary modeling for non-invasive fractional flow reserve; clinical literature in the European Heart Journal — Digital Health and the Journal of Cardiovascular Computed Tomography, 2020–2025.
Siemens Healthineers and Mayo Clinic. Partnership to develop AI-enhanced cardiovascular digital twins, announced September 2025.
GE HealthCare and Mayo Clinic. GEMINI-RT digital-twin initiative for radiation oncology planning, 2025.
U.S. Food and Drug Administration. Draft guidance supporting digital-twin simulations in medical-device trials and regulatory submissions, January 2025.
FutureHouse. “Demonstrating End-to-End Scientific Discovery with Robin.” Published in Nature, May 2026; system identifying ripasudil as a candidate for dry age-related macular degeneration. Code at github.com/Future-House/robin.