Can you trust a model you cannot see inside?
The honest starting point: nobody can fully see inside a large language model today, including the people who build them. Interpretability research, the effort to understand what is happening inside these systems, is making genuine progress but remains incomplete. So the trust question, as a business buyer, cannot be answered by inspecting the model's reasoning directly.
It can be answered a different way, and it is the same way you already answer it for people. You cannot see inside your estimator's head either. You trust their work because the tasks are defined, the sources are checkable, someone reviews the output, and there is a record of what was done. The practical question for an AI system is not "do we understand the model?" but "is the system built so that mistakes are visible and cheap?"
That reframing turns an unanswerable research question into a buying checklist. This post covers both halves: what interpretability research actually is, briefly and accurately, and what to look for in a system given that the research is unfinished.
What is interpretability research, in plain terms?
A trained model is not written line by line like normal software; it is grown through training, and what it learns is stored as billions of numbers with no labels. Interpretability researchers work out what those numbers are doing: identifying internal features that correspond to recognisable concepts, and tracing how they combine when the model answers a question. Think of it as neuroscience for neural networks. Anthropic publishes research in this area, including work mapping millions of internal features in production models and tracing the steps a model takes through a task.
Two accurate takeaways for a non-specialist. First, real progress: researchers can now identify and even adjust specific concepts inside a model, which was not possible a few years ago. Second, real limits: coverage is partial, the tools are research-grade, and no vendor can yet hand you a complete account of why a model produced a specific answer. Anyone selling "fully explainable AI" on top of a large language model is overclaiming.
If nobody can see inside, why is buying still rational?
Because opacity is not the deciding risk factor. Consequence containment is.
You already run a business on opaque components. Employees, subcontractors, even your accountant's judgement: none of them is inspectable from the inside, and all of them make mistakes. What makes them manageable is the scaffolding around them: defined scopes, checkable outputs, review before consequences, records after the fact. Businesses have run on this pattern for centuries.
The same scaffolding is what separates a safe AI deployment from a risky one. A model drafting a document that a named person reviews is a contained risk whatever is happening inside the model. The same model sending unreviewed emails to clients is an uncontained one. This is the core argument in what directors need to know about AI agents: capability is the vendor's problem, containment is yours.
What should a buyer actually look for?
Four properties. Together they make mistakes visible and cheap, which is the only trust that matters commercially.
| Buying criterion | Question to ask the vendor | What good looks like |
|---|---|---|
| Constrained tasks | "Exactly what can this system do, and what can it not do?" | A narrow, written scope: drafts these documents, answers from these sources. Not "it's a general assistant" |
| Sources shown | "When it states a fact, can I see where it came from?" | Answers cite the specific document and passage, and the citation is clickable and checkable |
| Approval steps | "What happens between the model's output and anything a client sees?" | A named human approves anything that leaves the business; the system cannot skip the gate |
| Logs | "Can I see afterwards what it was asked and what it produced?" | A reviewable record of inputs and outputs, retained, searchable, per user |
The pattern behind all four: each one converts a silent failure into a visible one. A constrained task means errors land in a known place. Shown sources mean a wrong claim can be caught in seconds. Approval gates mean the cost of a mistake is a rejected draft, not a client incident. Logs mean that when something does go wrong, you can find out what happened, which is the same logging discipline the NCSC small business guide recommends for the rest of your IT.
Notice that model choice barely appears. A modest model inside this scaffolding is more trustworthy, in the business sense, than a frontier model without it. There is more on picking the engine itself in which AI model for construction teams.
How do you test trust before you commit?
Run a pilot designed to surface mistakes, not to hide them.
Pick one constrained task, run the system alongside your existing process for a few weeks, and count: how many outputs needed correction, how long did checking take, did the error rate fall as prompts and sources improved? A vendor confident in their system will welcome this; a vendor who resists measurement is telling you something. The full method is in what a good AI pilot looks like, and the one non-negotiable is that errors found in the pilot are data, not failure. You are buying the error rate and the cost of catching them, not perfection.
Interpretability research will keep improving, and it genuinely matters: it is part of how the labs make models safer at source. But you do not have to wait for it to finish. Firms that deploy AI well today, AI Metric's builds included, trust the scaffolding, not the black box. Constrain the task, show the sources, gate the output, keep the logs. That is what trusting a system you cannot see inside has always meant.