Why a convincing explanation is not the same thing as a safe, fair or well-governed banking model
Artificial intelligence has created a peculiar governance problem for banks. The models that promise the greatest gains in fraud detection, credit assessment, customer service, compliance and operations are often the same models whose internal reasoning is hardest to reduce to a clean, human-readable explanation. That tension has made “explainability” one of the most frequently invoked safeguards in financial AI - and one of the most misunderstood.
Explainability matters. Regulators, customers, boards and model-risk teams need to understand why consequential systems behave as they do. But an explanation can be plausible without being complete, stable without being correct, or easy to read without faithfully describing the mechanism that produced the decision. The danger is that banks may mistake an explanation layer for evidence that a model is controlled.
The emerging supervisory view is more demanding: explainability should be treated as one control among many. The Bank for International Settlements has examined how regulators can approach AI explainability, while broader frameworks from the Financial Stability Board and NIST emphasise governance, validation, resilience, data quality and accountability alongside transparency. [BIS] [FSB] [NIST]
The Old Model-Risk Playbook Was Built for a Different Kind of Model
Banks are not new to model risk. Credit scoring, market-risk models, anti-money-laundering systems and stress-testing frameworks have long required development standards, independent validation, performance monitoring, documentation and governance. In the United States, the Federal Reserve’s SR 11-7 guidance helped institutionalise the idea that model risk arises both from incorrect models and from correct models being used incorrectly or inappropriately. [Federal Reserve SR 11-7]
Traditional model governance assumes that a model can usually be bounded: its inputs are reasonably known, its objective is specified, its output has a defined interpretation, and validation can compare its performance against an expected use. Generative and agentic AI weaken each of those assumptions. A general-purpose model can respond differently to small changes in prompts, incorporate external tools, generate unstructured outputs and operate across tasks for which it was not explicitly trained.
That gap is now visible in formal supervisory policy. In 2026, the OCC’s revised model-risk guidance explicitly excluded generative AI and agentic AI from scope because of their novelty and rapid evolution, while indicating that the banking agencies intended further work on banks’ use of AI. That exclusion is not a regulatory exemption; it is evidence that older model-risk frameworks do not map neatly onto newer systems. [OCC]
Explainability Is Not the Same as Interpretability
The language around AI governance often collapses several different ideas into one. Interpretability usually refers to how directly a human can understand the relationship between a model’s inputs, structure and outputs. Explainability is broader: it may include post-hoc techniques that produce a human-readable account of why a model generated a result even when the model itself remains complex.
That distinction matters because post-hoc explanations can become a second model layered on top of the first. A bank may have a highly complex decision engine and a separate explanatory method that estimates which features were influential. The explanation can be useful without being a literal reconstruction of the model’s internal process.
The BIS Financial Stability Institute has highlighted that explainability is context-dependent: different audiences need different kinds of explanations, and there is no single method that satisfies every regulatory, technical and consumer-protection objective. A model developer may need diagnostic detail; a supervisor may need evidence of control; a customer may need a clear reason for a consequential decision. [BIS]
The Fidelity Problem: A Good Story Can Still Be the Wrong Story
The most uncomfortable limitation of explainability is fidelity. An explanation that sounds coherent is not necessarily an accurate description of the causal path inside a complex model. This becomes especially important with large language models, which are exceptionally good at generating fluent rationales. Fluency can create the impression of introspection even when the generated explanation is itself another prediction.
For banks, that creates a governance trap. If a system denies a transaction, recommends closing an account or flags a customer as high risk, a natural-language rationale may make the decision appear auditable. But the governance question is not whether the model can produce a reason after the fact. It is whether the institution can demonstrate that the reason is stable, supported by the actual decision process, consistent with policy and legally sufficient for the use case.
This is why consumer-credit regulation provides a useful warning. The U.S. Consumer Financial Protection Bureau has stated that lenders using complex algorithms still need to provide accurate, specific reasons for adverse actions. Complexity is not a defence for an explanation that fails to tell the applicant why credit was denied. [CFPB]
A Transparent Model Can Still Be a Bad Model
Explainability also does not solve the underlying statistical and operational risks. A perfectly understandable model can be trained on biased data. It can rely on stale relationships. It can be vulnerable to manipulation. It can perform badly during market stress. It can optimise the wrong objective. It can produce disparate outcomes even when no explicitly protected characteristic is used.
The Financial Stability Board’s work on AI in finance places model risk alongside data quality, governance, cyber risk, third-party dependence and market correlations. That framing is important: transparency may help diagnose some problems, but it does not remove them. [FSB]
NIST takes a similarly multidimensional approach. Its AI Risk Management Framework treats trustworthy AI as involving characteristics such as validity and reliability, safety, security and resilience, accountability and transparency, explainability and interpretability, privacy and fairness. In other words, explainability is one dimension of trustworthiness, not a substitute for the others. [NIST]
The Trade-off Between Performance and Explainability Is Often Misstated
Banking debates sometimes frame the issue as a binary choice: use a simple model that can be explained or a sophisticated model that performs better but behaves like a black box. In practice, the trade-off is more complicated. Some high-performing models can be made reasonably interpretable for a specific use case, while some simple models can be badly specified or misunderstood.
The relevant question is proportionality. A low-risk internal productivity tool does not need the same level of explanation as a system deciding whether a household receives a mortgage. The European Union’s AI Act follows this logic by applying heightened obligations to high-risk uses, including certain systems used to evaluate individual creditworthiness. The regulatory architecture links risk level to requirements for governance, documentation, human oversight, robustness and traceability rather than treating explanation as a stand-alone solution. [European Commission]
When AI Becomes Agentic, Model Risk Becomes Action Risk
Agentic AI makes the explainability problem more serious because the system may no longer stop at producing a score or recommendation. An agent can interpret an objective, call tools, retrieve information, make intermediate decisions and execute actions. The risk therefore moves beyond “why did the model say this?” to “what was the model allowed to do because it said this?”
A treasury agent that incorrectly predicts liquidity pressure could move funds. A fraud agent could block legitimate payments. A compliance agent could escalate or close cases. An operational agent could change settings in connected systems. The model’s output and the institution’s real-world action can become part of the same automated chain.
This is one reason recent central-bank discussion of AI has placed heavy emphasis on operational resilience and third-party technology dependencies. The ECB has warned that financial institutions’ growing use of AI needs to be considered alongside cyber risk, concentration and critical infrastructure dependencies. [ECB]
Third-Party Models Create a Second Explainability Gap
Banks increasingly obtain AI capability through cloud platforms, foundation-model providers and specialist vendors rather than building every system internally. That creates a practical problem for model-risk teams: the institution may be accountable for an output generated by technology it cannot fully inspect.
The Bank of England and FCA’s survey of AI in UK financial services found widespread use of third-party implementations and material concentration among major cloud and model providers. This makes explainability partly a vendor-governance problem. A bank can demand documentation, testing rights and performance data, but commercial and technical constraints may still prevent full visibility into the underlying model. [Bank of England/FCA]
The governance standard therefore cannot be “we understand the source code.” For many modern AI systems that may be unrealistic or insufficient. Banks instead need evidence that the system behaves acceptably under the conditions that matter: normal use, edge cases, adversarial inputs, data shifts, outages and stressed environments.
What Good Validation Looks Like When Full Explanation Is Impossible
The practical answer is to shift some attention from explaining the model to validating the system around it. A bank does not need perfect access to every internal parameter to test whether an AI application is fit for purpose. It can examine outcomes, stability, failure modes, sensitivity, fairness, security, overrides and performance under controlled scenarios.
For higher-risk applications, that means combining traditional validation with techniques more suited to modern AI: benchmark testing, red-teaming, adversarial testing, counterfactual analysis, drift monitoring, retrieval-quality testing, prompt and tool-permission controls, and systematic review of hallucination or fabrication rates. The objective is not to produce one definitive explanation but to build multiple lines of evidence about behaviour.
The U.S. Treasury’s financial-services AI work reflects this broader lifecycle view, emphasising accountability, transparency and resilience rather than treating model explanations as a complete risk-management framework. [U.S. Treasury]
Human Oversight Can Also Become Performative
Banks often respond to explainability concerns by keeping a human “in the loop.” That can be valuable, but it can also provide false comfort. Human reviewers may see only the model’s recommendation and explanation, not the underlying uncertainty. If the system is correct most of the time, supervisors can develop automation bias and approve outputs mechanically.
Effective human oversight therefore requires genuine authority, time and information. Reviewers need to know when the model is uncertain, when data quality is weak, when the case is outside the system’s normal operating domain and what evidence would justify overriding the recommendation. A human approval click is not meaningful governance if the workflow is designed so that disagreement is practically impossible.
The Board-Level Question Is Not “Can We Explain the Model?”
Boards and senior executives should resist reducing AI governance to a single explainability metric. The more useful questions are operational: What decisions can the system influence? What data can it access? What actions can it initiate? How is performance independently tested? What happens when the model changes? Who can stop it? How quickly can the bank detect a failure?
This is particularly important because financial institutions are adopting AI rapidly. The Bank of England and FCA survey found that a large majority of responding firms were already using AI, while European supervisors have likewise reported widespread adoption among banks. As AI becomes ordinary infrastructure rather than a specialist experiment, governance needs to move from project-by-project review to enterprise-wide control. [Bank of England/FCA] [ECB]
A Better Standard: Explain What Matters, Test What Cannot Be Explained
The strongest model-risk framework for AI is unlikely to demand complete transparency in every circumstance. That standard would be unrealistic for some systems and could encourage superficial explanations that satisfy documentation requirements without improving safety.
A more defensible approach is risk-based. Banks should require explanations that are meaningful for the decision and audience; validate whether those explanations are faithful enough for their purpose; and supplement them with empirical testing wherever the model cannot be fully interpreted. High-impact decisions should receive stronger documentation, narrower permissions, more frequent monitoring and clearer human escalation.
This approach also recognises an important truth about banking: institutions have always managed systems they cannot predict perfectly. Credit portfolios, markets, customer behaviour and operational networks are all complex. The answer has never been to demand perfect understanding. It has been to build controls that make uncertainty survivable.
Conclusion: Explainability Is a Control, Not an Alibi
The debate over explainable AI can create the impression that the central problem is opacity. It is not. The central problem is accountability under uncertainty.
Banks need to understand enough about their AI systems to govern them, challenge them and explain consequential decisions. But they should not confuse a readable rationale with proof of correctness. A model can be explainable and biased, transparent and unstable, interpretable and badly governed. Conversely, a complex model can sometimes be used responsibly when it is tightly bounded, independently tested and continuously monitored.
The real standard for banking AI should therefore be higher than “Can the model explain itself?” It should be: Can the institution demonstrate that the system is appropriate for its purpose, that its limits are understood, that its behaviour is tested, that its actions are controlled and that someone remains accountable when it fails?
Explainability belongs inside that framework. It should never become a substitute for it.
References
2. Financial Stability Board - The Financial Stability Implications of Artificial Intelligence
3. NIST - Artificial Intelligence Risk Management Framework
4. U.S. Treasury - Treasury and Artificial Intelligence
5. Office of the Comptroller of the Currency - Revised Model Risk Management Guidance, 2026
6. European Commission - AI Act regulatory framework
7. Bank of England and FCA - Artificial intelligence in UK financial services: 2024
8. European Central Bank - Strengthening operational resilience for the age of AI
10. Federal Reserve - SR 11-7 Guidance on Model Risk Management