An artificial intelligence system that drafts a customer reply poses a different management problem from one that sends the reply, updates the customer record and issues a refund. The first produces information for someone to review. The second changes the business. That distinction should shape how companies approve AI projects, design access controls and measure results.
For business leaders, the practical question is how much authority a system needs to perform a particular task. Giving it access to more information and tools may make a demonstration impressive, but it can also increase the consequences of a mistake. A useful deployment starts with a specific operational purpose, grants limited permissions and provides a way to identify and contain errors. More capable models do not remove those requirements.
This matters to banks and other financial businesses as well as companies outside regulated sectors. An automated action can affect payments, confidential records, contractual commitments or a customer’s ability to obtain help. The controls should reflect those consequences, rather than assuming that every AI application belongs in the same risk category.
The shift from producing answers to changing systems
An AI agent can be understood as software that uses a model to select steps and interact with tools in pursuit of an assigned goal. Implementations vary considerably. Some only retrieve information and propose an action. Others can execute a sequence of operations across connected applications. The term agent therefore tells a manager less than a clear description of what the system can actually do.
Consider a hypothetical purchasing assistant. It might search an approved catalogue, compare delivery dates and draft a purchase request. Those functions can be useful without allowing it to add suppliers, change bank details or approve payments. Each additional capability introduces a separate decision about authority, evidence and accountability. Combining them in a single conversational interface does not make the underlying business controls unnecessary.
The OWASP discussion of excessive agency describes risks associated with excessive functionality, permissions and autonomy in systems built around large language models. Its recommendations include restricting tool permissions and enforcing access through downstream systems. The lesson for managers is concrete: an application should receive the authority required for its task, and that authority should be enforced outside the model’s generated instructions.
Define the task before buying the technology
A business case should specify which problem the system is expected to solve. Reducing the time needed to prepare a standard response is different from resolving a dispute or authorising an exception. If the task is poorly defined, evaluation can reward persuasive output even when the system has misunderstood the work or exceeded its role.
Teams should examine the existing process before automating it. They need to know where information comes from, which decisions require judgement and which exceptions already cause delays. A process with inconsistent customer records or unclear approval rules will carry those weaknesses into the new application. Automation can repeat an error more quickly, making the quality of the original workflow especially important.
Success measures should also distinguish speed from correctness. A pilot that reduces drafting time but increases rework may offer limited value. A system that closes more cases while making it difficult for customers to appeal may create costs elsewhere. The assessment should include the work transferred to reviewers and the time needed to investigate unusual results.
Access should follow the business role
Permission design begins with the resources the application can read and the actions it can perform. A service desk assistant may need access to an order’s delivery status without needing the customer’s full financial history. A reconciliation assistant may need to compare transaction records without being able to alter them. These distinctions are easier to maintain when access is deliberately narrow.
Shared administrator accounts are a poor foundation for this arrangement. They can make it difficult to identify which application performed an action and to remove one system’s access without disrupting others. Dedicated identities, restricted credentials and appropriately scoped tools give the business more options when a deployment needs to be paused or investigated.
The application should not be able to increase its own permissions simply because the model decides a broader action would help. Access requests need an established approval route. Where software acts on behalf of an employee, the design should also preserve the employee’s authorised scope instead of turning a limited user request into an unrestricted company-level operation.
External content is not an instruction from the business
Agents may read emails, documents, web pages or messages created by people outside the organisation. Those sources can contain misleading statements or instructions designed to redirect the application. A customer attachment should be treated as material to examine, rather than as authority to change how the company’s systems operate.
This creates a practical separation problem. The model may encounter useful facts and hostile instructions in the same document. Blocking a few suspicious phrases cannot establish that all external material is safe. Controls need to limit what the application can do even if the model misinterprets a source. Sensitive operations should require validated inputs and a separate authorisation decision.
In a hypothetical finance workflow, an invoice might contain text asking an assistant to replace the supplier’s bank account. The application can extract that request for review without executing it. The company’s established verification process should govern the account change. This avoids relying on the model to decide whether the invoice itself provides adequate evidence of authority.
Human review needs a specific purpose
Adding an approval button does not automatically create effective oversight. A reviewer needs enough information to understand the proposed action, the source of its inputs and any important uncertainty. If the interface shows only a confident summary, the employee may approve a decision without seeing what the system omitted or misunderstood.
The design should identify which actions require review and why. Low-impact, reversible operations may be handled differently from payments, changes to access rights or communications that commit the company. Reviewers need the time and authority to reject or escalate a proposal. Their performance should not be judged solely on how quickly they approve it.
Approval fatigue is another operational concern. If a system produces a large volume of routine requests, staff may begin to treat review as a formality. A better approach is to reduce unnecessary requests, make exceptional cases visible and keep material decisions within a manageable workload. Review quality should be tested during the pilot, rather than presumed from the presence of a human in the process.
Data access also creates privacy obligations
Useful automation does not always require copying every available record into an AI service. Companies should examine which fields are necessary, where information is processed, how long it is retained and who can access it. Retrieval permissions matter as much as the visible prompt, because an apparently simple question may cause the application to gather a much wider set of records.
The UK Information Commissioner’s Office guidance on AI security and data minimisation discusses these issues in the context of UK data protection. It provides a useful reference for evaluating necessity and security, while legal obligations depend on the jurisdiction and processing activity. A product’s ability to accept information does not establish that the organisation has a valid reason to supply it.
Logs need similar care. Detailed records can help explain an incident, but they may also contain personal information, credentials or commercially sensitive material. Retention, access and redaction should be designed together. A system that protects its main database but exposes the same information through unrestricted logs has not resolved the underlying problem.
Testing should cover the whole workflow
Model benchmarks provide information about particular tests; they do not establish that a connected business application is reliable. The deployment includes prompts, permissions, retrieval systems, tools and human interfaces. Errors can arise in any of those components, including places where the model itself performed as expected.
Tests should include ambiguous requests, missing information, conflicting records and attempts to push the application beyond its task. They should also examine repeated actions, interrupted connections and unavailable tools. A refund request that fails midway should not be retried blindly if the first attempt may already have succeeded. The application needs a reliable way to check the outcome.
The NCSC guidelines for secure AI system development organise security considerations across design, development, deployment and operation. CISA’s accompanying announcement emphasises that the guidance applies to AI systems generally, including those using external services. This lifecycle approach supports a broader evaluation than testing a model once before launch.
Record decisions and prepare for recovery
An operational record should help investigators answer what the system was asked to do, which information it used and which actions it attempted. Recording an outcome alone may be insufficient if the business cannot reconstruct the route to that outcome. However, collecting every possible detail can create unnecessary privacy and storage exposure. The record needs to serve an identified audit or recovery purpose.
Recovery is partly a business design issue. Some actions can be reversed directly, while others require a compensating action or contact with an affected customer. The team should know how to revoke access, stop queued work and check for incomplete transactions. It should also know who makes those decisions outside normal working hours.
Supplier arrangements deserve scrutiny. A change to a hosted model, tool integration or service configuration may affect behaviour without a visible change to the user interface. Contracts and operating procedures should address notification, testing, service interruption and the practical ability to move essential work to another process. A manual fallback needs enough staffing and instructions to work when it is required.
Measure value after including control costs
The NIST AI Risk Management Framework offers a voluntary reference for incorporating trustworthiness into the design, use and evaluation of AI systems. Applying a framework should support an operational decision, rather than become an exercise in collecting documents. The business still needs evidence that a particular deployment performs acceptably in its actual setting.
The financial assessment should include integration, monitoring, review, incident handling and ongoing testing. Time saved on a routine task can be worthwhile, but it should be compared with the full cost of delivering that saving safely. Expected benefits should remain estimates until the organisation has measured them against a credible baseline.
Staged deployment gives the business a way to learn without granting broad authority immediately. An application can begin by observing a workflow, then propose actions for review and later perform selected operations within limits. Progress should depend on evidence about errors, exceptions and recovery, rather than a predetermined schedule or enthusiasm for a more autonomous product.
Questions for business leaders
Does a more accurate model need fewer access controls
No. Accuracy and permission design address different issues. Even a capable model can receive misleading inputs or encounter unfamiliar conditions. The scope of its access should still match the authorised task and the consequences of a mistake.
Should every automated action require approval
That depends on its impact, reversibility and the surrounding controls. Requiring approval for every minor action may make review ineffective. Material actions need a clear basis for authorisation, with appropriate escalation where information is uncertain.
What should a pilot demonstrate
A pilot should establish useful performance on representative work, acceptable error handling and a workable recovery process. It should measure the additional burden on employees as well as the time the application appears to save.
The strongest case for an AI agent is a clearly defined task that benefits from limited, accountable automation. Companies can expand that role as evidence develops. The starting point is to decide what authority the system needs, who remains responsible and how the business will respond when something goes wrong.
References
1. OWASP — LLM06 2025 Excessive Agency
2. ICO — Security and Data Minimisation in AI
3. NCSC — Guidelines for Secure AI System Development