What happens when AI systems are left ungoverned and then… go rogue?
A recent incident, featuring Anthropic’s Mythos AI has given us an idea. The UK’s AI Security Institute (AISI) revealed how the Anthropic tech tried to create fake human profiles in an attempted cyber-attack.
Concerns about AI safety continued to grow when OpenAI revealed that its agents escaped a testing environment and breached the AI startup, Hugging Face. Competitors Meta and Google have also said their agents were involved in separate breach attempts.
These incidents highlight why organisations should carefully consider how technologies are deployed, controlled, observed and responded to.
They also raise an important question: who is accountable when an AI system takes an unexpected action?
Organisations need to factor in AI governance, cybersecurity, human oversight, and organisational accountability in their AI strategy.
In this blog:
- What is AI governance?
- What recent AI security incidents tell us about AI governance
- What organisations deploying AI agents need to establish
- Why does AI create a new accountability challenge?
- Who is accountable?
- Why is human oversight important for AI security?
- What are the cybersecurity risks of agentic AI?
- What should organisations do to govern autonomous AI?
What is AI governance?
AI governance is the framework of policies, processes, controls and responsibilities that organisations use to manage how AI is developed, deployed and used.
Effective AI governance determines who is responsible for an AI system, what the system is allowed to do, what risks must be assessed and how the organisation monitors its behaviour.
Katrina Gallagher, FDM Head of Privacy, highlights the importance of understanding how AI is being used and the different risks associated with different use cases:
“In order to understand the risks, the organisation needs to have visibility over where and how AI is being used, what information it can access, and what internal or external processes or decisions it may influence. The level and nature of the risk will depend on the use case, for example: a productivity tool will have a very different risk profile to an AI system making automated decisions about individuals or accessing the individual’s sensitive data.”
Three best practices for building secure AI systems
Despite the above, in most cases organisations are often concerned about the following:
- Data leakage particularly if sensitive or business confidential information is entered into third party AI tools.
- AI outputs can be inaccurate, biased, or fabricated and relying on these can lead to outcomes such as poor decisions, financial or reputational damage and potential regulatory consequences.
- The concern that employees can use AI without undergoing established procurement, security, or privacy assessments, leaving organisations blind to their own AI environment.
What recent AI security incidents tell us about AI governance
Recent incidents involving AI models have shown that autonomous systems can behave unexpectedly when given unsupervised access to tools, systems or the wider internet.
Recently, OpenAI scrapped plans to release a new model, saying it showed “higher levels of deception”. Additionally, the recent ASOS incident where app users received notifications from hackers also highlights how cyber criminals are using public and unconventional methods to put pressure on organisations following a data breach.
In the past we’ve seen these extortions happen in secret with the hackers hoping to get paid in exchange for their discretion.
However, in this case, the alleged hackers appear to have taken a different approach by making the incident public, potentially as a way of putting additional pressure on ASOS to enter negotiations.
This shows organisations need to consider not only what an AI model can generate, but also:
- What systems it can access
- What information it can retrieve
- What actions it can take
- What permissions it has
- How its activity is monitored
- When human approval is required
- How quickly the organisation can intervene
Without answers to those questions, an organisation doesn’t have an AI strategy.
Not sure where to start with an AI strategy? FDM can help organisations identify their gaps. Find out more.
Why does AI create a new accountability challenge?
Research shows that 87% of organisations lack full visibility into the AI tools being used across their business.
These findings highlight that organisational governance needs to keep up.
Company leadership should establish who is accountable for personal data being passed to AI agents and whether those controls can be evidenced. If the agent subsequently takes an unexpected action, that decision becomes part of the incident.
This creates what could be described as an accountability gap.
What banking leaders are getting wrong about AI
Organisations need to understand the risks, assign ownership, manage exposure, monitor effectiveness and provide evidence.
For example, an employee might say that the AI made the decision, a technology team might say that the employee approved the deployment, and a security team might point to the permissions that had already been granted.
If organisations had an effective AI accountability policy in place, they would prevent finger-pointing.
Who is accountable?
UK regulators have answered the accountability question. The FRC’s March 2026 guidance on generative and agentic AI in audit states that the human auditor is always accountable. A Canadian tribunal rejected an airline’s argument that its chatbot was a separate entity. The EU AI Act requires human overseers to remain aware of “the possible tendency of automatically relying or over-relying on the output”.
Organisations should create clear lines of accountability, ensuring that individuals understand who is responsible for overseeing AI systems, reviewing their outputs and intervening when something goes wrong.
Why is human oversight important for AI security?
Human oversight is one of the many important ways of managing autonomous AI.
Effective human oversight in AI systems is more than just technical expertise; it also requires a deep understanding of the ethical considerations of AI decision-making. Human oversight helps to mitigate the risks associated with AI, such as bias, discrimination, and operational errors.
Organisations should be able to understand what an AI system is doing, question its decisions and stop it when necessary.
Katrina believes, “The days when security awareness training could focus predominantly on tailgating, locking screens and being vigilant when taking calls in public are gone. While these practices still matter, the threat to organisations has evolved.”
She also shares, “Organisations need to train their employees to understand that phishing emails can be sophisticated, personalised, and convincing. Employees also need to be aware of AI-enabled impersonation, deepfakes, credential theft, and social engineering, as well as guidance on using generative AI securely. Further, organisations should make secure behaviour straightforward. Employees need to know which AI tools are approved, what information they can input, and where to seek guidance when unsure. Finally, a positive culture is also essential, where employees feel empowered to report suspicious activity or mistakes without fear of disproportionate blame. While prevention is critical, speed of reporting is also crucial.”
At FDM, we’re committed to developing AI-enabled talent who can confidently navigate this evolving tech landscape.
What are the cybersecurity risks of agentic AI?
Research shows that while over 88% of companies globally use AI in the workplace, only 40% report having organisational guidance on generative AI use.
With that, organisations should consider several areas when managing the cyber risk of agentic AI, including the possibility of agents being manipulated, taking unintended actions and operating with excessive permissions.
Katrina, explains why organisations remain vulnerable as AI becomes more deeply embedded in their technology environments:
“Organisations are vulnerable to AI-related cyberattacks because AI in cybersecurity is a double-edged sword. On one hand, it can be used to detect and address threats earlier and faster; on the other hand, attacks are quicker, easier to scale and more sophisticated.
Further, processes and governance have to keep up. Governance follows growth. Using AI to monitor your infrastructure is pointless if administrative policies are weak, or AI is used outside the organisation’s established governance processes.
Finally, organisations can often be vulnerable as a result of their attack surfaces. Organisations store their data across different systems, and while AI can monitor these environments, every new system or supplier can leave an organisation open to vulnerability.”
What should organisations do to govern autonomous AI?
Organisations should carefully consider how agentic AI is deployed, constrained, observed and responded to.
This provides a useful framework for organisations looking to introduce autonomous AI:
Identify what could go wrong
Before deploying an AI agent, organisations should consider how the system could fail, be manipulated or behave unexpectedly.
Threat modelling can help organisations identify potential attack paths and understand what could happen if an AI system is compromised or takes an unintended action.
Prompt carefully
The instructions given to an AI agent are an important part of its operating environment.
Organisations should define clear objectives, constraints and boundaries rather than assuming an AI system will always interpret an instruction in the intended way.
Prompt clarity should also be considered alongside technical controls.
Set the right level of oversight
Not every AI task requires the same degree of human involvement.
Organisations can establish different levels of autonomy depending on the consequences of an action.
For example:
Recommend — AI provides information or suggests an action.
Approve — a human reviews and authorises the action.
Execute — AI can perform the action independently.
This creates a practical framework for deciding where human intervention is required.
Make your AI activity easy to attribute
Organisations need to know when an action has been performed by an AI system and which system was responsible.
Clear attribution can support monitoring, investigation and accountability.
AI activity should therefore be appropriately logged, with organisations able to establish what the system did, when it did it and what permissions or tools it used.
Maintain the ability to ‘pull the plug’
Autonomous AI systems need an effective way to be stopped.
The ability to interrupt an agent becomes increasingly important when it can interact with external systems or perform consequential actions.
Organisations should therefore consider how they would disable an AI system or restrict its access if it began behaving unexpectedly.
Summary
AI governance and cybersecurity are increasingly becoming interconnected.
As organisations give AI access to more systems and information, security teams need to understand its capabilities and risks. At the same time, AI teams need to understand the security implications of the systems they are building.
This requires closer collaboration between technology, cybersecurity, legal, risk and business teams.
Katrina explains how this shift affects both employees and cybersecurity professionals. She shares, “Employees will need a basic level of AI and security literacy. Organisations should support their employees in understanding how data is used within appropriate tools, how to recognise AI-enabled social engineering, and to critically assess content rather than automatically trusting it.”
One of the most significant risks associated with AI is how data is collected, stored, and reused. Many AI systems rely on large datasets to learn and improve, and publicly available tools often retain conversation history or use inputs for model training.
An AI security incident should not simply result in a technical fix.
Organisations should also investigate the decisions and processes surrounding the incident.
For example:
- Why did the AI have access to the system?
- Was that access necessary?
- Who approved it?
- Were the risks assessed?
- Were appropriate safeguards in place?
- Could the same issue affect another AI system?
- Was there sufficient monitoring?
- Could the organisation stop the system quickly?
- Was it clear who was responsible?
These questions help organisations move from incident response to organisational learning.
Contact us to learn how our AI-ready consultants can help your organisation’s AI transformation.