Why AI Governance is an Accountability Problem, Not an AI Problem

Over the last few months, I’ve found myself having more conversations about AI than any other technology. Some are about productivity, some are about innovation and some are about regulation, and AI Governance. Almost all of them eventually arrive at the same place, trust.

This is not about whether AI can create value, I think that debate is largely over. AI has already demonstrated its ability to improve productivity, accelerate decision-making and help organisations do more with the resources they have. The challenge now is not whether organisations should adopt AI (they absolutely should), it’s whether they can trust it, and, more importantly, whether they can prove they should trust it.

A governance, risk and compliance survey I read recently found that 87% of organisations lack full visibility into the AI tools being used across their business, 86% believe many AI tools are not enterprise-ready, and 71% reported that AI had contributed to a failed audit or regulatory lapse. The report also found that governance is struggling to keep pace with adoption.

These findings should not be interpreted as a warning against AI. Far from it. In my view, they just highlight that the technology is moving quickly and that governance needs to keep up.

Effective AI governance is ultimately about understanding which AI agents are operating within your organisation, what they are permitted to do, who is accountable for them and whether those controls can be evidenced.

A lot of the current discussion around AI governance treats it as though it is an entirely new discipline. I’m not convinced. Throughout my career, organisations have repeatedly faced similar challenges whenever new technology becomes mainstream. Cloud computing introduced new questions around ownership, visibility and control. Digital transformation created new dependencies on software, data and third parties and connected OT blurred the lines between IT and critical operations.

Now AI is presenting us with a similar challenge. The technology has changed but the governance questions have not.

Cyber security asks:

Those questions are still relevant today. The only difference is that we are now asking them about AI agents. In fact, during a recent internal discussion around the future of AI governance, I found myself returning to these five questions. They were intended to help frame how organisations should think about AI governance. The more I considered them, the more I realised they apply just as easily to cyber security, operational technology and digital transformation.

This sounds like an obvious starting point but unfortunately, it often isn’t.

Many organisations know which AI platforms they have purchased. Far fewer know which agents are actively operating across the business, what functions they perform, which systems they interact with and who is ultimately responsible for them.

Visibility has always been the foundation of governance. If you cannot see it, you cannot govern it.

Not all AI agents carry the same level of risk. Some simply retrieve information, while others make recommendations. Increasingly, some can take actions on behalf of users and systems.

The critical question is how much authority the agent has been given. An AI agent with unrestricted access to data, systems or business processes creates a very different risk profile from one operating within tightly controlled boundaries. Which is simply a least-privilege discussion.

Every AI agent operates through an identity of some kind such as a user or a service account. It may have a privileged role or an application identity. It will certainly have a delegated permission set.

Understanding which identity an agent uses is fundamental to understanding its capabilities and potential impact. Identity security has become one of the most important disciplines in modern cyber security.

Every cyber security team monitors behaviour.

We monitor networks, users, workloads and we monitor privileged accounts. AI agents should be no different. Organisations need confidence that they continue to operate within expected boundaries over time.

Systems change and permissions evolve. Data sources move, business processes adapt and change.  So trust cannot be a one-time decision. It needs to be continuously validated.

This is the question that ties everything together.

Boards, regulators and their auditors increasingly want evidence. Claims of control are no longer sufficient. Organisations must be able to demonstrate ownership, visibility, monitoring and accountability through evidence that can withstand scrutiny.

The conversation is gradually moving towards whether you can you prove the controls are working.

So this is not a lot of questions about AI security. Ask exactly the same questions of:

The logic still works. Which are active, what are they allowed to do, who owns them, are they behaving as expected and critically can you prove all that?

That is simply governance. The same disciplines that help organisations manage users, systems and third parties are now being applied to AI agents. Visibility, ownership, monitoring and assurance become increasingly important as autonomous capabilities become part of day-to-day operations.

This conversation is becoming increasingly important because the regulatory environment is changing. In Ireland, organisations are preparing for the implementation of NIS2, supported by guidance from the National Cyber Security Centre and initiatives such as Cyber Fundamentals (CyFun). The emphasis is increasingly on governance, resilience, executive responsibility and ongoing risk management rather than technology deployment alone.

In the United Kingdom, organisations continue to align to the NIS Regulations, the Cyber Assessment Framework (CAF), operational resilience programmes and evolving regulatory expectations around digital risk and accountability. The common theme throughout these initiatives is understanding critical services, managing risk and demonstrating effective governance.

The language varies depending on the framework. The expectation is remarkably consistent. Understand your risks, assign ownership, manage exposure, monitor effectiveness and provide evidence.

Although many of these frameworks are described as cyber security legislation or cyber security guidance, the underlying objective is accountability. That expectation will inevitably extend to AI.

I sometimes see organisations position governance and innovation as competing priorities. Strong governance should accelerate AI adoption, not slow it down. Trustworthy AI will scale much faster than unmanaged AI.

The organisations gaining the most value from AI seem to be the ones creating confidence around how their agents operate.

I think business leaders will become less interested in what AI can theoretically do and become more interested in understanding what a specific agent owns, how success is measured and who is accountable when something goes wrong.

That is a healthy shift. It encourages adoption while ensuring responsibility keeps pace.

For organisations looking to expand their use of AI confidently, I would focus on five priorities.

Organisations that can answer the governance questions quickly, confidently and with evidence will be the organisations that gain the greatest value from their AI deployments. Because ultimately, trust is created by accountability, and accountability begins with visibility, ownership and evidence.