The AI governance challenge: A very human problem

There was a time when conversations about artificial intelligence (AI) in healthcare felt slightly theoretical. We talked about future possibilities, exciting innovations, and what might happen one day. That day has come.

The question has always been how we manage this safely, – something we discussed recently at one of our AI Knowledge Cafes. These were some of the highlights.

What became clear very quickly is that we are adopting AI more quickly than we can draw up rules and guidance to oversee it.

Who oversees AI?

This varies from organisation to organisation. Departments responsible for it can include:

• information governance
• cyber security
• clinical AI groups
• procurement
• newly formed AI steering groups

Some organisations have created sophisticated oversight structures, yet many frontline staff don’t even know they exist.

This creates an interesting challenge. Governance cannot simply be something that happens in a committee room. If staff do not know the rules, understand the risks, or know where to seek advice, governance risks becoming disconnected from practice.

Patient data changes everything

Perhaps the most significant discussion was about the use of patient information by Microsoft Copilot.

Some organisations are beginning to approve this, whilst others remain cautious. Decisions depend on local technical infrastructure, Microsoft tenant arrangements and security architecture. Organisations operating their own tenants often have greater flexibility, while those working within shared NHS.net environments are likely to encounter additional governance challenges.

There is also the question of what patients think? Concerns were raised about transparency, consent, opt-out mechanisms and how much the public understands about how AI use its data. In the end trust may prove just as important as technology.

The rise of shadow AI

The same thing happened in every organisation.

Staff are already using AI. Sometimes they are using approved tools. Sometimes they are experimenting with free tools, personal accounts, or applications that governance teams know little about. This growing phenomenon has become known as Shadow AI.

Prohibition alone is unlikely to work. People generally adopt new technologies because they help them solve problems. If organisations simply say, "don't use AI", staff may seek alternatives elsewhere. A more sustainable strategy may involve providing safer, governed options while simultaneously investing in education and support.

As one participant observed, the greatest cybersecurity risk is rarely the technology itself. It is usually human behaviour.

If AI gets it wrong, who gets the blame?

If an AI system produces a recommendation that contributes to patient harm, who is responsible?

• the clinician?
• the organisation?
• the software supplier?

These questions remain largely unresolved. There was concern that people would have to go over everything AI had done again, which would undermine any benefits gained.

The group agreed that AI should support decisions, not make them. Human judgement is still essential. Often it is seen in practice as “another member of the team”. It is anticipated that in the future AI will become autonomous, so what is the librarian’s role in helping people use it safely?

Education Is the real safety feature

When we talked about reducing the risks one answer came up over and over again training – not more policies, not more restrictions. Training.

Participants highlighted the importance of helping staff to:

• understand how AI works
• recognise its limitations
• write effective prompts
• identify bias
• apply critical skills
• manage data appropriately

This is particularly important because digital confidence varies enormously from one person to another. The discussion reinforced something many library and knowledge specialists have long understood: Technology literacy is rarely about technology alone.

It is about practical skills, access, confidence and critical thinking.

Libraries have entered the governance conversation

One encouraging theme was growing recognition of the role that library and knowledge services can play.

Many of us noted that AI policies remain heavily focused on cybersecurity and data protection whilst giving relatively little attention to copyright, evidence quality, research integrity and information literacy. This feels like an important shift.

As AI becomes embedded within healthcare, the ability to assess information quality may become just as important as understanding the technology itself.

The risk we may not be discussing enough

One particularly thought-provoking conversation focused on workforce development.

Across a number of sectors, there were reports that AI is increasingly doing many of the tasks traditionally done by junior staff. Drafting, summarising, information gathering, and other foundational activities are often the very experiences through which expertise develops. It’s not about productivity, but loss of expertise.

If AI removes too many early-career learning opportunities, organisations may unintentionally create future capability gaps.

The challenge therefore becomes not whether AI can do the work, but how people continue learning alongside it.

Final thoughts

Governance structures remain immature. Policies continue to evolve. National guidance is still developing. Most organisations are learning through experimentation, collaboration, and shared experience.

Yet amid all the uncertainty, one message emerged consistently. The biggest AI challenge is not the technology. It is helping people use it safely, ethically, and intelligently. And that feels less like a technical problem and more like a human one.

To join the discussion please sign up to the AI Literacy List.  Full notes can be found on the Current and Emerging Tech Group.

This post was drafted in Microsoft Copilot from the meeting notes and chat 20/06/26.  The prompt was used to “draft a blog post in the style of Northern Lights”.  It was referenced against standard customised prompt quality check and reviewed and edited by the author and checked by John Gale.