Cybersecurity – a threat and opportunity

If you read the news, you hear a lot about AI going wrong or being hacked. There is justified concern if you think about the British Library and concerns about US data sets. So, what problems are we facing just now?

The National Cyber Security Centre (NCSC) identifies seven classes of adversarial machine learning (AML) attacks that target the model itself rather than the surrounding IT systems.

1. Model characterisation

Attackers probe an AI system to learn how it works, where its weaknesses lie, and what kinds of inputs trigger particular outputs.

Healthcare example: Repeatedly querying an AI triage chatbot to discover which symptoms trigger urgent referrals and which do not.

2. Model inversion

Attackers use a model's outputs to infer information about its training data. In some cases, sensitive information may be partially reconstructed.

Healthcare example: This is a risk in medical AI systems for facial recognition (Fang et al, 2024) to be extracted and recreated.

3. Training data poisoning

The training data is deliberately altered so that the model learns incorrect patterns.

Healthcare example: Researchers demonstrated that medical large language models (LLMs) can be manipulated through training-data poisoning. In a 2025 Nature Medicine study, altering just 0.001% of training tokens with false medical information caused models to generate harmful clinical misinformation while still appearing to perform normally on standard benchmarks. This illustrates how small changes to training data can introduce unsafe behaviour that may not be detected through routine testing.

4. Malicious model training

Hidden behaviours or "backdoors" are embedded during model development while normal performance is maintained.

Healthcare example: Only takes 250 malicious documents to poison most large language models (Souly, A. et al, 2025).

5. Model input manipulation

Inputs are altered to mislead the model without changing the model itself.

Healthcare example: Subtle modifications to an X-ray image that cause an AI system to miss evidence of disease.

6. Model artefact manipulation

Attackers tamper with model files, weights, or other components used during deployment.

Healthcare example: A hospital downloads a compromised version of an open-source model, leading to altered outputs despite apparently normal operation. This attack is often targeted at supply chains (GenAI Security Project, 2025)

7. Model hardware attacks

The underlying hardware used to train or run AI systems is targeted.

Healthcare example: Interfering with AI infrastructure supporting imaging or genomics services, degrading performance or exposing data e.g. NVLink (Zhang et al, 2025).

What that means to libraries

Our discovery systems can be manipulated to give biased or misleading results. Chatbots can be misled. Data protection is required to guard against inversion attacks.

Most likely, libraries will obtain AI capability through suppliers rather than build systems themselves. This means the security, data provenance, and governance practices of vendors become important considerations. Model artefact manipulation and malicious model training demonstrate why assurance matters.

We are lower risk than other areas, but still a potential backdoor into other systems. So we need to understand the risks and how to mitigate them.

The opportunity for libraries

Libraries have always helped users evaluate information critically. The rise of AI expands that role.

In the past, information literacy focused on teaching people how to evaluate websites, books, and journals. Increasingly, libraries may need to teach AI literacy:

• how AI systems generate answers
• why AI can be wrong
• how outputs can be manipulated
• when verification is needed
• how to use AI safely and responsibly

In other words, libraries are not just adopters of AI. They are likely to become one of the key defences against its misuse.

This post was drafted in Microsoft Copilot from the meeting notes and chat 20/06/26, with reference to the NCSC and AI readiness toolkit.  The prompt was used to “draft a blog post in the style of Northern Lights”. It was tailored to healthcare and libraries. Standard customised prompt was used as a quality check and reviewed and edited by the author and reviewed by John Gale.