It's AI risks this week
CyberInsights #121 - From prompt injection to simple AI API exposure, it is all about risks to Artificial Intelligence and Machine Learning
AI, ML and the story of their security
Cutting edge technology leads to cutting edge risks. And some banal risks too.
What happens if you ask ChatGPT to repeat “Company” forever?
It repeats “Company” for a while. Then it spews out training data.
See the actual prompts here. [LINK]
It’s called ‘extractable memorisation’. It’s a technique where training data can be extracted from the model by simple queries, without knowing anything about the model.
Read this excellent paper about it [LINK]
“Models emit more training data as they get larger” says the paper.
If you want the TL;DR on the paper, go to section 5.3. Here is the summary:
The researchers were able to extract 10,000 unique training examples using about US$ 200 of queries
Long and divorced memorised output was produced
The data extracted, depending on the query, contained personally identifiable information (PII), NSFW (Not Safe for Work) data, literature, urls, UUIDs and even code
Exposed API Tokens offer full access to Large Language Models (LLMs) run by big tech
The next level of supply chain attacks is when your AI chatbot tells you what the hacker wants you to hear.
Huggingface.co is, loosely put, the GitHub for open source AI models. All of big tech have a model or two hosted there.
When the API tokens of Hugging Face are exposed, the LLMs hosted there are vulnerable. [LINK]
This means the API tokens for Meta, Microsoft, Google, VMware and more. The tokens also allowed write access, which means it could lead to AI poisoning.
In their defence, the Hugging Face team have said this:
Take Action:
AI attacks can come from prompts that spit out training data to API tokens for models being exposed.
It’s time for cybersecurity professionals to take a slightly more structured approach to managing AI risks. I have spoken about the NIST AI Risk Management Framework often. There is also the OWASP ML top 10 that you can refer to:
Read the two frameworks and build your AI risk competence.
For the cyber insurers, however, this does have an additional risk to consider. Make sure that the assessment prior to providing cyber insurance handles AI risks.