Coding AI hallucinates packages | No more new CVEs?
#182 - It's endemic | MITRE's contract to manage CVEs will expire this week
Vibe coders of the world, beware!
AI coding assistants generate hallucinated packages 1 in 5 times
AIs hallucinate. That’s how they are built.
You can increase or decrease the probability of hallucination when you set the “temperature” setting of your AI.
When AI is used for coding, hallucination creeps in. New research has shown that coding AI’s hallucinate and create new packages / libraries at random. When you ask it to write code, it tends to add libraries or packages that do not exist!!
This paper [available here] shows the hallucination rate:
While GPT4 has the lowest hallucination rate, they are more than 5 percent - a non negligible rate of hallucinations.
Why is that a cybersecurity problem?
It would not have been a cybersecurity problem and we would have just forwarded more memes about vibe coders, but for this attack vector:
This is a six step process:
The Prep
The hacker prompts the LLM to generate code. The AI hallucinates and creates a package with a new name.
The hacker queries and finds that the package does not exist
The hacker then proceeds to create a malicious package with the hallucinated name and uploads it to the repository.
The package is now available to all coders.
The attack
A normal user prompts LLM to generate code which it does with the same hallucinated package name
Now, the vibe coder would run dependencies without checking and install the malicious package
Take Action:
Well, don’t vibe code. Thoughtworks, a leading software company in their “Technology Radar”, has asked users other ‘Hold’ what they call ‘Complacency with AI generated code’. In its essence, don’t vibe code!
Do you have an AI policy for coding? Remember: This attack vector affects systems before code reaches the testing phase
No more new CVEs in the database
If MITRE’s contract status does not change soon, no new CVEs will get added to the database. The sharks will start circling.
The MITRE corporation manages the CVE database under a contract with the US government. A contract that expires this week without surety of renewal. Read more here.
If you have relied on the CVE website for your vulnerability assessments, then you are in a for a bit of a shock.
This trusty website, where all professional penetration testers and red teamers get their vulnerability data from, will not be updated once the contract expires. The site may continue for a while, but eventually will be shut down. Only the CVE Github repo will have data of historical vulnerabilities.
A major program relied on my all security professionals might come to an end as we know it!
Why is this happening?
MITRE has a contract with CISA. There is no clarity on why CISA is letting this contract (any many other cybersecurity contracts) expire. We really do not know why this is happening.
Is there an alternative?
The US government run NVD (National Vulnerabilities Database) will still continue to run. This is currently synced with the CVE database.
Take Action:
If you are using commericial vulnerability scanners, new vulnerabilities might not get an updated if they are using the CVE database as a source. Check if the sources can be switched to the NVD database. Check if there are other sources that can be linked to your scanners.
If you are a boots-on-the-ground red team member or penetration tester, then your work has just increased. Check multiple database for vulnerabilities and exploits. Build a list of vendor sites and feeds to follow for vulnerability management.