What is an AI code hallucination?

It is generated code that references something with no real referent — a function the library never exported, an npm package that was never published, a config key nothing reads, or an API response field the backend never sends. It differs from a logic bug: a logic bug does something real incorrectly, while a hallucination calls something that does not exist at all.

Why don't code reviews catch AI code hallucinations?

Review checks plausibility, not existence. Confirming a symbol is real means leaving the diff and opening the installed package or the docs for every unfamiliar name. Type checking helps only where strong types exist, so hallucinations survive in untyped seams like JSON.parse results and untyped API clients. And a green build proves the code compiles, not that the symbol resolves at runtime.

What is slopsquatting?

Slopsquatting is when an attacker publishes a real package under a name that AI models repeatedly hallucinate, so generated code installs the attacker's code. It differs from typosquatting, which relies on a human mistyping a real package name. The defence is to verify every AI-suggested dependency against the registry before installing, pin versions, and commit your lockfile.

How do I test for hallucinations in AI-generated code?

Verify new imports against the package registry, install from scratch in a clean environment rather than a warm cache, grep for each unfamiliar symbol inside the installed package source, execute every branch against the real integration instead of a mock, and assert on the behaviour any new config key is supposed to change. Executing the path catches the most for the least effort.

Which AI code hallucinations are hardest to find?

Phantom config keys and phantom API contracts, because neither throws an error. A setting nothing reads or a response field that is always undefined produces a working-looking app with a silent hole in it. These are the hallucinations most likely to be found by a client rather than by the team that shipped them.

AI Code Hallucinations: How to Catch Them Pre-Launch

AI code hallucinations are passages of generated code that reference things which do not exist — a function that was never in the library, a config key the framework never read, an npm package nobody published. They are dangerous precisely because they look correct: the naming is plausible, the syntax is valid, and the file often compiles. The only reliable way to catch AI code hallucinations is to execute the path that uses the invented thing, because a hallucination is an empirical claim about the outside world, and reading the code can never falsify it. Everything below is about building that execution step into your workflow before a client or user hits it.

What are AI code hallucinations?

A hallucination is not a logic bug. A logic bug is code that does something real, incorrectly. A hallucination is code that calls something which has no referent at all. The model has produced a statistically plausible token sequence — the method name looks like the method names in that library's docs — but the symbol was never implemented.

That distinction matters for QA, because the two failure modes need different detection. Logic bugs are found by asserting on output. Hallucinations are found simply by reaching the line. If the line never runs in your tests, in your click-through, or in the one happy path you demoed, the hallucination ships intact and fails the first time a real user takes that branch.

What do AI code hallucinations actually look like in a codebase?

They cluster into four recognisable kinds. Knowing which kind you are looking at tells you which check will catch it.

Kind	What the model invented	Where it fails	What catches it
Phantom API	A method or option that the real library does not export	Runtime, on first call	Executing that code path
Phantom package	An import for a package that was never published	Install or build	A clean install from a fresh lockfile
Phantom config	A key in a config, env, or schema file that nothing reads	Silently — the setting simply has no effect	Asserting on the behaviour the setting is supposed to change
Phantom contract	A response field, column, or event name the backend never sends	Runtime, as an undefined value flowing downstream	An end-to-end run against the real API

The phantom config and phantom contract cases are the expensive ones. Neither throws. A rate limit that is never applied, an audit flag that is never written, a user.organisation_id that is always undefined — these produce a working-looking app with a hole in it, and they are the hallucinations most likely to be found by a client rather than by you.

Why don't AI code hallucinations show up in code review?

Because review is reading, and a hallucination is only detectable by checking the code against reality. Three specific gaps let them through:

Review checks plausibility, not existence. A reviewer scanning a 400-line AI-generated diff is asking "does this look like how we do things?" — not "does this method exist in version 4.2 of this library?" The second question requires leaving the diff and opening the docs, for every unfamiliar symbol.
Type checking is only as strong as the types it has. A phantom method on a well-typed library is caught at compile time. A phantom field on a response typed any, parsed from JSON.parse, or coming from an untyped API client is not. Hallucinations survive in exactly the untyped seams where real integrations live.
A green build proves compilation, not execution. Compilation proves the code is well-formed. It does not prove the symbol resolves at runtime, that the package installs from a clean cache, or that the remote field is populated. "It built" is the single most common reason a hallucination reaches production.

This is the same structural blind spot described in why vibe coding creates bugs your test suite will never catch: AI raises the volume of code faster than it raises the volume of verification, and the gap is where hallucinations live.

How do you catch AI code hallucinations before launch?

Five checks, in the order that catches the most for the least effort:

Verify every new import against the registry before anything else. For each dependency the AI added, confirm the package exists, check its publish date and download history, and read who maintains it. A package created last week with no history is a red flag, not a new discovery.
Install from scratch, not from your warm cache. Delete node_modules, install from the lockfile in a clean environment, and build. A phantom package that resolved from a stale local cache will fail here — which is where you want it to fail, not in your client's CI.
Grep for every unfamiliar symbol in the dependency's own source. If the AI called client.batchUpsert(), find batchUpsert in the installed package, not in the model's explanation of it. Your node_modules directory is the authority on what the library exports; the chat window is not.
Run the path, don't read it. Every branch the AI wrote needs to be executed at least once against the real integration — not mocked. Mocks are the one place a phantom contract is guaranteed to pass, because you wrote the mock from the same hallucinated shape.
Assert on effect for anything configuration-shaped. For each new config key, flag, or policy, write one check that the intended behaviour actually changed. If you cannot write that assertion, you cannot tell a working setting from a key that nothing reads.

If you only have time for one of these, make it number four. Execution is the single check that catches all four kinds.

Which AI code hallucinations are a security problem?

Phantom packages, specifically. When a model repeatedly invents the same plausible package name, that name becomes a predictable target: an attacker can publish a real package under it and wait for generated code to install it. The practice has an accepted name — slopsquatting — distinguishing it from classic typosquatting, where the attacker relies on a human mistyping a real name. Here the attacker relies on a model inventing a name that was never real.

The defensive measure is unglamorous and effective: never install a dependency you have not looked up, pin versions, commit your lockfile, and treat any AI-suggested import as unverified until you have seen the package's registry page yourself. More on the adjacent failure modes in vibe coding security risks.

How does Klavity help catch AI code hallucinations?

Klavity is built around the one thing reading cannot do: running the app the way a user would. Sims drives your real deployed site with AI personas that take the branches your demo never touched — which is precisely how phantom APIs and phantom contracts surface, as a thrown error or an undefined rendered on screen. AutoSim turns those paths into self-healing end-to-end tests, so a hallucination caught once stays caught as the AI rewrites the file around it. And when a bug does reach a client, Snap captures it as a right-click report with the console error and network trace attached, so a phantom field shows up as evidence rather than as "it's broken on my end."

None of this replaces the registry check or the clean install — those take two minutes and should be habit. It replaces the assumption that a green build means the code refers to real things. For the wider picture of where automated checks fit alongside human review, see the complete guide to AI QA, and for the day-to-day workflow, testing AI-generated code.

The short version

AI code hallucinations are a verification problem, not a model problem. They will keep appearing for as long as generation is probabilistic, so the fix is not better prompting — it is a step in your process where generated code meets reality before a user does. Check the registry, install clean, grep the package, and above all run the path.

Scan your next client site free — Klavity runs your real app the way a user would and reports what breaks, before the hallucinations do.