Does AIsbom catch it?

Model scanners publish what they catch against malicious pickles. We wanted to know what ours misses. That's an awkward thing to publish, and it's also a number worth having — because the techniques are all public, written up in CVEs and conference talks that any attacker can read.

So we built the corpus and published the whole thing, including the parts that don't flatter us. Eleven documented pickle-scanner evasions, reproduced as inert artifacts, scanned with AIsbom's own engine. Here is what happens.

Two of those eleven are ⚠️ partial, which is the verdict we found hardest to write and care about most. It means AIsbom refused to call the file safe but never disassembled the payload — so you are correctly warned off the file, for the wrong reason. A scanner that counted those as wins would be lying by rounding.

You don't have to take our word for any of it:

pip install aisbom-cli py7zr
aisbom bypass-scorecard

That command regenerates every artifact from source, scans them in both modes, and prints the table below on your machine. The corpus ships inside the package — not in a private test directory — precisely so that "run it yourself" is a real answer and not an invitation to trust us. py7zr is needed only to build the 7z case; the scanner itself never unpacks 7z, which is exactly why that case scores a partial below.

The pickle scanner bypass scorecard

Blocklist is the default mode: flag known-dangerous globals. Strict is --strict, an allowlist where anything unrecognized is flagged. A technique counts as caught only if a scan actually surfaces it.

Two controls sit alongside these and are just as load-bearing. A plain os.system in an untampered PyTorch ZIP must always be caught — if that one ever fails, the harness is broken and the rest of the table is noise. And an ordinary model with only allowlisted globals must scan clean, because a scanner that flags everything is as useless as one that flags nothing.

The two pickle evasions we don't fully catch

These are the honest part of the table. Neither is an oversight we haven't got to; both are decisions, and we'd rather defend them in public than quietly count them as detections.

7z containers: named, not unpacked

nullifAI repacked a model with 7z instead of ZIP. PyTorch still loaded it; picklescan never opened it.

AIsbom reports CRITICAL (Non-Standard Container: 7z) — the right severity, earned from the container rather than the payload. The archive is named but never unpacked, so the os.system call inside is never disassembled and the reason we give you is not the real one.

ShadowPickle: the ceiling on static allowlists

ShadowPickle overwrites collections.OrderedDict — a name on essentially every scanner's allowlist — and resolves it via STACK_GLOBAL rather than an inline argument, so string-matching on the opcode argument sees nothing.

AIsbom does resolve STACK_GLOBAL and reads the pair off the stack, so it sees collections.OrderedDict. And that name is legitimately allowlisted, because real state_dicts are OrderedDicts. Both modes therefore return only MEDIUM (Pickle Present) — the baseline every pickle gets — rather than anything specific to this file.

The evasion we missed: CVE-2025-1889

Until this week the table had a third entry in the not-caught column, and it was worse than either partial: an outright ❌ missed.

CVE-2025-1889 is almost insultingly simple. Name the payload config.p. Discovery decided what to open from a file's suffix alone, .p wasn't on the list, and so the file was never opened. The scan reported No AI models found and exited 0.

A clean bill of health on a directory carrying a reverse shell. That is the worst failure mode a scanner has, because the user is not merely unprotected — they've been told they're fine.

The obvious fix is to add .p to the extension list. We didn't, because it protects nobody: the technique is any unexpected suffix, and the attacker picks the name. Adding .p would have flipped this row on our own published scorecard while leaving weights.dat wide open. That is exactly the kind of laundering the corpus exists to prevent.

So v1.3.2 identifies unclaimed files by content instead. Any file no extension claims gets its bytes checked, and if they're a pickle, it gets scanned.

What that cost us to get right

The first version of this check was wrong in a way worth publishing, because it's the trap anyone building the same feature will hit.

The natural rule is "disassemble the head; if it parses as pickle opcodes and reaches a STOP, it's a pickle." That rule is badly broken. . is the STOP opcode — so every stylesheet that opens with a class selector is a valid one-opcode pickle. Run against a real node_modules of 5,123 files, that rule claimed eleven of them: seven JavaScript files, a TypeScript declaration, a stylesheet and two man pages.

False positives in a security tool aren't cosmetic. Every phantom component lands in your AI Bill of Materials and trains you to skim past findings, which is how a real one gets missed. So the check now validates the pickle stack — a stream that pops from an empty stack or ends holding anything other than one object is rejected — and those eleven filenames are regression tests.

One more detail, because it's where the two evasion classes meet: validation runs on the prefix ending at the first STOP, not the whole buffer. A payload followed by a corrupt tail — the nullifAI shape — is still caught rather than thrown out along with its own garbage. Truncating a stream is otherwise a way to opt out of being scanned.

And the limit we're still carrying. Discovery reads a 64KB head and only re-reads with a bigger budget when that head looks like an unfinished pickle. Past 16MB it stops looking, so a payload behind a literal larger than that isn't discovered by content. Verified rather than asserted: caught at 16,773,120 bytes, not discovered at 16,781,312.

Why the misses can't quietly disappear

A scorecard you regenerate yourself is worth nothing if "the scorecard is failing" can be fixed by regenerating the scorecard. So there are two files, doing two different jobs.

baseline.json describes current detection and fails the build on drift in either direction. floor.json is a promise: the best verdict every case has ever reached. It only ever ratchets upward. Lowering it requires a hand edit to a committed file — which is the audit trail.

The gate is per-case and per-mode, not an aggregate. That distinction is not theoretical: when we deliberately broke detection to test the ratchet, the headline number didn't move at all, because strict mode still caught what blocklist mode had stopped catching. A count-based gate would have waved it through. The per-case gate named all three regressions and exited non-zero.

When CVE-2025-1889 flipped to detected this week, the floor rose with it. It cannot silently regress now without failing the build on every release.

Sources: the picklescan CVEs behind each case

Every case cites the research it reproduces. We verified each of these against the primary source before building the corpus rather than working from memory, because this page publishes them:

• ReversingLabs — nullifAI: 7z containers and deliberately broken streams.
• Sonatype — CVE-2025-1716, CVE-2025-1889, CVE-2025-1944, CVE-2025-1945.
• JFrog — three picklescan zero-days: CVE-2025-10155, CVE-2025-10156, CVE-2025-10157.
• Checkmarx — Free Hugs (Part 4): the Bdb.run gadget.
• ShadowPickle: allowlist abuse via STACK_GLOBAL.

These techniques were found and disclosed by other people's research, and several were fixed in picklescan before we ever wrote a corpus. This scorecard measures our tool against their work.

Run it against your own models

The corpus is synthetic and inert — every artifact is generated on your machine, and nothing in it is ever executed, only disassembled. The same engine scans real files:

# the published scorecard, on your machine (py7zr builds the 7z case)
pip install aisbom-cli py7zr
aisbom bypass-scorecard

# your models
aisbom scan ./models --strict

The full generated scorecard, including per-case detail, lives in docs/bypass-scorecard.md. If you find a technique we've missed — open an issue. A case we can't catch yet is worth more to us on this page than off it.

And the honest meta-point: every technique on this page exists because pickle is a programming language pretending to be a data format. Scanning it well is a rearguard action. If you control the format your models ship in, safetensors removes this entire class of problem — no opcodes, no gadgets, nothing to disassemble. The scorecard above is for everyone who doesn't get that choice.