In late August, Ars Technica reported on research showing that AI coding assistants would run shell commands they found written inside a project's llms.txt — the plain-text file a growing number of sites publish to give AI agents a short summary of what they do. Across a range of well-known companies, some of those files pointed at software packages or domains that had never actually been registered. When the researchers claimed a few of those unregistered names for themselves, machines inside large organizations connected back to them within hours: coding agents that had read a vendor's llms.txt and executed the install line it contained.
The reference isn't really the problem
It's tempting to read this as "some links were dead." The sharper issue is that these files can carry instructions an agent will act on — a pip install, an npx, a pipe-to-shell one-liner — and an agent reading one has no reliable way to tell whether the author meant it as documentation or as something to run. A package name that resolves to something harmless today can be abandoned and re-registered by someone else next quarter. The file doesn't have to be malicious when it's written to become a liability later.
What Entropy checks
Entropy's scanner already reads a set of public, well-known files for every vendor it grades. llms.txt and llms-full.txt are now part of that weekly pass. For each one it finds, it:
- resolves every domain the file references, and flags any that don't exist;
- checks every package name against the npm and PyPI registries, and flags any that aren't published;
- flags the file if it contains any install-or-execute command pattern at all —
curl … | sh,pip install,npm i -g,npx, and so on — whether or not the reference currently resolves.
Because the check re-runs every week, a reference that's fine today but taken over in six months still gets caught.
For now this is recorded as a flag for review, not folded into the vendor's grade. Whether an llms.txt problem should ever move a score is a real question, and we'd rather answer it with data from real vendor domains than bake it into the rubric by default.
The other half of the file
The same files often name the AI models a product is built on. Entropy surfaces those mentions on the vendor's profile as a dated, self-disclosed, unconfirmed note — shown with the exact text we matched, not a link to a file that changes — and a vendor who has claimed their profile can confirm or correct the list.
We run it on ourselves
Entropy publishes its own llms.txt and llms-full.txt. They're descriptive only: no install commands, no package names, every link pointing back to entropy.com. A test in our build runs the same parser against those files on every change and fails the build if an install command, a package reference, or an off-site link ever appears in one. As a scoring service rather than a model-backed product, we have nothing to list in the AI-models section — and llms-full.txt says as much.
For how the rest of a vendor's grade is put together, see the methodology page.