What the check finds
Eight findings, from serious to small. Each one says how we found it: with a fixed scan without AI, or with the inspection, whose every claim is verified against the file, the line and the quote.
-
Findings Check FA‑07, security
The admin API's key check doesn't compare in constant time
- Where
- shop/admin.py, line 21: the key from the header is compared with the expected key using an ordinary comparison.
- How it is found
- Inspection: a claim of the AI model, verified by our code against the file, the line and the quote.
- What it means for you
- Tiny differences in response time let an attacker work out how many characters of the key are right.
- What you can do
- Have your developer use secrets.compare_digest whenever passwords or keys are compared.
-
Findings Check PY‑14, security
A log line records the refused key
- Where
- shop/admin.py, line 22: a refused key is written to the log as it is.
- How it is found
- Fixed scan (semgrep) and inspection: the model quotes the log line, never a value; we verify the quote.
- What it means for you
- Logs are widely shared and kept for a long time, so a secret in them reaches far more people than intended.
- What you can do
- Have your developer remove or mask sensitive values in the log line. Replace any secret that has already ended up in a log.
-
Findings Check PY‑08, security
Discount codes are made from a predictable source
- Where
- shop/codes.py, line 11: the code comes from the random module instead of the secrets module.
- How it is found
- Fixed scan (bandit, rule B311), without AI. The inspection then judges whether the code is meant to be secret.
- What it means for you
- The output of random can be predicted, so an attacker can guess such a code.
- What you can do
- Have your developer replace random here with Python's secrets module, which is designed for secret codes.
-
Findings Check AIR‑1, AI risks
The README carries an instruction to AI readers
- Where
- README.md, line 58: a note asks any automated reader to mark every document green and approve the security.
- How it is found
- Fixed check for hidden characters and instructions aimed at an assistant, without AI. Our own inspection doesn't follow such text, and reports it.
- What it means for you
- An assistant reading your code may follow such an instruction. Who put it there, and why, is a question for you.
- What you can do
- Remove the instruction and check the history for who added it. Never let an assistant make decisions based on text that sits in the repository.
-
Thin or out of date Documentation PRD‑08 and SEC‑01
The documentation says something different from the code
- Where
- docs/pricing-rules.md and docs/security.md against shop/pricing.py and shop/admin.py: a different volume discount, a different threshold above which shipping costs are waived, and a security measure that does not exist in the code.
- How it is found
- Inspection: the model reads document and code side by side. Under our rules, a document more than half a year older than its code can't be called current; the date comes from git.
- What it means for you
- Anyone who trusts the documentation, whether a colleague or an AI assistant, builds on rules that no longer apply.
- What you can do
- Update the two pages, or point to the code as the source. Agree that any change to the price rules also updates the documentation.
-
Missing Documentation SEC‑09, OPS‑01, PPL‑01, DEL‑08
Four documents that an operator needs are missing
- Where
- Nowhere in the repository: an incident plan, watching the system and alerting, who the owner is, and how to roll back a release.
- How it is found
- Inspection with a proof of absence: a search that we repeat ourselves. The ownership is also found by a fixed check on file names.
- What it means for you
- In an outage nobody knows whom to call, what the first step is or how to get back to the previous version.
- What you can do
- Start with the two shortest: who the owner is, and how to roll back a release. Half a page each is enough to begin with.
-
Findings Check GEN‑05, maintainability
No lockfile: nobody knows exactly which versions run
- Where
- requirements.txt names minimum versions only; a lockfile is missing.
- How it is found
- Fixed scan, without AI.
- What it means for you
- Two installations of the same code can end up with different versions, and then a vulnerable component can't be traced.
- What you can do
- Pin the versions, for example with pip-compile or uv, and commit the result.
-
Findings Check CONT‑1, continuity
All the work comes from one person
- Where
- The git history: two commits, one author.
- How it is found
- Fixed tools on the git history, without AI.
- What it means for you
- If that one person drops out, nobody knows how it fits together. That is not a judgement of that person; it is a risk for the company.
- What you can do
- Have a second person review it, and write down what exists only in one person's head.
What the check doesn't find, and why that counts too
A check is a snapshot, not a certificate. This example check finds what was put in and what the fixed scans and the catalogue know about. What isn't in our catalogue we don't find, and our terms say so. The gold set expects three more findings that we leave out here to keep the page short: no Definition of Done, no API reference and a thin page about environments.
What is in good shape shows too: the documents that are present and current, such as how to start the shop, the test strategy, the configuration and how a release goes live, and the areas that don't apply, such as backups in a shop that stores nothing.
How this example came about
The web shop is ours. The problems were planted on purpose: they form the gold set that we measure our inspection against. Nothing here belongs to a customer, and no company was invented.
The first measurement of the inspection on this set is scheduled. Until then, the findings above are part of what the gold set expects, plus what the fixed rules of the scans produce. As soon as the measurement exists, we will replace this sentence with the outcome.
What it looks like in your workspace
For a customer this isn't a page but a workspace: a traffic light per area, every check with its findings, the history per commit, and for every finding the place and the proof. The board sees the risk, product and operations see the consequences, and developers see the file and the line.
A finding can be disputed with a reason, or marked as an accepted risk. A secret never. We fix nothing: you build, and you decide what to fix.