I Reviewed 3 Vibe-Coded PRs With Hardcoded Keys — The Problem Isn't the AI, It's That I Approved Them

There's a technique magicians use called misdirection: while your eye follows the hand that's moving, the other hand does something you never see. Vibe-coding does exactly the same thing to code review. The PR lands clean — correct types, green tests, tidy commits. Your eye follows the business logic. The other hand — the one holding AWS_SECRET_ACCESS_KEY = "AKIA..." on line 47 — slides right past you.

It didn't happen to me once. It happened three times in the same sprint.

Vibe coding security code review: the problem nobody wants to admit

When I wrote about the vibe-coding vs stress-coding process, I focused on the flow, the speed, how your relationship with code changes when an agent writes the first draft. What I didn't say — because it was embarrassing — is that during that same week I approved PRs I absolutely should not have approved.

Three PRs. Same root cause. Different context each time.

PR #1: Stripe integration. The model generated the complete webhook handler — working, with signature validation, the whole thing. Beautiful. In the config file, right next to STRIPE_WEBHOOK_SECRET, there was a hardcoded AWS_ACCESS_KEY_ID that nobody asked for. The model dropped it in "for context" when I gave it access to an example file.

PR #2: A data migration script. I ran the tests, they passed. The key was in a comment. Literally: // aws_secret: AKIA... like a margin note nobody was ever going to read.

PR #3: The most ridiculous one. A key inside a string in a test. A test that was never going to run in CI because it was a mocked unit test. But there it was, sitting in the repository.

I approved all three. An automated tool found all three, three days later.

Why your brain fails differently with AI-generated code

Here's the empirical data point that actually matters to me: it's not that AI-generated code is worse. In a lot of cases it's better — better typed, more consistent, cleaner than what I'd write at 11pm. The problem is how my reading process changes.

When you review code a teammate wrote, your brain goes into detective mode. Why did they do it this way? What were they thinking? There's an implicit theory of mind operating. You're looking for intent.

When you review AI-generated code, your brain shifts into validation mode. Does it work? Do the tests pass? Is the structure right? It's subtle, but it's different. You're checking output, not understanding process. And hardcoded keys aren't a logic error — they don't show up in tests, they don't break the build, they don't trigger a type error. They're just data sitting somewhere it shouldn't be.

Put another way: the model doesn't know that AKIA4EXAMPLE123456789 is a secret. To it, that's a string like any other. And you, in validation mode, read it the same way.

The experiment I ran this week

After the third PR, I decided to measure this more systematically. I took 10 PRs from the last month — 5 written by humans, 5 generated with AI assistance (Claude, Cursor, some scattered Copilot). I re-reviewed all of them with an explicit security checklist.

Results:

The number that hit me isn't the 3 secrets. It's that I reviewed them 40% faster. That's not efficiency. That's me paying less attention.

What actually happens during an AI code review

I have a theory. When code is clean, well-structured, with descriptive names and clear comments, your brain processes it as "trustworthy." It's the same bias that makes people trust a scam message more if it has good grammar.

Vibe-coding produces code that looks reviewed. That is itself a security problem.

What I should do — what I'm doing from now on — is run an explicit checklist before merging any AI-generated PR:

# checklist-ai-pr.sh
# I run this before approving any AI-assisted PR

echo "=== SECRETS SCAN ==="

# Look for AWS key patterns
git diff main...HEAD | grep -iE '(AKIA|ASIA|AROA)[A-Z0-9]{16}'

# Look for generic key patterns
git diff main...HEAD | grep -iE '(secret|password|token|key)\s*=\s*["\x27][^"\x27]{8,}'

# Look for hardcoded IPs that aren't localhost
git diff main...HEAD | grep -E '([0-9]{1,3}\.){3}[0-9]{1,3}' | grep -v '127.0.0.1' | grep -v '0.0.0.0'

# Look for URLs with embedded credentials
git diff main...HEAD | grep -iE 'https?://[^:]+:[^@]+@'

echo "=== NEW FILES ==="
# New files are where secrets show up most
git diff main...HEAD --name-only --diff-filter=A

echo "=== SUSPICIOUS COMMENTS ==="
# PR #2 had the key in a comment
git diff main...HEAD | grep '^+.*//.*AKIA\|^+.*#.*secret\|^+.*//.*password'

Not magic. Just making explicit what should be implicit but isn't when your brain is in validation mode.

The gotchas nobody tells you about

Gotcha 1: Tests can pass with fake keys that are meant to be replaced with real ones. The model sometimes generates code with EXAMPLE_KEY_REPLACE_ME in the tests and the real key in the config file. Tests pass because you mock the client. The real key stays in the repo.

Gotcha 2: The model learns from your context. If you hand it a .env.example so it understands the structure, it can reproduce those example values in the generated code. And in a lot of projects, those "example" values are the actual keys from the dev environment.

Gotcha 3: Comments are a no man's land. Secret scanners generally don't scan comments as aggressively. The model uses comments to "explain" configuration, and sometimes drops the actual value right in there.

Gotcha 4: Test files are the blind spot. Most secret scanning configurations exclude test folders. The model seems to know this (or at least acts like it does) and sometimes generates fixtures or mocks with data that looks very real.

That last point connects to something I wrote about watermarks in AI-generated code — the idea that model output has characteristics we can detect if we know what to look for. The problem is we're very focused on detecting "did an AI write this" and barely focused at all on detecting "what's actually inside it."

The process I'm adopting

After this experiment I changed three concrete things:

1. Pre-commit hooks on all new repos

# .husky/pre-commit
# Install: npm install --save-dev @secretlint/secretlint
npx secretlint "**/*"

// .secretlintrc.json
{
  "rules": [
    {
      "id": "@secretlint/secretlint-rule-preset-recommend"
    },
    {
      "id": "@secretlint/secretlint-rule-aws"
    }
  ]
}

2. I mentally separated "does it work?" from "is it secure?"

These are two different reviews. The first one I can do fast. The second one I do slow, with the script above, in a different state of attention. I don't mix them.

3. I added an explicit question to the PR template

## Security checklist
- [ ] No hardcoded secrets, tokens, or API keys
- [ ] Environment variables are in .env (never committed)
- [ ] If I used AI assistance: ran secretlint before opening the PR

It's blunt. It's obvious. It works because it makes explicit something the brain skips over in autopilot mode.

This also changes how I think about agents that research before coding — if the agent has access to your context so it can research better, it also has more surface area to accidentally leak secrets.

FAQ: vibe coding security code review

Is vibe coding inherently insecure? Not inherently, but it creates conditions that increase risk. AI-generated code can be technically correct and have serious security problems at the same time. The risk isn't in the quality of the code — it's in how your review process changes when you read it.

Should AI models detect and reject requests that include secrets? Some do, partially. But it's not their primary responsibility. Claude, for example, isn't going to commit a key for you — but if you pass it context that includes a key, it can reproduce that key in the output without "knowing" it's sensitive. Secret management is your responsibility.

What automatic tools do you recommend for detecting secrets in PRs? For GitHub repos: GitHub Secret Scanning (free for public repos, included in GitHub Advanced Security for private ones). For CI/CD: truffleHog, gitleaks, or secretlint. For pre-commit: the husky + secretlint combo I showed above. What matters is having at least one automatic layer that doesn't depend on your manual attention.

What if a key was already committed and pushed? First: rotate the key immediately, before you do anything else. Second: remove it from history with git filter-branch or BFG Repo Cleaner. Third: assume it was exposed even if the repo is private — the bots scraping GitHub are fast. There's no "it was only up for a moment." This connects to cryptography and the expiration date of secrets: a compromised key is a dead key.

How do I know if a PR was AI-generated or not? In many cases you won't — and that's exactly the point. A secure review process has to be the same regardless of whether a human or a model wrote it. Assuming the code is AI-generated when you're not sure puts you in the right state of attention.

Does the problem get better with newer models? Partially. Newer models are better at not reproducing obvious secrets and at suggesting environment variables instead. But if you give them context that includes sensitive data, they'll use it. The underlying problem isn't model capability — it's that we let our guard down when the output is clean and the tests pass. A better model doesn't fix that.

The problem is me, and that's actually good news

Saying "the problem is the AI" would be comfortable and completely useless. If the problem were the model, the solution would be to swap the model or stop using it. But the problem is my review process, and that I can actually change.

What I realized across those three PRs is that vibe-coding doesn't just change how code gets written — it changes how it gets read. And if you don't update your review process to compensate for that shift, you're running with your defenses down precisely when code is being generated the fastest.

The good news is the tools already exist. secretlint, truffleHog, GitHub Secret Scanning — none of them are new, none of them are expensive, none of them are hard to configure. My problem was that I wasn't using them consistently because my "manual" review process felt like enough. With AI-generated code, it isn't.

If you're using Cursor, Claude, Copilot, or any AI assistance tool to generate code in a repo that has real consequences — set up secretlint today. Before you close this tab. It's five minutes.

And the next time you see a PR with green tests and clean code, remember: that's exactly when you need to slow down, not speed up.

Has something like this happened to you? Do you have a different process for reviewing AI-generated PRs? I genuinely want to know — especially if you're doing something I'm not.