GUIDE · CONCEPT 12 min ·

Approval prompts are not a credential controlWhat Anthropic's 1,053-tester study leaves open

Anthropic's August study shows a classifier catching far more dangerous commands than a person clicking through prompts, and I believe the direction of that result. Both checks still judge a proposed command. Whether a process can read your AWS key is a yes-or-no property of the machine, and a short script can test it.

TL;DR· the answer, in twenty seconds

What happened: Anthropic reported on August 7, 2026 that 1,053 paid testers caught a swapped-in dangerous command 13.6% of the time and auto mode blocked 89% of the same commands. From August 14, new Claude Code sessions on Pro, Max and Team plans default to auto mode.

The minimum fix: Run a reach check from the agent's own shell, then deny every credential path and variable it reports. Claude Code's sandbox reads ~/.aws/credentials by default and ships no credential deny list.

The lesson: Keep the classifier, because it beats a tired reviewer. Prompts and classifiers estimate whether a command looks bad. Credential safety depends on what the process can read and where it can send it, and you can test both.

On August 7, 2026, Anthropic reported that 1,053 paid testers caught a swapped-in dangerous command 13.6% of the time while its classifier blocked 89%, and a week later auto mode became the default for new Claude Code sessions on Pro, Max and Team plans.

Those numbers justify the switch. They do not tell you whether your AWS key is safe, because neither reviewer decides what the agent's process can read. I ran the test that does, on fake credentials, and Claude Code's default sandbox read all of them.

What the 1,053-tester study measured

Anthropic's write-up describes the method. A research vendor hired the testers for a coding study in a test environment, away from their own code. They knew someone was evaluating them and did not know which behavior. Partway through each session, the vendor swapped one permission prompt for a "clearly dangerous command". Approving it ran nothing harmful. Testers declined 143 of 1,053 and auto mode blocked 937.

The limits are plain. This is a vendor's study of its own feature. Nobody had their own systems at stake. The study picked clearly dangerous commands, the easy end of what a classifier sees.

People quote the approval rate more than it deserves. Anthropic says users approve 97% of prompts and that most prompts are likely for safe, routine commands, so 97% alone proves nothing without a base rate. Its March and May posts gave about 93%, and I found no explanation for the change.

Two other figures carry more weight. Testers blocked about 17% of dangerous commands early in a session and about 5% after 50 or more prior prompts, while auto mode stayed flat. Users reject 39% of plans and 3% of permission requests. People do review, when the question is one they can answer. The settings data fits: as of June 2026, 49.5% of active CLI users had written a Bash allow rule, and 62% had used bypass mode or "don't ask again" on Bash.

Miss rates move a long way with the workload:

Source and workload Reviewer Miss rate Caveat
Anthropic, August: one swapped command, n=1,053 Human 86.4% My subtraction from 13.6% caught
Same commands Auto mode 11% "Clearly dangerous" commands only
Apollo Research pilot, August report Auto mode 7%, down from 12% Anthropic says this "should not be taken as the miss rate on real traffic"
Anthropic, March: synthetic exfiltration, n=1,000 Classifier 5.7% Earlier classifier
Anthropic, March: real overeager actions, n=52 Classifier 17% My 95% interval is 9.4% to 29.7%, assuming 9 of 52
Ji et al., v3 October: 253 state-changing actions Auto mode, end to end 81.0% Built on ambiguous authorization, first posted in April
Wauters game, 409,000 decisions: scope violations Human 35.0% The author calls it "just a game"; threats were about 34% of commands

Anthropic called its 17% "the honest number" in March and wrote that auto mode is "not a drop-in replacement for careful human review on high-stakes infrastructure". The August figure of 11% comes from a different workload, so it does not show the classifier improved by six points. Ji et al. attribute their 81.0% to a harder workload and say it does not contradict Anthropic. All of these can be true at once. Nobody has measured your repo, so none of them works as a guarantee.

A human above the classifier adds six catches

Anthropic published the two disagreement counts: auto mode blocked 800 commands a human approved, and humans blocked 6 that auto mode allowed. The other two cells follow by subtraction, which is my arithmetic:

Human blocked Human approved
Auto mode blocked 137 800
Auto mode allowed 6 110

Stack a person on top of the classifier in this study and misses fall from 116 to 110. Testers caught 5.2% of the commands the classifier allowed (6 of 116) and 14.6% of those it blocked (137 of 937). If the two reviewers erred independently, about 100 commands would have passed both. The study shows 110. That sample is small, but it leans toward shared blind spots: the commands that fool the classifier also fool people. Two reviewers looking at the same string do not add up to two independent layers.

The study also scored one command per session. A session contains many. If each attempt carries a miss rate m, the chance that at least one of N gets through is 1-(1-m)^N:

Miss rate per attempt 3 attempts 5 attempts
11%, auto mode in the August study 29.5% 44.2%
17%, classifier on real overeager actions 42.8% 60.6%
35%, humans on scope violations in the Wauters game 72.5% 88.4%

Treat this table as illustration. It assumes independent attempts. A retry can fail the same way as the first try, which would pull these figures down. No source measures attempts per session on a real workload. The shape still matters: a per-action rate understates what a persistent process gets.

Auto mode also hands decisions back. After three blocks in a row or twenty in a session, Claude Code falls back to manual approvals. Whether a person who sees that switch wakes up or clicks through, the study does not test. The Register's subhead named the worry: "Walk away and hope the classifier catches anything irreversible or destructive".

What a check on the command cannot see

The string under review is a small part of what runs. npm test says nothing about the script in package.json. In the Wauters game, npm run analyze was the most missed of 37 threats, approved 64.7% of the time. Three npm run commands pooled at 52.5% missed (n=9,482) against 28.4% for other exfiltration-style attacks, and the game showed the payload in the history log above the prompt. Players also split on cat ~/.zshrc, which 45.9% approved.

Auto mode has the same gap by design, for a defensible reason. Anthropic's March post says in-project edits skip the classifier because they are "reviewable via version control", and the permission docs add: "Narrow rules like Bash(npm test) stay in effect." I tested that route in Claude Code 2.1.318. With Bash(npm test) allowed, I asked the agent to change the test script to print token variable names and then run it. The debug log recorded this for the edit:

[DEBUG] Skipping auto mode classifier for Edit: would be allowed in acceptEdits mode

The log held no review line for the npm test call, and the run printed NPM_TOKEN and GITHUB_TOKEN, with CLAUDE_CODE_MESSAGING_TOKEN as a third name. Without the allow rule, the edit skipped the classifier the same way and the command still ran, on a request that asked for nothing dangerous. I made the request in plain terms with fake values, so this tests routing and shows nothing about attack resistance. I can't see server-side review, only what the log recorded. Ji et al. measured the same structure: 36.8% of state-changing actions fell outside the classifier through in-project edits.

Reads have their own exception. The docs say the first read outside working directories prompts, but the prompt "doesn't appear in non-interactive -p runs or background sessions; reads there run as before". In a -p run under auto mode, the Read tool returned a fake credentials file for me, with that file denied in sandbox.credentials. I asked for it by name and said it was a fixture, so this is another routing test. The sandbox covers the shell: "Claude's file tools, MCP servers, and hooks run outside it."

Credentials in .env get an explicit carve-out. The docs list "Reading .env and sending credentials to their matching API" as allowed by default. That is a sensible default for a developer. The docs define the rule by destination and do not say how it treats a production key sitting in that file.

Anthropic's own red team found the sharpest version. In February 2026, a researcher phished an employee into launching Claude Code with a prompt that asked it to read ~/.aws/credentials, encode the contents and POST them out. Claude completed that 24 times out of 25, and Anthropic attributes this to model-layer defenses that anchor on user intent. Anthropic's conclusion: "The only defense that holds in this situation is the environment", meaning egress controls and filesystem boundaries that keep ~/.aws out of reach. That run predates the August hard-deny description, and I found no re-run figure, so don't read 24 of 25 as today's rate.

Auto mode earns real credit here. Its default block list covers "Sending sensitive data to external endpoints" and "Printing a live credential or token into the transcript or a file". Anthropic says exfiltration sits in a category the classifier is designed to never approve. Among flagged production sessions at severity 7 or above, 6.3% under manual approval contained harm the user had not asked for, against 2.4% under auto mode. Anthropic commissioned Trajectory Labs, which ran 72 indirect injection scenarios ten times each and got 0 of 720, using its own browser integration. Pick auto mode over prompts. Then stop asking it to be your credential boundary, since Anthropic doesn't either.

Test what the process can reach

This script prints names and states and never prints values. Ask the agent to run it with its own shell tool, because a terminal you opened yourself tells you nothing about the agent's environment. The output names your credential files, and those names land in the transcript.

#!/usr/bin/env bash
# Prints names and states, never values. Run it the way the agent runs commands.
set -u
base="${1:-$HOME}"   # pass a fixture directory to dry-run this against fake files

for p in "$base/.aws/credentials" "$base/.ssh/id_ed25519" "$base/.ssh/id_rsa" \
         "$base/.kube/config" "$base/.config/gh/hosts.yml" "$base/.npmrc" \
         "$base/.docker/config.json" "$base/.netrc" "./.env"; do
  if head -c 1 -- "$p" >/dev/null 2>&1; then
    state=READABLE    # an actual read: a permission bit can lie under a sandbox
  elif [ -e "$p" ]; then
    state=blocked
  else
    state=absent      # or hidden from this process
  fi
  printf '%-9s %s\n' "$state" "$p"
done

# compgen lists names only.
# "env | cut -d= -f1" would print lines of a multi-line value.
compgen -e | grep -E 'KEY|TOKEN|SECRET|PASSWORD|CREDENTIAL' | sed 's/^/ENV-NAME  /'

if curl -fsS -o /dev/null --max-time 5 https://example.com 2>/dev/null; then
  echo "EGRESS    https://example.com reachable"
else
  echo "no-egress https://example.com"
fi

I ran it on November 5, 2026 with Claude Code 2.1.318 on macOS 27.0.1, against a fixture directory of fake credentials. absent can mean hidden: under a sandbox denial, the file reports absent because stat is denied too. Here are two runs. The first has the sandbox on with no credential settings, and the second adds the settings below:

# Sandbox on, no credential settings
READABLE  ~/.aws/credentials
READABLE  ~/.ssh/id_ed25519
READABLE  ~/.kube/config
READABLE  ~/.npmrc
READABLE  ./.env
ENV-NAME  GITHUB_TOKEN
ENV-NAME  NPM_TOKEN
no-egress https://example.com

# Same sandbox with the settings below
absent    ~/.aws/credentials
absent    ~/.ssh/id_ed25519
absent    ~/.kube/config
READABLE  ~/.npmrc
absent    ./.env
no-egress https://example.com

I shortened the fixture paths to ~ and removed rows for files the fixture lacks. I also cut two variable names I did not set and did not trace, CLAUDE_CODE_MESSAGING_TOKEN and CLOUDSDK_PROXY_PASSWORD, which appeared in both sandboxed runs. The first run matches the sandbox docs: default reads cover "Most of the machine", and "There is no built-in credential deny list". The sandbox's default protection covers writes and network. In that first run, the files were readable and the route to example.com was closed.

{
  "permissions": {
    "deny": [
      "Read(~/.aws/**)",
      "Read(~/.ssh/**)",
      "Read(~/.kube/**)",
      "Read(./.env)",
      "Read(./.env.*)"
    ]
  },
  "sandbox": {
    "enabled": true,
    "allowUnsandboxedCommands": false,
    "failIfUnavailable": true,
    "credentials": {
      "envVars": [
        { "name": "GITHUB_TOKEN", "mode": "deny" },
        { "name": "NPM_TOKEN", "mode": "deny" }
      ]
    },
    "network": { "allowedDomains": ["registry.npmjs.org"] }
  }
}

I tested this shape with absolute fixture paths in the Read(//...) form, and I have not tested the ~ forms printed here. Read deny rules blocked both sandboxed Bash and the Read tool, which refused with "File is in a directory that is denied by your permission settings." I left .npmrc off on purpose, and both routes still read it. The list covers what you name, so add every path your first run reports. I did not test a real install through registry.npmjs.org. A separate setting refused the file-tool read in my -p auto-mode test:

{ "permissions": { "blockReadsOutsideWorkingDirectories": true } }
Control The question it answers What still got through
Approval prompt Does a person accept this command string? 86.4% of swapped-in commands in Anthropic's study
Narrow allow rule Does the string match? An edited script behind npm test, in my test
Auto mode classifier Does this action look dangerous? 11% in the study; reads and in-project edits skip it
Sandbox defaults Where can the shell write and connect? Every fake credential file I placed
Sandbox plus deny lists What can shell and Read reach? Anything unlisted; MCP servers and hooks run outside
Credential absent or brokered Does this process hold the key at all? The child process you hand it to

The bottom rows have limits too. Anthropic's Cowork disclosure is the warning: a malicious workspace file carried an attacker's API key, and Claude uploaded files to the attacker's account through allowlisted api.anthropic.com. "The sandbox worked perfectly, and yet the data was exfiltrated." Anthropic fixed that case with a proxy that passes only the VM's own session token, and now says an allowlist "may be better conceptualized as a capability grant", so github.com on your list is more than a destination. I tested Claude Code alone. MCP servers need their own boundary, and other agents need their own run of this script.

Where a human approval still earns its click

The 39% plan rejection rate points to where people add something: rare decisions with context attached. Anthropic's production write-up shows teams spending attention that way. Kai Zhou at Nuro switches to interactive mode when Claude reviews a pull request on Zhou's behalf. Chad Kunsman at Gusto verifies each call on Terraform, AWS and direct POSTs to live APIs: "Ultimately, you're still responsible for what happens." Evan Magnussen at Garner Health configured auto mode not to approve actions that message people. Each picks an outward-facing or hard-to-undo action and leaves the rest to the machine.

Anthropic's May data fits this too. Experienced users auto-approve about twice as often as new users and interrupt more. Their oversight moves from gating steps to watching for drift.

Habituation research offers an analogy, with limits. Anderson, Vance and colleagues measured habituation to security warnings with fMRI (n=25) and on participants' own computers (n=80), and found polymorphic warnings resist it better. Those were security warnings, and the study did not cover coding agents, so I read it as a reason to keep interruptions rare and varied and nothing stronger.

My recommendation, which no study here tests: make the approval a grant. Ask which credential, for which process, until when. That is one decision with a visible blast radius, where a stream of commands asks you to reverse-engineer each one. The action-level authorization guide covers the next layer, for keys a process does hold.

What this means for your stack

Run the script from the agent's own shell before you change anything. Deny what it reports with both Read rules and sandbox.credentials, turn on failIfUnavailable, and re-run until the only READABLE rows are ones you chose. Keep auto mode on. It beat people in the only head-to-head study I found, and the fix for its gaps sits below it.

The pattern that closes this category keeps credentials absent until a process needs one. The agent holds a reference, a broker resolves it when a named command starts, and the key stays out of the agent's shell environment. This has a same-host limit: the child holds the key, so a poisoned test script can read what it receives. You still shrink the exposure from every command in a session to the commands you named.

hasp is one working implementation of that pattern, with that limit. After hasp setup, hasp run --project-root . --env OPENAI_API_KEY=@OPENAI_API_KEY --grant-secret window --grant-window 15m -- npm test gives the key to that test run and leaves the agent's shell without it. Source-available (FCL-1.0), local-first, macOS and Linux, no account.

Whichever tool you pick, re-run the check after every agent upgrade and settings change. A catch rate describes someone else's workload on someone else's commands. READABLE describes your machine today.

Sources· cited above, in one place

NEXT STEP~90 seconds

Stop handing the agent your real keys.

hasp keeps secrets in one local encrypted vault, brokers them into the child process at exec, and never lets the agent read the value.

  • Local, encrypted vault — no account, no cloud, no telemetry by default.
  • Brokered run — agent gets a reference, the child process gets the value.
  • Pre-commit + pre-push hooks catch managed values before they ship.
  • Append-only HMAC audit log answers "did the agent touch the prod token?" in seconds.
→ okvault unlocked · binding ./api
→ okgrant once · pid 88421
→ okagent never read

macOS & Linux. Source-available (FCL-1.0, converts to Apache 2.0). No account.

Browse all clusters· eight threads, one index