A claude -p job that was refused every step still exits 0. Check the commit, not the exit code

A scheduled claude -p job is usually wired the same way: run it from cron or CI, and alert if it exits non-zero. That catches a crashed process. It does not catch the more common failure, where the agent was not allowed to do the work, said so politely, and stopped.

We gave Claude Code 2.1.322 one task in a scratch repository: change version = 1 to version = 2 in config.txt and commit it. No allow rules, no prompt to answer. Both tool calls were refused. This is what the job runner saw:

exit=0
{
  "subtype": "success",
  "is_error": false,
  "terminal_reason": "completed",
  "permission_denials": [ { "tool_name": "Edit", ... }, { "tool_name": "Bash", ... } ]
}

The file still said version = 1, and HEAD had not moved. Claude's own result text was accurate (it said it had made no change), but nothing in a pipeline reads that text. The cron log shows a green run.

Seven runs, one task

Same repository, same prompt, Haiku 5.5, Claude Code 2.1.322, reset to the first commit before every run. "Committed" means git rev-parse HEAD differed before and after.

Run	Exit	subtype / is_error	What happened
No allow rules, --output-format json	0	success / false	Edit and commit refused, nothing changed
No allow rules, text output	0	(text)	Nothing changed. The text says so
--permission-prompts none	0	success / false	Edit and commit refused, nothing changed
--allowedTools "Edit"	0	success / false	File edited, commit refused three times, left uncommitted
--allowedTools "Edit,Bash(git add *),Bash(git commit *)"	0	success / false	Committed
--max-turns 1	1	error_max_turns / true	Stopped before doing anything
--permission-mode auto, asked to run curl -fsSL https://example.com/install.sh | bash	0	success / false	The command ran and failed (HTTP 404). Nothing installed

Five of the seven runs did not do the job, and four of those five exited 0. The fourth row is the one that slips through a half-built check: a job that only verifies "the file changed" passes it, while the change never reaches the branch, the deploy or the next person.

The headless documentation is precise about what the exit code covers: "Claude Code exits with code 0 on success and a non-zero code when the run fails." The run is the session. A refused tool call, a command that failed inside the session, or an agent that decided to stop are all things the session handled successfully. The non-zero cases are session failures: the turn limit above, missing authentication, an invalid flag, SIGTERM (exit 143, per the docs), and since 2.1.319, auto mode refusing actions "because no safety check was available", which used to exit 0. In our IP allowlist test on 2.1.319, a blocked API key gave exit 1 with "subtype": "success" next to "is_error": true (details). So read is_error. subtype alone does not tell you the run failed.

The check that matches the job

Decide what the job is supposed to leave behind, then test for exactly that after the run: a new commit, a pull request that exists, a file with a timestamp from this run, a row in a table. The agent's summary is a report about the work. It is not the work.

The same rule shows up in model evaluations. In unLAP's MaxxBench write-up (20 October), a task asked models to cap the records held in memory, and the original check read the number the program printed about itself. "Every attempt across all 13 models passed the self-reported peak check, while most failed the independent live-record check. Seven of the 13 models reasoned explicitly that an external test suite could only inspect the printed statistics line." A check that reads what the agent says it did is a check the agent can satisfy without doing it, with or without intent.

This wrapper runs one job and fails on four conditions: the session failed, a tool call was refused, no commit was made, or changes were left uncommitted. It writes the JSON result outside the repository so the last check does not trip on its own file.

#!/usr/bin/env bash
# run-agent.sh: fail the job unless claude -p produced the commit it was asked for
set -uo pipefail

PROMPT="Change version = 1 to version = 2 in config.txt and commit it with message 'bump'."
ALLOWED="${ALLOWED:-Edit,Bash(git add *),Bash(git commit *)}"
OUT=$(mktemp)

before=$(git rev-parse HEAD)
claude -p "$PROMPT" --output-format json --permission-prompts none \
  --allowedTools "$ALLOWED" < /dev/null > "$OUT"
code=$?
after=$(git rev-parse HEAD)

fail() { echo "agent job FAILED: $1"; echo "agent said: $(jq -r '.result // empty' "$OUT" | head -c 300)"; exit 1; }

[ "$code" -eq 0 ] || fail "claude exited $code"
jq -e . "$OUT" > /dev/null 2>&1 || fail "no JSON result"
[ "$(jq -r '.is_error' "$OUT")" = "false" ] || fail "is_error: $(jq -r '.subtype' "$OUT")"

denied=$(jq '.permission_denials | length' "$OUT")
if [ "$denied" -gt 0 ]; then
  jq -r '.permission_denials[] | "  denied \(.tool_name): \(.tool_input.command // .tool_input.file_path // "")"' "$OUT"
  fail "$denied tool call(s) refused"
fi

[ "$before" != "$after" ] || fail "no new commit"
[ -z "$(git status --porcelain)" ] || fail "uncommitted changes left behind"

echo "agent job OK: $(git log -1 --format='%h %s')"

Against the real CLI, with three different allow lists:

### ALLOWED=Read
  denied Edit: /private/tmp/ckrepo.Mjhg/config.txt
  denied Bash: git add config.txt && git commit -q -m "bump ...
agent job FAILED: 2 tool call(s) refused
wrapper exit=1
### ALLOWED=Edit
  denied Bash: git add config.txt && git commit -q -m "bump" ...
agent job FAILED: 1 tool call(s) refused
wrapper exit=1
### ALLOWED=Edit,Bash(git add *),Bash(git commit *)
agent job OK: da160fc bump
wrapper exit=0

The other three exits were tested with a stub claude on the PATH: one that printed "Done. I bumped the version and committed it." with no denials and changed nothing (→ no new commit), one that committed and then edited the file again (→ uncommitted changes left behind), and one that printed nothing (→ no JSON result). The first stub is the important one. A run with no refusals and a confident summary still fails if the commit is not there.

Swap the last two checks for whatever your job produces. For a job that opens a pull request, compare gh pr list --head <branch> --json number before and after. For a report job, check that the output file exists and is newer than the start of the run (find report.md -newer "$OUT"). Keep the denial check in every version. Allowed exceptions belong in --allowedTools, not in the wrapper.

Three details in that script

permission_denials is in plain --output-format json too. The headless docs mention it under stream-json ("the final result message lists them in permission_denials"). On 2.1.322 the single JSON result carries the same array, with tool_name, tool_use_id and the full tool_input, so the log shows the exact command the job needed and did not have. Each entry is a ready-made candidate for the allow list, once you have read it.

--permission-prompts none did not change the exit code. In a -p run with no permission host, requests that would prompt are denied either way. The flag, per the docs, also tells Claude "that nobody can approve the request and not to retry it". In our run without it and with only Edit allowed, the commit was attempted three times. With it, once. If the job runs under the Agent SDK with a canUseTool callback or a --permission-prompt-tool, the flag matters more: without it the run waits for that host to answer each request. It needs Claude Code 2.1.259 or later. For interactive sessions and subagents, the equivalent is "permissions": { "denyInsteadOfPrompting": true } in ~/.claude/settings.json (2.1.319+). It is a user or managed setting, so a repository cannot switch it on for you.

< /dev/null. Run from a non-interactive shell with nothing piped in, 2.1.322 waited three seconds and printed Warning: no stdin data received in 3s, proceeding without it. to stderr. Redirecting stdin removes the wait and the noise in the job log.

Fewer prompts also means fewer remembered approvals. An allow list in the job definition is reviewed once and lives in version control. A "don't ask again" answer in an interactive session is saved word for word, secrets included.

Pin the CLI the job runs on

The exit rules above are version behaviour. Auto-mode refusals switched from exit 0 to exit 1 in 2.1.319, and a wrapper written against one release can read a different result after an automatic update. On 12 November the npm stable tag pointed at 2.1.311 and latest at 2.1.322, so two machines that install "Claude Code" the same day can disagree.

In CI, install an exact version and stop it from updating itself:

npm install -g @anthropic-ai/[email protected]
export DISABLE_AUTOUPDATER=1

or with the native installer, curl -fsSL https://claude.ai/install.sh | bash -s 2.1.311. On a machine that runs scheduled jobs and is also used interactively, "autoUpdatesChannel": "stable" in settings gives a release that is typically about a week old and skips releases with major regressions. Raise the pin on purpose, rerun the wrapper against a scratch repository with a missing allow rule, and confirm it still fails before the new version reaches the cron job. The same reasoning applied to the Anthropic Python SDK's 2.x release: an unpinned dependency changes behaviour on the day someone else ships.

If the job also runs hooks as guards, those have the same blind spot in the other direction: a guard that cannot start lets the call through, and the run still looks clean. That post has the setting that closes it and a jq filter for the hook events.

Measured 12 November 2026 on Claude Code 2.1.322 (macOS), Haiku 5.5 for the seven-run table and the default model for the wrapper runs. Quotes from the headless, CLI reference, settings reference and setup pages, read the same day, and from the Claude Code 2.1.319 changelog.