A setting you pinned does not pin what it does. Probe your agent config after every upgrade

This file is the only Claude Code configuration in a scratch folder:

{ "sandbox": { "enabled": true } }

We gave two versions of Claude Code the same headless task with that file: use the Bash tool to run date > probe.txt, then reply DONE.

Version	npm channel	probe.txt written	permission_denials	Exit code	subtype
2.1.311	stable	yes	0	0	success
2.1.325	latest	no	1	0	success

The file is identical in both runs. What changed is version 2.1.324, released on 13 November, whose changelog says: "Changed sandboxed Bash commands to follow the normal permission flow by default; set sandbox.autoAllowBashIfSandboxed: true to keep running them without prompts". The settings reference now gives the default for that key as false and adds: "Before v2.1.324, the default was true." With the second key added to the file, 2.1.325 wrote probe.txt again.

In an interactive session you notice this within a minute, because a permission prompt appears where there was none. In a headless run nobody answers the prompt. The command is refused, the agent writes a polite final message explaining that the command did not run, and the session ends with exit code 0. Why the exit code is the wrong thing to alert on is covered in the post on claude -p jobs that exit 0. This post is about where the change came from, and how to catch the next one before a job runs on it.

The half of the rule we left out in September

When a launch week moved a pile of defaults in September, we summed it up in that post as: an explicit choice is preserved, and an unset value is a standing delegation to the vendor. That still holds for the value. Nobody overrode "enabled": true this week. The program reading it changed what it implies. Before 2.1.324 the line meant "run Bash in the sandbox and do not ask". Since 2.1.324 it means "run Bash in the sandbox", and whether to ask is decided by your allow rules and permission mode.

A setting has two halves: the value you wrote, and the version of the program that reads it. Committing the settings file pins the first half. Only the version pins the second, and for most teams the version moves on a schedule someone else sets. On npm this morning Claude Code's stable tag is 2.1.311 and latest is 2.1.325. A team on the stable channel has not met this change yet. It arrives on whatever day stable passes 2.1.324, and that day has not been announced.

Same version, same command, two results

Codex 0.169.0, released on 13 November, lists this under bug fixes: "codex exec keeps project configuration and MCP servers disabled in untrusted directories, even with --sandbox workspace-write. Use --trust with codex exec or codex review to explicitly save project trust."

To see what that does to a job, we made a git repository containing one file, .codex/config.toml, which declares an MCP server whose only action is to append the date to a marker file when Codex starts it. Then we ran codex exec --sandbox workspace-write "Reply with the single word OK." in it three ways:

• 0.167.0 in a new folder: the server started, the reply was OK, exit code 0. Codex also wrote trust_level = "trusted" for that folder into ~/.codex/config.toml without being asked.
• 0.169.0 in a new folder: the server did not start, no trust entry was written, the reply was OK, exit code 0. The only sign of the difference was one line on stderr: "Untrusted project-local configuration is disabled. Run codex exec --trust to allow project configuration and MCP servers for this and future chats in the project, including MCP commands outside the sandbox."
• 0.169.0 in the folder 0.167.0 had already run in: the server started. The entry the older version wrote is still in the file, and the new version honours it.

Put a CI runner next to a developer laptop and the effect is clear. A hosted runner starts from an empty home directory on every job, so from 0.169.0 it runs without the project's config and MCP servers, and exits 0. A laptop or self-hosted runner where an older Codex once ran in that folder keeps both. Same version, same command, and the outcome depends on history kept in a file nobody reviews. On the machine we tested on, ~/.codex/config.toml held 244 trusted project entries. 204 of them point to folders that no longer exist, 113 of those in temporary directories.

If a CI job does need the project config, --trust restores it, and the help text says what that costs: "Save trust for this project and future chats, including MCP commands outside the sandbox." A trusted project's MCP commands run outside the sandbox. On a job that checks out a pull request from a fork, --trust runs whatever command that pull request's .codex/config.toml declares. Keep it to jobs that run code you already trust, such as pushes to your default branch.

Two more this week, and the direction they share

GitHub Copilot, 14 December. Copilot removes Claude Fable 5, Claude Haiku 4.5 (except for individual subscribers on annual plans), Gemini 3.8 Flash, Gemini 4 Argon, Grok 4.6 and Kimi K3 from every Copilot surface on that date. GitHub's notice says the suggested alternatives are enabled automatically for Business and Enterprise "unless an administrator has turned off the global default or explicitly disabled the model." An admin who turned the global default off made a choice to decide model access deliberately. From 14 December the same choice means the people using those six models get no replacement until someone enables one in the model policies in Copilot settings. Kimi K3 has no named replacement at all.

Gemini 4 Hydrogen, generally available since 12 November. For API callers the version is the model string, so nothing changes until someone edits it. When someone does, the request body that stays the same is read by a model with different rules. Google's migration guide says Hydrogen "defaults to high rather than medium" for thinking, that "the max_output_tokens field caps combined thinking and output tokens", and, for temperature, top_p and top_k, "The server ignores in-range custom values if sent." A temperature: 0 your code sends for repeatable output is accepted without an error and has no effect. Inside Copilot, users of 3.8 Flash are moved to Hydrogen on 14 December whether they edit anything or not.

Three of the four leave a job doing less unless a person steps in: Claude Code asks where it used to run, Codex loads less in a folder it does not trust, and Copilot enables nothing new for an org that opted out of defaults. Each of them announces itself with a prompt, a warning or an admin notice. An unattended run has nobody to see any of them, so it shows up as work that did not happen in a run that reports success.

One probe per setting a job depends on

A changelog tells you that something changed. A probe tells you whether it changed for your setup. For each setting an unattended job relies on, write one short run that only succeeds while the setting still does what you meant, and run it after every upgrade, before the real job.

For the Claude Code sandbox, the probe is the run from the top of this post, pointed at the settings file your job uses:

cd "$(mktemp -d)"
claude -p "Use the Bash tool to run exactly: date > probe.txt. Then reply DONE." \
  --settings /path/to/job-settings.json --output-format json > probe.json
if [ -s probe.txt ]; then echo "sandboxed Bash ran without a prompt"
else echo "sandboxed Bash was not allowed; denials: $(jq '.permission_denials | length' probe.json)"; fi

On 2.1.325 with {"sandbox":{"enabled":true}} it prints sandboxed Bash was not allowed; denials: 1, and the JSON still says "subtype":"success","is_error":false.

For Codex in CI, the probe runs in a fresh clone, which is exactly what a hosted runner sees:

dir=$(mktemp -d) && git clone -q --depth 1 "$REPO" "$dir" && cd "$dir"
codex exec --sandbox workspace-write "Reply OK." < /dev/null 2> codex.err > /dev/null
grep -qF 'Untrusted project-local configuration is disabled' codex.err \
  && echo "project config and MCP servers were NOT loaded"

On 0.169.0 against our test repository it printed the warning line, exit code 0, and the MCP server never started. Run it on a laptop, too: a new temporary folder is untrusted even when its parent folder has a trust entry, so the probe reproduces what CI sees.

To decide which settings deserve a probe, filter the changelog for the version range you are about to cross by the keys your settings file sets. For Claude Code, from the version you run to the newest:

FROM=$(claude --version | cut -d' ' -f1)
curl -s https://raw.githubusercontent.com/anthropics/claude-code/main/CHANGELOG.md \
  | awk -v from="$FROM" '$0 == "## " from {exit} {print}' \
  | grep -E '^- Changed' \
  | grep -iF -f <(jq -r '[paths(scalars)][] | .[0]' /path/to/job-settings.json | sort -u)

Between 2.1.311 and 2.1.325 the changelog has 141 lines that start with "Changed". Filtered by the one top-level key in the sandbox-only settings file above, 13 remain, and the first of them is the line that flipped the behaviour. A settings file with permissions, env and model keys gets a longer list, still short enough to read before an upgrade. The changelog is not complete, and releases have shipped with no entry at all. Diffing the CLI itself covers those gaps.

Finally, make sure the probe and the job see the same version. A probe that passes on Tuesday says nothing about a job that auto-updated on Wednesday. Claude Code's setup docs describe autoUpdatesChannel: "stable" with minimumVersion set to the version you run ("The installation stays on 2.1.323 until the stable channel serves a newer version") and DISABLE_AUTOUPDATER for stopping background updates. In CI, install an exact version: npm install -g @anthropic-ai/[email protected] or @openai/[email protected]. Then move the version on purpose, with the probes as the gate.

When a probe fails after an upgrade, there is a decision to make, and that is the useful outcome. On Claude Code 2.1.325 the decision is whether sandboxed Bash should run unasked ("autoAllowBashIfSandboxed": true) or only the commands your allow rules name. For Codex it is whether a CI job should trust the project's config at all. Both answers are defensible. Finding out from a week of cron runs that reported success and did nothing is the outcome the probe exists to prevent.