Stage 3: Supervise the agent

What to check before you approve an action, when to stop the agent, how to review and undo its work, and when to move to auto mode

You are responsible for every change the agent makes, as if you had made it yourself. The agent is often right. When it is wrong, it can sound just as sure, so its confidence tells you nothing. This stage teaches you to judge its actions while they happen, and its work when it has finished.

You work in manual mode throughout this stage, as stage 1 set up. You move to auto mode at the end.

What a permission prompt shows

In manual mode, Claude Code stops before each action and shows you what the agent wants to do. In this workspace you see two kinds of action:

  • A shell command (the Bash tool), such as git push -u origin fix-typo.
  • Python code that the agent runs in its kernel (clikernel). The agent reads, searches and edits files this way. Even reading a file therefore asks for your approval. Python code can do anything that a shell command can do. A line that starts with ! is a shell command. Read kernel code as carefully as a shell command.

The prompt lets you approve the action once, approve it and stop Claude Code asking about that kind of action, or refuse it. Press Esc to refuse, then tell the agent what to do instead. Do not choose “don’t ask again” while you learn. It adds a permanent rule, and Claude Code then stops showing you that kind of action. For kernel code, the rule covers every later kernel call, and the kernel can run anything.

Some commands never reach a prompt. The safecmd tool approves a list of everyday commands. The list includes git status, git diff, git log, git add, git commit, git switch, git checkout, and the gh commands that only read. git checkout -- <file> discards your changes to that file, and safecmd approves it too. You see these commands in the conversation after they run.

Before you approve

Answer these questions before you approve an action. When the answer to any of them is no, or you are not sure, refuse the action and ask the agent to explain.

  1. Can you say what the action does? If you cannot, ask the agent to explain it before it runs.
  2. Is it part of what you asked for? A step you did not ask for, such as a new file, another notebook, a new package or a push, needs your agreement first.
  3. Does it change only the files you expect? Code changes go in notebooks under nbs/. Files in marisco/ change only through nbdev-export.
  4. Can you undo it? Commands that discard work or delete files cannot be undone. The cheatsheet lists the Git commands that discard work.
  5. Does it reach other people? git push, gh pr create, gh issue create and comments on GitHub publish what the agent wrote. Read the text before you approve.
  6. Does it contain a secret? Your Zotero API key, a password or a token must never appear in a command, a file or a message.

When to stop the agent

Press Esc to stop the agent, then tell it what is wrong. Stop it when it:

  • edits a file in marisco/ directly, instead of the notebook in nbs/;
  • works on main, or tries to push to main;
  • says that tests pass or that a fix works, and you did not see it run the test. Ask it to run the test and show you the output;
  • repeats a step that failed, or tries more and more complicated ways around an error;
  • deletes or weakens a test so that it passes;
  • changes a notebook in nbs/api/ while you work on a handler. Those notebooks are shared by every handler, and a change there needs the maintainer’s agreement;
  • changes a curation decision, such as a unit conversion, a nuclide mapping or a flag, without your agreement. Record decisions you are unsure about in a “FEEDBACK TO MARIS DATA TEAM” callout in the handler, for the MARIS data team to review;
  • says something about the data that you know is wrong. You know the data better than the agent does.

Review the result

When the agent says a task is done, check the result yourself before you push:

  1. git status lists the changed files. Each one must be a file you expected to change.
  2. Read every changed line. VS Code’s Source Control view shows notebook changes cell by cell, which is easier to read than git diff in the terminal.
  3. Run nbdev-test on each changed notebook yourself, and read its output. nbdev-test skips handler notebooks and prints No files were eligible for testing. Run a handler notebook by hand instead, as stage 4 describes.

On the PR page, check that CI passes and that the “Files changed” tab contains only the changes you reviewed.

Undo a mistake

How you undo a mistake depends on how far it has gone:

  • Not committed yet: Ask the agent to undo the change. To discard all changes to one file yourself, run git restore <file>. That change is then gone for good.
  • Committed, not pushed: Ask the agent to make a new commit that reverses the mistake. Do not ask it to rewrite commits.
  • Pushed to your branch: Push a commit that reverses the mistake. The change has not reached main, because main accepts changes only through a reviewed PR.
  • A secret was committed or pushed: Tell the maintainer at once. Ask the MARIS administrators to replace your Zotero API key. Removing the secret from the files does not remove it from the Git history.

Move to auto mode

Stay in manual mode until you have completed the check below. Then ask the agent to remove defaultMode from ~/.claude/settings.json. New sessions then start in auto mode. You can also switch for one session with Shift+Tab.

In auto mode, actions run without asking. A safety classifier checks each action in the background and blocks risky ones. It blocks force pushes, commands that discard uncommitted changes such as git reset --hard, merging a PR that no person has approved, and printing a credential. The classifier also blocks actions that you have ruled out in the conversation, such as “don’t push”. It can lose such an instruction in a long conversation. The protection on main cannot be lost.

Your review moves from each action to the result. After each task, review the result as described above. Return to manual mode with Shift+Tab for work you want to watch step by step, such as a change you do not yet understand.

Check your understanding

The maintainer prepares a branch that contains a few deliberate mistakes, such as a direct edit in marisco/, a changed unit conversion and a weakened test. Review it as described above, and list the mistakes you find.

Stage 3 is complete when you have found the planted mistakes, refused and redirected the agent at least once in a real session, and can answer these questions:

  • Which actions run without a prompt in manual mode?
  • Why must you read Python code sent to the kernel as carefully as a shell command?
  • Which changes must you never approve without the maintainer’s agreement?
  • How do you undo a change that you committed but did not push?
  • In auto mode, what stops a push to main, and what stops a force push?