CrashDiag: Mechanically Verified RL for Infrastructure Repair

In the last post, I wrote about moving from SRE-Zero to Indium AI Labs.

The short version was this: I am no longer trying to frame the work as one SRE benchmark. The larger direction is building verifiable reinforcement-learning environments for infrastructure and security agents.

This post is about the first concrete environment in that direction: CrashDiag.

CrashDiag is an environment for training and evaluating language-model agents that diagnose and repair infrastructure faults. The agent receives an incident observation, emits one bounded JSON action, and the environment decides whether the action actually repaired the system.

Public project home:

The point is not to reward explanations. The point is to reward system state.

The core idea

CrashDiag exists because infrastructure work has a property that many open-ended agent benchmarks do not: the outcome can often be checked mechanically.

If an application is down because an environment variable is wrong, the verifier can inspect the current environment and health state. If a process was OOM-killed, the verifier can check whether the process is running again and whether the service is healthy. If a dependency is mismatched, the verifier can inspect the installed and required versions. If a reverse proxy points at the wrong port, the verifier can check the proxy state and the application health endpoint.

That makes it possible to define reward without an LLM judge.

In CrashDiag, the model does not decide whether its fix worked. A verifier does.

The agent can produce a confident answer, but confidence does not matter. It can produce a nicely formatted explanation, but prose does not matter. It can name the right fault, but naming the fault is not enough. The action has to change the sandbox into a resolved state.

At the simplest level, the reward is:

1.0: the fault-specific state is fixed and the application is healthy
0.0: anything else

That sparse reward is intentionally blunt. It is harder to exploit through language because the model cannot argue its way into success.

Why this follows from SRE-Zero

SRE-Zero was useful because it exposed a recurring problem: tool-using agents can inspect useful evidence but still fail to convert that evidence into final resolution.

That failure mode showed up across several baseline runs. Evidence coverage and success were separable. An agent might read the right logs, inspect the right metrics, or touch the right fix path, then still stop early, submit the wrong remediation, or fail to close the incident.

CrashDiag keeps the strongest lesson from SRE-Zero but narrows the loop into something more trainable:

Inject a known infrastructure fault.
Present the model with an incident observation.
Require one structured action.
Execute that action.
Check the resulting sandbox state.
Assign reward from the verifier.

This is smaller than a full SRE incident-response trajectory, but that is the point. Before training an agent to run a long operational investigation, I want a clean environment where the relationship between observation, action, and verified system state is direct.

CrashDiag is the first environment in that ladder.

SRE-Zero asked: can an agent reason through incident response?

CrashDiag asks a sharper version: can a model learn to emit the action that actually repairs a broken system?

What is implemented

CrashDiag currently implements six injectable fault families with state-based resolution checks:

Fault	Difficulty	Mechanical failure	Recovery action
`oom_kill`	medium	process stopped with `OOMKilled`	`restart_app`
`bad_env_var`	easy	invalid `APP_ENV`	`rollback_env_var`
`broken_db_connection`	medium	invalid `DATABASE_URL`	`rollback_env_var`
`dependency_mismatch`	hard	installed and required versions differ	`fix_dependency`
`disk_full`	medium	disk usage exceeds health threshold	`clear_disk`
`port_proxy_misconfig`	easy	proxy and app ports differ	`fix_port_config`

The core loop uses a dependency-free MockSandbox with coupled process, environment, database, dependency, disk, proxy, and HTTP-health state.

That sandbox is not just a static prompt generator. Faults mutate state. Actions mutate state. The verifier observes state after the action. The reward comes from that state.

The repository also includes:

JSON-safe trajectory recording
single and batch episode orchestration
defensive OpenAI-compatible and vLLM agent output parsing
deterministic dataset generation
answer-free GRPO datasets
LoRA SFT using TRL SFTTrainer
GRPO using TRL GRPOTrainer
an executable mechanical reward function
local-weight, PEFT-adapter, and vLLM-endpoint evaluation paths
an isolated HTTP sandbox service
a hardened Dockerfile and Compose deployment
dependency-free SVG, JSON, and Markdown report generation
promotion gates for candidate runs

The project is not just a toy script around a prompt. It is becoming a full training and evaluation harness.

Bounded actions instead of arbitrary shell access

CrashDiag deliberately uses a bounded action space.

The model does not get unrestricted shell access. It emits a structured JSON action selected from operations the environment supports.

That design choice is important.

Unrestricted shell demos can look impressive, but they create several problems:

They are harder to verify.
They are harder to sandbox.
They are harder to reproduce.
They make reward attribution messy.
They create more ways for a model to do something unsafe or irrelevant.

Bounded actions make the environment stricter. If the model emits invalid JSON, an unknown action, or a malformed payload, the rollout fails. If the action executes but does not repair the fault, the rollout fails. If the prompt state cannot be reconstructed from the recorded seed and scenario, the rollout fails.

This gives the training loop a clean contract:

observation -> JSON action -> sandbox execution -> verifier reward

The model can still learn operational behavior, but it learns inside a space that can be audited.

That does not mean agents should remain limited forever. It means the action space should expand only as the sandbox, verifier, and safety model become strong enough to support it.

The mechanical verification guarantee

For each rollout, CrashDiag rebuilds the exact seeded, fault-injected scenario represented in the prompt.

Then it checks that the reconstructed observation matches the prompt state. This matters because a reward function should not accidentally score a different scenario than the one the model saw.

After reconstruction, CrashDiag parses the model output into an action, executes that action against the sandbox, observes the result, and calls the fault-specific is_resolved check.

Built-in faults require both conditions to pass:

The fault-specific state must be repaired.
The overall application health check must pass.

Several cases score zero immediately:

missing seed
prompt-state mismatch
invalid JSON
unknown action
action execution error
transport failure
unresolved fault state
unhealthy application state

The GRPO reward function does not import or call a reward model. The GRPO dataset does not need a target answer field. The environment can score candidate actions by executing them.

That is the central research reason CrashDiag matters.

Dataset design: SFT first, answer-free GRPO after

CrashDiag supports both supervised fine-tuning and reinforcement learning.

The SFT path teaches the model the interface:

what the incident prompt looks like
how the JSON schema works
which action families exist
how to express valid remediation attempts

SFT examples are mechanically validated before they are written. The expert repair is executed against a fresh sandbox, and only examples that actually resolve the injected fault are retained.

The GRPO path is different. GRPO rows are answer-free. They do not contain an expert completion or hidden answer field.

That distinction is important. In the RL phase, the model generates candidate actions. CrashDiag executes those actions and assigns reward from sandbox state. The dataset provides the scenario, not the answer.

This keeps the optimization target aligned with the environment:

SFT teaches format and starting competence.
GRPO optimizes against executable outcomes.
Evaluation replays held-out seeds mechanically.

The training objective is not "match this answer". It is "make the system healthy".

Schema v2: making the task harder

The original schema-v1 workflow was useful for proving the loop. But once the model sees enough direct examples, easy one-action tasks can become too direct.

Schema v2 is the harder CrashDiag curriculum.

It keeps the same six one-action fault families, but changes the prompt surface:

known-good values are redacted
stale incident history is included
harmless failed remediation can appear
hidden configuration is randomized
scenario setup remains mechanically reconstructable

This makes the prompt more operator-like. The model cannot rely only on obvious string matching. It has to choose the correct action from the observed state and the supported contract.

Schema v2 also revealed a useful design issue. The parent SFT policy often selected fix_dependency correctly but hallucinated a package version. Rather than reward arbitrary version strings, CrashDiag tightened the action contract. Dependency repair became declarative: restore the deployment's pinned version. The parser discards model-supplied dependency versions for that operation.

That is the kind of lesson I want from these environments. The benchmark should not only report scores. It should force the action contract to become safer and clearer.

The current GRPO v1 research candidate

CrashDiag has a private-bucket GRPO v1 research candidate.

The signed candidate package contains the inference adapter, tokenizer, model card, provenance manifests, and mechanically generated evaluation reports.

The candidate resolved:

175/192 hard schema-v2 episodes
91.15% hard success
96/96 original schema-v1 regression episodes
at least 62.5% success in every hard fault family
zero backend errors in the reported exact evaluations

That is encouraging, but it should be framed carefully.

This is a research candidate, not a final claim that GRPO caused the improvement. The exact 192-row parent-SFT hard baseline still needs to be run before attributing the candidate performance specifically to GRPO.

That distinction matters. A benchmark can produce a strong candidate result and still require more careful baselining before making a causal claim.

The useful point today is narrower:

CrashDiag now has the machinery to generate answer-free hard RL data, train a policy, execute candidate actions against a sandbox, produce exact held-out evaluations, and gate promotion on mechanical success.

That is the infrastructure I wanted.

Fail-closed promotion gates

CrashDiag does not treat every training run as progress.

The hard GRPO workflow includes fail-closed gates:

calibration must show useful reward variance
smoke training must show positive reward standard deviation
smoke training must show positive gradient norm
smoke training must change the adapter SHA from the parent
evaluation must cover all 192 hard rows
hard success must reach at least 70% overall
each fault family must reach at least 50%
schema-v1 regression success must reach at least 95%
backend error rate must be zero

If a gate fails, the run should fail. It should not quietly publish a _SUCCESS marker.

This is especially important for RL. A run can look busy while learning nothing. CrashDiag already caught an earlier schema-v1 GRPO smoke run with zero loss and zero gradient. That run is retained as an audit artifact, not treated as a trained model.

This is the standard I want for Indium AI Labs: failed experiments should teach us something, but they should not be promoted as results.

The remote sandbox service

CrashDiag includes an HTTP sandbox service for concurrent training workers.

The service gives each worker isolated session state and exposes the same strict action and mutation allowlists as the local backend. It can be run through Docker Compose, with a token-protected API and a hardened container setup:

non-root user
read-only root filesystem
no Docker socket mount
dropped capabilities
no-new-privileges
bounded session count
bounded operation count
bounded request timeout

For Kaggle GRPO runs, the workflow uses an authenticated HTTPS sandbox endpoint. The SFT notebook does not need the sandbox token because SFT trains from signed downloaded data. The GRPO notebook needs both the model artifact and the sandbox service because reward requires execution.

This separation is deliberate:

SFT is a data-and-adapter workflow.
GRPO is an execution-and-reward workflow.

The environment should make those boundaries explicit.

What CrashDiag is not claiming yet

The honest boundary is important.

CrashDiag currently uses isolated MockSandbox state for the local and Docker-backed service. It does not physically OOM a VPS, fill the host disk, alter host packages, or reconfigure a host proxy.

That means the current claim is not "we trained an agent on arbitrary real production infrastructure".

The current claim is:

CrashDiag provides a mechanically verified infrastructure-diagnosis RL environment with stateful sandbox dynamics, bounded actions, answer-free GRPO rewards, exact-seed evaluation, and signed training artifacts.

That is already meaningful, but it is not the endpoint.

A real-container or deployment-backed sandbox is still needed before claiming true infrastructure fault injection. The repository includes a CoolifySandbox direction, but deployment-version-dependent methods remain future work. No Coolify endpoints are guessed.

This is the right posture for the project. Mechanically verified environments only matter if the claims stay tied to what the verifier actually checks.

How this fits inside Indium AI Labs

Indium AI Labs is about verifiable, adversarial RL environments for infrastructure and security operations.

CrashDiag is the infrastructure-repair side of that direction.

IntrusionWatch is planned as the adversarial security side.

The shared thesis is:

Agents should be trained and evaluated against system outcomes, not response vibes.

CrashDiag is the first concrete implementation of that thesis. It starts with repair because repair has a clean operational structure: something is broken, the model chooses an action, and the verifier checks whether the system recovered.

IntrusionWatch will be harder because the system changes while the defender is reasoning. There is an active adversary. Availability and containment can conflict. But CrashDiag builds the foundation: sandboxing, action contracts, mechanical scoring, reproducible datasets, and promotion gates.

That foundation is reusable.

What comes next

The next step is not to declare victory from one candidate run.

The next step is to make the evidence stronger.

Immediate priorities:

run the exact 192-row parent-SFT hard baseline
compare parent SFT and GRPO v1 on identical hard prompts
tighten the port-action contract
expand fault families beyond the initial six
harden the remote sandbox service
move from mock state toward deployment-backed fault injection
publish reproducible reports without leaking private training artifacts
keep failed calibration and smoke runs as audit evidence
prepare the action/verifier interface for IntrusionWatch

The larger question remains the same one from blog 11:

Can we train reliable infrastructure and security agents using environments where success is determined by the real state of the system?

CrashDiag is my first serious attempt to answer that question.

It is not finished.

But it is now concrete enough to measure, train, fail, audit, and improve.