From SRE-Zero to Indium AI Labs

I am sunsetting SRE-Zero and beginning a much bigger chapter: Indium AI Labs.

SRE-Zero started as an experiment in training language models to respond to infrastructure incidents. The goal was to move beyond agents that simply answer technical questions and instead build agents capable of investigating and resolving failures inside an actual system.

While working on it, I kept running into a fundamental problem with the way many AI agents are evaluated:

Most agent benchmarks measure whether a response sounds correct, not whether the agent actually fixed anything.

That distinction matters.

In infrastructure and security operations, a convincing answer is not enough. The service either comes back online or it does not. The fault is either repaired or it remains unresolved. The intrusion is either contained or it reaches its objective.

This realization became the foundation for Indium AI Labs.

Why I am moving beyond SRE-Zero

SRE-Zero was useful because it helped me validate the core idea: infrastructure operations can provide mechanically verifiable outcomes for training and evaluating AI agents.

However, the project was still framed primarily as an SRE-focused model.

The larger research opportunity is not one specific model or one incident-response task. It is the construction of environments where agents must interact with real system state, take bounded actions, and succeed against an executable verifier.

Instead of continuing to expand SRE-Zero as a standalone project, I am carrying its central ideas into a broader research direction.

SRE-Zero is the predecessor and proof of concept.

Indium AI Labs is the research lab being built around the larger thesis.

The thesis behind Indium AI Labs

Indium AI Labs is focused on building verifiable, adversarial reinforcement-learning environments for infrastructure and security operations.

These environments are designed around a simple principle:

Reward should come from what happened to the system, not from another language model deciding whether the answer looked good.

An agent operating in one of these environments must do more than generate a plausible remediation plan. It must interact with the system, diagnose the failure, choose an action, and produce a measurable change in the environment.

A verifier can then inspect the resulting state:

Is the application healthy?
Is the process running?
Is the configuration valid?
Is the dependency available?
Is the database reachable?
Is the intrusion still active?
Has the attacker reached the objective?

The result should be determined programmatically.

No LLM judge. No subjective scoring rubric. No reward for writing a confident explanation. Just an outcome that can be independently reproduced and checked.

The first environment: CrashDiag

The first project under Indium AI Labs is CrashDiag.

CrashDiag trains language models to diagnose and repair infrastructure faults inside controlled sandbox environments.

A scenario begins with a known fault injected into the sandbox. The model receives an observation of the incident and selects a bounded action represented as structured JSON.

The environment executes the action, observes the resulting state, and determines whether the underlying fault has actually been resolved.

The model never decides whether its own action worked. The verifier does.

CrashDiag currently supports fault families involving areas such as:

incorrect environment variables
broken process state
database failures
dependency failures
disk-related failures
reverse-proxy and application health problems

Each scenario has a state-based resolution check. A rollout only succeeds when both the specific fault and the overall application health conditions pass.

Invalid actions, malformed output, unresolved faults, action failures, transport errors, and prompt-state mismatches receive no reward.

Why bounded actions matter

Giving an agent unrestricted shell access may make a demonstration look impressive, but it also makes training, verification, and security considerably harder.

CrashDiag instead uses a bounded action space.

The policy chooses from actions that the environment explicitly supports, and those actions are represented using a defined JSON schema.

This creates several advantages:

Actions are easier to validate. The environment can reject malformed or unsupported operations before execution.
Rollouts are reproducible. The same seeded scenario and action can be reconstructed and evaluated again.
Rewards remain tied to system state. The model cannot receive credit merely for describing a valid command.
The blast radius is controlled. Experiments can be run inside isolated sandboxes without exposing arbitrary system access.
Training data remains structured. Trajectories can be stored, inspected, replayed, and compared across models.

This does not mean agents should remain permanently limited to tiny action spaces. It means capabilities should expand alongside the strength of the sandbox, verifier, and safety model.

Training with mechanically verified rewards

CrashDiag supports a training pipeline combining supervised fine-tuning and Group Relative Policy Optimization.

Supervised fine-tuning can teach the model the action schema, expected reasoning patterns, and valid interaction format.

Reinforcement learning then optimizes the policy against executable environment outcomes.

The GRPO dataset does not need to contain an expert completion or a hidden answer field. The model generates candidate actions, those actions are executed against the sandbox, and reward is calculated from the resulting system state.

At its simplest, the reward is sparse:

1.0: the fault is resolved and the application is healthy
0.0: anything else

This makes the reward more difficult to exploit through persuasive language.

A model cannot obtain a higher score by claiming that the service recovered. It must cause the service to recover.

Why not use an LLM judge?

LLM judges can be useful when evaluating open-ended writing, preference alignment, or tasks without an objective ground truth.

But they introduce problems when the outcome can already be checked directly.

An LLM judge may reward:

a plausible but unexecuted remediation
a technically polished explanation
a familiar incident-response pattern
an incorrect answer expressed confidently
an action that would work in theory but failed in the actual environment

For infrastructure tasks, the system itself is the better judge.

If the agent claims it fixed a database connection but the application still cannot reach the database, the rollout failed.

If the agent produces a poorly worded response but successfully restores the service, the operational objective was achieved.

Mechanically verified environments allow training to optimize for the latter.

From infrastructure reliability to adversarial security

CrashDiag is the first step, but infrastructure diagnosis is only one part of the broader direction.

The next major environment planned under Indium AI Labs is IntrusionWatch.

Where CrashDiag focuses on diagnosing and repairing infrastructure failures, IntrusionWatch will focus on security incidents in which an attacker is actively progressing toward an objective.

The defensive agent may need to:

investigate suspicious activity
correlate events across services
identify compromised components
revoke credentials
isolate systems
block malicious traffic
preserve legitimate availability
contain the intrusion before the attacker succeeds

This introduces a fundamentally adversarial environment.

The attacker changes the system while the defender is reasoning about it. Delayed or incorrect actions have consequences. Overly aggressive remediation may stop the attacker while also destroying the service the defender was supposed to protect.

The verifier can evaluate the final state mechanically:

Was the attacker contained?
Was the attacker's objective prevented?
Did the defended service remain operational?
Were critical assets protected?
How much unnecessary disruption did the defender cause?

CrashDiag and IntrusionWatch will eventually share a common environment and training harness, while representing two different operational problem classes:

CrashDiag: restore a broken system.
IntrusionWatch: defend a system against an active adversary.

What happens to SRE-Zero?

SRE-Zero is not being erased from the story.

It represents the earlier stage of the research direction and the project that helped surface the central problem.

The progression is:

SRE-Zero
  -> environment-grounded incident response
  -> CrashDiag
  -> shared infrastructure and security RL harness
  -> Indium AI Labs

Rather than maintaining SRE-Zero as a separate identity, I am consolidating the work under Indium AI Labs.

This gives the research a clearer scope and creates room to investigate infrastructure repair, incident response, adversarial defense, agent robustness, reward design, and environment-grounded evaluation within one coherent lab.

What I am building next

The immediate focus is getting CrashDiag to a genuinely demonstrable and benchmarked state.

That includes:

expanding the number and difficulty of injectable faults
strengthening the sandbox isolation model
running reproducible SFT and GRPO experiments
evaluating models on held-out scenarios
measuring success separately across fault families
testing robustness against stale and misleading incident context
preventing reward leakage through prompts or datasets
publishing evaluation reports and trained adapters
designing a reusable environment interface for IntrusionWatch
comparing trained policies against prompting-only baselines

The goal is not to produce one impressive demo where an agent restarts a process.

The goal is to build a benchmark and training environment that can answer harder questions:

Does reinforcement learning improve real remediation success?
Does performance generalize to unseen configurations?
Can an agent distinguish the actual root cause from misleading evidence?
Does the policy remain effective when known-good values are hidden?
Can the model recover from failed remediation attempts?
How much does supervised fine-tuning contribute compared with GRPO?
Which capabilities transfer from infrastructure repair to security containment?

Why build this in the open?

CrashDiag is being developed publicly because verifiable agent research benefits from inspection.

The environment, fault definitions, reward logic, training pipeline, and evaluation methodology should be open to criticism.

A benchmark is only useful when researchers can understand what it measures and where it can be exploited.

Open development also makes it easier for others to:

add new infrastructure faults
build additional sandbox backends
evaluate different models
test alternative reinforcement-learning methods
identify reward-hacking opportunities
improve isolation and reproducibility
propose new environment families

The project is early, and many design decisions will change as the experiments reveal what works and what fails.

That is part of the point.

A new chapter

Retiring SRE-Zero was not a decision to abandon the original idea.

It was a decision to take the idea seriously enough to give it a larger foundation.

SRE-Zero began with a question:

Can a language model learn to respond to real infrastructure incidents rather than merely describe solutions?

Indium AI Labs is being built around the broader version of that question:

Can we train reliable infrastructure and security agents using environments where success is determined by the real state of the system?

CrashDiag is the first environment built to explore that question.

There is still a lot to build, benchmark, break, and improve.

But the direction is now clear.

Explore the projects

Contributions, technical feedback, research discussions, and attempts to break the environment are welcome.