Build the Environment, Then Optimize the Harness

In July 2026, alongside its Series B announcement, InstaLILY published a two-part research report, Build the Environment, Then Optimize the Harness, on how we evaluate and tune Lily, the company's coding agent. The report has no byline and was a team effort. I led the evaluation work in Part 1 and helped synthesize the final draft. This page is my version of the argument: how the environment in Part 1 is built, and what the optimizer in Part 2 found when it climbed it.

What I worked on

Lily is InstaLILY's coding agent. A small team of the company's most senior engineers built it on LangChain Deep Agents, and I worked on it with them over the summer of 2026. Part of my work was on the harness itself: extensions for subagent and multi-agent delegation, so Lily could split complex tasks the way InstaLILY's work needs. The larger part was a question every change to that harness raised. Did this make Lily better on our code, or did it only feel better?

Lily-Eval answers that question. It rewinds a codebase to just before each of its merged pull requests, asks an agent to redo the change blind, and grades the result against what actually shipped. I led the design of Lily-Eval, which I built with a few colleagues; my main piece was the package that turns a repository's merged pull requests into graded tasks. Part 1 of the report is that system and its first head-to-head. Part 2, the optimizer that climbs it, was the wider team's work.

The argument

The usual way to improve a coding agent is to tune it directly: edit the system prompt, add a tool, run a handful of tasks, and keep whatever looks better. The report argues for a different order. First build an environment that reproduces your own engineering work and can be reset and replayed. Then measure how much its scores move when nothing has changed. Only after that should an optimizer touch the harness.

Public benchmarks can't do that job. SWE-bench tests a generic agent on generic open-source repositories, and a company deploying a coding agent has a narrower question: can this agent ship code that meets our bar, on our codebase, under our platform's rules? Our environment comes from merged pull requests, each one a task whose answer key already exists, so any task can be replayed as often as an optimizer needs. The report says it in two sentences:

A benchmark tells you where you are. An environment lets you climb.

Every merged PR is a graded exam

A merged pull request is an exam we already hold the answer key to. The change shipped, so the before and after give ground truth for free. The report calls the procedure the rollback contract. For each case:

1. Rewind the repository to the commit just before a real, shipped change.
2. Describe the problem in plain English, the way a coworker or end user would: business language, no file paths, no hints about how the original fix worked.
3. Let the agent implement blind. It never sees the reference solution or the grading tests.
4. Score against the shipped version as the answer key.

Because the answer key is free, the environment is also resettable. The same change can be replayed as many times as needed with the right answer always known, which is what lets an optimizer train against the suite instead of taking it once.

Most PRs don't make the cut

The tempting move is to mine every merged PR and call it a dataset. Curation goes the other way. One invariant governs everything: a case is valid only if its gold behavior fails before the change and passes after it. Before any test enters the suite, it has to show that a do-nothing agent scores zero and the real shipped fix scores one.

The subtlest filter is about what kind of failure counts. The "before" failure has to be a behavioral one. A test that fails only because a function doesn't exist yet would reward an agent for creating an empty function with the right name. Beyond that, automated gates reject mega-merges with no single intent, umbrella UI changes too broad to attribute, and low-value hygiene PRs, and every rejection is recorded with its reason.

A surviving PR becomes one record with two payloads that pull in opposite directions on purpose. The request the agent sees is vague about implementation and written in business language. The validation assertions are exact about behavior, and the agent never sees them. Each record compiles into a sandboxed task in Harbor, the Terminal-Bench team's task format: an instruction file, an environment built at the PR's base commit, a verifier that writes the reward, and the shipped fix kept aside as an oracle that only proves the task is solvable.

A score you can trust

Real product changes aren't graded by unit tests alone, so the tasks aren't either. Every check in the suite is one of three kinds, trusted in proportion to how hard it is to fool.

Tier	Asks	Graded by
Outcome	Does the feature behave correctly?	Deterministic code. The floor; it can zero a case on its own.
Conformance	Does the change follow the platform's rules?	Deterministic structural checks. Adjusts the score, never gates alone.
Judged	Is it well-crafted code a senior engineer would approve?	An LLM judge panel on a fixed rubric. Reported, never trusted alone.

The judges are kept apart from the work they grade. The one that writes functional tests sees only the plain-English request, never the agent's code, and the one that scores craft never runs tests. The reason is the optimizer waiting in Part 2. It will exploit any soft spot in its reward, so if a persuasive diff could talk a judge into a high score, a harness search would learn to write persuasive diffs.

Lily vs. Claude Code

The suite's first job was a head-to-head between Lily and Claude Code, both running the same model, claude-opus-4-8 at maximum effort: 46 tasks, 3 attempts each, 276 trials. Lily led on the deterministic outcome bar, 56% to 49% pass@1, and won more tasks outright. Claude Code led on mean reward and on judged craft, and did it faster. Where the two disagree, the behavioral bar is the more authoritative one, because it is the deterministic floor, and the report publishes both.

The result that mattered more than the ranking was the noise. The same agent on the same task at the same settings does not reliably produce the same verdict: 26% of task verdicts flipped between attempts of the same agent, with no infrastructure failures involved. A gap between two agents means nothing until it beats what that variance produces on its own, which is the question Part 2 opens with.

We also read every task that failed every attempt and sorted the failures into families, with the rubric fixed before reading. Most were not the model failing to reason. They were habits it never applied: reuse the field name the codebase already has instead of inventing one, run the code before claiming it works, change the one place the request names. A missing habit is something the harness around a model can enforce, and that is what the optimizer went looking for.

What Part 2 found

Part 2 freezes Lily's model and searches everything around it: prompts, context, tools, review loops and stopping rules. The idea comes from the Meta-Harness paper, which optimizes the scaffold around a frozen model instead of the model itself, and Lily-Eval supplies the reward. The wider team built the optimizer on top of the environment, and it found three things.

Scores are noisy. Two runs of the identical agent differ by about ±0.05 on the full 46-task suite and about ±0.11 on a 14-task slice. A system-prompt candidate that gained +0.14 on that slice scored below baseline on an independent re-run. Without a rule that a gain must survive a re-run it was not selected on, it would have looked like a win. The same audit downgraded the head-to-head above (Lily ahead on behavioral pass@3, Claude Code ahead on mean reward) to "leans, not leads".

The only changes to survive were code in the harness, not prompt text. Fifteen candidates went through the gates and two were promoted. The stronger one was a review hook that makes the agent show evidence before it finishes, aimed at the missing habits above.

The gain carried to a held-out codebase the search never touched: 11 of 20 tasks solved against 8 of 20, still on the gpt-5.5 exploration model rather than Lily's deployment model. On eleven of the twenty held-out tasks the score didn't move under the new harness, which fits the report's view of where a wrapper stops:

The model chooses your ceiling; the harness decides how close to it you operate.

What I took from it

An optimizer will find whatever the scoring rewards, including noise. If the team had pointed it at the suite before measuring the null band and requiring re-runs, the +0.14 prompt candidate would have shipped. With those rules in place, thirteen rejections out of fifteen read as the gates doing their job.

Going in, we assumed the dataset was the valuable artifact. Partway through, a few of us realized the durable thing was the machinery that turns an organization's repositories and shipped history into a replica of how it builds software. Anyone can accumulate tasks. The replica is resettable, spans several codebases by construction, and is versioned with the code it models, which is what lets an optimizer climb it. The tasks themselves stay private, both because the codebases are proprietary and because published tasks invite contamination. The report says InstaLILY will release a package that builds this kind of environment for any organization, and making the pipeline work beyond InstaLILY's own repositories is where I spent the end of the summer.

If another team asked me where to start with agent evals, I would tell them to spend the first weeks on the tasks and the error bars before any tuning. Confirm each task separates a do-nothing run from the real fix, run the same agent several times to see how far its scores drift, and decide in advance what a gain has to survive before you believe it. The question I most want answered next is whether the curation rules and the three grading tiers hold on a repository with a different stack and different review habits.