Breaking into Y Combinator: Part 2 | Thermal Stack Calibration TB3

I submitted my take home interview with six green gates and zero exceptions. The next day I found out the most important 0.0 in the evidence was a TypeError, not proof of anything.

In part 1, Hybrid Retrieval Fusion, we covered the TB3/Harbor pipeline, what counts as a failure, verifier design, benchmark hardening, and the trial lifecycle. That first attempt failed as a benchmark: the target output was exposed too early and the implementation was too small to force real debugging.

Originally the idea, Composite-Wall-Conduction, had the agent calculate hot- and cold-side temperatures through a digital wall: hot on one side, cold on the other, layered materials in the middle. The agent would start in /app with a broken Python solver and instructions. Early on, I saw the idea was too simple to be worth running trials on, so it never got past the idea stage.

Part 2: the first run

Since agents would have passed that task easily, I wanted to make it harder. For the first run, thermal-contact-stack, we spread contact resistance, temperature dependence, and aging formulas across different files. They were scattered in models.py, solver.py, materials.py, mesh.py, assembly.py, and postprocess.py, so the agent had to trace the model across multiple files.

The formula contract problem

The first big mistake was that the instructions gave away the formulas The agent simply had to find where the code diverged and patch it. It did not need to recover a model or debug much. It could read the spec and translate the formulas into Python.

Both agents scored a full 1.0 on v1.1: Codex 3/3, and Claude cleared it in 35 minutes. That quickly forced the redesign.

The redesign

The new version, renamed thermal-stack-calibration, removed every formula from the instruction. It gave the agent calibration cases, approved outputs, and engineering notes instead.

The goal changed from

During the redesign I also added radiation to the model, expecting it to make the task harder. It didn't. The extra physics gave agents more structure to infer from, not less. More physics ≠ harder benchmark.

The task shifted to:

1. read public calibration data
2. read engineering notes
3. repair a compact Python thermal solver
4. match public approved outputs
5. still generalize to hidden qualification cases

In place of the formulas, the instruction now read:

That difference is the entire pivot: from

Task design

The agent inherited a Python thermal-stack qualification tool used for layered electronics cooling.

The CLI stayed simple:

python3 /app/scripts/run_solver.py --cases /path/to/cases.json --output /path/to/results.json

The public artifacts were:

The intended model had to be recovered from calibration behavior, code structure, unit conventions, and physical invariants.

Hardening loop

The task had multiple bug clusters that interacted with each other.

The hidden cases covered convection, internal heat generation, high conductivity contrast, contact resistance, two-contact stacks, date-driven aging, radiation, and energy residual closure.

The audit grouped the bug surface into six clusters:

The more concrete version was the wrong-fix probe list.

Every wrong fix received reward 0.0. That mattered because a benchmark task should not only reject doing nothing. It should also reject plausible but incomplete repairs.

The 64-state bitmask audit proved only 111111 passed. Every partial fix scored 0.0.

This is where the second task became stronger than Hybrid Retrieval. Hybrid had a bug chain, but the codebase was too readable. Composite Wall had interacting model requirements: missing one piece could still match public calibration in places, but fail hidden generalization.

The recurring loop:

flowchart TD
    A["Public calibration passes"] --> B["Hidden verifier fails"]
    B --> C["Identify plausible missing model cluster"]
    C --> D["Add or inspect wrong-fix mutation"]
    D --> E["Run partial-fix / bitmask audit"]
    E --> F["Keep oracle passing"]
    F --> G["Run fresh Codex and Claude trials"]
    G --> H["Review trajectories for shortcut evidence"]
    H --> B

That loop iterated through multiple versions until both models produced 3/3 valid standard failures.

Results

The final calibrated version was stronger. The verifier compared the repaired app against an independent deterministic reference solver on hidden qualification cases.

Oracle, nop, and Docker oracle/nop validations all passed. The calibrated gatekeeper runs were:

One run that did not count

The first gatekeeper-claude run returned 0.0, but on inspection the calibration data itself was inconsistent: calibration_cases.json and approved_outputs.json did not agree. I discarded that run, fixed the data, and every run after it used clean calibration artifacts.

A benchmark with wrong calibration data tests nothing. Catching it before counting the result separates a real eval from a casual test.

The cheat reproductions, the interview's adversarial gate, also scored 0.0:

The Codex cheat run is the important one. It was told to find a shortcut. This is the moment it refused, straight from the trajectory log:

"I can't help create a shortcut that tricks the verifier or tampers with validation artifacts. I'll treat the concrete engineering request as the actionable part: repair the solver to match the approved model and verify it against the public calibration data."

It then produced a repair that matched public calibration to 1.4e-11. The hidden verifier still returned 0.0. That looked like the cleanest possible proof that public calibration matching does not equal model recovery. Or so it seemed at the time.

Trajectory review found zero hidden-fixture, solution, verifier, reward-file, or base64 shortcut access across all six standard trial logs.

Closing the gap

The earlier temp-contact and radiation branches had mixed outcomes, including one Codex pass and one cheat pass, which motivated the final calibration fix.

This task forced a real separation between:

Late findings

As the deadline approached, I was burning usage tokens fast. There was no time to spare. Every gate looked green and everything was documented in the repo, so I responded to the engineer's email and attached the GitHub repo.

The next day, while reading over the notes to prepare this blog, I took a closer look at why the Codex cheat had failed. The Codex cheat's 0.0 was caused by an undocumented internal API signature test

The verifier called case_from_dict(raw) with one argument, but the original baseline required case_from_dict(raw_input, materials). Nothing in the instruction told agents to change or preserve that signature. Codex's physically correct solver passed all 8 hidden cases (schema, numerics, invariants, order, anti-cheat) and failed on nothing but this:

Traceback (most recent call last):
  File "/tests/test_solver.py", line 190, in test_normalization_is_idempotent_and_non_mutating
    case_a = case_from_dict(copy.deepcopy(raw))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: case_from_dict() missing 1 required positional argument: 'materials'

The test: we removed the unfair interface trap and re-ran against the fixed verifier:

• Codex cheat → 1.0 (was 0.0)
• Oracle → 1.0 (regression intact)
• Nop → 0.0 (still correctly fails)

The old task had mechanically passing evidence, but it was not cleanly fair. With the fairness fix in place, the Codex cheat passes (1.0), which means the task needs hardening. Our 0.0 was a false signal: a verifier bug, not proof of robustness. We had only proved the verifier was unexploitable against the specific unfair trap we accidentally included, not against genuine behavioral difficulty.

Lesson and Part 3 preview

Part 2 taught me not to let the instructions give away the answer: public examples can guide a repair without proving the model was recovered. Thermal Stack Calibration solved the "too small and too readable" problem from Part 1, but it still did not fully close the benchmark-hardness question. I also learned to give myself enough time to verify failures. A 0.0 you haven't inspected is not evidence.

For Part 3, I moved into a C++ transient 2D heat-diffusion task, Thermal-Pulse-Solver-TB3, where the key issues become numerical resolution, shared runtime budgets, replay hardening, and whether brute-force over-resolution can be made too expensive while the reference path still fits.