Phase 0: derisk on SWE-ZERO

Two models, don’t confuse them. TerminalWorld involves an agent (the policy that issues bash commands; this is the model we benchmark) and a simulator (a separate model that emits terminal outputs given a command and prior session state, used as a training-time stand-in for Daytona). The arms below describe how the agent is trained; the simulator always uses standard assistant-only SFT.

Train Qwen3-1.7B-Base on subsets of SWE-ZERO-12M-trajectories, eval on the 100-task SWE-bench Verified slice from marin#4898. SWE-ZERO is the right substrate because (1) its mini-swe-agent rollouts use a narrow, read-mostly POSIX vocabulary — small enough that simulating it is tractable; (2) arm (a) baselines already exist at three scales from marin#5611 — 10K → 7%, 100K → 9%, 1M → 11%; (3) v5p-8 fits the 10K SFT run in ~1.5 h, so the iteration loop is cheap.

The three arms — a clean loss-mask decomposition

The three arms are an exact partition of the loss mask over identical SWE-ZERO data: every non-system token is either an assistant turn (a command) or a user turn (task + observation), and the arms differ only in which set gets loss.

with (a) ⊕ (c) = (b) — verified at the token level. Each is a one-line chat-template change (qwen3_chat_template.py, _full_transcript, _user_only); nothing else differs.

(a) and (b) are agents (graded on SWE-bench pass@1 below). (c) is a simulator — it never learns to emit commands, so it is graded on env-token perplexity in the world-model probe, not pass@1. (c) is also the apples-to-apples “pure simulator” — same agent-format data as (a)/(b), unlike the env-rewritten dedicated simulator.

Note on the old (c). Earlier this slot was “SFT against a learned simulator” (train an agent on env-substituted data). That’s a downstream experiment, not a loss-mask variant, so it has moved to Phase 0.5.

Dataset composition

Why the loss mask matters so much: in real SWE-ZERO trajectories the terminal output dwarfs the agent’s commands. Tokenizing 2,000 trajectories from the 1M training subset (Qwen3 tokenizer, content tokens):

User splits into the task (1st message, ~759 tok) and observations (~393 tok/message mean, median only 128 — a heavy right tail of large file dumps). User tokens are ~3.8× the command tokens.

That ratio sets what each loss mask actually trains on (real data, pre-8K-truncation, incl. ~4 structural delimiter tokens/message):

Two consequences this grounds in data:

• Why (a) forgets the world model: it puts loss on only ~19% of content (commands) and zero on the 73% that is terminal output — so its env-modeling ability gets no gradient and decays with scale.
• Why (b) dilutes the policy: commands are only ~21% of (b)‘s loss tokens (1,743 / (6,588+1,743)), so switching (a)→(b) cuts the per-command gradient share to about a fifth — the dilution behind (b)‘s lower pass@1.

(Pre-truncation: the mean trajectory is ~8,976 tokens, just over the 8,192 SFT window, so right-truncation trims the tail of longer trajectories; the in-training role ratio is similar but skews slightly toward earlier turns.)

Recipe

(a) and (b) are byte-identical except for the train-time chat template: shared base model, raw data (swe-zero-12m-jsonl-10k-2f328e1d), shuffle seed 42, optimizer (LR 2e-5, cosine, warmup 0.03, wd 0.1, bs 8), v5p-8 us-east5, arch max_seq_len=32768, data max_seq_len=8192, eos_token_id=[151643, 151645]. (Mask coverage on a tiny synthetic example reads 17.5% (a) → 54.4% (b); on real data it is ~19% / ~92% — see the table above, the ratio is example-dependent.) Exported chat_template.jinja is byte-identical between checkpoints (the {% generation %} markers are training-time only).

Six harness fixes from #5611 are load-bearing — without any of them arm (a) gets 0%:

1. eos patch ([151643, 151645]) so vLLM stops on <|im_end|> at Qwen3 turn boundaries.
2. Pass max_tokens explicitly to litellm.completion.
3. SFT model_config.max_seq_len=32768 so RoPE scales out past 8K (else !!! collapse).
4. action_observation_template = "Observation: {{output.output}}" (default renders the dict repr).
5. Strip Daytona’s bash: cannot set terminal process group stderr noise.
6. max_turns=50, max_output_tokens=4096.

Eval protocol

Harbor + mini-swe-agent v1 + Daytona, 100-task slice from #5611. vLLM native on v6e-4 us-east5, max_model_len=32768, tensor_parallel_size=4, temp 1.0. Sharded 10 × 10 tasks with 4 concurrent Daytona instances per shard = 40 in-flight. Daytona’s 100-instance cap means arms run sequentially, not in parallel.

Scoring rule (pass@1, first-trial). For each task we take the first trial by started_at that reflects the model’s own behavior — a graded result (0 or 1) or a ContextWindowExceededError (the agent overran the 32K window without committing a patch, which counts as a model failure, 0 — not ignored). Only genuine infrastructure failures (Daytona sandbox errors — 2 trials total across all runs, both in (b) 1M rep1) are ignored, falling the task through to its next trial. We do not take best-of-N over retries, and no task was all-infra, so the denominator stays 100. Retries only ever occurred in the (b) 1M re-runs; every other cell is single-trial, where first-trial = the only trial.

Results (2026-05-28; (b) 1M re-run 3× 2026-06-02)

All three scales now have (a) and (b) numbers on the 100-task slice.

† (b) 1M is the mean of 3 eval reps (same checkpoint, temp = 1.0 resample): 4 / 9 / 5 (pooled 18 / 300 = 6.0%), after the original single 4 / 100 looked suspiciously low. All other cells are single runs. Scored under the first-trial rule above.

The 10K → 100K trend was consistent with the dilution hypothesis (gap narrowing 2 → 1 pp). At 1M the gap re-widens to 5 pp — directionally opposite. Three plausible reads:

1. Eval-sampling noise — now bounded. The original (b) 1M run scored 4 / 100; re-sampling the same checkpoint 3× at temp 1.0 gives 4 / 9 / 5 (mean 6, sample SD ≈ 2.6, SEM ≈ 1.5 pp). So 4/100 was a low decoding draw — but even the mean (6) sits below (a)‘s single-run 11, so the reversal is real, just smaller than one seed implied. This bounds eval (decoding) variance; the training-seed variance of a fresh SFT run is still unmeasured, as is (a) 1M’s own resample.
2. Real env-token contamination at scale. If env-token loss is teaching bad habits, more data could amplify rather than dilute. Would invert the smooth 10K → 100K trend, but the trend was based on only two points.
3. Hyperparameter / LR mismatch. Same LR/schedule as arm (a) 1M, but the wider loss mask effectively trains on different gradient distributions; the optimal LR/decay for (b) might differ.

Behaviorally, (b) 1M is healthy — 0 empty assistant turns (vs 28 at 10K), 28% clean-submit rate, read/edit/exec balance close to (a)‘s (45.5% / 26.5% / 4.1%). It’s not broken; it’s just less effective. Whatever’s happening at 1M isn’t the 10K starved-policy pathology returning.

(b) 10K solves: django__django-{14855, 15368, 15467}, pydata__xarray-4629, pytest-dev__pytest-8399. (b) 100K solves: django__django-{12050, 13109, 13363, 14855, 15277, 15467}, pydata__xarray-4629, pytest-dev__pytest-10081. (b) 1M solves (per rep): rep0 (4) django__django-{12050, 13109, 15467} + pydata__xarray-4629; rep1 (9) those plus django__django-{13363, 14855, 15277, 15368, 15987} (−12050 recurs); rep2 (5) django__django-{13109, 14855, 15368, 16527} + pydata__xarray-4629. Only django-13109 and pydata__xarray-4629 solve in all three reps. Against (a) 1M’s 11 solves, django-13109 is the robust (b)-unique win (a) never gets; (a) uniquely holds 5 (pylint-6903, pytest-8399, scikit-learn-15100, sphinx-8721, sphinx-9698). django-16527 is the one task the first-trial rule demotes: in rep1 its first attempt overran the context window (model failure → 0); a later retry passed, but best-of-N is not used.

HF datasets: (a) 10K · (b) 10K · (a) 100K · (b) 100K · (a) 1M · (b) 1M.

Behavioral comparison

The pass@1 gap is small, but the way (b) and (a) reach those numbers differs. Computed over jointly-attempted tasks (93 at 10K, 91 at 100K — (a) has a few Daytona setup failures of its own):

(b) reads more, edits less, and at 10K occasionally emits literally nothing (the empty-turn pathology). The empty-turn pathology resolves entirely at 100K; the read/edit imbalance compresses but persists. Representative failure modes seen in individual (b) trajectories that did not solve their task:

• django__django-12050: 1 sed in 50 turns, then repeated python -c "from ... import resolve_lookup_value" import attempts failing because the fix was never applied.
• django__django-14855: 47 greps, 0 seds — the model searched the codebase non-stop, never committing.
• sympy__sympy-13480: 40 of 50 bash blocks open with # — syntactically valid no-ops.

One (b)-unique win is genuinely informative: on pytest-dev__pytest-8399 (b) made a single targeted sed on src/_pytest/unittest.py:147 to swap a fixture-name prefix, while (a) was pulled into the wrong file (src/_pytest/runner.py) for 100 turns. On a literal pattern-match bug, (b)‘s read-heavy style stayed close to the bug report and won.

Status

• (a) and (b) agents at 10K, 100K, and 1M: complete, pass@1 on HF (table above).
• (c) user-only simulator: 10K training in progress; graded on env-PPL in the world-model probe, not pass@1. Scales to 100K/1M if the predicted ordering (c) ≤ (b) < (a) holds at 10K.
• The downstream train-against-the-simulator experiment (old (c)) now lives in Phase 0.5.

Next steps

1. Finish the (c) user-only sweep (10K → 100K → 1M) and complete the loss-mask decomposition on the world-model probe. (c) vs the dedicated env-rewritten simulator isolates whether the rewrite pipeline buys anything.
2. Pin the pass@1 effect size with a second SFT seed at 10K. The (a)/(b) gaps are inside the single-seed noise band; a second seed separates “small real regression” from “noise coincidence.”
3. (b’): weighted-token full-transcript SFT — down-weight env tokens by ≈ |actions| / |envs| so action-token loss share stays ~100%. If b’ ≈ a, the (b) effect is pure budget dilution; if b’ < a, env-token loss is actively contaminating the shared representation.
4. Phase 0.5: use the simulator (the (c) model, or the dedicated one) as a training environment — static env-substitution first, then closed-loop rollouts.

Reproduce

All code pinned at commit da8d12e on the terminalworld branch of AlienKevin/marin. The only difference between arms (a) and (b) is the chat template (which tokens get loss); everything else is identical.

• Eval (agents (a)/(b)): exp5611_eval_qwen3_1_7b_swe_zero_8k.py (defines the 100-task slice via ALL_TASK_NAMES), launched 10-sharded with scripts/launch_eval_arm_b.sh. Simulator (c) is graded on env-PPL — see the world-model probe.
• Dedicated env-rewritten simulator + env-substitution pipeline (now Phase 0.5): exp5611_sft_qwen3_1_7b_swe_zero_sim_10k_8k.py, scripts/rewrite_swe_zero_for_sim.py, scripts/substitute_env_with_simulator.py.
• Checkpoints (gs://marin-us-east5/checkpoints/): (a) exp5611_sft_qwen3_1_7b_swe_zero_{10k_8192tokens_arch32k_v5p8-e5118c, 100k_…_v5p16-ee03cb, 1m_…_v5p32-a26bea}; (b) …_10k_…_echo_v5p8-8028d4, …_100k_…_echo_v5p-3beb0f, …_1m_…_echo_v5p32-a2b7ba. All hf/step-* final exports, eos-patched to [151643, 151645].
• Eval outputs + HF datasets: linked in the results table above.