AI Portability: What Changes When the Evaluation Setup Changes

An optimization can help on the measured task and remain unproven elsewhere. The useful question is which part of the setup must stay fixed for the result to transfer.

Correction, 22 October 2026: The earlier article assigned causes to benchmark differences without a controlled comparison, treated a reported control as a universal proof and presented HarnessDev's transfer statistic as a general baseline. It also combined evidence from separate code and search harnesses. This revision separates those claims.

SWE-2: a score table does not diagnose a training failure

Cognition's release report lists SWE-2 at 92.8% on Terminal-Bench 2.1 and 27.3% on Terminal-Bench 4. It also reports FrontierCode 1.1 Main at 50.0%, compared with Fable 5.1 at 50.9%, and a 64% cost advantage for that comparison.

Those observations should stay attached to their respective evaluations. They do not, by themselves, prove that the cost penalty in reinforcement learning caused the lower Terminal-Bench 4 score. Nor does the FrontierCode comparison establish equal capability on every coding task.

A causal test would need to vary the suspected training choice while controlling other relevant factors. For model selection, a simpler first step is to run both candidates on a representative task set with the same acceptance criteria and an explicit execution budget.

PC-ALM: a theorem and an experiment answer different questions

The PC-ALM paper uses an augmented Lagrangian with layer constraints, activation updates and dual multipliers. Its linear fixed-weight analysis establishes convergence under a spectral stability condition. The empirical section evaluates deep, narrow networks under stated architectures and inference budgets.

Keep those forms of evidence distinct. A theorem proves its conclusion under its assumptions. An empirical sweep describes the configurations actually measured. Neither can silently inherit the scope of a different architecture.

When evaluating a training method, retain the activation function, width, depth, parameterization, optimizer and inner-step budget. A nearby accuracy result is incomplete without the work required to obtain it. Compare wall time and memory on the hardware you would use before describing the method as a practical replacement.

FLM: a useful negative control with a reproducibility boundary

The FLM repository couples a fixed connectome to a frozen language backbone through a trained readout. Its documentation says the matched direct-input control in the separate study performed slightly better and does not establish an advantage from fly anatomy.

The same documentation distinguishes its public conversational training recipe from private study artifacts and per-token results. AI Mastery has not reproduced that study. The repository's availability therefore should not be presented as independent verification of the earlier article's exact three-seed totals.

A control can challenge a proposed explanation without proving a universal negative. Here, the appropriate conclusion is that the reported experiment did not establish the claimed topological advantage.

HarnessDev: keep the experimental unit intact

HarnessDev reports feedback and held-out scores moving in the same direction for 34 of 64 adjacent version switches. That 53.1% is a study result, not an expected success rate for an arbitrary optimization loop.

Its executor-transfer analysis also separates domains. The Opus SWE-Pro score changes from 69.3 to 33.0 under a Gemini executor. The duplicate-query change from 10.1% to 88.2% concerns the Opus search harness. Combining them into one measured causal chain was incorrect.

Record the harness, executor, prompts, tool protocol, limits and stopping rules together. Changing the executor changes a dependency of that system, even if the harness code still runs.

Design a transfer check before accepting the claim

This is a proposed evaluation procedure. The four sources do not form a common benchmark or establish that portability matters more than capability in every deployment. Our benchmark-reading review covers the earlier step: preserving the dataset and metric before comparing a result.