AI Portability Is the Real Bottleneck, Not Capability

Isolated capability wins are no longer the bottleneck. The bottleneck is portability — whether a result holds when the workload, architecture, or evaluation setup changes. Four releases in roughly one month make this concrete: each claims a meaningful advance, and each shows exactly where that advance stops transferring.

Where the Edges Show Up

The clearest demonstration is SWE-2. Cognition's model matches Fable 5.1 on FrontierCode 1.1 Main at 64% lower cost, and on Terminal-Bench 2.1 it leads the field outright at 92.8%. That looks broadly strong until Terminal-Bench 4, where SWE-2 scores 27.3% against Fable 5.1's 55.8% and GPT-6 Astra's 57.9% — a 30-point deficit that puts it barely ahead of its own K3 base at 21.5%. The RL reward formulation (R = S − λC, where S is binary task success and C is weighted inference cost and rollout time) compresses turns by 58% and cost by 81% on well-scoped tasks. That recipe does not transfer to long-horizon open-ended work.

PC-ALM's near-backprop result is similar in structure. On Fashion-MNIST with width 32, depth 32, ReLU activations, PC-ALM scores 77.75% against backprop's 78.66% — genuinely close, with gradient cosine similarity rising from 0.604 under standard predictive coding to 0.909. The theoretical guarantee that PC-ALM converges to exact backprop adjoints holds only for linear networks under a per-mode spectral stability bound. Nonlinear networks carry no such guarantee. The empirical sweep across widths, depths, and activation functions is the real evidence — and it covers residual MLPs on MNIST and CIFAR-10, not the transformer-scale workloads where the result would need to hold to matter for production training pipelines.

The fly connectome experiment is the starkest case. Embedding the complete MaleCNS v1.0 connectome — 166,700 neurons, 25.6 million directed edges — into a frozen 1.2B-parameter backbone improved on the raw backbone by 0.022 nats/token. But a parameter-matched control with no graph at all beat the fly-wired variant across all three random seeds. The 278,528-parameter readout has enough capacity to learn a useful correction regardless of what signal it receives; the ablation proves the readout is doing the work, not the connectome.

Making the Problem Measurable

ByteDance's HarnessDev provides the most direct measurement of transferability as a quantity. Its core finding: of 64 adjacent version switches during Evolution, feedback scores and held-out scores moved in the same direction only 34 times — 53.1%, barely better than a coin flip. Only 2 of 9 declared final versions were held-out optimal. The executor-swap experiment adds a second dimension: moving Opus 4.8's harness from its own executor to a fixed Gemini 3.1 Pro collapsed its SWE-Pro score from 69.3 to 33.0, because one harness had hard-coded a 120-step limit tuned for the original executor, causing its duplicate-query rate to jump from 10.1% to 88.2% — a coupling no internal test detected.

The table below consolidates where each advance holds and where it breaks:

A Framework for Distinguishing Reusable from Local

The pattern is consistent: a result is real within its evaluation frame and loses advantage when one variable changes — task distribution, network class, graph presence, or executor identity. Three questions any claimed advance should answer before being treated as reusable:

Does the minimal falsifying comparison exist? FLM answered this correctly with its no-graph control; most benchmark announcements do not run the structurally equivalent ablation. Does the advance survive executor substitution? HarnessDev's 53.1% directional agreement is the baseline expectation when this test is skipped. Is the theoretical basis coextensive with the empirical claim? PC-ALM's proof covers linear networks; the claim is about 1,000-layer ReLU MLPs. Both can be true, but conflating them overstates the guarantee.

The strongest counter-argument is that early-stage results are supposed to be narrow — fundamental research establishes a foothold before generality follows. Nothing here disputes the validity of these results within their stated scope. The problem is that market signals — benchmark rankings, cost comparisons, architecture announcements — do not naturally carry scope annotations. SWE-2's 64% cost advantage and its 30-point Terminal-Bench 4 deficit are both true; only one tends to travel.

What would have to be true for this argument to be wrong: PC-ALM would need to hold within two points of backprop on attention-based architectures at scale; SWE-2's RL recipe would need to close the Terminal-Bench 4 gap without separate training; and harness Evolution would need to reliably outperform feedback-score noise under executor substitution. Any one of those outcomes would meaningfully revise the portability picture. Until then, the scaffolding around a model determines deployed performance at least as much as the weights do — and the 34-of-64 figure is the most honest summary of where the field currently stands on measuring that gap. HarnessDev's value is precisely that it forces the scope annotation into the measurement itself, making the portability question answerable rather than assumed away.