Amazon Nova Forge Multi-Turn RFT: Composite Reward Design

AWS has published an implementation guide for designing composite reward functions in Amazon Nova Forge's multi-turn reinforcement fine-tuning (RFT) system, covering reward architecture, sandboxed code execution, and the instrumentation needed to detect when a reward component quietly stops contributing to training. The guide targets practitioners building agentic models who need to move beyond single-turn grading — a shift that matters because, as the authors demonstrate through a real training run, aggregate loss and reward curves can look healthy while a heavily weighted reward component contributes exactly zero gradient.

Multi-turn RFT differs from supervised fine-tuning: rather than requiring annotated reasoning paths, it learns from evaluation signals on the model's own outputs, optimizing cumulative reward across an entire trajectory. Nova Forge uses Group Relative Policy Optimization (GRPO), which ranks K rollouts per conversation and updates model weights according to the normalized within-group advantage. Any reward component that returns the same value for every rollout in a group contributes nothing to the advantage, and therefore nothing to the gradient — regardless of its assigned weight. That constraint shapes every design decision in the guide. For teams thinking through what the agentic era means for model training pipelines, this is a concrete and non-obvious constraint.

Reward Architecture and the BYOO Execution Path

Single-turn RFT in Nova Forge routes reward logic through an AWS Lambda function via reward_lambda_arn. Multi-turn tasks exceed Lambda's 15-minute invocation limit, so Nova Forge provides a Bring Your Own Orchestration (BYOO) path instead. Setting rollout.delegate: true instructs Nova Forge to delegate each rollout to an environment container — in the guide's example, deployed on Amazon ECS as part of a SageMaker HyperPod cluster. That container manages the full multi-turn interaction: running a user simulator, executing model-generated code, calling verifiers, and returning both an aggregate_reward_score and an optional metrics_list of per-component scores to the training loop. Nova Forge also offers a serverless multi-turn RFT option now in general availability for teams that prefer not to manage the container environment.

The worked example trains Amazon Nova Lite 2.0 on a collaborative-coding task across 500 unique programming problems. The model receives an underspecified coding request; a user simulator holds the full specification privately and reveals details only when queried. The model either asks a clarifying question or commits code each turn; committing ends the episode and triggers unit-test execution against the submitted code.

Composite Reward: Four Components

Two principles govern the structure. First, asked_before_coding is un-gated from correctness — it rewards the target behavior independently but still requires eventual code commitment to close the "ask forever, never commit" loophole. Second, guessed_immediately makes guessing strictly worse than asking within any GRPO group, preserving the within-group variance the optimizer requires to produce a gradient.

An earlier version gated the asking bonus behind correctness and added an efficiency term rewarding shorter conversations. Training collapsed: correctness was near zero on hard tasks, so the asking bonus almost never fired, and the efficiency term had a degenerate optimum in a single non-committal turn. Every completion resembled every other, within-group variance vanished, and learning stopped while aggregate metrics looked unremarkable.

Sandboxed Code Execution Inside the Reward

The correctness component executes model-generated code against hidden unit tests inside the reward handler. The guide treats model output under RL as unvalidated and specifies concrete precautions: a per-run random nonce (secrets.token_hex(8)) that makes the expected result unforgeable, RLIMIT_CPU set to the timeout value, RLIMIT_AS capped at 2 GB, RLIMIT_NPROC capped at 64 processes, execution in a temporary directory with a stripped PATH environment (/usr/bin only, no credentials, no network), and subprocess-level timeout enforcement. The harness also validates the number of tests actually executed against the number expected, preventing the model from diluting the score by injecting trivially-passing tests. This parallels the adversarial-output concern raised in discussions of agentic AI security in live environments.

Diagnosing Dead Reward Components

The most operationally significant section covers a failure mode the authors encountered directly: their correctness component returned the same value on every rollout because the test harness never successfully executed the model's output. The model's clarifying-question rate rose from approximately 34% to 96% — a visible behavioral shift — while code correctness barely moved. Aggregate reward, policy loss, advantage, and completion-length curves all looked plausible, with no indication that the highest-weight component was contributing nothing.

The prescribed diagnostic is tracking each component's within-group standard deviation through metrics_list, not the aggregate reward curve. A component with near-zero within-group variance is not training regardless of its weight; a flat aggregate reward mean is ambiguous, but a flat within-group variance is not. The guide also recommends sorting transcripts by the suspect component rather than total reward, ablating each component to confirm it actually affects the objective, and monitoring for dense behavioral shaping terms that saturate and crowd out the sparse outcome reward — once the asked_before_coding term saturated in this run, the correctness term lacked enough gradient signal to move the policy further.

Multi-turn RL reward engineering is an instrumentation problem as much as a design problem. GRPO's dependence on within-group variation creates a class of silent failures — collapsed strategies, gated bonuses behind unreachable conditions, dead scorers — that look identical to slow-but-healthy training in every standard metric. Teams adopting Nova Forge's BYOO path take full ownership of that surface, and the guide makes clear that per-component advantage variance tracking is not optional tooling; it is the mechanism by which reward failures become detectable at all.