400 Examples Beat 51,200: AWS SFT Data Strategies That Cut Compute

AWS's machine learning blog has published the second installment in its supervised fine-tuning data series, authored by Senior Applied Scientist Krishnateja Killamsetty and Senior GenAI Strategist Elyse Zhang. Where Part 1 covered schema compliance, quality checks, and train/eval splits, this post assumes a clean, correctly formatted dataset and addresses four decisions that determine whether fine-tuning delivers: how much data is enough, which examples carry the most signal, how to generate synthetic examples that hold up, and how to mix domain data with general data. For engineers already running SFT pipelines and thinking about how pipeline architecture drives model performance, the guidance is concrete enough to act on immediately.

Learning Curve Analysis as a Diagnostic

The post centers data readiness evaluation on a learning curve methodology requiring only a single training run. Rather than training multiple models at different dataset sizes, practitioners save intermediate checkpoints every 10–20 percent of training steps. Because each checkpoint at step N has seen roughly N/total_steps of unique data during the first epoch, evaluating each checkpoint against a held-out benchmark approximates a full data-scaling curve without repeated runs.

The saturation threshold is specific: when doubling training data yields less than 1–2 percentage points of improvement on the primary metric, more data of the same distribution is unlikely to help. The post recommends 2,000 high-quality samples as a starting ballpark for a typical SFT task, while noting that simple format changes can work with as few as 500 samples and complex multi-step reasoning tasks may require 10,000 or more.

The key empirical result is from the Data Repetition Beats Scaling paper. Under a fixed compute budget, training on 400 reasoning examples for 128 epochs outperformed single-epoch training on 51,200 examples by 12–26 percentage points on AIME and GPQA benchmarks. This directly challenges the pretraining intuition that more data monotonically improves outcomes—SFT scaling is governed by coverage and instruction-set depth, not raw volume.

Subset Selection: When Fewer Examples Win

Once the learning curve flattens, intelligent subset selection outperforms adding more of the same data. Methods cited include DEITA, DELIFT, and coreset selection, which score candidates on quality, diversity, and marginal teaching value. The AlpaGasus result is the headline number: filtering to the top 20 percent of samples by quality produced faster training and higher scores than training on the full dataset.

The post also frames subset selection as a natural mitigation for catastrophic forgetting. Fewer, more targeted gradient updates pull less on the model's pretrained weights, which can reduce or eliminate the need for data mixing—a meaningful compute saving.

The practical workflow: if the curve saturates early, curate a high-value subset for all future retraining cycles. If performance is still improving at 100 percent of the data, collect more examples targeting failure cases identified through error analysis—not more samples from the same distribution.

Synthetic Data Augmentation: Three Strategies

For datasets too small or too narrow for subset selection to help, the post outlines three augmentation approaches. Distillation from a stronger model—generating traces or responses from a teacher such as DeepSeek-R1 and keeping outputs only on correct samples—is described as the method behind most open reasoning datasets. Self-generation using STaR's approach samples multiple candidate responses per problem at high temperature from the base model itself, retaining only those whose final answers are correct or internally self-consistent across samples. For high-stakes domains such as medical, legal, or safety-critical applications, expert-written examples remain the recommended gold standard, with LLM-assisted paraphrasing used to amplify them into multiple stylistic variants while preserving logical steps.

Self-Instruct and Evol-Instruct provide complementary coverage at the instruction level. MAmmoTH's finding applies here: stylistic diversity in correct solutions—two differently structured proofs rather than two copies of the same proof—improves generalization more than content variety alone.

Two non-negotiable quality principles apply to all augmented data: verification against the same quality bar used for human-curated examples, and deduplication before augmented examples enter the training set, because synthetic generation is flagged as one of the most common sources of subtle duplicates.

Data Mixing: Ratios, Risks, and Automated Optimization

Data mixing addresses catastrophic forgetting by interleaving target-task data with general-capability samples at the batch level. Amazon Nova experiments found a 50/50 split provides a strong starting point between specialization and capability retention. A highly specialized task such as medical coding can run at 80 percent target data; light adaptations such as tone changes benefit from a closer split, with 5–10 percent safety data in either case. Qwen2.5-Coder results show that a 70:20:10 ratio of code, text, and math outperformed 100 percent code training even on coding benchmarks, illustrating that positive transfer is real when data sources share underlying reasoning patterns.

A critical warning: token-level ratios and sample-level ratios can diverge dramatically. A 5 percent general-data mix by sample count can represent over 80 percent of gradient signal by token count if general sequences are substantially longer. Monitoring token-level ratios is mandatory.

Dynamic Data Mixing, which adjusts source weights continuously during training based on each source's remaining contribution, typically outperforms the best static mixture by 2–5 percent. The post recommends specific learning rate starting points for mixing experiments: 1e-5 for LoRA and 5e-6 for full-rank fine-tuning. For multimodal datasets, video data should remain above 20 percent of the mix to preserve general benchmark performance.

The throughline across all four strategies is the same: treat each decision as an empirical question resolved by a diagnostic run. Teams that measure saturation, curate for quality, verify synthetic examples, and monitor token-level mixing ratios consistently extract more capability per compute dollar than those applying standard recipes unchanged. For practitioners running regular retraining cycles, the agentic era is reshaping how these workflows are structured in ways that make dynamic, measurement-driven pipelines the default rather than the exception.