Qwen2.5-0.5B Fine-Tuned With DPO After Auditing HH-RLHF Length Bias

Direct Preference Optimization has matured into a production-grade alignment technique, but most public tutorials skip the step that matters most before training: determining whether the dataset itself encodes spurious shortcuts that the model will learn instead of genuine human preference. A workflow published on MarkTechPost closes that gap, documenting an end-to-end pipeline that audits Anthropic's HH-RLHF dataset for structural biases, then fine-tunes Qwen/Qwen2.5-0.5B-Instruct with DPO via TRL and LoRA — all within a single reproducible notebook.

The practical stakes are higher than they appear. As teams exploring what the agentic era means for data science are discovering, alignment quality upstream of deployment defines the behavioral envelope downstream. A model that learns to prefer longer responses over genuinely helpful ones has learned a length heuristic, not a preference signal. This pipeline is explicitly designed to catch and quantify that failure mode before it bakes into model weights.

Dataset Construction and Bias Auditing

The pipeline draws from four named HH-RLHF subsets: helpful-base, helpful-rejection-sampled, helpful-online, and harmless-base. From each, it samples 120 training pairs and 30 test pairs, yielding a balanced cross-subset corpus before filtering. Each raw example is parsed by splitting on \n\nHuman: and \n\nAssistant: delimiters using a compiled regex, then validated to ensure chosen and rejected responses share an identical conversational prefix — pairs failing that check are dropped.

The audit stage computes word counts for chosen and rejected completions per subset and calculates mean(chosen_words − rejected_words) — the length delta — as the primary bias signal. A TF-IDF vectorizer (unigrams and bigrams, min_df=2, up to 20,000 features, sublinear TF scaling) feeds a logistic regression classifier trained to distinguish chosen from rejected responses. The classifier reports accuracy and ROC-AUC on a held-out half of the pair indices, then repeats on permuted labels to establish a chance baseline. If the observed AUC exceeds the permuted baseline by more than 0.02, a real lexical shortcut is flagged. The top-20 absolute logistic coefficients are reported numerically; feature strings are suppressed because the source corpus may contain offensive text.

Training Configuration and LoRA Setup

Token-length filtering enforces a maximum prompt length of 256 tokens and a maximum total sequence length of 512 tokens, applied after chat-template rendering with tok.apply_chat_template. DPO hyperparameters: beta=0.1, max_steps=30, per_device_train_batch_size=1, gradient_accumulation_steps=8 (effective batch size of 8), learning_rate=5e-6, and warmup_ratio=0.1. Logging occurs every 5 steps. The random seed is fixed at 17 throughout data shuffling, model initialisation, and NumPy's default RNG.

LoRA is configured with rank r=16, lora_alpha=32, lora_dropout=0.05, no bias terms, and task_type="CAUSAL_LM". When LoRA is active, the frozen base model serves as the implicit DPO reference model, eliminating the need to load a separate reference copy and cutting peak memory accordingly. A version-compatibility layer inspects DPOConfig field names at runtime and converts warmup_ratio to warmup_steps when the installed TRL build does not accept the ratio form directly, making the same notebook run across TRL releases without manual intervention.

Per-Source Reward Accuracy and Length Shortcut Detection

Post-training evaluation goes beyond aggregate metrics. A per_source_reward_accuracy function samples up to 40 held-out pairs, computes log-probabilities for both chosen and rejected completions under the policy and reference model, and derives the DPO reward margin as beta × [(log π(chosen|prompt) − log π_ref(chosen|prompt)) − (log π(rejected|prompt) − log π_ref(rejected|prompt))]. A positive margin counts as a correct preference decision.

The diagnostic of central importance is the agreement rate between "model prefers chosen" and "chosen response is longer": a value near 0.5 indicates no length shortcut; a value approaching 1.0 signals the policy is ranking primarily by token count rather than preference quality. The pipeline explicitly warns that if harmless-base reward accuracy falls while the helpful subsets improve, the model is likely exploiting the length asymmetry surfaced during the pre-training audit. At MAX_STEPS=30 on CPU, reward accuracy near 0.5 is the expected outcome — the configuration is a smoke test; meaningful results require GPU and substantially more steps.

Bias auditing and DPO training are inseparable concerns, not sequential phases. Teams fine-tuning models for alignment — including those building on Anthropic's broader agent infrastructure — should treat the lexical diagnostic and per-source reward decomposition as mandatory checkpoints. All hyperparameters are named constants, the seed is fixed at 17, and the version-compatibility layer means the same notebook runs across TRL releases without intervention.