IMDb Sentiment Analysis: DistilBERT LoRA vs TF-IDF with Calibration and Semi-Supervised Learning

A tutorial published by MarkTechPost on August 9, 2026 walks through a complete sentiment classification pipeline on the Stanford NLP IMDb Large Movie Review Dataset, pitting a TF-IDF + Logistic Regression baseline against DistilBERT fine-tuned with Low-Rank Adaptation (LoRA) via the Hugging Face PEFT library. The value of the exercise is not the final accuracy numbers alone—it is the systematic treatment of calibration, truncation failure modes, and semi-supervised self-training that practitioners rarely see benchmarked together in one reproducible notebook.

The tutorial surfaces three dataset pitfalls before any model sees a training example: label ordering in the raw splits (the first and last five labels confirm the data arrives pre-sorted by class, demanding a shuffle before subsampling); review-length skew (the 99th-percentile review length substantially exceeds the 256-token ceiling, with the notebook flagging that a rough words-to-tokens factor of 1.3 means a meaningful fraction of reviews are silently truncated); and zero exact duplicates across train and test boundaries after MD5-level deduplication. Engineers who skip these audits risk reporting inflated or unstable benchmark numbers—a lesson that applies equally when selecting small language models from Hugging Face for production deployment.

Architecture and Training Configuration

The LoRA configuration targets the q_lin and v_lin projection matrices inside DistilBERT-base-uncased, using rank r=16, lora_alpha=32, and lora_dropout=0.05. The classifier head and pre-classifier layer are added to modules_to_save so they train in full precision alongside the low-rank adapter weights. With MAX_LEN=256, N_TRAIN=5000, N_EVAL=2000, two training epochs, a batch size of 16, and a learning rate of 3e-4 with a 6% warmup ratio and weight decay of 0.01, the run is scoped for a free GPU tier. A FULL_RUN flag switches the experiment to 25,000 training examples, 25,000 evaluation examples, and three epochs—the notebook estimates roughly 40 minutes on a T4 for that configuration. Early stopping uses patience of 2 evaluated at epoch boundaries.

The TF-IDF baseline uses unigrams and bigrams (ngram_range=(1,2)), sublinear TF scaling, min_df=2, up to 300,000 features, and LogisticRegression with C=8.0 and up to 2,000 solver iterations. The pipeline trains on cleaned text with <br /> HTML tags stripped before vectorization.

Benchmark Results and Calibration

The notebook reports all three rows in a consolidated summary DataFrame at execution time; the source material does not reproduce fixed numerical outputs, so figures vary by runtime and subsample. What is fixed is the Expected Calibration Error measurement: the tutorial computes ECE over 10 confidence bins using max(p, 1-p) as the confidence signal and renders a reliability diagram. An ECE of 0 indicates perfect calibration. A threshold sweep from 0.05 to 0.95 in 91 steps identifies the cutoff that maximises accuracy on the evaluation set, exposing whether the default 0.5 decision boundary is suboptimal for this class-balanced dataset. The tutorial prints accuracy at the 0.5 threshold alongside best-threshold accuracy explicitly, giving engineers a direct read on how much threshold tuning is worth relative to model improvement.

Truncation Failures and Occlusion Saliency

Section 9 of the notebook is the most practically actionable block. The 600 longest reviews by word count are isolated, and two scoring passes are run: one over the first 180 words and one over the last 180 words. The resulting accuracy gap between head and tail directly answers whether sentiment language is front-loaded or back-loaded in long IMDb reviews. The tutorial states the practical recommendation explicitly: if the tail window wins, feed a head-plus-tail concatenation or raise MAX_LEN rather than accepting left-only truncation.

Occlusion saliency complements this by dropping one word at a time from a 60-word excerpt of the highest-confidence correct prediction, measuring the delta in P(positive) for each omission. The 15 highest-magnitude contributors are plotted as a horizontal bar chart—green for words that push the prediction toward positive and red for those that push it toward negative. This is a model-agnostic interpretability technique that requires no attention-weight access, relevant for practitioners who want token-level explanations without committing to a specific attribution library, and a useful complement to automated prompt optimisation workflows that also need to understand which input tokens drive outputs.

Semi-Supervised Self-Training

The unlabeled IMDb split provides a third dataset partition. The merged DistilBERT model generates probability estimates for 3,000 unlabeled reviews; examples where P(positive) > 0.95 or P(positive) < 0.05 are retained as pseudo-labeled training data. These high-confidence synthetic labels are concatenated with the original 5,000 supervised examples and used to retrain the TF-IDF + Logistic Regression pipeline under identical hyperparameters. The notebook reports the accuracy delta between the original baseline and the pseudo-labeled augmented classifier, along with an explicit caveat that self-training amplifies teacher biases and that gains are bounded by the quality of the teacher model.

The broader signal is architectural decision support: LoRA adapters on a distilled encoder deliver measurably stronger ROC-AUC and calibration than a bag-of-words classifier on 5,000 examples, while still leaving room for pseudo-labeling to recover some of that gap for the classical approach at near-zero inference cost. For teams weighing PEFT fine-tuning against classical pipelines—especially under data-scarcity constraints—this is one of the more complete empirical comparisons available in tutorial form, and the FULL_RUN flag makes it straightforward to rerun at production scale.