PROVE: Xiaomi's Perception-Aligned Video Removal Metrics RC-S and RC-T
In this article
Object removal models have improved faster than the metrics used to judge them. Diffusion erasers now reconstruct shadows, reflections, and occluded structure convincingly, yet PSNR, SSIM, LPIPS, ReMOVE, and CFD frequently rank their outputs incorrectly. The root cause is structural: erasure is an ill-posed, one-to-many task, so no single ground truth exists. A team from MiLM Plus at Xiaomi Inc. has released PROVE (Perceptual RemOVal cohErence), accepted at ACM MM 2026, to close that gap. PROVE introduces two reference-free metrics — RC-S for spatial coherence and RC-T for temporal consistency — alongside PROVE-Bench, a two-tier real-world video dataset. Code ships under Apache 2.0; the dataset is on Hugging Face.
Why Existing Metrics Fail
On ROSE-Bench, progressively blurring the masked region causes neither ReMOVE nor CFD to degrade — both eventually surpass their unblurred baselines, meaning they actively reward a worse edit. Under the same blur perturbation on RORD-Val, ReMOVE identifies the clean image as preferable in only 60.06% of cases and CFD in 49.27%, effectively near-chance. CFD's SAM-based hallucination term compounds the problem: a correctly restored bicycle seat is flagged as a hallucination, scoring worse than the unedited input.
Full-reference metrics carry the opposite bias. Cutting diffusion inference steps improves PSNR and SSIM even as perceptual quality collapses — regression to the mean is rewarded over genuine erasure. Residual shadows occupy few pixels and incur almost no PSNR penalty. Temporal Consistency and Temporal Flickering operate on full-frame features, diluting the edited region by unchanged background; injected Random Drop and Random Replace corruptions on DAVIS barely move them, and sometimes move them in the wrong direction.
How RC-S and RC-T Are Computed
Both metrics use sliding-window Maximum Mean Discrepancy over DINOv2 features, evaluated locally around the erased region rather than globally across the frame.
RC-S splits the mask into independent connected components. Each component's bounding box is expanded by one-third of its side length, the crop is passed through DINOv2, and the mask is downsampled to feature resolution. A w×w window slides across the feature map, computing squared MMD with a Gaussian RBF kernel between masked and local background features at each position. Scores are averaged per target, then across targets. A blurred patch has a different feature distribution from its surroundings, and MMD catches that.
RC-T crops adjacent frames jointly under the union of their masks to prevent spatial misalignment, then computes MMD only inside the intersection — the region restored in both frames. An ablation confirms that removing the crop step makes RC-T insensitive to injected corruption.
Ablations quantify each design choice: DINOv2 produces 0.59 average Kendall's τ versus 0.51 for DINOv3 and 0.44 for SAM; dropping the sliding window costs 0.11 τ; replacing MMD with cosine similarity costs 0.07 τ.
Benchmark Validation Against Human Rankings
Twenty participants ranked model outputs per sample, with Borda count aggregation producing the ground-truth ordering. Results across six benchmarks are unambiguous.
| Metric | Avg Kendall's τ | Avg Spearman's ρ | Runtime (RTX 4090) | Benchmarks Ranked First |
|---|---|---|---|---|
| RC-S | 0.59 | 0.66 | 134.6 ms/frame | 5 of 6 |
| ReMOVE | 0.26 | 0.29 | 180.7 ms/frame | — |
| CFD | 0.16 | 0.18 | 1842.8 ms/frame | — |
| Background-only FR variants | negative (most datasets) | negative (most datasets) | varies | 0 of 6 |
On RORD-Val, RC-S prefers the clean image over blurred and region-swapped variants in 100% of cases. RC-S is also 13.7× cheaper than CFD at 134.6 ms/frame, making nightly CI gates on RC-S and RC-T practically feasible without paired ground truth — a meaningful advantage for teams evaluating models across large editing catalogs where reference video collection is prohibitively expensive.
PROVE-Bench and Current Leaderboard
PROVE-Bench splits into two tiers. PROVE-M contains 80 paired videos: tripod-captured input footage and target-free footage shot within a two-minute window, with SAM3 masks refined frame by frame, three-stage quality control, and Ken Burns-style motion augmentation applied synchronously to the input-target-mask triplet. Each clip runs 81 frames at 1080p. PROVE-H adds 100 hard videos without ground truth, targeting scenarios that stress existing models — crowds, flowing water, flames, textured terrain, multi-puddle reflections, and fast motion — using unrefined SAM3 masks intentionally to reflect real-world annotation noise.
On the public leaderboard, SVOR at 1.3B parameters leads the combined RC-S ranking at 0.5197, while EffectErase leads RC-T at 0.2525. The codebase requires Python 3.10+, PyTorch 2.6+, Transformers 4.51+, and DINOv2-giant weights; evaluation is invoked through run_prove_metrics.py, with binary masks where white pixels mark the removed object.
PROVE establishes a concrete, reproducible evaluation contract for a task that previously lacked one. As benchmark gaps appear across fast-moving generative domains, PROVE gives researchers and applied engineering teams a human-aligned signal that is cheaper, faster, and demonstrably more accurate than the alternatives it replaces.