The Evolution of Generative Inpainting and the Metric Gap
The field of image and video inpainting has transitioned from simple patch-matching techniques to complex deep-learning architectures. Modern diffusion erasers can now handle intricate scenarios, such as removing a moving vehicle while reconstructing the dynamic reflection on a nearby glass building. However, this progress has exposed a fundamental flaw in standardized evaluation: the "ill-posed" nature of the task. Because there is no single "correct" way to fill a hole left by a removed object, comparing a generated result against a single ground-truth reference often leads to misleading scores.
Standard metrics like PSNR, SSIM, and even more recent perceptual metrics like LPIPS (Learned Perceptual Image Patch Similarity) or ReMOVE frequently penalize high-quality generative outputs. If a model generates a realistic patch of grass that differs slightly in texture from the original background, these metrics register a high error, even if a human observer would find the edit perfect. This misalignment has hindered the development of more effective models, as researchers lack a reliable "north star" to guide their optimizations. Xiaomi’s PROVE framework, which has been accepted for presentation at ACM MM 2026, aims to bridge this gap by moving away from global reference-based comparisons and toward local, feature-based distribution matching.
Technical Architecture: Redefining Spatial and Temporal Evaluation
The PROVE framework introduces two primary metrics: RC-S (RemOVal cohErence – Spatial) and RC-T (RemOVal cohErence – Temporal). These metrics are designed to work without the need for a reference video, making them applicable to real-world scenarios where "clean" footage of the background was never captured.
RC-S: Spatial Coherence through DINOv2 Features
RC-S focuses on the spatial integration of the edited region within its surrounding environment. The methodology begins with a connected-component analysis that identifies and isolates independent removal targets within a frame. For each target, the system generates a bounding box that is expanded by one-third of its side length to capture the immediate context of the background.
The core of RC-S lies in its use of DINOv2-giant, a self-supervised vision transformer model. Unlike models trained on supervised classification tasks, DINOv2 provides dense, semantically rich features that are highly sensitive to structural consistency. The framework applies a sliding-window Maximum Mean Discrepancy (MMD) calculation using a Gaussian Radial Basis Function (RBF) kernel. By comparing the distribution of features within the masked (edited) region to the distribution of features in the local background, RC-S can determine if the "fill" matches the surrounding environment in terms of texture, lighting, and semantic logic.
RC-T: Temporal Consistency in Dynamic Scenes
Evaluating video requires more than just frame-by-frame analysis; it necessitates a measure of how consistently the edited region behaves over time. RC-T addresses this by analyzing adjacent frames. To prevent misalignment caused by camera or object motion, adjacent frames are cropped jointly based on the union of their masks.
The MMD is then computed specifically within the intersection of these masks—the region that has been restored in both frames. This approach ensures that the metric is sensitive to "flickering" or drifting textures, which are common failure modes in video inpainting. Ablation studies conducted by the Xiaomi team demonstrated that without this specific cropping and intersection logic, temporal metrics often become "blind" to injected corruptions, failing to distinguish between smooth transitions and jarring artifacts.
PROVE-Bench: A Multi-Tiered Evaluation Standard
To validate the effectiveness of the PROVE metrics, the researchers introduced PROVE-Bench, a two-tier benchmark dataset that provides a rigorous testing ground for object removal models.
- PROVE-M (The Ground-Truth Tier): This tier consists of 80 paired videos. Each pair includes an "input" video containing an object and a "target-free" video of the exact same scene shot within two minutes of the original. These were captured using tripods to ensure maximum alignment. The dataset utilizes SAM3 (Segment Anything Model 3) masks that have been manually refined frame-by-frame. To simulate realistic camera movement, the researchers applied Ken Burns-style motion augmentation synchronously to the triplets (input, target, and mask). Each clip consists of 81 frames at 1080p resolution.
- PROVE-H (The Hard Tier): Recognizing that real-world editing often involves complex environments, PROVE-H includes 100 high-difficulty videos without ground-truth references. These videos feature challenging elements such as flowing water, flames, dense crowds, highly textured terrain, and multi-puddle reflections. Unlike PROVE-M, this tier uses unrefined SAM3 masks to test how models—and the evaluation metrics—handle imperfect input data and fast-moving subjects.
Performance Metrics and Comparative Study
The Xiaomi research team conducted extensive testing to compare PROVE against existing metrics, including ReMOVE and CFD (Content-Fidelity-Diversity). The results were benchmarked against human rankings provided by 20 participants, whose preferences were aggregated using a Borda count.
In terms of correlation with human judgment, RC-S significantly outperformed its predecessors. It achieved an average Kendall’s $tau$ (tau) of 0.59 and a Spearman’s $rho$ (rho) of 0.66. In contrast, ReMOVE managed only 0.26 and 0.29, respectively, while CFD trailed further behind at 0.16 and 0.18. These figures indicate that RC-S is more than twice as effective at predicting which edited image a human would prefer.
Furthermore, RC-S demonstrated exceptional robustness on the RORD-Val dataset. In a test involving blurred and region-swapped variants of images, RC-S correctly preferred the clean, high-quality image in 100% of the cases. ReMOVE and CFD showed much lower accuracy, with CFD failing to distinguish between high-quality edits and blurred versions nearly 50% of the time.
Ablation studies also highlighted the efficiency of the PROVE framework. Despite its reliance on the massive DINOv2-giant model, RC-S is 13.7 times faster than CFD. The study also confirmed that DINOv2 is superior for this task than newer iterations like DINOv3 or other models like SAM, as DINOv2’s feature space appears uniquely suited for measuring local distribution shifts in natural images.
Industry Implications and Future Trajectories
The release of PROVE marks a significant milestone for the mobile and consumer electronics industry, particularly for companies like Xiaomi that integrate AI-powered photo and video editing tools directly into their smartphone operating systems. The ability to objectively measure the performance of "Magic Eraser" features allows for more rapid iteration and better quality control during the development phase.
From a broader perspective, PROVE addresses a critical bottleneck in the "AI for Content Creation" (AIGC) pipeline. As generative models become more capable of creating synthetic content, the industry requires standardized, perception-aligned tools to ensure that this content meets human standards of quality. By open-sourcing the PROVE framework under the Apache 2.0 license, Xiaomi is providing the research community with a standardized harness that could become the new industry benchmark for inpainting tasks.
The metrics also have implications for the detection of AI-generated content. Because RC-S and RC-T are highly sensitive to local distribution mismatches in the deep feature space, they could potentially be adapted to identify "forged" or edited regions in videos where the generative model has failed to perfectly match the underlying statistical properties of the original footage.
Implementation and Open-Source Accessibility
The PROVE framework is currently available as a PyTorch repository on GitHub. It is designed to be an "evaluation harness" rather than a user-facing product feature. Developers and researchers can integrate it into their workflows via a command-line interface.
The technical requirements for running PROVE include Python 3.10 or higher, PyTorch 2.6+, and Transformers 4.51+. The system also requires the pre-trained weights for DINOv2-giant. To use the metrics, users must provide the video frames along with binary masks where white pixels indicate the area where an object has been removed.
Current leaderboards on the PROVE-Bench show that the SVOR (1.3B parameter) model currently leads in spatial coherence (RC-S), while the EffectErase model holds the top spot for temporal consistency (RC-T). These benchmarks provide a clear hierarchy for existing models and set a high bar for future developments in the field of video object removal.
As the industry moves toward ACM MM 2026, the PROVE framework is expected to see widespread adoption among researchers aiming to validate the perceptual quality of their generative models. By shifting the focus from pixel-perfect reconstruction to perceptual coherence, Xiaomi has provided a more nuanced and accurate way to judge the "invisible" art of object removal.
