Energy-Based Video Captioning with Statistical Energy Distance: A Unified Framework

Abstract

Encoder-decoder models for video captioning trained with token-level cross-entropy often exhibit weak semantic grounding and sensitivity to spurious correlations, particularly when audio provides misleading cues. We introduce EBM-EoD-VC, a unified framework that integrates sample-level energy-based modeling with distribution-level energy statistics into a standard Transformer pipeline. The model learns a joint embedding space regularized by energy distance, distance correlation, and partial distance correlation to control for audio as a confounder. On the compact MSVD dataset, a well-tuned cross-entropy baseline remains competitive. On the larger, noisier MSR-VTT dataset, our method achieves consistent gains in CIDEr and SPICE, especially when audio is used. These results demonstrate that energy-based regularization tightens video-caption alignment and improves semantic adequacy, with benefits most pronounced in open-domain, multimodal settings.