Looking through the codebook: Generative anomaly segmentation with multi-contrastive learning

초록

Most existing semantic segmentation models are based on discriminative approaches. Such models often fail to detect out-of-distribution (OoD) objects because they primarily learn class-specific decision boundaries without explicitly modeling the underlying data distribution, which leads to overconfident misclassification of unseen ob jects. In contrast, generative models aim to capture the underlying data distribution, resulting in more effective anomaly detection. Recent methods still rely on discriminative segmentation networks with generative modules used only as auxiliary components, which prevents them from leveraging generative modeling for pixel-wise like lihood estimation and effective OoD separation. Among generative models, codebook-based approaches such as vector-quantized variational autoencoders (VQ-VAEs) discretize the latent space into a finite set of codevectors, enabling a fully generative formulation in which each pixel is modeled by its likelihood under the learned la tent distribution. Motivated by this property, we propose a purely generative anomaly segmentation method that integrates a VQ-VAE, a weighted top-K scoring strategy, and multi-contrastive learning. By treating the segmenta tion class-specific codevectors of VQ-VAE as in-distribution (ID) representations, we introduce a codevector-wise top-K criterion. This criterion scores each pixel based on its top-K nearest codevectors and their associated scores, thereby reflecting relative similarities within the same classes. Furthermore, we implement a codevector-based multi-contrastive learning strategy with specially sampled void labels. This implementation effectively structures the latent space among classes and ensures that anomalies are not aligned with ID codevectors. Extensive exper iments demonstrate that the proposed method detects anomalies effectively and performs robustly in single-and cross-domain scenarios.

키워드