Variation-aware proxy learning for semantic segmentation

초록

In semantic segmentation, accurately modeling intra-class variation is essential for capturing fine-grained details and resolving ambiguity near class boundaries. While existing proxy-based embedding methods represent each class with a single prototype, they struggle to reflect diverse intra-class structures, especially in complex scenes. In this paper, we propose a novel representation learning framework called Variation-Aware Proxy Learning, which introduces a representative proxy to encode shared class semantics and multiple variation vectors to capture fine-grained intra-class variations. These components are integrated through a factorized similarity score, enabling more expressive and discriminative embedding structures. To further enhance learning in ambiguous regions, we introduce focal modulation and design a new Compositional Similarity Loss composed of attraction and repulsion terms that adaptively amplify the contribution of hard examples. Our method is model-agnostic and requires no additional inference-time cost. Extensive experiments across multiple segmentation benchmarks-Cityscapes, COCO-Stuff10k, iSAID and ADE20K-and diverse backbones including CNNs and Transformers, demonstrate consistent improvements in mIoU and boundary precision, particularly in challenging regions with high intra-class variability.

키워드