Skip to main navigation Skip to search Skip to main content

Frozen CLIP: A Strong Backbone for Weakly Supervised Semantic Segmentation

  • Bingfeng Zhang
  • , Siyue Yu*
  • , Yunchao Wei
  • , Yao Zhao
  • , Jimin Xiao*
  • *Corresponding author for this work
    • China University of Petroleum (East China)
    • Beijing Jiaotong University

    Research output: Contribution to journal › Conference article › peer-review

    97 Citations (Scopus)

    Abstract

    Weakly supervised semantic segmentation has witnessed great achievements with image-level labels. Several recent approaches use the CLIP model to generate pseudo labels for training an individual segmentation model, while there is no attempt to apply the CLIP model as the backbone to directly segment objects with image-level labels. In this paper, we propose WeCLIP, a CLIP-based single-stage pipeline, for weakly supervised semantic segmentation. Specifically, the frozen CLIP model is applied as the backbone for semantic feature extraction, and a new decoder is designed to interpret extracted semantic features for final prediction. Meanwhile, we utilize the above frozen backbone to generate pseudo labels for training the decoder. Such labels cannot be optimized during training. We then propose a refinement module (RFM) to rectify them dynamically. Our architecture enforces the proposed decoder and RFM to benefit from each other to boost the final performance. Extensive experiments show that our approach significantly outperforms other approaches with less training cost. Additionally, our WeCLIP also obtains promising results for fully supervised settings. The code is available at https://github.com/zbf1991/WeCLIP.

    Original languageEnglish
    Pages (from-to)3796-3806
    Number of pages11
    JournalProceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
    DOIs
    Publication statusPublished - 2024
    Event2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024 - Seattle, United States
    Duration: 16 Jun 2024 → 22 Jun 2024

    Keywords

    • CLIP
    • Semantic Segmentation
    • Weakly-supervised

    Fingerprint

    Dive into the research topics of 'Frozen CLIP: A Strong Backbone for Weakly Supervised Semantic Segmentation'. Together they form a unique fingerprint.

    Cite this