Fully and Weakly Supervised Referring Expression Segmentation With End-to-End Learning

Hui Li, Mingjie Sun, Jimin Xiao, Eng Gee Lim, Yao Zhao

Research output: Contribution to journalArticlepeer-review

3 Citations (Scopus)


Referring Expression Segmentation (RES), which is aimed at localizing and segmenting the target according to the given language expression, has drawn increasing attention. Existing methods jointly consider the localization and segmentation steps, which rely on the fused visual and linguistic features for both steps. We argue that the conflict between the purpose of identifying an object and generating a mask limits the RES performance. To solve this problem, we propose a parallel position-kernel-segmentation pipeline to better isolate and then interact the localization and segmentation steps. In our pipeline, linguistic information will not directly contaminate the visual feature for segmentation. Specifically, the localization step localizes the target object in the image based on the referring expression, and then the visual kernel obtained from the localization step guides the segmentation step. This pipeline also enables us to train RES in a weakly-supervised way, where the pixel-level segmentation labels are replaced by click annotations on center and corner points. The position head is fully-supervised and trained with the click annotations as supervision, and the segmentation head is trained with weakly-supervised segmentation losses. To validate our framework on a weakly-supervised setting, we annotated three RES benchmark datasets (RefCOCO, RefCOCO+ and RefCOCOg) with click annotations. Our method is simple but surprisingly effective, outperforming all previous state-of-the-art RES methods on fully- and weakly-supervised settings by a large margin. The code and dataset will be released on https://github.com/detectiveli/PKS.git.

Original languageEnglish
Pages (from-to)5999-6012
Number of pages14
JournalIEEE Transactions on Circuits and Systems for Video Technology
Issue number10
Publication statusPublished - 1 Oct 2023


  • Referring expression segmentation
  • end-to-end
  • position-kernel-segmentation
  • weakly-supervised


Dive into the research topics of 'Fully and Weakly Supervised Referring Expression Segmentation With End-to-End Learning'. Together they form a unique fingerprint.

Cite this