Transformer-Based Language-Person Search with Multiple Region Slicing

Hui Li, Jimin Xiao*, Mingjie Sun, Eng Gee Lim, Yao Zhao

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

16 Citations (Scopus)

Abstract

Language-person search is an essential technique for applications like criminal searching, where it is more feasible for a witness to provide language descriptions of a suspect than providing a photo. Most existing works treat the language-person pair as a black-box, neither considering the inner structure in a person picture, nor the correlations between image regions and referring words. In this work, we propose a transformer-based language-person search framework with matching conducted between words and image regions, where a person picture is vertically separated into multiple regions using two different ways, including the overlapped slicing and the key-point-based slicing. The co-attention between linguistic referring words and visual features are evaluated via transformer blocks. Besides the obtained outstanding searching performance, the proposed method enables to provide interpretability by visualizing the co-attention between image parts in the person picture and the corresponding referring words. Without bells and whistles, we achieve the state-of-the-art performance on the CUHK-PEDES dataset with Rank-1 score of 57.67% and the PA100K dataset with mAP of 22.88%, with simple yet elegant design. Code is available on https://github.com/detectiveli/T-MRS.

Original languageEnglish
Pages (from-to)1624-1633
Number of pages10
JournalIEEE Transactions on Circuits and Systems for Video Technology
Volume32
Issue number3
DOIs
Publication statusPublished - 1 Mar 2022

Keywords

  • Transformer
  • language-person search

Cite this