Using publications and domain knowledge to build research profiles: An application in automatic reviewer assignment

Humayun Kabir Biswas*, Md Maruf Hasan

*Corresponding author for this work

Research output: Chapter in Book or Report/Conference proceedingConference Proceedingpeer-review

29 Citations (Scopus)

Abstract

Peer-review has been a common practice for quality control in scholarly publications for decades. The ubiquity of the Internet and, subsequently, the availability of easy-to-use Web-based systems (both free and commercial) has made the peer-review process fast, cost-effective and convenient. In a typical scenario, authors upload papers online and manually assign topic-areas; reviewers also sign up by letting the system know about their area of expertise. A rudimentary Paper-Reviewer matching is usually performed by the system and validated by the Program-Chair (for conferences) or by the Editor-in-Chief (for journals). As argued in relevant literature, the peer-review process suffers from several flaws including author's or reviewer's bias in choosing topic-areas and expertise, as well as inter-reviewer agreement, etc. In this research, we explore automatic reviewer assignment for papers by solely considering the content of the papers and the true profile of the reviewers. In this research, we experimented with three approaches to calculate paper-reviewer relevance using the Vector Space Model. We used a set of 10 papers, 30 reviewers and the real paper-reviewer assignment information from a real-conference; and justifed the result of automatic paper-reviewer assignment based on the above three approaches. We noticed that the overlap between real-assignment and automatic-assignment is poor (with only 55-66% of the reviewers being in common). Such a result was not surprising to us, since we are aware that reviewers often express their frustrations claiming that some papers assigned them are not in line with their preferences and expertise. The data-set we used was rather small and suffered from data-sparseness problem and therefore we tried to analyze the automatic-assignment rationales through unbiased human judgment to identify the effect of the above-mentioned approaches in automatic reviewer assignment. We concluded that combining domain-knowledge with automatically extracted keywords (i.e., ontology-driven topic inference using automatically-extracted keywords) could potentially identify the most relevant candidate-reviewers for a paper.

Original languageEnglish
Title of host publicationICICT 2007
Subtitle of host publicationProceedings of International Conference on Information and Communication Technology
Pages82-86
Number of pages5
DOIs
Publication statusPublished - 2007
Externally publishedYes
EventICICT 2007: International Conference on Information and Communication Technology - Dhaka, Bangladesh
Duration: 7 Mar 20079 Mar 2007

Publication series

NameICICT 2007: Proceedings of International Conference on Information and Communication Technology

Conference

ConferenceICICT 2007: International Conference on Information and Communication Technology
Country/TerritoryBangladesh
CityDhaka
Period7/03/079/03/07

Cite this