TY - GEN
T1 - Using publications and domain knowledge to build research profiles
T2 - ICICT 2007: International Conference on Information and Communication Technology
AU - Biswas, Humayun Kabir
AU - Hasan, Md Maruf
PY - 2007
Y1 - 2007
N2 - Peer-review has been a common practice for quality control in scholarly publications for decades. The ubiquity of the Internet and, subsequently, the availability of easy-to-use Web-based systems (both free and commercial) has made the peer-review process fast, cost-effective and convenient. In a typical scenario, authors upload papers online and manually assign topic-areas; reviewers also sign up by letting the system know about their area of expertise. A rudimentary Paper-Reviewer matching is usually performed by the system and validated by the Program-Chair (for conferences) or by the Editor-in-Chief (for journals). As argued in relevant literature, the peer-review process suffers from several flaws including author's or reviewer's bias in choosing topic-areas and expertise, as well as inter-reviewer agreement, etc. In this research, we explore automatic reviewer assignment for papers by solely considering the content of the papers and the true profile of the reviewers. In this research, we experimented with three approaches to calculate paper-reviewer relevance using the Vector Space Model. We used a set of 10 papers, 30 reviewers and the real paper-reviewer assignment information from a real-conference; and justifed the result of automatic paper-reviewer assignment based on the above three approaches. We noticed that the overlap between real-assignment and automatic-assignment is poor (with only 55-66% of the reviewers being in common). Such a result was not surprising to us, since we are aware that reviewers often express their frustrations claiming that some papers assigned them are not in line with their preferences and expertise. The data-set we used was rather small and suffered from data-sparseness problem and therefore we tried to analyze the automatic-assignment rationales through unbiased human judgment to identify the effect of the above-mentioned approaches in automatic reviewer assignment. We concluded that combining domain-knowledge with automatically extracted keywords (i.e., ontology-driven topic inference using automatically-extracted keywords) could potentially identify the most relevant candidate-reviewers for a paper.
AB - Peer-review has been a common practice for quality control in scholarly publications for decades. The ubiquity of the Internet and, subsequently, the availability of easy-to-use Web-based systems (both free and commercial) has made the peer-review process fast, cost-effective and convenient. In a typical scenario, authors upload papers online and manually assign topic-areas; reviewers also sign up by letting the system know about their area of expertise. A rudimentary Paper-Reviewer matching is usually performed by the system and validated by the Program-Chair (for conferences) or by the Editor-in-Chief (for journals). As argued in relevant literature, the peer-review process suffers from several flaws including author's or reviewer's bias in choosing topic-areas and expertise, as well as inter-reviewer agreement, etc. In this research, we explore automatic reviewer assignment for papers by solely considering the content of the papers and the true profile of the reviewers. In this research, we experimented with three approaches to calculate paper-reviewer relevance using the Vector Space Model. We used a set of 10 papers, 30 reviewers and the real paper-reviewer assignment information from a real-conference; and justifed the result of automatic paper-reviewer assignment based on the above three approaches. We noticed that the overlap between real-assignment and automatic-assignment is poor (with only 55-66% of the reviewers being in common). Such a result was not surprising to us, since we are aware that reviewers often express their frustrations claiming that some papers assigned them are not in line with their preferences and expertise. The data-set we used was rather small and suffered from data-sparseness problem and therefore we tried to analyze the automatic-assignment rationales through unbiased human judgment to identify the effect of the above-mentioned approaches in automatic reviewer assignment. We concluded that combining domain-knowledge with automatically extracted keywords (i.e., ontology-driven topic inference using automatically-extracted keywords) could potentially identify the most relevant candidate-reviewers for a paper.
UR - http://www.scopus.com/inward/record.url?scp=34748888311&partnerID=8YFLogxK
U2 - 10.1109/ICICT.2007.375347
DO - 10.1109/ICICT.2007.375347
M3 - Conference Proceeding
AN - SCOPUS:34748888311
SN - 9843233948
SN - 9789843233943
T3 - ICICT 2007: Proceedings of International Conference on Information and Communication Technology
SP - 82
EP - 86
BT - ICICT 2007
Y2 - 7 March 2007 through 9 March 2007
ER -