Performance Evaluation of a Bi-directional Feedback Approach for Expertise Recommendation in Academic Research

.


Introduction
Recommender systems are described as systems that provide users with an ordered list of items and information that help them to decide which items to consider or look at, based on the individual user's preferences [12]. The word "item" is the general term used to denote what the system recommends to users. A recommender system normally focuses on a specific type of item (for example, music, movies or news) and accordingly its design, its graphical user interface, and the core recommendation technique used to generate the recommendations are all customized to provide useful and effective suggestions for that specific type of item [10]. Recommender systems have been deployed in various fields of life such as e-commerce and academia. It has contributed to increase in sales and greater customer satisfaction in e-commerce. This has culminated in recommender systems appearing as a significant feature in most websites pertaining to e-commerce and social networks. It is also deployed in the academia for expertise recommendation. An expertise recommender system offers a means by which scarce resources in the form of human experts can be identified and accessed [16]. The Expertise Recommender (ER) is a system designed to facilitate identifying individuals who have the necessary expertise to solve a specific problem [5]. In expertise recommendation systems, the target is "expert" in place of "items", such as movies or books as in the case of general recommender systems. Some websites that provide these services include www.linkedin.com as well as www.researchgate.net among others which helps researchers interact with one another. Expertise recommendation systems have been shown in literature to be a useful tool in knowledge management that is very essential in academic research.
The tacit nature of expertise has posed a serious challenge in terms of how to differentiate between the actual expert and the self-acclaimed expert. Little attention has also been given to types of users and their ability to provide reliable judgment about the experts. Also, the use of feedback mechanism for evaluation of recommender systems has been unidirectional in the sense that the recommender system uses only the feedback provided by users against items such as movies and music which are inanimate for evaluating recommended items. However, this evaluation approach is deficient when the item to be recommended is human. This is 28 Performance Evaluation of a Bi-directional Feedback Approach for Expertise Recommendation in Academic Research because humans possess the ability to speak and reason as in the case of human experts. Thus, the recommended experts are not given the privilege to also express their opinions concerning those who seek their expertise. This research is therefore premised on the need to have a bi-directional or two-way feedback mechanism which collates feedback from both the expertise seeker and expertise provider and uses it to enhance recommendation.
The study is limited to the recommendation of experts among academic researchers in tertiary institutions; the experts possess required expertise in their respective research areas. In the study, an expert is defined as one who has obtained a Doctor of Philosophy (Ph.D.) degree in the target research area in addition to research publications. Thus, as a Ph.D. degree holder, the expert has obtained the highest qualification and as such may require little or no supervision when carrying out any research in the same area in which he obtained a Ph.D. qualification. Anyone with qualification less than Ph.D. degree is regarded as a trainee. The level of expertise is determined by the number of publications in the field.
The rest of this paper is organized as follows. Section 2.0 is concerned with the materials and methods used to actualize the proposed bi-directional feedback. Section 3.0 deals with results and discussions while section 4.0 concludes the paper.

Approaches to Recommender Systems Development
Recommender systems are now popular both commercially and in the research community, where many approaches have been suggested for providing recommendations [14]. In many cases, a system designer that wishes to employ a recommendation system must choose between a set of candidate approaches. These approaches have in turn led to different classifications depending on the perspective of the recommender system's developer.
According to [9], recommender systems can be classified broadly into six categories depending on the information they utilize in recommending items. These include content-based filtering systems, collaborative filtering systems, demographic filtering systems, knowledge-based recommender systems, utility-based recommender systems and hybrid recommender systems. However, three major classifications of recommender system are widely publicized in literature and these classifications are based on the design approach. They are the content-based approach, the collaborative filtering approach and the hybrid approach.
The content-based approach to recommendation has its roots in the information retrieval community and employs many of the same techniques such that text documents for instance are recommended based on a comparison between their content and a user profile [3]. Two important sub-problems in designing a content-based recommender system have been identified in [8]. The first is finding a representation of documents. The second is to create a profile that allows for unseen documents to be recommended. In view of these sub-problems, content-based filtering system are made to select items based on the correlation between the content of the items and the user's preferences as opposed to a collaborative filtering system that chooses items based on the correlation between people with similar preferences.
Collaborative filtering (CF) is a popular recommendation algorithm that bases its predictions and recommendations on the ratings or behavior of other users in the system. The term "collaborative filtering" was coined by Doug Terry at Xerox PARC in the early nineties [9]. It is the process of evaluating information using the opinion of other people [13]. Either collaborative or content-based filtering has its shortcomings when deployed independently. The need to provide more quality recommendations and overcome the weaknesses of the different independent methods has led to the combination of different techniques in a manner called hybridization [5].

Related Works
Academic research is a unique and dynamic area that requires special attention. It is therefore appropriate that recommender system, as applied in other spheres of life, would require adjustments to fit into the domain of academic research, taking cognizance of its uniqueness and dynamism. A survey of available literature on expertise recommendation shows that academic expertise systems are less common, but have become more prevalent due to the large amount of available networked data on academics and their publications [6].
However, certain studies have concentrated on recommending experts within the academia. From the social network perspective, [16] developed a system called ArnetMiner which aims at extracting and mining academic social networks. It employs a unidirectional feedback mechanism which only collects suggestions from the users on how to assign papers to various researchers and not as a means of evaluating the claimed or extracted information about the expertise of the researcher. The study noted that 'people search' is an important problem because users often require identifying relevant people rather than documents. For instance, in an academic research setting, researchers may seek other researchers with complementary skills for a given project.
Reference [7] developed a system called AcademTech. AcademTech is an example of an academic expertise finder that draws on data from the web. It was built on the Terrier Information Retrieval platform, and claims to be the first expert search system to use a faceted search interface. However, the available facets are quite limited -university, location, and number of publications.
When dealing with expertise recommendation in the area of academic research, the relevance or quality of publications of the researchers is usually considered. Thus, the keywords in each research publication is extracted and utilized to represent knowledge level of the researcher. However, the knowledge of the researcher is not wholesomely represented through the use of the keywords. This is because the keywords are basically used as a link to their main knowledge-base which is embedded. Nonetheless, in practice, researchers' expertise in a topical area could be aggregated and measured by their publications, projects, and judgments given by their colleagues [15]. Adopting the keyword approach to expertise representation, [17] proposed what was referred to as a "triangulated approach" to expertise recommendation. This approach on the first part employs data mining techniques to gather relevant information about the experts. The second part was for the outputs of the data mining to be reviewed by the experts themselves, providing a second means of validation. The third part to the triangulated approach was the use of profiles and the gathering of feedback from searchers to ensure that recommendations provided were satisfactory. The system used automated searching as a foundation from which the initial data was captured and against which the data was regularly cross checked. Reference [5] developed the 'Virtu' academic expertise finder system which was built on the Apache Solr platform using data from Mendeley, a social network and bibliographic management system. It takes a task-based approach to expertise recommendation; exposing and giving the user control over dimensions of expertise that are more or less desirable depending on the type of expert-finding task. The system supports information interaction and exploration through a number of browsing and filtering tools, including facets and sliders. Following this concept of using keywords as a representation of the expertise of the researchers, a two-layer network model has been proposed [20], which combines the social network and semantic expertise information. These layers are concept layer and the researcher layer. The concept layer contains expertise of researchers in the form of words or phrases. The researcher layer contains profile of researchers. A link between concept and researcher layer means that a researcher shows expertise in the corresponding area.
So far, available literature has shown that expertise management can be internetworked or intra-networked. The internetworked approach implies that experts are located from outside the domain network using agents [19] or from the social network perspective [11]. However, the intra-networked approach involves searching for experts within a particular system domain [1], which is the approach adopted in this study.

Materials and Methods
The study adopted a database driven approach to achieve recommendation. The proposed model is as shown in Figure  1. The model provides a bidirectional feedback structure that is utilized for the evaluation of users and experts. It is composed of expertise seeker, expertise provider, recommendation engine, expertise, expert and trainee. These are explained as follows.
a) Expertise seeker: The expertise seeker, in this context, is a user of the system, who is in need of an expert in a particular field of interest and thus searches through the system to locate the expert who possesses the desired expertise.
b) Expertise provider: This is the expert whose expertise is being sought by an expertise seeker. Thus, the expert renders the desired service as may be stated by the expertise seeker.
c) Recommendation engine: This generates a list of available experts in the system with respect to the field specified by the expertise seeker. This is achieved using system criteria. It is also responsible for manipulating the feedback provided by the users in form of ratings by mappings feedback to the target expert within the system. d) Expertise: This is the requisite knowledge possessed by the expert and it is represented by the keywords in the research publications of the expert. The frequency of occurrence of each of the keywords in the publications is calculated during recommendation. e) Expert: This is the expertise provider. To be identified as an expert by the system, a user must have obtained Doctor of Philosophy (Ph.D.) degree in academic qualification in addition to research publications. f) Trainee: Any registered user of the system who possesses any academic qualification less than Ph.D. is automatically confined to the trainee compartment at the end of registration.
The operation of the model is shown through the activity diagram in Figure 2. A user accesses the home page of the system through the browser. If the user has an existing account, he logs into the system or creates an account if the user is new. When logged in, the system authenticates the user and determines the category -that is the user's class. Two classes of users are identified in the diagram namelythe expert and the trainee. It is assumed that the experts are those who have Doctor of Philosophy (Ph.D.) degrees and above while the trainees are those below the level of Ph.D. by academic qualification in their research area. Information required from the users includes, but not limited to: Name, Country, E-mail, Institution, Department, Qualifications, and Preferred Language. Major information required from the research publications of the experts include: publication title, keywords in each publication, authors and their e-mail addresses. When all necessary personal information about the experts has been provided, the system registers and allows the user access to the system during which he can select the task to undertake. The experts are identified by the system based on available user information; especially the qualification and willingness to collaborate or mentor.

30
Performance Evaluation of a Bi-directional Feedback Approach for Expertise Recommendation in Academic Research  Any user that is not in the category of experts is automatically regarded as a trainee by the system. This is a kind of lock that forces the system to recommend desirable experts at all times leading to high precision value. Depending on the task selected, recommendations are made following some certain criteria. These criteria include: number of collaborations, rating score and keyword frequency which is derived from the research publications of the experts in the target field. The system generates recommendation for collaboration using Equation (1) while Equation (2) is used to generate recommendation for mentoring. Algorithm 1 also shows the recommendation procedure. (1) where R EC represents Recommendation; R s represents Ratings score; N c represents Number of collaborations; and K f represents Keyword frequency. However, when a user selects mentoring, the recommendation criteria change to Equation 2.
where N m is the number of mentoring while other parameters maintain their meaning as specified above. The expert with highest R EC value is ranked first while others follow in descending order.

Proposed System's Technology
The proposed system is built on three-tier architecture namely: presentation tier, middle tier and data tier. The presentation tier interacts with the middle tier (application layer) through a client application programming interface (API). It carries out data layout and formatting functions. The middle (application) tier is the nucleus of the system and provides various services such as language analysis, instance pooling of knowledge bases, and management of the distribution of tasks across computing systems. It is accessed through client application programming interfaces that are provided for writing client code that interacts with the system. Finally, the data tier is responsible for storing configuration information. The application tier was implemented using Hypertext Preprocessor (PHP 5.2.6) technology which is embedded in hypertext markup codes. This tier presents the user interface and allows data input, simple validation and analysis. Finally, the data tier was implemented using the Structured Query Language (SQL) which is built on a relational database server MySQL 5.0.51b. The technology diagram is as shown in Figure 3.

Assumptions of the Proposed System
The assumptions made in this study are:i.
An academic staff researcher in tertiary institution that does not possess a Doctor of Philosophy (Ph.D.) degree in his field of research is still a trainee researcher and as such cannot be an authority in that research field. This assumption is informed by the fact that such a researcher must be subject to the supervisory expertise of a senior researcher in his field, who must have obtained a Doctor of Philosophy (Ph.D.) degree in order to qualify for the award of higher degree. ii.
An academic researcher who has obtained a Doctor of Philosophy (Ph.D.) degree in a particular research field is considered to be a trained researcher who may independently carry out a research in his field of expertise or require little external input. Hence, such a researcher is considered an expert in that particular field. iii.
The number of times a researcher collaborates with others in different research activities is an indication of his willingness to engage in more collaborative work. iv.
The frequency of occurrence of a keyword from a particular field in the publications of an academic researcher is an indication of his expertise in that field with respect to the keyword [20].

The Proposed Bi-directional Feedback Approach
When a research is initiated by a user, either for mentoring or a collaborative purpose, the system notes that an activity has been initiated and identifies the initiator. At the end of each activity, both the expert and the initiator of the activity are expected to provide feedback in form of ratings and these ratings are used to judge the competence of the experts in their respective field of expertise as well as the contribution of the user to the research activity.
Each user of the system has two different rating values stored against his User_Id. One of the ratings is provided against his User_Id when he acts as an initiator of the research (that is the expertise seeker) and the other is provided against his User_Id when the system recommends him to another initiator in which case he acts as an expert (that is expertise provider) as shown in Figure 4. The expertise rating is used to evaluate the expert's claim of expertise in a particular field while the initiator rating is used to evaluate his conduct and contribution in the course of the research activity. The users are made to see remarks such as Excellent, Very Good, Good, Fair, and Poor on the interface when providing feedback. These remarks are transformed to figures on the database which are then calculated during recommendation.
The values attached to each remark differ between expert and trainee by the value 0.05 as shown in Table 1. This difference though may not represent the actual values in reality but is introduced to underscore the variation in knowledge level among academics as reflected through their qualification. The facets for interacting with the system are shown in Figure

Results and Discussions
Precision and Recall were used as performance metrics. In this study, recall represents the coverage of relevant experts the recommender system can obtain. In other words, it measures the capacity of the system to obtain all the relevant experts present in the database. On the other hand, precision shows the recommender system's capacity for showing only relevant experts, while trying to minimize mixing them with irrelevant ones. The equations for both precision and recall derived from the confusion matrix are given in Equations (3) and Equation (4)  where: TP represents relevant experts recommended; FN represents irrelevant experts recommended; FP represents relevant experts not recommended; and TN represents irrelevant experts not recommended.

Dataset Description
Two datasets were used for the system testing. The proceedings contained about one hundred and forty-eight authors and fifty-eight publications in all which were used for initial testing. A list of all authors, their e-mail addresses, titles of their publications, keywords from the publications and their qualifications were extracted from these conference proceedings. The e-mail address was used as primary key for the publications. Those who did not supply full information about themselves in their publications were arbitrarily assigned such missing information for the purpose of system testing.
The second dataset was downloaded from http://lis.irb.hr/challenge/files/dataset/VL.NetRSChallenge Dataset.zip. It contains data on authors registered on VideoLectures.Net and their information. A total of eight thousand and ninety-two (8,092) records of authors and their e-mail addresses are available in the dataset. The qualifications of the authors were manually assigned because such information was not included in the original dataset.
To evaluate the system, a total of twenty iterations were carried out using the second dataset. Precision and recall were taken for top-n at 5,10,15,20,25,30,35,40,45, and 50 respectively. A total of two hundred and thirty-eight(238) users, out of eight thousand nine hundred and two (8,902) users on specific field 'data mining', were designated as experts in the database against which the search was conducted. The first set of ten iterations was executed on the proposed system which deployed the criteria proposed in this study to achieve expertise recommendation.  228, 223, 218, 213, 208, 203, 198, 193, and 188. The FP value is zero (0) for all. For each of the iterations, precision and recall as well as average precision and average recall were calculated and recorded as shown in Table 2. The second set of ten iterations was carried out using the recommendation procedure in Algorithm 2 adapted from the existing expertise recommender system of [21]. The choice of [21] was because other recent studies for expertise recommendation in academic research did not employ feedback for evaluating expertise. The TP values are 2, 4, 7, 9, 11, 13, 16, 19, 21, and  The precision and recall as well as average precision and average recall for each of the second set of iteration were calculated and recorded as shown in Table 3. Furthermore, we compare the performance of both systems by computing the Receiver Operating Characteristics (ROC). Reference [22] stated that "ROC curves emphasize the proportion of items that are not preferred that end up being recommended" and it is calculated using Equation 5. The difference ROC values for the proposed and existing system are given in Table 2 and Table 3     A comparison of the two systems indicates that the proposed system performed better in coverage with average precision value of 1 than the existing system which has average precision value of 0.448. Also, higher recall was recorded by the proposed system compared to that of the existing system which is an indication that the proposed system offered a better quality recommendation than the existing system. This comparison is further illustrated using precision-precision chart and recall-recall chart for both existing and proposed systems as shown in Figure 9 and Figure 10, respectively. To compute the quantitative performance difference between the two systems, the harmonic mean for both the existing and proposed systems was calculated using F1 Measure and the formula is given in Equation 6. The harmonic mean gives a combined output of both precision and recall for each system.
The harmonic mean for the existing system is given as: The harmonic mean for the proposed system is given as: This implies that the proposed system outperformed the existing system by 0.025 which is equivalent to 2.5%.

Conclusions
This paper shows a step-by-step and consistent procedure towards actualizing expertise recommendation in academic research. It has provided a bidirectional feedback model that will ensure viable means of evaluating experts as well as users so as to prove their competence and genuineness. The user classification approach introduces a refinement in the expertise recommendation system, as reflected through average ROC value of zero, by reducing the knowledge gap between recommended experts and the expert seeker and this enhances collaboration in academic research. This research also underscores the need to pay special attention to expertise recommendation as a peculiar case of recommendation systems especially as regards expertise in the area of academic research. The results from this study have shown that deploying user classification in expertise recommendation system in general can be of great help in filtering experts from non-experts and especially in the academic domain. Again, the study went further to show that since seeking for expertise and providing expertise are human-to-human activities, it is expedient that the feedback structure deployed for such systems should be bidirectional.
In future, the study will develop a quantitative measure for the bidirectional model and test for acceptance using existing technology acceptance model [18], [2]. Furthermore, the security of users on the network with respect to user and usage information will be addressed. Finally, this paper has introduced an invaluable means of ensuring reliable recommendation of experts in academic research which would translate into enhanced collaborative research activities among academics.