Measuring Researcher-Production in Information Systems

Cecil Chua, Lan Cao, Karlene Cousins, Detmar Straub
2002 Journal of the AIS  
While many studies have assessed IS researcher-production, most have focused on either ranking IS journals or assessing prolific researchers using a restricted time frame and a small "basket" of journals (i.e., those journals selected for sampling). We found no research that has assessed the IS specificity of journals (i.e., the suitability of journals for publishing IS research) nor any that evaluated IS researcher-production measures. Based on a coding of over 26,000 articles and more than
more » ... 00 authors, this study attempts such an evaluation by (1) determining the rate of publication of IS researchers in 58 journals perceived by at least one IS institution as IS specific, (2) profiling prolific and typical IS researchers using descriptive statistics, (3) evaluating the convergent validity of various researcher-production measures, (4) assessing the reliability of these researcher-production measures by varying baskets of journals and time periods, and (5) comparing the sensitivity of measures across prolific and typical researchers. The study demonstrates that many journals perceived to be of high quality by IS researchers are not specifically targeted to information systems. Changing the evaluation procedure has a significant impact on measures of typical and prolific IS researchers. For typical IS researchers, measures of production are strongly convergent and are not sensitive to changes in journal baskets. However, for prolific researchers, measures of production are not convergent and highly sensitive to changes in journal baskets. The evaluation of both prolific and typical IS researchers is also highly sensitive to temporal effects. The differences in convergent validity and reliability demonstrate that prolific researchers are more sensitive to minor variations in the assessment procedure. Based on the empirical findings, the study closes with recommendations both for the evaluation of researcher-production and for developing institutional target journal lists, i.e., lists of journals viewed favorably by an institution. 2 The number of methodologies employed appears to be related to the size of the faculty. 3 Allen Lee, former editor-in-chief of MIS Quarterly, prefers the term "contributing disciplines," indicating that other disciplines add to IS research, but are not the only basis (reference) for our work. To promote future use of this term, we make mention of this useful substitution here.
doi:10.17705/1jais.00026 fatcat:kpixts7vtnbmlcg7krmhjchvbq