Central to any innovation process is the evaluation of proposed projects and allocation of resources. We investigate whether novel research projects, those deviating from existing research paradigms, are treated with a negative bias in expert evaluations. We analyze the results of a peer review process for medical research grant proposals at a leading medical research university, in which we recruited 142 expert university faculty members to evaluate 150 submissions, resulting in 2,130 randomly assigned proposal-evaluator pair observations. Our results confirm a systematic penalty for novel proposals; a standard deviation increase in novelty drops the expected rank of a proposal by 4.5 percentile points. This discounting is robust to various controls for unobserved proposal quality and alternative explanations. Additional tests suggest information effects rather than strategic effects account for the novelty penalty. Only a minority of the novelty penalty could be related to perceptions of lesser feasibility of novel proposals.
Score: Main score (out of 10) given by evaluators as an overall assessment of the potential impact of a research proposal. Points allocation: Scores given to proposals by evaluators across various dimensions. Feasibility.
The authors included 37 impact-evaluation studies in the review. The studies cover 25 countries across sub-Saharan Africa (nine studies), South Asia (10 studies), Latin America and the Caribbean (10 studies), East Asia and the Pacific (four studies), Eastern Europe (two studies) and North Africa (two studies). Two-thirds of the interventions evaluated came from low-income or lower-middle-income countries. Of the estimates, 80% were based on experimental interventions.