Academic Merit 2.0

Talking about shortcomings of impact factor (IF) as a measure of the quality of the work researchers do often dies out at ‘yeah, but what better system should be used’? In a world that favors productivity above all, such an idea is likable. To judge academics on simple measures as IF is convenient.

At the University of Utrecht, this is not the case. The Open Science officer, professor Paul Boselie, is on a mission to advance ways the researchers are evaluated towards a system better fitting todays’ needs. His efforts produced a splash among the Dutch academics in June 2021 upon university-wide removal of the IF as a promotion and hiring criterion. The Utrecht academic echo chamber erupted in disagreement.

Instead of the easy to use but largely unfair numerical metrics, he used a more narrative system. Introducing the Recognition and Rewards Vision started the thorny journey towards better promotion and hiring process. Here, Paul suggests to leverage a TRIPLE model consisting of six components:

• Team spirit - the default to all academic work. Not every activity must be joint, naturally. But an open, collaborative environment towards shared goals is immensely beneficial.
• Research - wide range of activities including desk, experimental, fundamental, and applied research.
• Impact - includes an interplay between science and society. Targeted actions and activities impacting public policy, impact on peers, societal and community impact, co-creation with citizens, professional practice, economic impact through entrepreneurship, or pedagogical impact.
• Professional performance - tasks and roles outside teaching and research strongly connected to the respective academic discipline that can create great societal value.
• Leadership - facilitates and enables academic work in an open, transparent, inspiring, inclusive, and safe environment. It stimulates the impact and is intertwined with research, education, and professional performance. Essentially, it means being a good leader, setting communal goals, addressing problems.
• Education - full spectrum of teaching activities connecting disciplinary, practical, and scientific knowledge used to enhance student learning process.

The research, professional performance, and education is where academics generate specific output. These have an impact on science, society, and leadership in academia. A few more underlying principles in academic evaluation are - recognition and reward of quality over quantity, recognition of openness in all domains, stimulation of dynamic careers, hands-on and value drive.

Does this model increase the difficulty and time demands on evaluating academic candidates for promotion or tenure? Indeed. Nevertheless, with all issues perpetuating the dissatisfaction with the present state of the matter, it may be necessary. 

Another independent example of necessity to leveraging a more narrative approach to academic evaluation represents this interview with Randy Schekman. A Nobel prize winner in physiology or medicine for revealing machinery regulating the secretion and transport of proteins in human cells. He suggests a similar, well crafted narrative driven evaluation of academics. For example, one used by the US national academy of sciences judging grant applications.

The process is set to help academics describe what their work has brought to the world and consists of:

• Document with a brief CV.
• A list of 12 most important papers.
• 50 word bulleted text about what you discovered.
• 250 word text explaining more about those discoveries.

The committee deciding about the funding cannot judge the papers, except in their narrow field of expertise. But a well written broad narrative is a fitting initial way to evaluate if someone has done something interesting. Further proof of substance in your work is a signature of an authority from the field. This authority can’t be related to you closely. No collaborating professors, or former PIs etc. Once passed through this initial gate, one is invited for a Zoom, before an in depth, in-person interview. Here, the committee focuses on qualitative, instead of quantitative measures. 

Interestingly, already ten years ago, professor Randy Scheckman has abandoned the pressure to publish in the ‘top-tier’ journals. It encouraged scientists to chase trendy fields instead of focusing on more important work. This issue was amplified by the editors of these journals, favoring publications likely to make a bigger splash.

In a world where every research discipline goes so deep that funding, or tenure deciding committees cannot possibly give an expert opinion on your work, a different approach must be used. The two independent professors suggesting the narrative process are a great example of steps in the right direction.

Since the academic world contains countless worlds with their own nuances and specifics, there is no one size fits all system. However, a rounded approach encompassing criteria outside measurable output is going to be vital to the evolution of academic institutions and funding bodies to foster a research culture that will be more advanced looking into the future.

Do you have ideas on how to improve the current system in your field? Do share in the comments.

#GradSchool #ECRchat, #research, #AcademicLife, #PeerReview, #AmReading, #PhDchat, #MakePublishingGreatAgain #Frelsi #EchoChamber #justdoingmywork #merit2