Funding research is vital to all societies and strongly depends on the investment of time and expertise you provide as external evaluators.

Research assessment directly impacts research culture, and your work ensures a culture of integrity, diversity, and quality. We thank you for your contribution to this crucial aspect of research!

We know that peer review is complex and subjective, so funders work hard to promote a fair process while minimizing the burden on applicants and evaluators.

Focusing on shorthand or quantitative criteria such as journal-based metrics, H-index, university rankings, author list position in a publication and the reputation of a researcher's institution is a familiar, but problematic practice in research evaluation.

We have learned that this emphasis fosters tendencies to game the system, favouring short-term achievement at the expense of longer-term goals, such as reproducibility, research integrity and broader service to science and society.

Funding research has provided the means to incredible advancement in human knowledge.

A redesigned system will improve the quality of science, and requires expanding the definition of research output, quality, and impact!

As a result, the way we assess research proposals is moving forward - just like research itself! This more holistic assessment system is being piloted by different funders, with positive feedback from applicants and evaluators!

So how can we all collaborate in the scientific community to achieve more holistic and responsible evaluation procedures, shifting the system through collective individual effort? How can we capture researcher's contributions to science and society in a broader sense and apply them to our evaluation?

Based on experiences from numerous institutions and good practice examples worldwide, here are six important points to consider when adopting a more holistic perspective on research evaluation.

First. Align your decision making to the strategic objectives and specific criteria of the funding institution or funding instrument. Funding institutions are happy to provide guidance and help if something is not clear.

Second. Be clear about the context and limitations of any quantitative metrics used and balance them with qualitative parts of the proposal.

These can include recognizing the clarity, originality, timeliness and methodology of the proposal, looking at the team composition and their ways of working together, or judging the value of non-traditional achievements or experience.

Third. Look broad instead of narrow to capture the full range of a researcher's contributions. Beyond publications consider also outputs, outcomes, and activities such as:

• Contributions to Open Science
• Teaching, leadership, and mentoring
• Active research collaborations and stakeholder engagement
• Service to the research community such as memberships in committees and other similar activities
  
  

  On-screen text
  scientific boards
  editorial boards
  national or international groups

• and societal interactions such as participation in outreach events and other similar activities.
  
  

  On-screen text
  articles in lay publications
  development of policy briefs
  public involvement in the research process

Fourth. Be aware of unintended biases that arise from scientific and cultural stereotypes such as gender, ethnicity, seniority, affiliation and discipline. These biases can be found in everyone and are held even by the most egalitarian researchers. Slow down, be vigilant, and check for evidence!

Fifth. Good panel discussions rely on teamwork. Foster personal and group accountability for responsible research assessment during evaluation! Be a gracious giver and receiver of feedback!

And finally: If you are not sure whether you have a conflict of interest or not ask the funding institution for guidance! They will be happy to help.

Your contribution as evaluators of research is more important than ever before!

Through adopting a broader perspective, staying aware of biases, and fostering a more holistic evaluation process, you can lead the way towards better research, as well as a fairer and more inclusive research culture!

On-screen text
Co-produced by
Luxembourg National Research Fund and DORA

Script
Sean Sapcariu and Anna Hatch

Production
scienceRELATIONS

Animation
Harald Schröder

Special thanks to
Stephen Curry (DORA), Marc Schiltz (FNR) and the DORA funders exchange group