Plea for assessments, against bean counting – part 3

Valorization of research has become an increasingly important pillar in research evaluation. The LERU report “Research universities and research assessment” does acknowledge this development. The report does not take a strong stand but limits itself to a cautious preliminary assessment of impact assessment. It gives an overview of the British, US, and European approach to “impact” as evaluation criterion. In the UK, one fifth of the grade in the new Research Excellence Framework will be awarded based on a combination of the “reach” of the impact and its “significance”. Universities are asked to present case studies as empirical evidence of societal impact of their research. The LERU report points to the resource intensiveness of this approach as well as to the novelty of this type of measurement for academia. Panel members will have to be develop expertise in this area. Also, the way research may have wider impact in society will vary strongly by research field.

In the US a different, large-scale data oriented route has been taken with the STAR METRICS project funded by NIH, NSF and the White House Office of Science and Technology. There is no lack of ambition for the US project. According to Francis Collins, Director of NIH, STAR METRICS will “yield a rigorous, transparent review of how our science investments are performing. In the short term, we’ll know the impact on jobs. In the long term, we’ll be able to measure patents, publications, citations, and business start-ups”. LERU warns that this might be too optimistic. “Already anecdotal evidence suggests that a number of anomalies appear to be occurring. There is concern about coverage especially in disciplines that focus on highly selective and tightly focused conference proceedings, traditional journals being deemed to slow. In addition, it is thought that there may be perverse effects on young new investigators.”

Not mentioned in the report is the role of commercial companies in research assessment. This is a growing market and the increasing pressure on university budgets has the paradoxical effect of making research assessments and bibliometric analyses even more important. As a result, commercial companies have developed aggressive strategies to attract universities as clients. Some universities have developed spin-off companies, and CWTS itself is in fact an exemplar of such a hybrid of a research centre and commercial service provider. This has been the state of affairs from the very beginning of scientometrics as a field of research. So there is nothing new here. Still, universities need to be aware of  potential conflicts of interest between the companies producing information about research and themselves. A good strategy might be to always maintain ownership of the data produced by the university and to promote open access where possible. Universities are starting to develop campus-wide policies and they might have profited from LERU advice on this topic.

Last, but not least, the LERU report does not discuss the changing demographics of the research population at universities and the acute need for universities to develop a more future oriented career policy. According to many specialists, the way universities develop their human resource management might very well decide how they will fare. An important question is how research evaluations are affecting the development of research careers and to what extent they are producing perverse effects. The fact that this is not mentioned at all in the LERU report is a missed opportunity in an otherwise balanced and carefully written policy report.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: