The need for change in the governance of science – II

Turbulent times at the Trippenhuis, home of the Royal Netherlands Academy of Arts and Sciences (KNAW). Last Thursday and Friday the Academy opened its doors for the Science in Transition conference: two days of debate between representatives of science, industry, and policy-making aimed at revising some of the checks and balances of the scientific and scholarly system. We already blogged about some of the most problematic aspects of current quality control mechanisms last week. Interestingly, there was remarkable consensus among conference participants on a number of points relating to these mechanisms. Most keynotes, commentators, and members of the audience seemed to want to avoid:

• Research agendas that are not driven by content and relevance;
• Excessive competition and careerism;
• A publish or perish culture that favors quantity over quality, promotes cherry picking of results and salami slicing, and discourages validation, verification and replication;
• An ill-functioning peer review system that lacks incentives for sound quality judgment;
• One-size-fits-all evaluation procedures;
• Perverse allocation models and career policy mechanisms (in which for instance number of students directly affect the number of .fte spent on research and young researchers are hired on short-term contracts funded through external grants [‘PhD and Post-doc factories’).

But of course there was still a lot left to debate. As a result of the succesful media campaign and the subsequent hype around Science in Transition, some speakers felt that they needed to ‘stand up for science’. Hans Clevers, president of the KNAW, and Jos Engelen, chairman of the Netherlands Organisation for Scientific Research (NWO) were noticably unhappy about the portrayal in the media of science ‘in crisis’. Both stressed that Dutch science is doing well, judging for instance from the scores on university rankings. Both radiated their aggravation about painting an ambiguous picture of science to outsiders, because of the potential risks of feeding already existing scepticism and mistrust. At the same time it was telling that these key figures in the landscape of Dutch governance of science were supportive of the debate and the fundamental points raised by the organisers.

Like Clevers and Engelen, Lodi Nauta (dean of the faculty of philosophy in Groningen) too, argued that not everything is going astray in science. According to him there are still many inspiring examples of solid, prize-worthy, trust-worthy, interdisciplinary, societally relevant research. But Nauta also signaled that there is much ‘sloppy science’. Not all symposium participants agreed on how much, and if there is indeed a general increase. Peter Blom, CEO of Triodos Bank, made an important aside. He thought it rather arrogant that whilst basically every other sector is in crisis, science should think it could distance itself from these economic and socio-political currents. But many participants took a cautionary stance: If there is indeed such a thing as a crisis, we should not lose sight of the nuances. It is not all bad everywhere, at the same time, and for everyone. Some argued that young researchers suffer most from current governance structures and evaluation procedures; that certain fields are more resilient than others; and that compared to other countries the Dutch scientific and scholarly system is not doing that badly at all. Henk van Houten, general manager of Philips Research, on the contrary, argued that ‘university as a whole has a governance issue’: The only moment that universities have actual influence is when they appoint professors at particular chairs. However, these professors are subsequently mainly held accountable to external funders. One is left to wonder which governance model is to be preferred: this one, or models companies like Philips put in practice.

At the heart of the debate on being open about the present crisis lies a rather dated desire to leave ‘the black box of science’ unopened. Whilst Lodi Nauta for instance argued – with Kant – that an ideal-typical image of science is necessary as a ‘regulatory idea’, the Science in Transition initiators deemed it pointless to keep spreading a fairytale about ‘a perfect scientific method by individuals with high moral values without any bias or interests’. Van Houten (Philips) and Blom (Triodos) also argued that science does not take its publics seriously enough if it sticks to this myth. Letting go of this myth does not amount to ‘science bashing’ – on the contrary. It is valuable to explain how science ‘really’ works, how objective facts are made, where the uncertainties lie, which interests are involved, and how science contributes through trained judgment and highly specialized expertise.

A hotly debated matter also relates to ‘black-boxing’ science: Who gets to have a say about proper quality assessment and the shaping of research agendas? André Knottnerus, chairman of the Scientific Council for Government Policy (WRR) pointed at ambivalence in discussions on these matters. We tend to only take criticism on performance measurement seriously if delivered by researchers that score high on these same measures. There were also differences of opinion about the role of industry in defining research agendas. (i.e. detrimental effects of pharmaceutical companies on clinical research. Obviously Philips was invited to serve as a counter-example of positive bonds between (bio-medical) research and commercial partners). And what about society at large? Who speaks for science and to whom are we to be held accountable, Sheila Jasanoff asked? (How) should researchers pay more attention to mobilizing new publics and participatory mechanisms, and productive democratisation of the politics of science?

Most speakers were of the opinion that we should move away from narrow impact measurement towards contextually sensitive evaluation systems. Systems that reward mission oriented research, collaboration and interdisciplinarity, which not only accommodate short-term production but also the generation of deep knowledge. These (ideal-typical?) systems should allow for diversification in talent selection, and grant academic prestige through balanced reward mechanisms and ‘meaningful metrics’. Though the symposium did a lot of the groundwork, how to arrive at such systems is of course the biggest challenge (see also Miedema’s ‘toolbox for Science in Transition’ for concrete suggestions). This is assuming it is possible at all. But perhaps we need this ideal-typical image as a ‘regulatory idea’.

Advertisement

The need for change in the governance of science

Tomorrow, a two-day conference will be held, Science in Transition, at the beautiful headquarters of the Royal Netherlands Academy of Arts and Sciences, the Trippenhuis in Amsterdam. Five researchers with backgrounds in medical research, history, and science & technology studies, have taken the lead in what they hope will become a strong movement for change in the governance of science and scholarship. The conference tomorrow builds on a series of three workshops held earlier this year about “image and trust”, “quality and corruption”, and “communication and democracy”. On the eve of the conference, the initiators published their agenda for change. In this document, 7 issues are defined as key topics and a large number of questions about the necessary direction for change are formulated. These issues are: the image science has in the public view; public trust in science; quality control; fraud and deceit; new challenges in science communication; the relationship between science, democracy and policy; and the connection between education and research.

With this list, the agenda is rather encompassing and broad. The thread running through the document as well as through the supporting “position paper” is discontent with the current governance of the scientific and scholarly system. The position paper is strong in that it is based on the professional experience of the authors, some of whom have been leading and managing research for many years. At the same time, this is also the source of some obvious weaknesses. The situation in the medical sciences is here and there a bit too dominant in the description of reality in science, whereas the situation in the humanities and social sciences is rather different (although equally problematic). Because the agenda is so broad, the position paper in its current version tends to lump together problems of quite different sources as if they are all of a kind. The subtleties that are so important in the daily practices of scientists and scholars tend to disappear from view. But then again, some of this may be inevitable if one wishes to push an agenda for change. A quite strong feature of the position paper is that it does not try to justify or deny the problematic aspects of science (of which fraud and corruption are only the most visible forms) but attempts to confront them head-on.

This is the reason that I think Science in Transition is an excellent iniatiative which deserves strong support from all participants and users in the current system of knowledge creation. Certainly in the Netherlands, which is the focus of most experiences the initiative builds on, but also more globally, the current ways of governing the increasingly complex scientific system hit their limits. Let me focus on the matter of quality control, the issue with which we deal regularly in this blog. The peer review system is straining under increasing pressure. Data intensive research requires new forms of data quality control that are not yet in place. Fraudulent journals have become a major source of profit for shady publishers. Open access of both publications and research data is increasingly needed, but at the same time it threatens to introduce corrupt business models in science and may harm the publication of books in the humanities (if not done carefully). Simplified but easily accessible indicators, such as the h-index and the Journal Impact Factor, have in many biomedical fields acquired the mantle of a goal in itself. Editors of journals feel pressured to increase their impact factor in sound and less sound ways. The economics of science is dominated by a huge supply of easily replaceable temporary labour force and for many PhD students there is no real career possibility in meaningful research. Peer review tends to favour methodological soundness above scientific or societal relevance. The publicly funded budgets are not always sufficient to perform the research as thoroughly as is needed. The current publication cultures tend to prefer positive results over negative ones (especially dangerous in the context of pharmaceutical research).

I realize that this short summary of some of the better known problems is as generalizing as the position paper. Of course, these problems are not acute in every field. Some journals are not afflicted with impactitis, but manage to focus on pushing the research front in their area. Universities behave differently in the ecology of higher education and research. Many researchers are delivering a decent or excellent performance. Scientific specialties differ strongly in epistemic styles as well as in publication cultures. And the solutions are certainly not easy. Nevertheless, the governance of science requires some fundamental adaptations, including a possible revision of the role of universities and other institutions of higher education. Science in Transition deserves to be applauded for having put this complex problem forcefully on the agenda.

I am also enthusiastic about the project because it resonates so well with the research agenda of CWTS. We have even created a new working group which focuses on the detailed, ethnographic, study of actual evalution practices in science and scholarship: EPIC (evaluation practices in context). We need to have a much more detailed understanding of what actually goes on in the laboratories, hospitals, and research institutes at universities. This is the only way we can supplement generalizing and normative statements about trends in scientific governance with “thick descriptions” of the complex reality of current science.

The more complex the research system has become, the more important quantitative information, including indicators, is for the researchers, research managers and science policy makers. This requires more advanced methodologies in the field of scientometrics (and not only in bibliometrics), such as science mapping, the topic of another CWTS working group. It requires more accurate data collection, including better accounting systems of the costs of scientific research. (Currently, universities actually do not know how much their research actually costs.) But it also requires vigilance against “management by indicators”. If young PhD students aim to publish mainly in order to increase their performance indicators so that they can have a career, as many a senior researcher in a hospital has experienced, we know that the system is in trouble.

Accounting systems are sometimes certainly necessary, but these should be put in place in such a way that they do not derail the primary processes (such as knowledge creation) that they are supposed to support. In the scientific system in the Netherlands, we therefore need a renewed balance between performance measurement and expert judgement in the quality control mechanisms. This is what we mean with our new CWTS motto: meaningful metrics. The future of scientometrics is not in the production of ever more indicators, but in more effectively supporting researchers in their endeavour to create new knowledge.

Who is the modern scientist? Lecture by Steven Shapin

There are now many historical studies of what’s been called scientists’ personæ–-the typifications, images, and expectations attached to people who do scientific work. There has been much less interest in the largely managerial and bureaucratic exercises of counting scientists-– finding out how many there are, of what sorts, working in what institutions. This talk first describes how and why scientists came to be counted from about the middle of the twentieth century and then relates those statistical exercises to changing senses of who the scientist was, what scientific inquiry was, and what it was good for.

Here’s more information, including how to register

Date: Thursday 28 November 2013

Time: 5-7 pm

Place: Felix Meritis (Teekenzaal), Keizersgracht 324, Amsterdam