CWTS in new European consortium

Good news came our way yesterday! CWTS will be partner in a new project funded by the Swedisch Riksbankens Jubileumsfond: Knowledge in science and policy. Creating an evidence base for converging modes of governance in policy and science (KNOWSCIENCE). The project is coordinated by Merle Jacob (Lund University, Sweden). Other partners in the consortium are Dietmar Braun (Lausanne University, Switzerland), Tomas Hellström (Department of Business Administration, Lund University), Niilo Kauppi (CNRS, Strasbourg, France), Duncan Thomas & Maria Nedeva (Manchester Institute of Innovation Research, Manchester Business School, UK), Rikard Stankiewitz (Lund University), and Sarah de Rijcke & Paul Wouters (CWTS).

KNOWSCIENCE focuses on deepening our understanding of the interplay between policy instruments intended to govern the structural organization of higher education and research (HER) and the informal rules and processes that organisations have developed for ensuring the validity and quality of the knowledge they produce. KNOWSCIENCE refers to this as the interplay between structural and epistemic governance, and argue that an understanding of this relationship is necessary for building sustainable knowledge producing arrangements and institutions and securing society’s long-term knowledge provision.

The main research question guiding the project is ‘how do policy and the science systems co-produce the conditions for sustainable knowledge provision?’ Specifically we ask:

(a) How are HER policy steering mechanisms enabled, disabled and transformed throughout the HER sector via the academic social system?

(b) What are the most significant unintended consequences of HER policy on the HER system? and

(c) What types of policy frameworks would be required to meet these challenges?

The announcement on the RJ website can be found via this link.

Advertisement

In Search of Excellence? Debating the Merits of Introducing an Elite Dutch University Model

Report by Alex Rushforth

Should the Netherlands strive for excellence in its university systems? Will maintaining quality suffice? This was the topic of a recent panel debate at the WTMC annual meeting on 21 November 2014 in De Balie, Amsterdam. Organised and chaired by Willem Halffman, the session focused on an article published by Barend Van Der Meulen in the national newspaper De Volkskrant, which advocated the need to produce two excellent universities which excel on internationally published rankings, thereby creating a new top-tier in the Dutch higher education system.

Both van der Meulen and Halffman presented their views, with an opposing position also coming from Sally Wyatt. Completing the panel, CWTS’s very own Paul Wouters provided results from recent empirical work about rankings.

Barend van der Meulen’s call for an elite university stemmed from the fact Dutch universities perennially sit outside of the top-50 in Shanghai and Times Higher Education rankings. For him the message is clear: the Netherlands is repeatedly failing to enhance its reputation as an elite player among global universities, a position which ought to cause concern. Van der Meulen stated that his call for an elite university model is part of a need to create an expanded repertoire of what universities are and what they should do in the Netherlands. The pursuit of rankings through this vehicle is therefore tightly coupled with a rejection of the status quo. Rankings are a social technology which ought to be harnessed for quality improvement and as tools through which to promote democratic participation by equipping students and policymakers with tools to make judgments and exert some influence over universities. Alternative modes of evaluation like peer review provide closed systems in which only other academics can make judgments, leaving university activities unaccountable to external modes of evaluation. This ‘ivory tower’ situation reminiscent of the 1980s is an image Van Der Meulen wishes to escape from, as ultimately it damages credibility and legitimacy of universities. The reliance on public money for research and education makes the moral case for university improvement and accountability particularly pressing in the Netherlands. For Van Der Meulen, the ‘good enough’ university (see Wyatt’s argument below) is not enough, given that excellence is imposing itself as a viable and increasingly important alternative.

First to oppose the motion in favour of elite universities was Willem Hallfman, whose talk built on a reply co-authored with Roland Bal, also in De Volkskrant. In the talk Halffman questioned the very foundations of the idea that ‘excellence’ ought to be pursued. Drawing unflattering comparisons between the research budget of Harvard University and that of the entire Netherlands, it was argued that competing within a global superleague would require a radical expansion of existing research budgets and wage structures across the Dutch university system, which he felt unrealistic and unreasonable against a backdrop of crisis in public finances. As well as reproducing national elites, Halffman also questioned the desirability of ranking systems which promote academic stars and the consequences this brings to institutions of science in general and Dutch universities in particular. Football-style league tables provide poor models on which to rate universities, as in contrast with sport where a winner-takes-all logic is central, for universities embodying a broad repertoire of societal functions, it is not clear what ‘winning’ means and how this would be made visible and commensurable through performance indicators.

Sally Wyatt recounted her personal experiences of the shock she encountered when studying and working in British universities in the 1980s, having grown-up in Canada within a period of prosperity and social mobility. These experiences fired a series of warning shots not to go down a road of pursuing excellence. When a move to the Netherlands came about in 1999, it promised her an oasis away from the turmoil the British university system had faced as a result of Thatcherite policy reforms. With the emergence of the Research Assessment Exercise (RAE) and its ranking logic comes also a rise in managerial positions and policies, decline in working conditions, and a widening gender gap. Gone also was a latent class system engrained in the culture of universities, with dominant elite institutions the site of social stratification reproduced across generations, which rankings merely encourage and reinforce. Despite erosion of certain positive attributes in universities since her arrival in the Netherlands, Wyatt argued that the Dutch system still preserves enough of a ‘level-playing field’ in terms of funding allocation to merit fierce resistance to any introduction of an elite university model. For Wyatt sometimes it is better to promote the ‘good enough’ than to chase an imperialist and elitist vision of ‘excellence’.

Drawing on work on university and hospital rankings carried-out with Sarah De Rijcke (CWTS), Iris Wallenburg and Roland Bal (Erasmus MC, Rotterdam), Paul Wouters’ talk advocated the need for a more fine-grained STS investigations into the kinds of work that goes into rankings, who is doing it, and in what situations. What is at stake in studying rankings then is not simply the critique of this or that tool, but a more pervasive (and sometimes invisible) logic/set of practices encountered across public organisations like universities and hospitals. Wouters advocated a move towards combining audit society critiques (which tend to be top-down) with STS insights into how ranking is practiced across various organisational levels in universities. This would provide a more promising platform through which to inform debates of the kind playing-out over the desirability of the elite university.

So the contrast between positions was stark. Are rankings – these seemingly ubiquitous ordering mechanisms of contemporary social life – something the Netherlands can afford to back away from in governing its universities? If they are being pursued anyway, shouldn’t policy intervene and assist a more systematic pursuit up the rankings which would enable more pronounced successes? Or is it necessary to oppose the very notion that the Netherlands needs to excel in a ‘globally competitive’ race, particularly given the seeming arbitrariness of many of the metrics according to which prestige gets attributed via ranking mechanisms? Despite polarization on what is to be done, potential for extending STS’s conceptual and empirical apparatus to mediate these discussions seemed to strike a chord among panelists and the audience alike. No doubt this stimulating debate touches on a set of issues that will not be going away quickly, and is one on which the WTMC community is surely well placed to intervene.

Ethics and misconduct – Review of a play organized by the Young Academy (KNAW)

This is a guest blog post by Joost Kosten. Joost is PhD student at CWTS and member of the EPIC working group. His research focuses on the use of research indicators from the perspective of public policy. Joost obtained an MSc in Public Administration (Leiden University) and was also trained in Political Science (Stockholm University) and Law (VU University Amsterdam).

Scientific (mis)conduct – The sins, the drama, the identification

On Tuesday November 18th 2014 the Young Academy of the Royal Netherlands Academy of Sciences organized a performance of the play Gewetenschap by Tony Maples at Leiden University. These weeks, Pandemonia Science Theater is on tour in the Netherlands to perform this piece at several universities. Gewetenschap was inspired by occasional troubles with respect to ethics and integrity which recently occurred in Dutch science and scholarship. Although these troubles concerned grave violations of the scientific code of conduct (i.e., the cardinal sins of fraud, fabrication, and plagiarism) the play focusses on common dilemma’s in a researcher’s everyday life. The title Gewetenschap is a non-existent word, which combines the Dutch words geweten (conscience) and wetenschap (science).

The playwright used confidential interviews with members of the Young Academy to gain insight into the most frequently occurring ethical dilemma’s researchers have to deal with. Professor Karin de Zwaan is a research group leader who has hardly any time to do research herself. She puts much effort in organizing grants, attracting new students and organizing her research group. Post-doc Jeroen Dreef is a very active researcher who does not have enough time to take organizational responsibilities serious. A tenure track is all he wants. Given their other important activities, Karin and Jeroen hardly have any time to supervise PhD student Lotte. One could question the type of support they do give her.

At times, given the reaction on scenes of the drama piece, the topics presented were clearly recognized by the audience. Afterwards, the dilemma’s touched upon during the play are presented by prof. Bas Haring. The audience discusses the following topics:

• Is there a conflict between the research topics a researcher likes himself and what the research group expects her/him to do?
• In one of the scenes, the researchers were delighted because of the acceptance of a publication. Haring asks if that exhibits “natural behaviour”. Shouldn’t a researcher be happy with good results instead of a publication being accepted? One of the participants replies that a publication functions as a reward.
• What do you do with your data? Is endless application of a diversity analysis methods until you find nice results a responsible approach?
• What about impact factors (IF)? Bas Haring himself says his IF is 0. “Do you think I am an idiot?” Which role do numbers such as the IF play in your opinion about colleagues? There seems to be quite a diversity of opinions. An early career research says everone knows these numbers are nonsense. An experienced scientist points out that there is a correlation between scores and quality. Someone else expresses his optimism since he expects that this focus on numbers will be over with ten years. This causes another to respond that in the past there was competition too, but in a different way.
• When is someone a co-author? This question results in a lively debate. Apparently, there are considerable differences from field to field. In the medical fields, a co-authorship can be a way to express gratitude to authors who have played a vital role in a research project, such as people who could organize experimental subjects. In this way, a co-authorship becomes a tradeable commodity. A medicine professor points out that in his field, co-authorships can be used to compare a curriculum vitae with the development of status as a researcher. Thus, it can be used as a criterion to judge grant proposals. A good researcher should start with first position co-authorships, later on should have co-authorships somewhere in between the first and last author, and should end his career with papers in which has co-authorships in the last position. Thus, the further the career has been developed, the more the name of the other should be in the final part of the author list. Another participant states that one can deal with co-authorships in three different ways: 1. Co-authors should always have full responsibility for everything in the paper. 2. Similar to openness which is given at the end of a movie, co-authors should clarify what each co-author’s contribution was. 3. Only those who really contributed in writing a paper can be a co-author. The participant admits that this last proposal works in his own field but might not work in other fields.
• Can a researcher exaggerate his findings if he presents them to journalists? Should you keep control over a journalist’s work in order to avoid that he will present things differently? Is it allowed to present untruth information in order to help support your case, just to avoid that a proper scientific argumentation will be too complex for the man in the street?
• Is it allowed to to present your work as having more societal relevance than you really expect? One of the reactions is that researchers are forced to express the societal relevance of their work when they apply for a grant. From the very nature of scientific research it is hardly possible to clearly indicate what society will gain from the results.
• What does a good relationship between a PhD-student and a supervisor look like? What is a good balance between serving the interests of PhD students, serving organizational interests (e.g. the future of the organization by attracting new students and grants), and the own interest of the researcher?

The discussion did not concentrate on the following dilemma´s presented in Gewetenschap:

• To what extent are requirements for grant proposals contradictory? On the one hand, researchers are expected to think ‘out-of-the-box’ while on the other hand they should meet a large amount of requirements. Moreover, should one propose new ideas including the risks which come along, or is it better to walk on the beaten path in order to guarantee successes?
• Should colleagues who did not show respect be served with the same sauce if you have a chance to review their work? Should you always judge scientific work on its merits? Are there any principles of ‘due process’ which should guide peer review?
• Whose are the data if someone contributed to them but moves to another research group or institute?

 

Developing guiding principles and standards in the field of evaluation – lessons learned

This is a guest blog post by professor Peter Dahler-Larsen. The reflections below are a follow-up of his keynote at the STI conference in Leiden (3-5 September 2014) and the special session at STI on the development of quality standards for science & technology indicators. Dahler-Larsen holds a chair at the Department of Political Science, University of Copenhagen. He is former president of the European Evaluation Society and author of The Evaluation Society (Stanford University Press, 2012).

Lessons learned about the development of guiding principles and standards in the field of evaluation – A personal reflection

Professor Peter Dahler-Larsen, 5 October 2014

Guidelines are symbolic, not regulatory

The limited institutional status of guiding principles and standards should be understood as a starting point for the debate. In the initial phases of development of such standards and guidelines, people often have very strong views. But only the state can enforce laws. To the extent that guidelines and standards merely express some official views of a professional association who has no institutional power to enforce them, standards and guidelines will have limited direct consequences for practitioners. The discussion becomes clearer once it is recognized that standards and guidelines thus primarily have a symbolic and communicative function, not a regulatory one. Practitioners will continue to be free to do whatever kind of practice they like, also after guidelines have been adopted.

Design a process of debate and involvement

All members of a professional association should have a possibility to comment on a draft version of guidelines/standards. An important component in the adoption of guidelines/standards is the design of a proper organizational process that involves the composition of a draft by a select group of recognized experts, an open debate among members, and an official procedure for the adoption of standards/guidelines as organizational policy.

Acknowledge the difference between minimum and maximum standards

Minimal standards must be complied with in all situations. Maximum standards are ideal principles worth striving for, although they will not be accomplished in any particular situation. It often turns out that there will be many maximum principles in a set of guidelines, although that is not what most people believe is “standards.” For that reason I personally prefer the term guidelines or guiding principles rather that “standards.”

Think carefully about guidelines and methodological pluralism

Advocates of a particular method often think that methodological rules connected to their own method defines quality as such in the whole field. For that reason, they are likely to insert their own methodological rules into the set of guidelines. As a consequence, guidelines can be used politically to promote one set of methods or one particular paradigm rather than another. Great care should be exercised in the formulation of guidelines to make sure that pluralism remains protected. For example, in evaluation the rule is that if you subscribe to a particular method, you should have high competence in the chosen method. But that goes for all methods.

Get beyond the “but that´s obvious” argument

Some argue that it is futile to formulate a set of guidelines because at that level of generality, it is only possible to state some very broad and obvious principles with which every sensible person must agree. The argument sounds plausible when you hear it, but my experience suggests otherwise for a number of reasons. First, some people have just not thought about a very bad practice (for example, doing evaluation without written Terms of Reference). Once you see, that someone has formulated a guideline against this, you are likely to start paying attention to the problem. Just because a principle is obvious to some, does not mean that it is obvious to all. Second, although there may be general agreement about a principle (such as “do no unnecessary harm” or “take general social welfare into account”), there can be strong disagreement about the interpretations and implications of the principle in practice.  Third, a good set of guiding principles will often comprise at least two principles that are somewhat in tension with each other, for example the principle of being quick and useful versus the principle of being scientifically rigorous. To sort out exactly which kind of tension between these two principles one can live with in a concrete case turns out to be a matter of complicated professional judgment. So, get beyond the “that´s obvious” argument.

Recognize the fruitful uses of guidelines

Among the most important uses of guidelines in evaluation are:

– In application situations, good evaluators can explain their practice with reference to broader principles

– In conferences, guidelines can stimulate insightful professional discussions about how to handle complicated cases

– Books and journals can make use of guidelines as inspiration for the development of an ethical awareness among practitioners. For example, google Michael Morris´ work in the field of evaluation.

– There is great use of guidelines in teaching and in other forms of socialization of evaluators.

Respect the multiplicity of organizations

If, say, the European Evaluation Society wants to adopt a set of guidelines, it should be respected that, say, the German and the Swiss association already have their own guidelines. Furthermore, some professional associations (say, psychologists) also have guidelines. A professional association should take such overlaps seriously and find ways to exchange views and experiences with guidelines across national and organizational borders.

Professionals are not alone, but relations can be described in guidelines, too

It is often debated that one of the major problems in bad evaluation practice is the behavior of commissioners. Some therefore think that guidelines describing good evaluation practice are in vain until the behavior of commissioners (and perhaps other users of evaluation) are included in the guidelines, too. However, there is no particular reason why the guidelines cannot describe a good relation and a good interaction between commissioners and evaluators. Remember, guidelines have no regulatory power. They express merely the official norms of the professional association. Evaluators are allowed to express what they think a good commissioner should do or not do. In fact, explicit guidelines can help clarify mutual and reciprocal role expectations.

Allow for regular reflection, evaluation and revision of guidelines

At regular intervals, guidelines should be debated, evaluated and revised. The AEA guidelines, for example, have been revised and now reflect values regarding culturally competent evaluation that was not in earlier versions. Guidelines are organic and reflect a particular socio-historical situation.

Sources:

Michael Morris (2008). Evaluation Ethics for Best Practice. Guilford Press.

American Evaluation Association Guiding principles

On citation stress and publication pressure

Our article on citation stress and publication pressure in biomedicine went online this week – co-authored with colleagues from the Free University and University Medical Centre Utrecht:

Tijdink, J.K., S. de Rijcke, C.H. Vinkers, Y.M. Smulders, P.F. Wouters, 2014. Publicatiedrang en citatiestress: De invloed van prestatie-indicatoren op wetenschapsbeoefening. Nederlands Tijdschrift voor Geneeskunde 158: A7147.

* Dutch only *

How does science go wrong?

We are happy to announce that our abstract got accepted for the 2014 Conference of the European Consortium for Political Research (ECPR), which will be held in Glasgow from 3-6 September. Our paper is selected for a panel on ‘The role of ideas and indicators in science policies and research management’, organised by Luis Sanz-Menéndez and Laura Cruz-Castro (both at CSIC-IPP).

Title of our paper: How does science go wrong?

“Science is in need of fundamental reform.” In 2013, five Dutch researchers took the lead in what they hope will become a strong movement for change in the governance of science and scholarship: Science in Transition. SiT appears to voice concerns heard beyond national borders about the need for change in the governance of science (cf. The Economist 19 October 2013; THE 23 Jan. 2014; Nature 16 Oct. 2013; Die Zeit 5 Jan. 2014). One of the most hotly debated concerns is quality control, and it encompasses the implications of a perceived increasing publication pressure, purported flaws in the peer review system, impact factor manipulation, irreproducibility of results, and the need for new forms of data quality management.

One could argue that SiT landed in fertile ground. In recent years, a number of severe fraud cases drew attention to possible ‘perverse effects’ in the management system of science and scholarship. Partly due to the juicy aspects of most cases of misconduct, these debates tend to focus on ‘bad apples’ and shy away from more fundamental problems in the governance of science and scholarship.

Our paper articulates how key actors construct the notion of ‘quality’ in these debates, and how they respond to each other’s position. By making these constructions explicit, we shift focus back to the self-reinforcing ‘performance loops’ that most researchers are caught up in at present. Our methodology is a combination of the mapping of the dynamics of media waves (Vasterman, 2005) and discourse analysis (Gilbert & Mulkay, 1984).

References

A revolutionary mission statement: improve the world. Times Higher Education, 23 January 2014.

Chalmers, I., Bracken, M. B., Djulbegovic, B., Garattini, S., Grant, J., Gülmezoglu, A. M., Oliver, S. (2014). How to increase value and reduce waste when research priorities are set. The Lancet, 383 (9912), 156–165.

Gilbert, G. N., & Mulkay, M. J. (1984). Opening Pandora’s Box. A Sociological Analysis of Scientists’ Discourse. Cambridge: Cambridge University Press.

Research evaluation: Impact. (2013). Nature, 502(7471), 287–287.

Rettet die Wissenschaft!: “Die Folgekosten können hoch sein.” Die Zeit, 5 January 2014.

Trouble at the lab. The Economist, 19 October 2013.

Vasterman, P. L. M. (2005). Media-Hype. European Journal of Communication , 20 (4 ), 508–530.

Selling science to Nature

On Saturday 22 December, the Dutch national newspaper NRC published an interview with Hans Clevers, professor of molecular genetics and president of the Royal Netherlands Academy of Arts and Sciences (KNAW). The interview is the latest in a series of public performances following Clevers’ installment as president in 2012, in which he responds to current concerns about the need for revisions in the governance of science. The recent Science in Transition initiative for instance stirred quite some debate in the Netherlands, also within the Academy. One of the most hotly debated issues is that of quality control, an issue that encompasses the implications of an increasing publication pressure, purported flaws in the peer review system, impact factor manipulation, and the need for new forms of data quality management.

Clevers is currently combining the KNAW-presidency with his group leadership at the Hubrecht Institute in Utrecht. In both roles he actively promotes data sharing. He told the NRC that he stimulates his own researchers to share all findings. “Everything is for the entire lab. Asians in particular sometimes need to be scolded for trying to keep things to themselves.” When it comes to publishing the findings, it is Clevers who decides who contributed most to a particular project and who deserves to be first author. “This can be a big deal for the careers of PhD students and post-docs.” The articles for ‘top journals’ like Nature or Science he always writes himself. “I know what the journals expect. It requires great precision. A title consists of 102 characters. It should be spot-on in terms of content, but it should also be exciting.”

Clevers does acknowledge some of the problems with the current governance of science — the issue of data sharing and mistrust mentioned above, but for instance also the systematic imbalance in the academic reward system when it comes to appreciation for teaching. However, he does not seem very concerned with publication pressure. He argued on numerous occasions that publishing is simply part of daily scientific life. According to him, the number of articles is not a leading criterium. In most fields, it’s the quality of the papers that matters most. With these statements Clevers clearly puts himself in the mainstream view on scientific management. But there are also dissenting opinions, and sometimes they are voiced by other prominent scientists from the same field. Last month, Nobel Prize winner Randy Schekman, professor of molecular and cell biology at UC Berkeley, declared a boycott on three top-tier journals at the Nobel Prize ceremony in Stockholm. Schekman argued that Nature, Cell, Science and other “luxury” journals are damaging the scientific process by artificially restricting the number of papers they accept, by make improper use of the journal impact factor as a marketing tool, and by depending on editors that favor spectacular findings over soundness of the results. 

The Guardian published an article in which Schekman iterated his critique. The journal also made an inventory of the reactions of the editors-in-chief of Nature, Cell and Science. They washed their hands of the matter. Some even delegated the problems to the scientists themselves. Philip Campbell, editor-in-chief of Nature, referred to a recent survey of the Nature Publishing Group which revealed that “[t]he research community tends towards an over-reliance in assessing research by the journal in which it appears, or the impact factor of that journal.”

In a previous blog post we paid attention to a call for an in-depth study of the editorial policies of Nature, Science, and Cell by Jos Engelen, president of the Netherlands Organization for Scientific Research (NWO). It is worth reiterating some parts of his argument. According to Engelen the reputation of these journals, published by commercial publishers, is based on ‘selling’ innovative science derived from publicly funded research. Their “extremely selective publishing policy” has turned these journals into ‘brands’ that have ‘selling’ as their primary interest, and not, for example, “promoting the best researchers.” Here we see the contours of a disagreement with Clevers. Without wanting to read too much into his statements, Clevers on more than one occasion treats the status and quality of Nature, Cell and Science as apparently self-evident — as the main current of thought would have it. But in the NRC interview Clevers also does something else: By explaining his policy to write the ‘top-papers’ himself he also reveals that these papers are as much the result of craft, reputation and access, as they are an ‘essential’ quality of the science behind it. Knowing how to write attractive titles is a start – but it is certainly not the only skill needed in this scientific reputation game.

The stakes are high with regard to scientific publishing  — that much is clear. Articles in ‘top’ journals can make, break or sustain careers. One possible explanation for the status of these journals is of course that researchers have become highly reliant on on external funding for the continuation of their research. And highly cited papers in high impact journals have become the main ‘currency’ in science, as theoretical physicist Jan Zaanen called it in a lecture at our institute. The fact that articles in top journals serve as de facto proxies for the quality of researchers is perhaps not problematic in itself (or is it?). But it certainly becomes tricky if these same journals increasingly treat short-term news-worthiness as an important criterion in their publishing policies, and if peer review committee work also increasingly revolves around selecting those projects that are most likely to have short-term success. Amongst others Frank Miedema (one of the initiators of Science in Transition) argues that this is the case in his booklet Science 3.0. Clearly, there is a need for thorough research into these dynamics. How prevalent are they? And what are the potential consequences for longer-term research agendas?

The need for change in the governance of science – II

Turbulent times at the Trippenhuis, home of the Royal Netherlands Academy of Arts and Sciences (KNAW). Last Thursday and Friday the Academy opened its doors for the Science in Transition conference: two days of debate between representatives of science, industry, and policy-making aimed at revising some of the checks and balances of the scientific and scholarly system. We already blogged about some of the most problematic aspects of current quality control mechanisms last week. Interestingly, there was remarkable consensus among conference participants on a number of points relating to these mechanisms. Most keynotes, commentators, and members of the audience seemed to want to avoid:

• Research agendas that are not driven by content and relevance;
• Excessive competition and careerism;
• A publish or perish culture that favors quantity over quality, promotes cherry picking of results and salami slicing, and discourages validation, verification and replication;
• An ill-functioning peer review system that lacks incentives for sound quality judgment;
• One-size-fits-all evaluation procedures;
• Perverse allocation models and career policy mechanisms (in which for instance number of students directly affect the number of .fte spent on research and young researchers are hired on short-term contracts funded through external grants [‘PhD and Post-doc factories’).

But of course there was still a lot left to debate. As a result of the succesful media campaign and the subsequent hype around Science in Transition, some speakers felt that they needed to ‘stand up for science’. Hans Clevers, president of the KNAW, and Jos Engelen, chairman of the Netherlands Organisation for Scientific Research (NWO) were noticably unhappy about the portrayal in the media of science ‘in crisis’. Both stressed that Dutch science is doing well, judging for instance from the scores on university rankings. Both radiated their aggravation about painting an ambiguous picture of science to outsiders, because of the potential risks of feeding already existing scepticism and mistrust. At the same time it was telling that these key figures in the landscape of Dutch governance of science were supportive of the debate and the fundamental points raised by the organisers.

Like Clevers and Engelen, Lodi Nauta (dean of the faculty of philosophy in Groningen) too, argued that not everything is going astray in science. According to him there are still many inspiring examples of solid, prize-worthy, trust-worthy, interdisciplinary, societally relevant research. But Nauta also signaled that there is much ‘sloppy science’. Not all symposium participants agreed on how much, and if there is indeed a general increase. Peter Blom, CEO of Triodos Bank, made an important aside. He thought it rather arrogant that whilst basically every other sector is in crisis, science should think it could distance itself from these economic and socio-political currents. But many participants took a cautionary stance: If there is indeed such a thing as a crisis, we should not lose sight of the nuances. It is not all bad everywhere, at the same time, and for everyone. Some argued that young researchers suffer most from current governance structures and evaluation procedures; that certain fields are more resilient than others; and that compared to other countries the Dutch scientific and scholarly system is not doing that badly at all. Henk van Houten, general manager of Philips Research, on the contrary, argued that ‘university as a whole has a governance issue’: The only moment that universities have actual influence is when they appoint professors at particular chairs. However, these professors are subsequently mainly held accountable to external funders. One is left to wonder which governance model is to be preferred: this one, or models companies like Philips put in practice.

At the heart of the debate on being open about the present crisis lies a rather dated desire to leave ‘the black box of science’ unopened. Whilst Lodi Nauta for instance argued – with Kant – that an ideal-typical image of science is necessary as a ‘regulatory idea’, the Science in Transition initiators deemed it pointless to keep spreading a fairytale about ‘a perfect scientific method by individuals with high moral values without any bias or interests’. Van Houten (Philips) and Blom (Triodos) also argued that science does not take its publics seriously enough if it sticks to this myth. Letting go of this myth does not amount to ‘science bashing’ – on the contrary. It is valuable to explain how science ‘really’ works, how objective facts are made, where the uncertainties lie, which interests are involved, and how science contributes through trained judgment and highly specialized expertise.

A hotly debated matter also relates to ‘black-boxing’ science: Who gets to have a say about proper quality assessment and the shaping of research agendas? André Knottnerus, chairman of the Scientific Council for Government Policy (WRR) pointed at ambivalence in discussions on these matters. We tend to only take criticism on performance measurement seriously if delivered by researchers that score high on these same measures. There were also differences of opinion about the role of industry in defining research agendas. (i.e. detrimental effects of pharmaceutical companies on clinical research. Obviously Philips was invited to serve as a counter-example of positive bonds between (bio-medical) research and commercial partners). And what about society at large? Who speaks for science and to whom are we to be held accountable, Sheila Jasanoff asked? (How) should researchers pay more attention to mobilizing new publics and participatory mechanisms, and productive democratisation of the politics of science?

Most speakers were of the opinion that we should move away from narrow impact measurement towards contextually sensitive evaluation systems. Systems that reward mission oriented research, collaboration and interdisciplinarity, which not only accommodate short-term production but also the generation of deep knowledge. These (ideal-typical?) systems should allow for diversification in talent selection, and grant academic prestige through balanced reward mechanisms and ‘meaningful metrics’. Though the symposium did a lot of the groundwork, how to arrive at such systems is of course the biggest challenge (see also Miedema’s ‘toolbox for Science in Transition’ for concrete suggestions). This is assuming it is possible at all. But perhaps we need this ideal-typical image as a ‘regulatory idea’.

The need for change in the governance of science

Tomorrow, a two-day conference will be held, Science in Transition, at the beautiful headquarters of the Royal Netherlands Academy of Arts and Sciences, the Trippenhuis in Amsterdam. Five researchers with backgrounds in medical research, history, and science & technology studies, have taken the lead in what they hope will become a strong movement for change in the governance of science and scholarship. The conference tomorrow builds on a series of three workshops held earlier this year about “image and trust”, “quality and corruption”, and “communication and democracy”. On the eve of the conference, the initiators published their agenda for change. In this document, 7 issues are defined as key topics and a large number of questions about the necessary direction for change are formulated. These issues are: the image science has in the public view; public trust in science; quality control; fraud and deceit; new challenges in science communication; the relationship between science, democracy and policy; and the connection between education and research.

With this list, the agenda is rather encompassing and broad. The thread running through the document as well as through the supporting “position paper” is discontent with the current governance of the scientific and scholarly system. The position paper is strong in that it is based on the professional experience of the authors, some of whom have been leading and managing research for many years. At the same time, this is also the source of some obvious weaknesses. The situation in the medical sciences is here and there a bit too dominant in the description of reality in science, whereas the situation in the humanities and social sciences is rather different (although equally problematic). Because the agenda is so broad, the position paper in its current version tends to lump together problems of quite different sources as if they are all of a kind. The subtleties that are so important in the daily practices of scientists and scholars tend to disappear from view. But then again, some of this may be inevitable if one wishes to push an agenda for change. A quite strong feature of the position paper is that it does not try to justify or deny the problematic aspects of science (of which fraud and corruption are only the most visible forms) but attempts to confront them head-on.

This is the reason that I think Science in Transition is an excellent iniatiative which deserves strong support from all participants and users in the current system of knowledge creation. Certainly in the Netherlands, which is the focus of most experiences the initiative builds on, but also more globally, the current ways of governing the increasingly complex scientific system hit their limits. Let me focus on the matter of quality control, the issue with which we deal regularly in this blog. The peer review system is straining under increasing pressure. Data intensive research requires new forms of data quality control that are not yet in place. Fraudulent journals have become a major source of profit for shady publishers. Open access of both publications and research data is increasingly needed, but at the same time it threatens to introduce corrupt business models in science and may harm the publication of books in the humanities (if not done carefully). Simplified but easily accessible indicators, such as the h-index and the Journal Impact Factor, have in many biomedical fields acquired the mantle of a goal in itself. Editors of journals feel pressured to increase their impact factor in sound and less sound ways. The economics of science is dominated by a huge supply of easily replaceable temporary labour force and for many PhD students there is no real career possibility in meaningful research. Peer review tends to favour methodological soundness above scientific or societal relevance. The publicly funded budgets are not always sufficient to perform the research as thoroughly as is needed. The current publication cultures tend to prefer positive results over negative ones (especially dangerous in the context of pharmaceutical research).

I realize that this short summary of some of the better known problems is as generalizing as the position paper. Of course, these problems are not acute in every field. Some journals are not afflicted with impactitis, but manage to focus on pushing the research front in their area. Universities behave differently in the ecology of higher education and research. Many researchers are delivering a decent or excellent performance. Scientific specialties differ strongly in epistemic styles as well as in publication cultures. And the solutions are certainly not easy. Nevertheless, the governance of science requires some fundamental adaptations, including a possible revision of the role of universities and other institutions of higher education. Science in Transition deserves to be applauded for having put this complex problem forcefully on the agenda.

I am also enthusiastic about the project because it resonates so well with the research agenda of CWTS. We have even created a new working group which focuses on the detailed, ethnographic, study of actual evalution practices in science and scholarship: EPIC (evaluation practices in context). We need to have a much more detailed understanding of what actually goes on in the laboratories, hospitals, and research institutes at universities. This is the only way we can supplement generalizing and normative statements about trends in scientific governance with “thick descriptions” of the complex reality of current science.

The more complex the research system has become, the more important quantitative information, including indicators, is for the researchers, research managers and science policy makers. This requires more advanced methodologies in the field of scientometrics (and not only in bibliometrics), such as science mapping, the topic of another CWTS working group. It requires more accurate data collection, including better accounting systems of the costs of scientific research. (Currently, universities actually do not know how much their research actually costs.) But it also requires vigilance against “management by indicators”. If young PhD students aim to publish mainly in order to increase their performance indicators so that they can have a career, as many a senior researcher in a hospital has experienced, we know that the system is in trouble.

Accounting systems are sometimes certainly necessary, but these should be put in place in such a way that they do not derail the primary processes (such as knowledge creation) that they are supposed to support. In the scientific system in the Netherlands, we therefore need a renewed balance between performance measurement and expert judgement in the quality control mechanisms. This is what we mean with our new CWTS motto: meaningful metrics. The future of scientometrics is not in the production of ever more indicators, but in more effectively supporting researchers in their endeavour to create new knowledge.