Evaluation of a EU research funding line
The study evaluated approximately 155 projects in Key Action 2 of the Specific Programme ‘Quality of Life and Management of Living Resources‘ under the Fifth Framework Programme of Research that deals with human health. In this Key Action, projects were funded in three thematic areas: Development of improved or novel mono-component, multi-component and combined vaccines Strategies to identify and control infectious diseases Aspects of public health and care delivery systems Most of these projects were still underway at the time of the evaluation.
The study’s goals were to assess the contribution of the funded projects to the scientific progress in the control of infectious diseases, to identify the results and (possible) impacts of the funded projects, and to determine the factors inside and outside the projects that had an effect on their scientific and economic results and impacts. Building on this, recommendations were made to improve the awareness in the relevant target groups concerning the results of the projects.
The evaluation made use of a combination of methods including the analysis of documents and interviews. The methodological framework was innovation research and meta-evaluation as aspects of ‘research on research’. The projects were classified according to several criteria, e. g. their goals and actors involved, and were integrated into a database. To evaluate the quality and results, objective indicators like numbers of publications or patents that result from the project work were used as far as possible. Exceptionally interesting projects were analysed in more detail as examples for "best practice".
The study has resulted in an overview of the research in Key Action 2 and the projects funded in this Key Action that includes their scientific and technological contribution to their respective thematic area. Suggestions were made to improve the awareness of the impacts of the research projects and to support the utilisation of the projects' results. Four main lessons learnt are presented here: The selection of highly motivated and methodologically skilled participants is of central importance for the success of a consortium as is a strong and experienced coordinator as well as good communication between the partners. More emphasis should be given to the involvement of industry and SMEs as partners because this is an adequate means to bring products onto the market quickly. Academic researchers can learn about the needs of the industry by direct interaction with industry partners and by exchange of personnel within the consortium. Interaction with related EU and international research should be strengthened. Whereas dissemination to the scientific community was very extensive, more emphasis should be given to inform the public and the political level on the practical consequences of a project e.g. by press releases, information events dedicated to policy-makers or even TV broadcastings. Although a number industry partners were attached to the projects, more could be done to exploit the results with the industry, e. g. in the form of information events. Other examples in KA2 projects are publications not only in scientific but also in business journals.
A strong science base has been developed in the KA2 projects. Resulting from the limited budgets and funding periods, a danger was seen by many projects to lose the elaborated knowledge after the finalisation of the project. Measures should be implemented like mailing lists or thematic expert networks, probably in the framework of the new European Centre for Disease Prevention and Control (ECDC), in order to preserve the contacts of the Commission to successful coordinators and contractors and to keep track of the well-functioning networks. As the development of new drugs and vaccines bears many risks, more flexible funding schemes should be allowed to assure that probable drawbacks can be coped with. The experiences with the present impact evaluation show that such projects are possible in principle and can lead to useful and applicable conclusions. It should be considered for similar projects in the future to draw more on the project leaders as a source of information that is not systematically given in the periodic reports.
For future similar impact evaluations, a two-step approach is suggested: A more systematical internal registration and assessment of routine data which are already available yet dispersed at different Commission services. The methodology of the actual project could support this. A more comprehensive external evaluation to assess more in-depth the impacts of the programme with additional instruments and research questions specifically targeted to the requirements of further research policy making. The respective experiences of the present project should be used.
Duration: 01. Jan. 2003 – 31. July 2004
Funding: European Union (FP5)
Coordination: Bernhard Bührlen at Fraunhofer ISI
-----
Picture: Helicobacter pylori, by Janice Carr (CDC)