University differentiation in Europe : results from the AQUAMETH project
Although the role of universities in the knowledge society is increasingly significant, there remains a severe lack of systematic quantitative evidence at the micro-level, with virtually all policy discussion based on country level statistics or case studies. This project redressed the balance by examining original data from universities in six European countries – Italy, Norway, Portugal, Spain, Switzerland and the UK (the source of the 2007 book see below), enlarged afterwards to 6 new countries (including Australia which joined on a voluntary basis with their own funds). It has provided micro-based evidence on the evolution of the strategic profile of universities in terms of scientific research, contract research, education and the third mission.
The project started as a small exploratory effort considered as highly risky by the Scientific Council. Very soon (after the first presentation at PRIME conferences), new teams joined to build a consortium of 10 PRIME participants involved in the construction of university-based data bases, in the development of new econometric techniques and in their exploitation.
As such the construction of the databases was considered as a challenge, especially taking into account the descriptors asked for and collected at the level of individual universities (see box). Its success has been an important element of the selection of the consortium for the EUMIDA project which gathers data on some 2500 universities in 27 member states (the consortium is made of 5 core teams, all members of PRIME, surrounded by 27 national experts).
A second major achievement has been the development of new econometric approaches based upon the work by Leopold Simar. The Aquameth project has for the first time applied to higher education and S&T a family of new nonparametric techniques, based on the integration of statistical tools with established frontier techniques. These new techniques not only permit the estimation of confidence intervals, but also solve for the main limitations of DEA (sensitivity to outliers, need for large samples of curse of dimensionality) : robust nonparametric techniques (order m), conditional robust non parametric analysis, and bootstrap.
These developments are so visible that it has raised a strong interest within the PhD community. The project has organised two successive summer schools, each with over 25 doctoral students, having to operate a strong selection between applicants (Pisa 11-17 July 2007, Pisa 7-11 July 2008).
They are also visible within the econometric community : After presentations that were granted the best paper award, Pisa was selected to organise the XI EWEPA Conference (European Workshop on Efficiency and Productivity Analysis) (23-26 June 2009).
Furthermore, in view of fostering their use at the level of university managers, these new methods have been incoporated in a software (in a beta version) developed by Philippe van den Eckaut, Harold Fried and collaborators, under a small contractual provision of the project. The software incorporates almost all new developments in nonparametric efficiency analysis and permits the comparison between the efficiency of a given university and a reference group, extracted from the overall dataset on the basis of a principle of stochastic dominance. In other words, each individual university can be compared with dominating universities (i.e. those that employ less inputs and produce the same level of output or more) and dominated universities (i.e. those that employ more inputs and produce the same level of output or less). These two groups are clearly comparable with the target unit, because they are structurally similar in the input structure, but perform differently. Therefore the benchmark is really informative and can support the collective learning of university managers.
A third achievement has been to demonstrate the richness of such an approach. This has been done through a highly visible book (see box below), numerous academic publications, and through active interaction with stakeholders. A paper was presented at the OECD blue sky conference (Ottawa, September 2006). Members of the Aquameth team have been invited to take part to the EUROSTAT Working Party on S&T Indicators and on Higher Education, on a systematic basis. Important exchanges have also taken place with IPTS and UNESCO. Altogether only in 2006-2007, members of the project presented results of it in 39 conferences, book chapers and articles. It has continued since and only a small selection is listed below. Furthermore the project published a synthesis of Aquameth approach and results (January 2009) : “A micro characterisation of the European university landscape : evidence from the Aquameth project”. It is one of the rare papers in social sciences signed by more than 10 authors (26 in total from 20 institutions !), the corresponding author being a young researcher, Cinzia Daraio. It presents key results from 488 universities coming from 11 countries. We cannot account here for the richness and variety of results. Probably one facinating result is, whatever criteria used, the very limited differentiation that describes the European landscape. The exceptions of the UK, and to a lesser extent Switzerland and the Netherlands, are associated to a few universities whose research productivity is at the upper extreme of the European distribution. Another striking phenomenon is that trade-offs between “activities” are important, and for instance ‘research intensive’ universities may not belong to the highest performers in term of undergraduate education. The national comparative treatment that remains at this stage is in great part due to the rather limited (though unique) set (less than 500 universities). The new EUMIDA project, with around 2500 universities, will enable deepen and characterise forms of differentiation, and analyse relative performance depending on intrinsic features of universities (as organisations).