The University Outcry – Merits and Demerits of Rankings and Performance Evaluation University Reform and Far-reaching National Policy No Global Excellence Without National Competition - Discuss Japan
Discuss Japan > Back Number > No.21 > The University Outcry – Merits and Demerits of Rankings and Performance Evaluation University Reform and Far-reaching National Policy No Global Excellence Without National Competition
Society, No.21  Mar. 27, 2014

The University Outcry – Merits and Demerits of Rankings and Performance Evaluation University Reform and Far-reaching National Policy No Global Excellence Without National Competition

Why evaluate academia? 

UEYAMA Takahiro, Professor, Keio University

UEYAMA Takahiro, Professor, Keio University

Sustaining the world of knowledge is difficult in any era. As well as pinning high expectations on the benefits of the technologies and new ideas that emerge from universities, the patrons of society are always casting a stern eye over academia. In The Higher Learning in America (1918), Thorstein Veblen commented on this ironic state of affairs, “[…] idle learning has sought shelter in the university as the only establishment in which it could find a domicile, even on sufferance, and so could achieve that footing of consecutive intellectual enterprise running through successive generations of scholars which is above all else indispensable to the advancement of knowledge.”

Before growing to require the enormous financial foundation of today, academia maintained strained ties with patrons for its survival and was continually shaken by the evaluations of the external parties that supported its universities.

In 1977, the presidents and academics at research universities in the United States were gripped by a great sense of fear about the fate of higher education. Faced with severe cuts in the generous federal research grants that had been the norm since the end of the war, they came together to publish a report with the support of the Ford Foundation. In the report, Derek Bok, then president of HarvardUniversity, wrote that government support for universities was moving toward competitive funding, and that as a result of convergence and selection, projects tended to focus on narrow themes. Budget structures were checked down to the smallest detail and changes were hardly ever accepted. In addition, keeping detailed records and drafting documents related to the research occupied more than 20% of researchers’ time. The cuts to personnel costs had got the priorities wrong, increasing the administrative workload and hindering research. Academic research careers had been robbed of any attraction and excellent students were no longer applying to doctoral programs. If things continued like this, higher learning in the United States would be under threat, causing the national power of the country to stagnate. Is it not as if we were listening to someone talk about the university environment in Japan today?

This was also the time when American universities started to reach out more to society. They disclosed information about themselves, welcomed external evaluations, deepened relations with industry, and became more proactive about promoting academic life in society. Turning to Japan, has academia here provided sufficient information to stakeholders in the ten years since the start of the transition to national university corporations? Have the administrative authorities exercised creative policies for higher education? This short article is an attempt to answer these questions.

The response to global university rankings and Americanization

Society bashing universities is not a recent phenomenon. Thinking back, there was, of course, the period of student unrest, but also when I was at university, the boredom of lectures where the same teaching notes were recycled year in and year out was fair game for ridicule in mass media. You often heard researchers in the natural sciences, particularly in the engineering fields, at corporate research institutes making dismissive statements about university level research. However, the recent tenor of the argument in Japan has escalated into strong criticism of both the university structure and the system itself.

One of the triggers is presumably that in 2011, the University of Tokyo tumbled to 30th place in the world university rankings compiled by Times Higher Education. It seemed that Japanese academia, the first to modernize in East Asia, and the first to achieve success with the introduction of the Western university system was about to lose its leadership position, not only globally, but also in Asia. This aroused strong discontent, not only among politicians who are sensitive to the economic strength and reputation of the country, but also among the corporations that have to face up to the global front lines in their business activities.

This phenomenon is not limited to Japan. In a global competition one might well call the second coming of mercantilism, university world rankings have come to be regarded as an unparalleled index of national power. The watershed came with the Academic Ranking of World Universities published by ShanghaiJiaoTongUniversity in 2003. Today, several organizations including LeidenUniversity, the Consejo Superior de Investigaciones Científicas (CSIC) in Spain, and the European Commission publish world university rankings using their own criteria. Worldwide, there are probably more than thirty types of university rankings with varying degrees of influence. Among them, the university rankings by the Times, first in cooperation with the QS World University Rankings from 2004, and then with Thomson Reuters from 2010, are perceived as the most authoritative.

Now, what circumstances and mechanics are at work behind these academic rankings, which are not unlike the competition for stars among restaurants in the gastronomic world, and which have increased as quickly as bamboo shoots after rain in the past ten years?

I view the trend as a counter to the Americanization of the world of knowledge. Since the Second World War, countries have kept an eye on the integration of globalization and Americanization with awareness of how to protect the culture, ambience and political systems of their own countries coming to the fore. Now, academia has been placed on the front lines. Originally, it was military affairs, the economy, and diplomacy, but now that academia, the source of the foundation of knowledge that supports the activities of the nation, is squaring up to global forces, interest in universities as the symbols of academia has been stirred up. Universities have acted as stores of knowledge, sharing the wisdom of humanity, but now we are starting to see competition between nations baring its teeth.

In particular, the fact that China and some of the European countries have fired the first shots in the university rankings is perhaps best seen as a symbolic manifestation of the current clash between global forces. This state of affairs is extremely interesting to the author who has observed transitions in academia and their global impact with a focus on the United States. The full-scale transformation of American academia started in the 1980s, and in the 1990s, the American idea of reconfiguring next-generation economic, military, and diplomatic knowledge through the university organization and systems came into its own. Countries have now started to plainly see the foundations for the American intellectual strategy in the university system started more than twenty years ago.

In particular, the competition with American universities has become a major threat to the main countries in the European Union (EU). There is a pressing need to see things in a new light as the universities, which until now have shouldered the history and culture of their individual countries, are expanding beyond the EU, and in the context of systematic change that is converging on English as the common language. At the same time, they have had to face up to the reality of American universities luring away excellent people. It began with star researchers moving between universities in the EU, but with the introduction of English as the international language, the distance to the American universities has also shrunk. There is no doubt that university administrators in the European countries believed that the rankings would be essential for rebuilding their knowledge strategies. In actual fact, the flow of people from the top schools in Europe, including Oxford and Cambridge in Britain, to American universities still shows no sign of letting up.

Today, the role of universities as the source of knowledge is more important than ever. Therefore, the focus is on research competency as a measure of the ability of universities to produce knowledge, and university performance is measured on the basis of the competency of its researchers. The ranking of Shanghai Jiao Tong University is a typical example of the importance of the so-called sciencemetrics (indicators such as more researchers writing more articles, employing researchers who write high-impact articles, advancing joint research globally), which plays an important role in these rankings. As far as the measurement of the indicators is concerned, many people claim that they do not accurately reflect the competency of a university. For example, many international joint authors have been dragged in against their will, and there have also been cases where researchers have cited each other’s works in order to bolster the index.

Setting aside the degree of intention on the part of the government, recent interaction with Japanese academia has also taken a similar course. After the collapse of the economic bubble and a period of economic woes, the government enacted the TLO Act (Act on the Promotion of Technology Transfer from Universities to Private Business Operators) in 1998 in order to accelerate collaboration between industry and academia, and in 1999, the Japanese version of the Bayh-Dole Act, which aimed to promote the rights of universities to retain patents. In the late 1990s and the 2000s, it became necessary to consider restructuring the system of academia. The phenomenon of university rankings should be perceived in the context of such transformation of the global world of knowledge.

However, in light of the above, is it not more surprising that Japan has not designed a world university ranking of its own? I do not endorse the simplistic thinking that introduces rankings to universities, which are places of learning. Even so, I cannot help feeling let down by the fact that the administrative authorities in Japan, who have been pursuing university reform, have not come up with this idea. It feels like a flaw in the creative strategy for designing technology and higher education based on a far-reaching national policy.

American university rankings and national competition

Turning to the United States, how do American universities and administrative authorities view the recent fad for global university rankings? In actual fact, the earliest efforts to evaluate and rank universities were made in the United States. As early as 1925, Raymond Hughes at MiamiUniversity published an attempt to compare the performance of graduate schools in the United States based on peer reviews. He continued with the evaluations and published a similar ranking in 1935. Hayward Keniston in 1957, as well as K. Roose, C. Andersen, G. Carter and C. Conrad and R. Blackburn in the 1970s and 80s, published successive evaluations. In addition, government agencies such as the National Science Foundation (NSF), the National Institutes of Health (NIH), and the National Aeronautics and Space Association (NASA), which provide research grants to universities, as well as the Ford Foundation, the largest supporter of higher learning in the United States, have commissioned endless successions of surveys for the “strategic” allocation of research support grants.

In light of these historic facts, it would seem that there is essentially no difference between the recent global rankings and the experience in America. At present, the Executive Committee of the American University Alumni Association, Forbes, the National Research Council, and university evaluation centers publish numerous national university rankings in the United States. But, the university staff and administrative authorities in the United States hardly show any interest in the university rankings published by the Times and others. Rather, they look at the national rankings and evaluation of universities. Why is that?

Of course, even if they were to look at the Times rankings, three quarters of the top twenty rankings are occupied by American universities, so there is not much of a threat from universities in other countries. It is unlikely that the managers of an American university hoping to elevate their university to the top ranks in the United States would consult the global rankings even if they had ambitions to lure the top researchers in their fields, not only from universities in the United States, but around the world, to do so. Rather, according to American universities, it is precisely the intensive competition between universities at the national level that supports the international rankings of American academia.

Something else that must be noted is that the university evaluations are based on the disclosure of robust internal information about the universities. For example, these days many people probably consult the U.S. News & World Report for the university rankings. First published as Best Colleges in 1983, this ranking mainly evaluates undergraduate courses, unlike earlier evaluations that rated programs at graduate school level. In addition to interviews with the presidents, provosts and directors of admissions offices at various universities, assessments by graduates and experts in the industrial world, and evaluations by admissions officers, the indicators include average class size, staff salary levels, the percentage of doctorates, the ratio of students to faculty member, graduation rates, the number of first-year students repeating classes, and other data, even the financial situation of the university and the performance of the college fund are included for a comprehensive evaluation based on a wide variety of internal data.

Incidentally, this kind of information is difficult to come by where Japanese universities are concerned. Regardless of whether the university is public or private, they do not even publish basic data such as withdrawal rates or graduation rates. People outside the university know absolutely nothing about how subsidies such as management expenditure grants to national universities, or subsidies paid out to private universities are allocated across the different departments in the university, or what the departments use the money for, or which departments procure a lot of competitive funds, how the funds are allocated within the university, or other important information. The financial situation of a university, which holds important clues to its future, is essentially a black box.

Some publishers in Japan are publishing so-called university rankings, but since the data disclosed by universities are limited, the analysis is not persuasive. The fact is that they fall far short of indicators like the overseas rankings, which have a major impact on university administration and university management policy. The ranking of universities in Japan is, unfortunately, not decided by the track record and educational performance of researchers at universities, but by the standard deviation values of students sitting university entrance exams. The rankings of Japanese universities are trivialized by the tendencies of eighteen- and nineteen-year-old students taking entrance exams. Even if a university wants to improve its ranking, the officials only try to attract excellent applicants in Japan, and there is no incentive to improve the level of research at the university. So, how should we view the global rankings under these circumstances?

No global excellence without domestic competition

In the first place, the rankings should be premised on the recognition that the ranks of the survey subjects change. However, in Japan, no distinction is made between research universities and teaching universities. At the top of the hierarchy, we have the University of Tokyo and then it goes all the way down to what I might call marginal universities, and there is never any change. In short, the concept of rankings has essentially never existed in Japan. There is not even any awareness or environment for universities to compete against each other in the aim for annual rankings.

Here, I should point out a special feature of the American university rankings that is worth mentioning: the focus is on assessments by field and by department. Ever since the work of Professor Hughes in 1925, the format has been to take mutual decisions about the competency of graduate school programs in each field of learning, and today the ranking of each department over time is clear at a glance. Consequently, rather than comprehensive indicators to determine which university is at the top, the American rankings are used as criteria for education and research at each department and as the driving force for the overall management of the university.

With the help of the ranking indicators, university managers find out which areas of their universities are strong. In addition, they are able to take management decisions such as where they should invest the resources, or what kinds of researchers to invite in order to improve their reputation. For scholars as well, the annually fluctuating ranking of departmental strengths provides reference points when making judgments about transfers to another place of employment. In 2000, when Lawrence Lessig, the lawyer who made his name by advocating the Creative Commons, announced that he was transferring from HarvardLawSchool to Stanford, American media made a fuss and speculated that the reason was that Stanford had bypassed Harvard in the previous year’s rankings by the U.S. News & World Report. (In actual fact, Lessig moved to be near his partner’s university.)

The interdepartmental rankings are also essential for formulating a long-term vision for universities. If the differences in competencies are identified by department, it becomes easy for university managers to review the internal resources of their universities, and on this basis, to produce an original vision for their university and to paint a picture of what direction management should take in the next ten years. Historically, this is how American universities have used the university rankings. To put it in today’s language, this is the concept of Institutional Research (IR).

Even if the overall global rankings for a university are lined up side by side, it is difficult for university managers to identify the clues to produce a strategy for their university because the rankings move up and down. On the other hand, if there is clarity about departmental strengths and weaknesses, it is possible to make management decisions about investing university funds in viable departments, or handing over comparatively minor academic disciplines to other universities.

The reputation that a university has acquired is also reflected in the research grants that researchers obtain. In the natural sciences, in particular, where huge sums of money are needed, staying with a university where it is possible to get the equipment that will allow the research to move forward and where there is high potential for joint research, or one that has the equipment and systems to attract excellent graduate students as well as a good reputation are very effective ways of obtaining research grants. In that sense, improving the rankings by discipline is linked to an improved standing for the university as a whole. We might even say that the university evaluations have solidified into a foundation for governance and management at each university.

But what about Japan? The drop in the global rankings has done nothing to change competition between universities in Japan. For example, KyotoUniversity or OsakaUniversity have not thought of luring people away from the top schools in the United States to overtake the University of Tokyo. Essentially, the principal research universities in Japan do not regard the other universities in Japan as potential competitors aiming for the top spot in the country. Where there is no competition between national universities, there is also no global excellence. This is a keenly felt conviction with me.

What is creative evaluation?

So, is the purpose of a university evaluation to check that universities are implementing teaching and research above a certain standard? The university evaluation systems implemented in the past ten years seem to monitor whether or not all universities meet the standards for research and teaching at a suitable level for higher education. The administrative authorities decide the standard of education that should be achieved, and check whether or not the standards have been met. This is a manifestation of an amazing lack of confidence in universities.

But, is it right to pursue policies that force organizations like universities with extremely varied purposes and social values into the straitjacket of uniform evaluation? Policy should be something more creative. It should not stop at the inertia of checks and management, but bring diversity to the evaluation criteria, and on this basis, confirm the potential of each actor to take decisions while visualizing the intricacies of the different directions for each university. At the same time, policy should also stimulate the private interests of each university that wants to improve its reputation and create a blank map for the whole world of knowledge.

Perhaps politicians and the media overreact to the trends in the simplified rankings. However, for the administrative authorities with responsibility for the future of Japan as a nation, producing an attractively dynamic and complex blank map is not comparable to simple evaluation and management, but it is a far more creative task. This is where the strategic power of advanced nations is put to the test. For Japan, which has been thrust into the advanced knowledge-based society, rebuilding a world of knowledge filled with creativity is an urgent task. I believe that the future of the nation hinges on whether or not the leadership is able to undertake this difficult task.

Translated from “Tokushu: Daigaku no Himei – Rankingu to Gyosekihyoka no Kozai / Daigakukaikaku to Kokka no Taikei –Kokunaikyoso ga naitokoroni Gurobaruna Takuetsu wa nai (Feature: The University Outcry – Merits and Demerits of Rankings and Performance Evaluation / University Reform and Far-reaching National Policy – No Global Excellence Without National Competition),” Chuokoron, February, pp. 54–61. (Courtesy of Chuo Koron Shinsha) [February 2014]