Discuss Japan > Back Number > No.21 > The University Outcry – Merits and Demerits of Rankings and Performance Evaluation
Universities losing their ideals to internationalization and the mass-production of academic papers

You Can’t Rank Academics
LinkedInTumblrDeliciousYahoo Bookmarks
No.21 ,Society  Mar 27, 2014

The University Outcry – Merits and Demerits of Rankings and Performance Evaluation
Universities losing their ideals to internationalization and the mass-production of academic papers

You Can’t Rank Academics

The pros and cons of converting everything to figures

INOKI Takenori, Specially-appointed Professor, Aoyama Gakuin University

INOKI Takenori, Specially-appointed Professor, Aoyama Gakuin University

We find ratings and ranks in so many fields these days. Ratings – numerical indices that put things in order – have some kind of quality that stimulates people’s curiosity and arouses competition. When we are shown international rankings or levels such as the per-country rating list of the Olympic gold/silver/bronze medals won, we have the impression that the strength of something has been measured “objectively,” and nationalism of sorts rises in us. Ratings offer us immobile criteria with which to make judgments, which may become the grounds for drafting a policy or a change of politics. But this “ratings fever” tends to create radical competition based on collectivist psychology, making us lose sight of the true purpose of things. Let me offer just two examples from my familiar field of education and research.

Late last year, the Organisation for Economic Co-operation and Development (OECD) released the 2012 results of its Programme for International Student Assessment (PISA) survey, which it conducts every three years. Media reported that according to these results, Japan not only ranked high in reading, scientific application and mathematical application skills, it also improved these scores from the previous survey. The Ministry of Education, Culture, Sports, Science and Technology (MEXT) commented that this was due to its policy shift away from Yutori education (“pressure-free education”). There were no detailed analyses of what caused MEXT to change its policy or how the shift away from Yutori education affected the survey results.

What exactly does it mean when the survey says that the people of countries and regions that ranked highly had better scientific capabilities and reading skills than those of other countries? The issue of “average” and the issue of “distribution” – the number of young people who display extraordinary talent – are issues that should be in two completely different realms. The traits of the region or school subject to the sample would change the scores, and the results would change significantly depending on what questions are posed. We should not become emotional over trivial changes in ratings.

Yet figures and ranks take off on their own. And as “objective data,” they become extremely political tools on a practical level.

Adapting to the rating categories

Another example is the college ratings produced by numerous institutions and media. I admit that there needs to be a way for people outside a college to know what research and education the college undertakes. Allowing people outside the college (such as applicants) to have ready access to information on the quality of a college’s education or research is not only necessary for the optimal college selection, but it also gives the college a reason to make efforts to improve its quality. In economic terms, when you have inequality in terms of product information possessed by both the seller and the buyer, fair trade will not last. It is the same logic as, say, ratings serving as credibility indices of bonds to guarantee their fair trade in the market. We should not neglect the importance of evaluating colleges.

But when it comes to college ratings, the social side effects are significant. World college ratings usually include U.S. and British universities in the majority of the upper ranks. One wonders what categories are involved in deriving these total scores. I undertook a study and found that it was not only about how many times research papers cite the college, its ratio of students to faculty, or the number of foreign students and faculty. Factors that exercised significant weight were its reputation among economic circles and evaluations from fellow experts. This means that this rating is merely a quantified and tallied score of practical benefits within established knowledge and assessment from a “family” circle, rather than an estimate or assessment of creative “knowledge in the making” concerning human research that forecasts a long span of time into the future. The rating is, in other words, based on an extremely conservative, one-sided and short-term measure of evaluation.

One of the reasons why I cannot erase my suspicions is in the fact that rating colleges and giving them ranks trivializes the essential issue of colleges. A typical example is how prominent colleges announce their goal and intention to “rank within the top X within Y years,” which sounds like something athletes would say about how they plan to work out. MEXT’s goal to “have X people win the Nobel Prize within Y years” also contains the preconception that wisdom can be produced by spending money, and I am sure that I am not the only one who feels a sense of gloom over this. A nation can obviously support scientific and technological research as a national policy. But when it comes to the humanities and social sciences, state-led research rarely produces decent results.

The raison d’être of colleges ultimately lies in their capability to freely debate the truth (at times a truth that is inconvenient for the majority), rather than to undertake education and research under top-down command or guidance. Yet they cannot stipulate in advance what creative and intellectual activities are. That is why colleges in a free society must keep the door to the truth open at all times. When they start forgetting this important role, they no longer have a reason to be colleges.

What makes things even worse is that college operations, their research topics and education tend to be motivated toward matching the standards of these ratings (toward raising standards that are the evaluation criteria of ratings). For example, increasing the number of foreign students raises rating scores, so colleges think about increasing foreign students. And to do so, many Japanese colleges today are desperately attempting to offer more classes in English for foreign students. The issue of colleges has now become an issue of language.

Internationalization and English fervor

This English issue has, in fact, been a problem for the Japanese ever since the Meiji period (1868–1912). The same issue has been the subject of debate on so many occasions. One of these debates involved the idea of making English (or its simplified form) an official language, and another involved seeing the decline in English competency as a problem. Looking back at these specific examples as we keep in mind similar debates seen in contemporary Japanese industrial circles or in some colleges, we find that there is a lot to learn.

Let us first look at the debate on English as an official language. Mori Arinori, a diplomat and education official in the Meiji period and the first Minister of Education to serve in the Ito Hirobumi cabinet, suggested (1872) that Japan develop a simplified English system, and was one of the people who advocated (1873) that English be an official language. His idea was that the irregular conjugations seen in English verbs and nouns are too complicated, so they should be rationally standardized – in other words, instead of speak, spoke and spoken, it would be speak, speaked, speaked; or instead of think, thought, thought, it would be think, thinked, thinked. Mori sent his suggestion to Japanese and American experts for their comments, and one of them, Yale College linguistics professor William Whitney, returned to Mori a courteous but quite scathing criticism based on his expertise (June 29, 1872). Professor Whitney believed that while people tend to consider language as an instrument of communication, it is, in fact, a means of thinking that intimately relates to a society’s culture and history, and I am sure that I do not need to cite the details of his criticism against Mori Arinori here.

Mori, overwhelmed by the amount of western literary documents that entered Japanese society within a mere five to six years after the Meiji Restoration, and by their technical level, went further in that he believed that there was no way for the Japanese people to compete politically and economically other than by turning them into English speakers. Ideas similar to this are not all that rare, even in today’s society. Mori lamented that our Japanese language was such a weak language compared to English, which commercially superior people were using, and stated that the nation as it was would not be able to adapt to the age of steam and electricity.

Fukuzawa Yukichi sharply criticized Mori’s ideas (“Jinbouron,Gakumon no Susume (An Encouragement of Learning), Vol. 17, 1876). Fukuzawa said, “There is a scribe who says that the Japanese language is inconvenient and cannot create a written text or speech, and who is a complete fool in saying that we should use English and write English. As I see it, this scribe has not yet used the Japanese language sufficiently since his birth as a Japanese. A nation’s language gradually increases according to the amount of things and events that occupy the nation, and its writing should not have any disabilities. Anyway, the Japanese people of today need to learn how to use their current language skillfully to improve their speech.”

Fukuzawa did not mention Mori’s name here. But when he spoke at a meeting at Enzetsukan on June 7, 1874, the year following the announcement of Mori’s idea, Fukuzawa referred to Mori’s “English as the official language” argument and thoroughly refuted it: “I object to this. There is not a single reason why a nation’s citizen cannot use his own language, in which he freely converses, to talk to the public. Our nation has had – and has to this day – a great custom of speech. Have you never heard of lectures from temple priests? Even if you have not heard of priests’ lectures, you have heard of war stories at the theater, or comedy stories. All of these are about a person facing a mass audience to say what he thinks, which is exactly what speech is.”

Would more English raise the quality of colleges?

So what do these historical examples tell us? One clear point is that seeing language as a mere means of communication and artificially forcing it on people using top-down power does not necessarily help the development of society. In terms of current issues, this is like the idea of trying to increase the number of foreign students by creating a college curriculum to meet foreign student needs and increasing the number of classes offered in English, even in the humanities and social science fields, which does not sufficiently take into consideration the major side effect of lowering the ability to think. Language moves and changes on a great force called social necessity; it does not achieve goals that administrations and the ruling people suggest and want to achieve. Really, I am sure there are a lot of students who are smart enough to think that they would rather take English-speaking classes in a natively English-speaking country than listen to English lectures offered in Japan.

The experiences and observations of the people of the Meiji period once again offer an insight into this point as well. Natsume Soseki begins his Gogaku Youseihou (1911) with a discussion on the Japanese people’s decline in English skills and its causes. What we must note in his argument is that he focuses on the relationship between foreign language skills and national strength. Soseki pointed out that when a nation gains power and its education and industry develop, its motivation to learn foreign languages weakens.

In higher education institutions in Japan immediately after the Meiji Restoration, there were no Japanese textbooks on geography, history, mathematics or animal/botanical sciences; everyone studied from foreign language textbooks. There were times when students wrote answers in English. Even in Soseki’s time, it was not rare to see Japanese lecturers teaching math in a foreign language, so we see that English education back then had a fundamentally different density than that of later times. When Japan solidified its fundamentals of national survival, Japanese education was normalized, and Japanese teachers lectured using Japanese textbooks. With this, the level of English skills among Japanese students naturally dropped. In other words, the drop in English skills was an obvious result of a rise in Japan’s national power. How well a language is accepted is closely associated with national power, and the language of a strong nation, like currency, has greater circulation.

Soseki’s point coincides with what I felt about English acceptance when I traveled abroad. One aspect was the fact that small nations in Europe have numerous people who are fluent in English. This is because their economic life requires that they speak English as a social necessity. On the other hand, in the United States of forty to fifty years ago where I spent time as a graduate student, there were unexpectedly few people who were fluent in a foreign language. They probably had no need to learn a foreign language.

Similar effects are found in Asian nations (though with some exceptions). In developing countries or countries with a history of colonialization, many people in leadership positions are fluent in a foreign language. In other words, if a foreign language is deemed indispensable to surviving in that country, more people naturally learn that language. We can see a clear example of this in sumo wrestlers of foreign nationalities and their Japanese conversational skills.

MEXT, meanwhile, is encouraging colleges to offer more classes in English for the sake of foreign students and to train “global human resources.” (We will avoid discussing the issue of what that means here.) But I wonder what kind of human resources would come to study in Japan in the hope of taking English classes? Would highly competent students who do not speak Japanese but who can communicate in English not go and study in the United States, United Kingdom or any other “authentic” English-speaking country? It may not necessarily be so in polytechnic fields, but at least people in the humanities and social science fields who wish to come and study in Japan mostly have advanced Japanese proficiency already.

Measuring colleges’ humanities research performance by weight?

I just noted “may not necessarily be so in polytechnic fields.” While this distinction between the science and humanities fields does not have a particularly clear meaning, we need to consider the differences in the two fields’ characteristics when evaluating colleges or considering budgets. This is because in our current state, where science is far more strongly associated with money than humanities and takes precedence in anything, much of college evaluations and what a college decides in terms of budget allocations are judged based on science standards. Let us look at the nature of the science and humanities fields, which never go together in terms of evaluation and budget.

Firstly, regarding the overall evaluation of colleges. Society often evaluates education/research in humanities and social sciences carelessly using criteria for natural sciences and engineering. For example, one criterion for evaluating college research performance in many science fields is based on how many academic papers have been published in authoritative, international (and English) academic journals that work on the system of peer review. But the evaluations of people in the same field tend to become conservative, as their mainstream factions form a key consensus. This system of peer review is starting to take root in the humanities and social science fields as well, but while it plays a role in eliminating papers of poor quality, it lacks the power to discover excellent ones due largely to its conservative nature.

And even in terms of how many times other researchers have cited a paper, there are not necessarily all that many other researchers in the humanities field who are capable of undertaking evaluations. In other words, there are fields that do not have standard methods of submitting research, how research is done and how it is verified. A mathematician acquaintance of mine says that even in the field of advanced mathematics, which is hardly political, “Judging performance based on the frequency of citations as a key index is dangerous. There are so many outstanding pieces of research in mathematics that other researchers won’t cite.”

Language plays an important role when we think of cultural “pleats.” Similar issues do exist in the research fields of social science as well, despite differences in their levels. We cannot ignore differences in social structures and come up with a quick generalization. Every language is deeply rooted in the culture and civilization of its society, so it goes without saying that the humanities would have many fields of study that do not have international journals of high quality that use English as a common language. Their academic concepts would be defined by their country’s language, so when transferred to the standard language of English, the conceptual differences – the differences in cultures that arise as “pleats” – would be lost.

This is evident when you think of how one would lecture on a country’s history or philosophical history in English. You could probably carry out introductory overviews in English. But could you – taking Japanese philosophical history as an example – explain and expect people to understand in English what makes Keichu’s works so great, or why Kamono Mabuchi won so much respect, or why Motoori Norinaga’s works were amazing achievements? To whom would you direct that effort in any event? Could you explain the heart of the Hinokami debate between Motoori Norinaga and Ueda Akinari in English? And even if, by some slim chance, you were able to do so accurately, who is going to read that English paper?

But that does not mean that the works of the humanities and social science fields are not “evaluated.” Each college should be hiring and promoting individual researchers according to their research and education skills and personalities. Everyone knows that human resources evaluation based on distorted criteria results in the downfall of an organization. My point is that the humanities and the sciences are significantly different in their notions of achievements, what they measure them with, and the time span they measure them in, and we need to take these differences into consideration.

While basic training to produce researchers is required in the humanities and social sciences, many research subjects in these fields do not compete with fellow researchers of the same era and field. And studies that fellow researchers do not accept with open arms still have the possibility of becoming excellent studies. These questions lead to the question, “What authority is academia ultimately falling back on?”

A professor once said that to evaluate academic performance in the humanities field, we ought to put all their books on a scale and weigh them. This is indeed a statement that we can nod our heads to. Of course, this “weight value” theory is neither a necessary nor sufficient condition for evaluation. But in the sense that it serves as an alert against introducing easy measurement standards to a difficult task called evaluation, it is worth listening to.

Neither majority vote nor autocratic


The truth is not something that power or the majority vote decides. The method of evaluating research by giving greater value to those that were cited more often clearly indicates the atmosphere of the modern democratic society. But rousing one’s motivation for intellectual discoveries with an eye on politics or the market is different from the non-egoistic passion for truth that burns inside some few individuals. It is not about which is right or wrong. Political ambition and economic desires are themselves important engines for social advancement, and we cannot deny that. My point is that we should not confuse the desire to use knowledge to one’s advantage with the pure desire to understand.

Great scholars are mostly “exceptional” people who possess the intellectual and psychological energy to overcome worn-out rules and poor research environments. To isolate these exceptional people from system reforms and budget allocation battles that consume time and energy and to protect them are more important in the long history of academia than making them compete to mass-produce papers. I am sure that many researchers in the humanities and social science fields would honestly wish for time ahead of money.

Many science fields can be compared with the devices industry in the production business in that they are capital-intensive types that cannot operate without machinery. On the contrary, most humanities fields only require books, the costs of fieldwork, paper and pencils, the costs of hosting research seminars, and a computer to process data. Their biggest capital is time to think a lot. “Time is money” is their only principle.

What goes on today at colleges central to humanities/social science research that have been allocated ample research grants? Although I don’t know all the circumstances, I hear that the “strong” colleges that were focally assigned research grants are busy trying to use this budget on symposiums, business trips, inviting foreign professors, and renting offices off-campus. In any case, we can easily imagine how much research time is taken away from young researchers who usually take on such work. This is where we see the tragedy and drama produced from equating science studies, which have turned into a device industry, with the humanities, which require time as their greatest resource.

No stability of mind without a stable occupation

In April 2004, national universities were made into corporations and strengthened their nature of merit-based, competitive ideals in both the research and education aspects. As the principles of evaluation kicked in, they began allocating resources based on evaluation results. The fact that this corporate shift of national higher education and research institutions was undertaken as part of administrative/financial reforms and not by choice based on internal necessity has since started to have various effects, both large and small, on university operations. Strong colleges would become even stronger, and modest national universities that have contributed to the local area are now pushed into a tight corner in terms of budget and human resources, which is one of these effects.

We would assume that the corporate national universities have now heightened their independence, but since there are no significant changes in their income structure, MEXT involvement appears to have strengthened more than before. Japanese national university corporations, whose income is basically paid for by taxes, are run on the national budget (flow). The ImperialUniversity that was established in late nineteenth century Japan was not given land or assets (stock), so it was self-evident that it would come to rely on the national government for its financial aspects. It therefore began as an organization with a completely different financial groundwork than the universities in Europe and the United States. Considering that major universities in the United States already existed before the nation called the United States of America was established, the level of economic and psychological autonomy of U.S. universities, which have their own assets, is highly convincing.

Yet Japanese national universities, even under their current corporate state, basically gain their income (current cost) from the operation subsidies granted by MEXT every fiscal year. Income from side businesses is extremely low, particularly for national universities. The financial groundwork of Japanese national universities is extremely unstable and weak without operation subsidies.

Then how is it for private universities? Over three-quarters of undergraduates and one-third of graduate students are private university students in Japan. As these figures clearly indicate, private universities are quantitatively responsible for the major portion of Japanese higher education, particularly undergraduate education. The Act on Subsidies for Private Schools (enacted 1976) stipulates that up to half of ordinary expenses could be paid by subsidies, but the actual rate of subsidies remains at around 11%. Private education subsidies are extremely low compared with what the law guarantees.

Unless major Japanese universities work to develop as many assets as possible on their own and move toward running free research and educational operations based on income from those assets, they will not only fail to take advantage of the true benefits of their corporate nature, but will also fail to diversify. Won’t the so-called “flow of drafting estimate requests” of Department      University      MEXT     Ministry of Finance tend to produce leaders of universities and research institutions who would submit to not taking risks? Aren’t they entering a state of “No stability of mind without a stable income source”?

Since turning corporate, universities now have more papers to submit to MEXT than before. MEXT’s likely excuse is that in these financially tight times, it needs to see reasons that will convince it to acquire a budget. A friend of mine who recently finished serving in the position of Dean of Economics told me that the work upon which he and the head of graduate courses had to spend the most time was rewriting the new definition of the mission of the Economics Department and Graduate School over and over again in accordance with MEXT guidance. This is a guidance that may lead MEXT to demand colleges to draft academic teaching guides similar to those for elementary and secondary education. The deans and executives of each department are, therefore, forced to put a lot of effort into such work, sacrificing their primary roles of educating and researching.

There are no academic entrepreneurs

Running a university is not an easy task in practical terms that can make a researcher into a business owner overnight. So universities need to delegate their management aspects to an academic entrepreneur on advisory work. In this case, you could imagine an academic entrepreneur as “an administrative person who has completed a graduate course and has research experience.”

A university needs experts on asset management and other business operations. Rather than have researchers negotiate with MEXT in an unfamiliar context, it should develop a system where an academic entrepreneur can extensively seek methods of enhancing its alumni organization, collecting donations, running a multilateral business, or whatever else. In this sense, there is a lot to learn from the business management of foreign universities in recent years. A higher education/research institution needs not only strong backup from public finance, but also profitable businesses other than education and research in order to maintain its autonomy.

Major universities overseas all seem to be suffering from financial issues. U.S. universities are no exception, running patent license programs or going corporate with Internet education. Even at HarvardUniversity, there has been talk of putting corporate logos on its syllabus and course catalog. The primary funding source of U.S. public universities used to be subsidies from the state and federal governments, but the ratio has dropped in recent years, and they now rely to a large extent on the National Science Foundation, tuition income and donations from the private sector. Private universities have also raised their tuition and are relying more on federal subsidies and private donations.

Universities also need to actively collect donations from alumni and industrial circles. It seems like an unrealistic policy in the short term when we consider it solely in terms of the recent sluggish situation of the Japanese economy. But in the long run, they will need to develop a system where a corporate organization manages its basic assets from donors who “offer money but won’t butt in.”

Time is the capital good of academia

The new national university corporation now has a Board Meeting headed by the chancellor, a Management Council headed by the chancellor, and an Education Research Council headed likewise by the chancellor (and formed basically of members selected from within the university). At least in form, it is now a one-leader organization where the chancellor/chairman heads everything. But in reality, neither this Education Research Council nor the Management Council that was set up to incorporate voices from outside the university (defined as an organization “for discussing important matters concerning the management of the national university corporation” such as school rules and wages of the directors and faculty) are staging any heated discussions. Half the members of the Management Council are people outside the university, comprising business owners, local autonomy executives, cultural figures and experts. The fact that the opinions of these people are reflected in the management is one of the biggest changes made under this corporate shift of universities. Whether discussions in this council gain a practical impact or end up being insubstantial will largely define the future operations of the corporation. 

Will the organization of the national university corporation end up being mere pie in the sky, or will essential reforms penetrate? We will have to wait for future action in order to carry out our evaluations. Issues of, say, “How will it operate in such a way that the corporate head can demonstrate his leadership?” or “In what way will it demonstrate autonomy?” need to be tackled head on.

The issues are too diverse and complex to summarize and discuss for over 700 four-year colleges in Japan today under the single topic “college issues.” In this paper, however, I have discussed just a small portion of the issues faced by Japanese research universities today from the perspectives of evaluations and ratings.

The current state of science school systems being applied to the humanities as we see in evaluations and budget allocations still contains much to be reviewed and discussed. We need to consider the differences in the notions of achievements between the humanities and science, and reconsider how research grants are allocated and how budgets are developed and spent.

The greatest concern among young researchers these days is the lack of time they are able to spend concentrating on research. Particularly in the academic fields of the humanities and social sciences, their basic capital good is free time. That is why spending so much time on writing documents for the big “reform” significantly hurts humanities researchers.

I can only pray that the past twenty years of college reforms do not end up being empty years for Japanese academics.

Translated from “Tokushu: Daigaku no Himei – Rankingu to Gyosekihyoka no Kozai / ‘Kokusaika’ to Ronbunryosan niyotte Ushinawareru Daigaku no Rinen – Gakumon ni Rankudzuke nado dekinai (Feature: The University Outcry – Merits and Demerits of Rankings and Performance Evaluation / Universities losing their ideals to internationalization and the mass-production of academic papers – You Can’t Rank Academics),”Chuokoron, February, pp. 22–31. (Courtesy of Chuo Koron Shinsha) [February 2014]