Quality assessments do not improve quality itself

PISA results can be used as arguments by politicians, but offer little aid or instruction in the development of our school system.

The PISA survey, conducted every three years, measures the level of general knowledge among schoolchildren. When the results from the first PISA study were published at the beginning of the 21st century, Finland was among the leading countries in every category. In recent years, however, the country has not been quite as successful, although Finland still remains close to the top.

New PISA assessments always kick off a debate on the state of Finnish comprehensive education. Yet many researchers say that the survey does not provide us with answers related to the developmental direction that comprehensive education should be taking. However, the survey results do help politicians find topical arguments regarding education.

“The best quality indicators are the kind that provide schools with practical help in teaching,” says Associate Professor of Education Sciences, Jaakko Kauko.

“The best quality indicators are the kind that provide schools with practical help in teaching,” says Associate Professor of Education Sciences, Jaakko Kauko.

“In politics, change is made possible by external signals and crisis awareness. PISA results are rocket fuel to political debate,” says Associate Professor Jaakko Kauko, from the Faculty of Education at the University of Tampere.

Unlike any other education quality indicator, the PISA assessment, created by the OECD, has gained the status of a global milestone. Still, it cannot function as instructions on how to fix an education system. Florian Waldow (Humboldt University of Berlin) has noted a great example of this in his study, where he describes how the public debates on educational reforms in Germany and Sweden reflected opposing images of Finland’s PISA results. PISA measures schoolchildren’s general abilities: literacy, performance in mathematics and sciences, and as the latest addition, their problem-solving skills.

“PISA assessments do not measure how well a school is able to achieve a country’s national objectives. It is not directly linked to a national curriculum, nor does it measure how well a school can pass on the curriculum’s content,” says Kauko.

There are several international examples of how it has become more important to measure the quality of education rather than discuss what quality actually means. According to Kauko, quality assessments can easily be seen as a be-all and end-all solution to problems in education policy-making.

“ We are not short of information. Plenty of indicators are being developed in various countries to measure the quality of education, and new mechanisms are constantly being created to control and produce indicators assessing a number of elements,’ Kauko explains.

“But we are not utilising this information to its fullest – simply producing it will not support our efforts to improve the quality of education,” he adds.

These conclusions were reached in the research project Transnational Dynamics in Quality Assurance and Evaluation Politics of Basic Education in Brazil, China and Russia. The project was funded by the Academy of Finland and studied the effects of quality policies on education in Brazil, China and Russia. Kauko was the principal investigator of the research consortium and led the Brazilian studies. In addition, a large number of international researchers from the Finnish universities of Tampere and Turku took part in the project.

According to Kauko, the logic is often this: Assessments show that something is not working. The proposed solution is to have even more assessments and closer control.

“In other words, there is this fallacy that by measuring a thing we can improve its quality. In the worst-case scenario, this notion will lead us away from the value-driven educational debate, Kauko points out.

Furthermore, it will stop teachers from utilising the acquired information in their work, and the assessments will be used more as means of control, Kauko points out.

Kauko believes that in education administration in particular, quality assurance tools can be of significant use. But the study discovered that staff at schools often feel unsure as to why this information is needed.

“The demand for information comes from above, and schools must respond to this demand instead of using the information to improve their quality of education. This is a big problem,” says Kauko.

But alternative operating models do exist. For example, in Scotland, a school inspector is also an educational developer.

“Finland differs from the mainstream approach, because here assessments are performed based on a philosophy, according to which assessments are not intended as a means of control. Information is gathered to be utilised by schools, and the aim is to conduct assessments for the purpose of using them for improvements,” Kauko says.

Another finding when studying the quality assessments of general public education in Brazil, China and Russia was that quality control methods are also used to achieve goals outside the field of education. Educational expectations may be directly linked to international politics. The OECD’s PISA survey is a good example of how such assessments can be used in an attempt to raise a country’s status and gain global recognition.

For example, Brazil has set a goal of reaching the average level in PISA by 2022.

In the studied countries – the same as elsewhere – education is tied to huge social objectives on how education might be used to change society and solve social issues.

By measuring a thing, we are simultaneously determining that this thing equals quality. If we study learning results, then the main focus will shift to these results. Similarly, if we are measuring the number of pupils who must repeat a grade, this number will become the quality indicator for schools.

Kauko thinks that information gained through quality assessments should be viewed as no more than fragments. This, however, is easier said than done. In policy-making, numbers tend to start live a life of their own.

Finnish education policy research has a documented case of how assessments can completely change the direction of social debate regarding education.

“In the 1990s, criticism began to emerge of the country’s comprehensive school. In the late autumn of 2001, the Confederation of Industry and Employers held a large conference in Helsinki. The general atmosphere at the event was that something must be done to fix our statutory education system, because it was failing. A few weeks after the conference, the first PISA results arrived, placing Finland in the lead in all categories. At this point, the fledgling debate died down,” Kauko explains.

What is a good quality indicator like? “The kind that provides practical grassroots-level help to teachers at schools,” Kauko says.

However, tying quality assessments with resource allocation should only be done after careful consideration.

“Every time we start using an indicator to assess an activity, especially if it is linked to resources or recruitment, we begin to receive a specific set of results. The funding criteria for universities is a good example of this,” Kauko says.

The existence of indicators creates a conflict. Being aware of quality assessments being made can easily shift the focus of an activity from its main goal to meeting the indicator’s criteria instead.

“Indicators do not illustrate true quality,” Kauko points out.

“The premise is that a perfect indicator does not exist. Moreover, in order to understand an indicator, we must understand its context. That is why it may be useful to think of it in terms of the aid it can provide at the grassroots level,” he says.

Learning as part of society

• Future learning was studied by analysing three emerging economies, i.e. three BRICS countries, as a joint research project Transnational Dynamics in Quality Assurance and Evaluation Politics of Basic Education in Brazil, China and Russia carried out at the universities of Tampere and Turku.
• The study was part of the Academy of Finland’s programme ‘The Future of Learning, Knowledge and Skills’.
• The study identified three dynamics that began to occur repeatedly in the studied countries after quality assurance became part of their education systems. The main finding was that in education policy-making, quality assessment had become more important than discussing what quality actually is.
• The research results have been compiled into a book ‘Politics of Quality in Education: A Comparative Study of Brazil, China, and Russia’, which can be read online free of charge.

Finland’s placements in the 2015 PISA results
Science performance 5th
Reading performance 4th
Mathematics performance 13th
Collaborative problem solving 7th

Text: Tiina Lankinen
Photograph: Jenni Toivonen