Academic gap: How much did it change between 2006 and 2012?

In summary

Using the metric proposed in my last post, this post presents changes of the “academic gap” across countries with PISA three-subject average between 2006 and 2012. Countries’ experience vary widely, while there cannot be well-defined patterns to call from past performance – there were gainers (Japan, Poland, Italy) and losers (Finland, New Zealand, Sweden) in academic gap among the best performing countries, as there were also gainers (Brazil, Colombia, Turkey) and losers (Slovak Republic, Iceland) among lower performing countries. However, overall, the span of academic gap across countries was significantly reduced – from 54 to 240 in 2006, from 73 to 206 in 2012 – which is a global positive outcome.

Academic gap: How did it change between 2006 and 2012?

In my last post, I proposed a metric appropriate to measure the “academic gap” between a group most at risk academically (the 15 year-olds in the lowest 10th percentile of the distribution of PISA scores, considering the three subjects together) and the OECD standard for average performance across all its member countries, the benchmark of 500 points. The use of this metric, beyond the comparison across countries in a given year, serves as its further validation. In this post, I analyze changes in the performance of the most academically disadvantaged group over the longest time span allowed by the PISA data at this point in time. Has the “academic gap” shown a significant reduction in most countries?

Academic gap: Measuring progress over time

PISA tests have started in 2000. Since the beginning, it has covered three main academic subjects, i.e. reading literacy, mathematics literacy and science literacy, each subject being alternatively the core subject (with deep content), the other two being then secondary ones (with somewhat lighter content). In order to establish its standard average of 500 score points on the soundest basis possible, the OECD made it for each subject when the given subject was the core one. This resulted in the 500 benchmark being established for reading in 2000, for mathematics in 2003 and for science in 2006. After each of these dates, even the scores based on lighter content could be scaled against the set benchmark (but it was not recommended to consider this similarly for the tests held before the benchmark was established). As a consequence, and because we operationalized the “academic gap” with an average of the three subject scores, the longest period available at this time is between 2006 and 2012. (I will update this post when the next measure for 2015 is released, scheduled for early December 2016.)

Has the academic gap decreased between 2006 and 2012?

The academic gap may decrease, a sign of improvement in the academic situation of 15 year-olds, remain fairly stable, a sign of status quo, or increase, a sign that young people in a new generation of 15 year-olds sink somewhat deeper in academic troubles. Results are shown in Table 1 column 3, for the 40 countries retained in the analysis in the last post. When the change in the academic gap is positive, it means that the academic gap in effect decreases by the number of points shown, i.e. a decrease of the academic gap of 35 points in Brazil – in other words, the 15 year-olds at the 10th percentile in 2012 show an academic gap reduced by 35 points compared to the one their 2006 peers experienced. When the change in the academic gap is negative, it means that the academic gap increases by the number of points shown, i.e. an increase of 34 points in Finland – in other words, the 15 year-olds at the 10th percentile in 2012 were 34 points further away from the 500 standard than their peers were in 2006. When change (either positive or negative) is less than 10 points, I consider it too small to be significant enough to draw any conclusion about improvement or deterioration of the situation – this is in consideration of the statistical degree of precision that can be achieved with a sample survey like PISA (I recognize this is a non sophisticated way of setting a “significance” limit but I do not wish to get into statistical sophistication in a post like this).

Table 1 – Changes in Academic Gap and overall gains and losses from 2006 to 2012

[wpdatatable id=1 table_view=regular]

Note: In the table, Costa Rica and the United States appear with no data in the change score columns because Costa Rica started to participate in PISA in 2009 and the United States did not report scores in reading in 2006, with the result that no average of the three subjects could be computed for 2006.

• Between 2006 and 2012, the academic gap has decreased in 18 of the 38 countries for which we can measure such change with PISA. It has increased in 6 countries and remained more or less stable in the other 14 countries (i.e. a change in academic gap between -9 and 9).
• In four countries (Brazil, Columbia, Turkey and Israel), the academic gap was reduced by 30 points or more. Remember: 39 points represent gains attributable to about one more year of schooling (on average among OECD countries) and 70 points represent the difference between two consecutive levels of proficiency. The 30 to 35 points of improvement in these countries are rather significant accomplishments. Yet, the road to greater equity is still the longest for them – Brazil and Colombia have the highest academic gaps in 2012, Israel is 37th of 40 countries and Turkey is 30th.
• The next two countries (Japan and Poland) demonstrate high improvement of the situation of the most academically disadvantaged between 2006 and 2012 with a reduction of their academic gap by 28 and 27 points, respectively. They were already starting from a much better situation in 2006 than the four countries above: in Japan, the academic gap was then 113 points (ranked 9th) and in Poland, it was 120 points (ranked 14th). These are larger relative gains (respectively 25 % and 23%) than in the previous four countries. It shows that significant further reductions of inequality can be achieved even in countries already achieving relatively decent results on this inequality scale.
• But what happened at the other end of the change challenge is also quite telling: high equality, as measured by a low academic gap, is hard to maintain over time. In 2006, Finland was showing a 54 point academic gap (ranking 1st). Then, by 2012, it had experienced the largest aggravated performance – a loss of 35 points in its academic gap. Despite this, it remained in an envious international position, just slipping to the 4th place in the academic gap ranking by 2012. But it begs the question about what may be happening to the highly praised “Finish model” (“discovered” with PISA 2000). Other countries such as Sweden and New Zealand, although starting from similar heights as Finland, also made significant slips in their position: with a 30 point aggravation in its academic gap, Sweden slipped from 13th to 32nd, and New Zealand, with a 16 point aggravation slipped from 8th to 19th.

Has the academic gap evolved in sync with global educational performance?

Another question of importance for policies: in what global context of change in national performance between 2006 and 2012 have the changes in the academic gap occurred? Table 1 columns 4 and 5 document this issue. Chart 1 also helps respond to this question. Let me first explain what is represented in the chart:

• I examine whether there is a relationship between the change in the academic gap between 2006 and 2012 and the change in the average score during the same period. In other words, my question is: when the academic gap decreases, is it concomitant with a similar or different change in the average performance in the country? The points on the chart are the international country codes with the following coordinates: the change in the average score on the horizontal axis and the change in the academic gap on the vertical axis.
• A country placed at the intersection of the two axis has experienced no change in its academic gap, nor any change in the average score of its 15-year-olds students. In consideration of the statistical degree of precision of the data in the context of a sample survey, I have designed at the centre of the chart a green box: I would consider that countries falling within this box have not experienced a change along both dimensions that would be significant enough to warrant commenting, other than saying that they basically experienced a performance status quo. These countries are: Norway, Lithuania, Ireland, Greece, Belgium (actually Flanders), France, United Kingdom (actually England and Northern Ireland), Korea, Denmark, Luxembourg, Netherlands, Canada, Slovenia and Hungary.

• The two crossing axis define four quadrants:
  • North-East: a situation in which countries have experienced both gains in average performance and a significant reduction in the academic gap – overall, the most envious situation; 16 countries are in this situation, but with very different magnitudes.
  • North-West: a situation in which countries have experienced a reduction in the academic gap while their average performance declined; this is a situation where we find two countries, Czech Republic and Austria, but, although the gains in academic gap are substantial, the negative change in overall average is insignificant (-2 points).
  • South-West: a situation in which countries have experienced both an increase in the academic gap and a lower overall average score; six countries are in such a situation: Finland, Sweden, Slovak Republic, New Zealand, Iceland and Australia (in a marginal way, though).
  • South-East: a situation in which countries would see a deterioration of the academic gap while the average performance of its students would improve; no country has experienced such a dramatic record for the less fortunate students.
• I have drawn two lines: one, the solid blue line, is the diagonal across along which all points show equality between the change in the academic gap and the change in the overall average performance – an equal line. The second line, the dotted red one, is the line that goes across all country marks, that minimizes all country distances to that line – a trend line.
• The trend line sits above the equal line in the North West quadrant. This means that the dominant tendency among countries is that the reduction in the academic gap tends to be larger (in points) than the increase in the three-subject average, a possible sign that a policy focus on raising the performance of the most academically disadvantage also pays off in terms of the global performance.
• Four countries are on the blue line in the North-East quadrant: Turkey and Israel, on the high part, and Portugal and Spain, somewhat lower. These four countries have managed to raise students’ performance in general and reduce the academic gap as well by a similar number of points. They were all starting from a significantly below OECD “standard” performance (500 points) situation in 2006, but come closer in 2012.
• Six countries are on or close to the trend line: Japan, Poland, Italy, Russian Federation, Estonia and Latvia. their improvement in the academic gap is coincidental (being careful not to draw a causal relationship in absence of further investigation) with a gain in overall performance a bit lower.
• Six other countries appear above the trend line: Brazil, Colombia, Mexico, Chile, Germany and Switzerland. In these countries, the reduction in the academic gap is rather impressive, but not quite matched by gains of same magnitude in the overall average. Germany and Switzerland were already rather high performers, somewhat limiting the scope for high gains on the average. The other four American countries were much lower in overall performance and the recent gains can be considered as much promising.
• At the other end of the trend line, in the South-West quadrant, five countries are below or on the trend line: Finland, Sweden, New Zealand, Slovak Republic and Iceland. They all show an aggravation of the academic gap that is somewhat larger than their also negative change in the three-subject average. They come from a variety of PISA performance background, demonstrating that there is nothing predetermined by past performance, likely leaving much room for appropriate policy intervention. In 2006, Finland and New Zealand were among the highest performers of the OECD countries, with the shortest academic gap for Finland and a decent ranking in this respect for New Zealand. The other three countries were not as well placed (although Sweden was actually close to New Zealand, just above the 500 points), yet lost significantly in both academic gap and overall average performance.

A careful analysis of the data thus leads to important, policy relevant issues to be addressed nationally. The international comparison allows to benchmark individual country’s performance against a relevant country comparator and to draw policy lessons. It remains interesting to note that, overall, the span of national academic gaps was reduced significantly – from 54 (Finland) to 240 (Colombia) in 2006, from 73 (Korea) to 206 (Colombia) in 2012 – which is a global positive outcome.

Non solum data – Data sine monito oculo nihil sunt.

---

Through this blog, we invite constructive comments and constructive article contributions – review our blog policy and have your say!