Academic gap: Any progress since 2012?

In summary

In two earlier posts, I proposed a metric to measure the “academic gap” – the distance between students performing the lowest and the recognized international benchmark for “average or standard performance“. I measured the evolution of the academic gap between 2006 and 2012.

In this post, I’m updating the profiles of the countries with the newest data released last December from the PISA project. I am focusing on the countries whose performance declined significantly between 2006 and 2015, and those in which the academic gap was significantly reduced in 2015 compared to 2006. I chose 2006 as it is the best year to serve as a base for score change analysis.

Of the 40 countries we retain in this analysis, half have shown a fairly stable situation. The academic gap does not present either a substantial decline or a significant improvement between 2006 and 2015. There may have been some more or less significant moves in the academic gap demonstrated by performance measured in intermediate years (2009 and 2012), but performance in 2006 and 2015 are rather on the same level. Among the other half countries, eight show a marked deterioration, an aggravation of the academic gap – acknowledging it is from different initial situations: Australia, Finland, Hungary, Korea, Netherlands, New Zealand, Slovak Republic and Sweden. The other twelve demonstrate a noted improvement in the relative performance of the academically weakest students: Brazil, Chile, Colombia, Germany, Israel, Italy, Japan, Mexico, Norway, Portugal, Russian Federation and Spain.

Academic gap: Any progress since 2012?

In early December 2016, the OECD released the results of the most recent round of the PISA survey which took place in 2015. With this last round, the PISA project completed a second cycle of all three major subjects it covers, i.e. reading, mathematics and science.

In a previous post, I shared my perspective and criticisms of the measure of inequality in student academic performance developed by the UNICEF Office of Research – Innocenti. I proposed an alternative measure of inequality in education which I feel more appropriate: an “academic gap” that measures how far behind are the 10% weakest students in each country.

---

The “Academic gap” measures how far behind from the PISA “standard” 500 benchmark are students whose average scores on the three subjects (reading, mathematics and science) fall in the lowest 10th percentile of the performance distribution measured in PISA – considered the 10% weakest students in each country.

One recalls that 70 points score represent the difference between two consecutive proficiency levels. We consider a significant increase or reduction in the academic gap a change of 10 points score or more.

---

In 2006, six countries were showing their (10%) weakest students performing within one-and-a-half proficiency level of the “standard” 500, i.e. a gap of less than 105 points score: Finland, Korea, Canada, Estonia, Australia and the Netherlands (in this order). These countries had the smallest academic gap and were the most equitable in performance: lowest performing students were not too far behind those performing average in OECD countries.

At the other end of the distribution, six countries presented an academic gap above two-and-a-half proficiency levels, i.e. a gap of 175 points or more: Turkey, Chile, Israel, Mexico, Brazil and Colombia. These countries had the largest academic gap and were the least equitable in performance: lowest performing students were very far behind those performing average in OECD countries.

Previously, we looked at the progress/deterioration of the academic gap between 2006 and 2012 and all six countries with the largest gap in 2006 have shown significant decrease of the academic gap: from 23 points in Chile to 35 points in Brazil. Other countries, although from a lower inequality starting level, also demonstrated a reduction in the academic gap: Germany, Italy, Poland and Japan with a gap reduction of 20 to 28 points.

At the other end, six countries presented a significant increase in their academic gap. One country, Finland, the most equitable in performance in 2006, experienced the highest increase in the academic gap with 35 points. The other four countries are New Zealand, Sweden, Iceland and the Slovak Republic, coming from diverse initial situations in 2006 with gaps ranging from 110 to 145 points. Their academic gap increased by 16, 30, 13 and 18 points score respectively.

The relationship between an academic gap reduction or increase and an overall performance improvement or deterioration aligns rather well: generally, the academic gap evolves in the same – positive or negative – direction. Different national situations are just matters of magnitude in variations of the respective change in performance, overall vs. that of weakest students (see chart in the second post).

What further happened to the academic gap after 2012?

With the most recent release of PISA, we are able to analyze the further development of the academic gap to 2015. Let’s ask ourselves four questions and respond with the data:

• In which countries did the academic gap deteriorate between 2006 and 2015 and what was the pattern of evolution over that period?
• In which countries did the academic gap improve between 2006 and 2015 and what was the pattern of evolution over that period?
• How did all countries map themselves in the evolution of their academic gap between 2012 and 2015?
• Was the evolution of the academic gap between 2006 and 2015 in sync with the overall change in performance between 2006 and 2015?

Where did the academic gap deteriorate between 2006 and 2015?

In eight countries, the academic gap is larger in 2015 than it was in 2006: Finland, Korea, Netherlands, New Zealand, Australia, Sweden, Hungary and Slovak Republic. This is represented in Chart 1 below. The line at the top of the chart represent the international “standard” score of 500. The higher the countries’ year points/line, the smaller the academic gap.

In all countries except New Zealand and Sweden, the gap continued an increase that started as early as 2009 in Finland and Australia, or 2012 in Korea, Netherlands, Hungary and Slovak Republic. In these six countries, the academic gap increased from 19 (Netherlands) to 44 points (Finland). In New Zealand and Sweden, the earlier increase (between 2006 and 2012) either just stabilized (in New Zealand) or was partially compensated by a reduction in the gap (Sweden), by 2015. It is worth noting that, despite recent unfavourable evolution, Finland and Korea remain in the top ten country performances (among the 40 countries in this analysis) in 2015. The Netherlands fall back to the second quartile of countries in the ranking of academic gap while New Zealand and Australia remain in this group.

Where did the academic gap improve between 2006 and 2015?

The situation of under-performing students was significantly better in 2015 compared to 2006 in twelve countries: Japan, Russian Federation, Norway, Germany, Spain, Portugal, Italy, Chile, Israel, Mexico, Colombia and Brazil. Patterns of improvement for these countries are presented in Chart 2.

Of these countries, only the Russian Federation, Spain, Israel, Mexico and Colombia made gains (or at least kept prior gains) assessment after assessment since 2006. In the other countries, the improvement in the academic gap, when comparing performances between 2006 and 2015, presents “falling back” situations in some years.

Yet, the long term gains in the gap were largest in Colombia and the Russian Federation (45 and 30 points, respectively) and smallest in Germany and Brazil (12 points). Japan remains among the top ten countries presenting the smallest academic gap and the Russian Federation is accessing the tenth position. Norway, Spain and Portugal have moved up from third to second quartile of countries, while Germany maintained its position in the second quartile. Italy moved up from fourth to third quartile and the other countries (Chile, Israel, Mexico, Colombia and Brazil) remained in the fourth quartile, even if their academic gap has shown improvement by 2015 compared to 2006.

How did all countries map themselves in the evolution of their academic gap between 2012 and 2015?

Chart 3 gets us back to a familiar mapping representation of countries in four quadrants and a diagonal line. On the horizontal axis is the variation of the academic gap between 2006 and 2012 – the gap I analyzed extensively in the last post. From left to right (Finland to Colombia), countries are placed from large increase to large reduction in the academic gap.

On the vertical axis, I extend the evolution of the gap to 2015, still from the same base year 2006. From bottom to top (Finland to Colombia also), countries are placed also from large increase to large reduction of the academic gap, but here between 2006 and 2015.

I will not comment on the situation of countries in the centre, within the green box, as they experience variations of less than ten points on either the horizontal or vertical dimension. Countries placed on the diagonal (blue) line, or very close, have experienced no change in the academic gap between 2012 and 2015. This was the case for Israel, Mexico, Italy (name hidden behind Mexico) and Spain where the reduction in gap between 2006 and 2012 was just maintained in the last three years, on the positive side. On the increase-in-the-gap side, it was also the case for New Zealand and Iceland where the increase did not aggravate after 2012.

Then, for the other countries, the further away from the diagonal line, the larger the further evolution of the academic gap between 2012 and 2015. In the North-East (upper right) quadrant, above the diagonal line, four countries have continued the progress made earlier in reducing the performance gap of their weakest students – Colombia, Russian Federation, Portugal and Norway (in red). In the same upper right quadrant, but below the diagonal line, several countries have lost on their progress momentum – moderately (10 points or less) for Japan, Chile, Germany and Latvia, more in Estonia, Brazil and Poland (all in yellow). Four countries have just lost all gains made by 2012: Turkey, Czech Republic, Austria and Switzerland.

On the opposite quadrant, the South-West (bottom left), the analysis focuses on whether or not the increase in the gap between 2006 and 2012 was further accentuated. This is the case in the five countries mapped in this quadrant below the diagonal line, and the farther away from the diagonal, the larger the further increase of the gap: Hungary (the gap increased by 21 points), Australia (17 points), Netherlands (15 points), Finland (9 points) and Slovak Republic (7 points). On the other side of the diagonal stands Sweden where the academic gap is still larger in 2015 than it was in 2006, but reduced by 13 points compared with what it was in 2012.

On this map, Korea is one country that stands alone as it was showing non-significant change of the situation of the weakest students between 2006 and 2012, but then a large deterioration after 2012, 38 points – while still remaining among the ten best country performers in this respect.

Was the evolution of the academic gap between 2006 and 2015 in sync with overall change in performance between 2006 and 2015?

I like to conclude this examination with an update on the mapping provided in Chart 1 of the last post. It looked at the question of whether the evolution of the academic gap was in sync with the evolution of the global performance of the country. With the 2012 results, the main message was that, indeed, there was a fairly strong relationship between gains/losses in overall performance and improvement/deterioration of the relative situation of the weakest students – countries lined up fairly closely to the diagonal. In the post, I explained the meaning of the countries’ positions in the four quadrants – actually, as the correlation was high (R²=0.84), all countries mapped along a line South-West/North-East.

How has the situation changed when one takes the result of 2015 into account? Let’s see Chart 4:

 

At first glance, the map looks much the same. The trend line (red dotted line) also shows a slight (counter-clock) rotation and countries are even closer to the trend line, an indication given by an even higher correlation (R²=0.91). Score point changes in academic gap were not only correlated with score point changes in the overall average of the three subjects. They were also slightly higher when improvements were obtained and slightly lower when performance deteriorated. On average across countries, every positive gain of 10 score points in overall performance were accompanied with a reduction of the academic gap of 13 points, and on the other side, every decline of 10 score points in overall performance were accompanied by a 13 point larger handicap of the weakest students.

Save