PIAAC: Measuring adult literacy and skills

by Gatien, posted in Adult learning, Education, Non-series

See note below for original citation

The Programme for International Assessment of Adult Competencies (PIAAC) – the Organisation for Economic Cooperation and Development (OECD ) also calls it the Survey of Adult Skills – is an international effort to assess human skills and further develop our understanding of human capital. It started in 2012 in 22 countries (listed below) and coordinated by the OECD. The list of participating countries will be extended to new participants during this decennial as new rounds of the PIAAC assessment are being implemented (participants of the second round of PIAAC are also listed).

 


Intro video from the OECD | http://www.oecd.org/site/piaac/

 

Following in the footsteps of the International Adult Literacy Survey (IALS – 1994) and the Adult Literacy and Lifeskills Survey (ALL – 2003), PIAAC (2012) proposes to measure skill levels in a technology-rich environment (shortened to TRE) and to compare them across many OECD countries. With a wealth of common socio-economic indicators, PIAAC opens the door for extensive comparative research on education and its social outcomes (also read Education and Health: building indicators in international comparison, RISS – IJHS, 4(1)), which is extremely useful for further research on the determinants of health.

Surveys like PIAAC , which include psychometric elements, are complex to develop. To appropriately measure one’s skills, these tests must minimize any variable that would affect the performance of the interviewees. Tests, taken in person in the home environment, require a continued effort for an extended period of time, hence have to account for tiredness and boredom – variables we wish to exclude from the skills assessment as they may alter an individual’s performance. Other indicators are in place at the time of the assessment to keep record of the conditions of the test (length of time per item, overall duration of the test, presence of relatives or children).

Two elements compose the PIAAC survey:

  • A background questionnaire with a common core for all countries (to standardize socio-economic characteristics of individuals) and a limited set of questions established by each country.
  • Three psychometric tests with sets of “items” that will serve to establish a score for (1) literacy skills, (2) numeracy skills and (3) problem-solving skills in TRE.

The whole survey and its various components were developed under the leadership of Education Testing Services (ETS), an American-based company that built its experience with the previous national and international surveys (IALS & ALL), and the Graduate Record Examination (GRE) and the Test of English as a Foreign Language (TOEFL). These psychometric items remain the intellectual property of ETS and may only be consulted or used with their permission. ETS always welcomes researchers to submit their project. Funding may also be available to expand the use and reach of the programme’s database.

In most countries, national agencies and ministries were responsible for the implementation of PIAAC: from sampling to collection and presentation of data. Some countries resorted to private firms (ex: Singapore). Each country was also able to add their own (limited) set of questions to the background questionnaire, in the perspective of further national work on adult skills. This extra set of data may be available for research use through each national agency or ministry.

Assessment design

To limit the duration of the test, each individual would only be tested on two of the three type of skills: one would not be tested on her/his literacy, numeracy and problem-solving skills (as shown in Figure 1). This scheme allows interviewers to alleviate some of the mental stress put upon the interviewees and, presumably, test more accurately individual skills.

 

2014-08-27 de Broucker - figure 1 (EN)

Figure 1: Generally, seven possibilities to do the PIAAC assessment (computer-based). The psychometric tests are colored in yellow. Individuals who declare very little or no computer experience in the Background Questionnaire are tested on their literacy and numeracy skills but not on their problem-solving skills in TRE.

 

Yet, to preserve the comprehensiveness of such skills assessment and allow for comparison, a set of 10 plausible values would act for the missing empirical data in further analysis. The plausible values for one type of skills are chosen randomly within a range determined by (1) the score obtained in the two other tested skills for each individual and by (2) the scores of other individuals obtaining similar scores in the common area of testing (literacy, numeracy or problem-solving). The Australian Bureau of Statistics offers more information on the making and use of these plausible values.

It is also important to note that, when comparing between countries, the sample in each national dataset is further weighted to their respective population. Researchers often use Canada as a point of reference for accurate data, in consideration of the fact that it has the largest national sample – it was built to provide analytically significant estimate at the provincial/territorial level and for specific sub-populations. This allows for precise global estimates of literacy, numeracy and problem-solving levels and for better comparison.

Accessing and analyzing PIAAC data

PIAAC data are publicly available either through the OECD online software (the International Data Explorer) or by downloading Public User Data Files (PUF). PUF are available for download for every participating country (except Australia*) and can be analyzed with SPSS, SAS, Excel or STATA. Data compendia and instructions are also presented.

*Access to Australian PUF may be requested through the Australian Bureau of Statistics. Australian data are available in the OECD online software.

Software developed by stakeholders of PIAAC such as the OECD and the International Association for the Evaluation of Educational Achievement (IEA) were made to guide analysts in the computation of PIAAC results. They process the plausible values and sample weights automatically, which makes them very useful for quicker access to compiled PIAAC data.

For data visualization and analysis:

Both software are publicly available.

Quick useful resources

PIAAC YouTube channel, including an Overview of PIAAC by Irwin Kirsch (ETS) and some results for the USA (published in 2013).

The OECD website also lists a handful of resources and links for interested data analysts.

The Canadian report on PIAAC: Skills in Canada: First results from the Programme for the International Assessment of Adult Competencies, 2012 (final)

“Skills assessment, PIAAC and the adult literacy agenda” was published in the peer-reviewed International Journal of Lifelong Education (Volume 33, Issue 2) in 2014.

Feel free to leave a comment below if you want to recommend resources about PIAAC available at your organization.

National agency or ministry

Please note that the agencies’ website are not necessarily available in English. As much as possible, the English name of the institution is listed here whenever provided by the institution itself. Italicized countries participate to the second round of PIAAC assessment.

Should you see any mistake in this list, please do not hesitate to leave a comment below: I will be notified automatically.

 

The globe in the featured image (top) was taken from Wikimedia Commons under the Creative Commons Attribution-Share Alike 3.0 Unported license.

 

Note: I published this article in the blog of the Interdisciplinary Journal of Health Sciences, an open-access academic journal of the University of Ottawa. To cite this article, please use the original reference:

de Broucker, Gatien. (August 27, 2014). Introduction to PIAAC – measuring adult literacy and skills [blog]. Interdisciplinary Journal of Health Sciences. Retrieved from:

Print Friendly, PDF & Email
The following two tabs change content below.
My Twitter profileMy LinkedIn profileMy YouTube channel

Gatien

Health economist at Johns Hopkins University
My Twitter profileMy LinkedIn profileMy YouTube channel

Latest posts by Gatien (see all)

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.