At their best, international comparative tests such as the Southern and Eastern African Consortium for Monitoring Education Quality (SACMEQ) are important for a number of reasons. First, they provide a measure of learner performance that is objective and scientifically valid. Second, they give diagnostic insights into weak areas of the curriculum that require particular attention. Third, when administered periodically, they indicate the extent to which school systems change over time. And fourth, they indicate how each system is doing compared with countries of the same size, stage of development, expenditure on education, and the like.
In the last 17 years South Africa has participated in three or four iterations of three such test programmes: the Trends in Maths and Science Study (TIMSS), which tested maths and science in Grades 8 and 9, the Progress in Reading Literacy Study (PIRLS), which tested reading in Grades 4 and 5, and SACMEQ, which tested maths and language in Grade 6. In addition to testing learners, SACMEQ also administers tests to the teachers of these learners. While the results of all these programmes have precipitated something of a national crisis in South Africa because of our poor performance relative to many of our much poorer neighbours, they have provided invaluable information and benchmarks against which to measure improvement. Most encouraging, the 2011 iteration of TIMSS indicated a very significant improvement in both maths and science scores in Grades 8 and 9 across all provinces.
All the benefits of these programmes depend on the rigour of test design, administration and analysis: if there is the slightest doubt about the scientific validity of any of these steps the results are immediately brought into question, and trust in the programme evaporates. And, as is the case in every area of public and private endeavour, trust takes a lifetime to build, but can be lost through a single mistake.
This is the great danger that we run in accepting the latest round of SACMEQ results in their present form. As Nic Spaull has said, accepting the huge improvement in SACMEQ scores across all 14 participating countries is akin to believing that all the finalists in the Olympic 100m final could run the race in 5 seconds. When gains of this magnitude appear, there is every reason to doubt the validity of the measuring device. The fact that the scores for South African teachers dropped significantly while learner scores showed massive gains further undermines confidence in both sets of outcomes.
At this point, and considering the concerns raised by experts in this field, it will be wise to suspend judgement on the results until a thorough re-analysis has been conducted. This analysis should be done by a team of internationally recognised psychometricians who have no links to participating countries. It is an essential step to restoring trust in this important measure of the health of primary schooling in the key subjects of English and mathematics. It will also ensure that the considerable investment we have made in the enterprise over nearly two decades remains a productive asset and not yet another waste of public funds.
For more on this issue see:
www.education.gov.za/Newsroom/MediaRelease/tabid/347/ctl/Details/mid/3963/ItemID/4103/Default.aspx
pmg.org.za/committee-meeting/23250/
nicspaull.com/2016/09/14/serious-technical-concerns-about-sacmeq-iv-results-presented-to-parliament/
