Leaning Tower of PISA: Why has International Student Testing failed to improve education systems?

 

The Program of International Student Assessment (PISA), managed by Andreas Schleicher and the OECD Education Skills Office in Paris, France, is still regarded as the “gold standard” in comparative student assessment and enjoys a rather charmed life. Every three years, educational leaders, commentators, and engaged teachers eagerly await the results of student testing and its so called ‘league tables’ ranking the performance of 15-year-olds from some 79 participating jurisdictions. A new book, Dire Straits: Education Reforms, Ideology, Vested Interests and Evidence, produced by two Spanish researchers, Montserrat Gomendio and José Ignacio Wert, is sure to rock that edifice and punch holes in the credibility of the OECD’s education branch.

Student assessment and accountability are essential and yet elusive in global K-12 education, both within countries and internationally, so school reformers put faith in ILSAs like PISA to provide solid evidence on how students were actually performing in the core skills of reading, mathematics and science. Across the globe, educational leaders and policy-makers looked to PISA to provide evidence and guidance to allow us to form a consensus on what works in different countries, and particularly on what can be learned from student achievement gains in top-performing nations. That has not happened according to one of the book’s authors, Montserrat Gomendio, OECD’s former deputy director for education and head of its Centre for Skills. It’s all spelled out in a devasting critique in the current Winter 2023 edition of Education Next.

PISA is OECD Education’s crown jewel in an organization dedicated to providing reliable data and policy advice, encouraging comparative analysis and learning exchanges worldwide.  From the first cycle of PISA (2000) to the last (2018), the number of participating countries increased from a rather homogeneous group of 32 OECD countries to some 79, owing largely to the addition of many low- and middle-income countries. Flush with its own success, the OECD made a boastful claim: “PISA has become the world’s premier yardstick for evaluating the quality, equity and efficiency of school systems, and an influential force for education reform.”

PISA’s own data tells the tale. “After almost two decades of testing, student outcomes have not improved overall in OECD nations or most other participating countries,” according to Gomendio. She recognizes that, up until 2018, a global recession, the rise of social media, and environmental disasters did present “headwinds for school-improvement efforts.” Failing to achieve its mission, she points out, led to “blame games.” That was precipitated by the dawning realization that student outcomes had flatlined from 2000 to 2018. In response, OECD Education officials pointed fingers at its own member states for not taking advantage of the PISA data and carrying out the recommended policy changes.

Policy recommendations from PISA are built upon two different approaches – quantitative analyses of student outcomes and a range of features of education systems and qualitative analyses of low- and top-performing countries. It is commonly agreed PISA’s quantitative analyses of cross-sectional samples and correlations cannot be used to draw causal inferences. It’s qualitative analyses, particularly with regard to Nordic countries, also suffer from serious drawbacks such as cherry-picking. Other weaknesses, she points out in Education Next, have gone largely unnoticed.  One of the biggest question marks is the reliability of student results on such “low stakes” tests. In the case of Australia, for example, the National Council on Educational Research (NCER) found that a majority of Australian students (73%) may not have taken the PISA test seriously and would have invested more effort if it counted towards their marks.

Quality and Equity – Confronting the Contradictions

PISA seeks to measure two complementary dimensions of education systems: quality and equity. Measuring quality on the basis of average student test scores is far easier than assessing equity. To do so, PISA employs a multidimensional concept using metrics such as the relationship between socioeconomic status and student performance, the degree of differences in student performance within and between schools, and many others. None of these variables, Gomendio points out, “tell the full story” and “each of them leads to different conclusions.” So, ultimately PISA’s prism on equity is ultimately too narrow and somewhat unreliable.

PISA’s analysis of school choice and policy recommendations on that issue draw fire from Gomendio and Wert. Claims that students in private schools do not perform better that those in public schools (after correcting for socioeconomic status), are problematic. Analyses lumping private schools together with government-funded, privately managed charter schools skews the results. It also makes it impossible to disaggregate the data. That explains why PISA analyses are at odds with other international assessments, as well as research studies, which show that “school choice often does lead to better student outcomes without necessarily generating segregation.” In addition, the small number of countries with early tracking (streaming into academic and applied/vocational) show “little (if any) differences in student performance and employability rates for vocational-education students.” It is clear that PISA would benefit from thinking outside the box, paying attention to academic research and looking at the broader picture.

            The new book Dire Straits, written by two Spanish researchers, confronts squarely PISA’s implicit bias in favour of Finland and other Nordic countries. The authors are particularly critical of PISA’s analyses of Finland and Germany. In PISA’s first cycle, they call into question the lionizing of Finland for its “quality and equity” and labelling of Germany as a “heavily tracked system” that promoted inequity and “should be avoided.”  

Nordic societies like Finland do get a free ride with PISA because they were egalitarian long before the inception of PISA. Egalitarian societies like Finland possess natural advantages since teachers work with a more uniform student population and are better positioned to implement inclusive policies across the board to all students. More stratified societies in Europe and Latin America, for example, require more differentiated approaches to meet the needs of the full spectrum of students. More recognition should be accorded to stratified societies with income inequalities that tend to have bigger challenges closing the equity gap. In the case of Canada, for example, it is useful to examine how our country manages to maintain reasonable student achievement standards, while alleviating the equity gap, particularly in relation to the United States.

Identifying Exemplars, Applying the Right Lessons

PISA completely missed the boat on the rise of student outcomes in Singapore and its East Asian neighbours and the relative decline of Finland. A few decades ago, Singapore had an illiterate population and very few natural resources. The country made a decision to invest in human capital as the engine of economic growth and prosperity, and, in a few decades, it became the top performer in all international assessment programs. Part of that improvement can be attributed to implementing tracking in primary school in an effort to decrease its high dropout rate. Once this was achieved, the country delayed tracking until the end of primary school. So far, PISA has not provided a coherent and differentiated analysis of the “Singapore Miracle.”

Teacher quality is more salient than PISA recognizes in its analyses. In the case of Singapore and the East Asian countries only top-performing students can enter education-degree programs, whereas poorer performing Latin American countries tend to have teachers drawn from the weaker academic ranks. Professional recruitment programs are mostly weak and teacher evaluation mechanisms almost non-existent.  Teacher unions are not always helpful in improving the quality of instruction.  In the case of Latin America, teacher unions exercise considerable clout and have succeeded in securing lower class sizes, generating more teaching positions. Top-performing East Asian countries, on the other hand, tend to have weaker unions and there are, consequently, fewer political costs involved in running larger class sizes or in implementing rigorous teacher evaluation systems.  Increases in education spending get invested in reducing class sizes, contrary to the PISA recommendation, and in the face of robust evidence that it does not improve student outcomes.

Conclusions

            Ideology, education governance and conflicts of interest all serve to undermine the overall effectiveness of evidence-based, PISA-informed policy prescriptions. Education authorities tend to be risk-averse when it comes to implementing targeted policy prescriptions and resisting the pressures to increase spending levels, driven by the core interests, most notably local education authorities and teacher unions.

Three key lessons jump out in the latest book on PISA. First, decreases in class size and increases in teacher salaries do not work in improving student achievement but such policy recommendations run headlong into the vested interests of unions and the preference of active parents alert to any diminution in the amount of resources received from public funds. Secondly, some influential factors are “strongly context-dependent” (such as school autonomy and site-based management) and are difficult for policymakers to interpret. In such cases, applying policies universally can yield dire consequences. Finally, attempts to measure equity, including those of PISA analysts, tend to be inconclusive and partial, leading to recommendations more often than not “heavily influenced by ideology.”  This has led to a universal recommendation to apply comprehensive policies and avoid those that are regarded as ‘discriminatory’ (such as ability grouping and early tracking). Such policies lead to the worst outcomes in terms of equity among more stratified societies.

Pointing fingers and apportioning blame has become all-too-common in OECD’s highly influential PISA reports.  What’s clear from the latest critique, levelled by the two former PISA insiders, is that flatlined student outcomes and policy shortcomings have much to do with PISA’s implicit biases (ideology), structural impediments (union advocacy), and conflicts of interest (service provider capture). That is why, according to the critics, PISA is failing in its mission.

Judging from the latest book, PISA has made little difference in improving school systems.  Is PISA failing in its mission? With so much evidence from student testing, why do education systems tend to brush aside the runaway success of top-performing Asian countries and, perhaps most importantly, why do so many systems continue to struggle?

 

Read Full Post »

Confronting ‘Doing Our Best’ COVID-19 Education: Are the Kids Really Going to be Alright?

‘Everyone is doing their best’ was the prevailing narrative during the COVID-19 school disruption.  That may explain why school authorities either suspended system-wide student tracking or chose to conceal data collected relating to student progress and engagement.  A June 2020 CBC Investigation into this issue in four Atlantic Canadian provinces came up almost empty  and revealed that no one was able to provide any credible information on how many students went missing during pandemic distance learning.

“Doing Our Best” education may well have lasting consequences for students. Coming out of a three-month suspension of in-person, face-to-face teaching and learning, we are beginning to confront the hard realities: the coronavirus generation has fallen months behind, most housebound children were bored and disengaged, and struggling students have lost the most ground.

What we know about the real COVID-19 impact on children and teens did not emanate from education officialdom. With senior education leaders and school districts remaining tight-lipped, public opinion survey pollsters stepped up to fill the vacuum and assist intrepid education reporters trying to penetrate the wall of silence. Back on May 10, over a month ago, the Angus Reid Institute broke the code: “Canadian children are done with school from home, fear falling behind, and miss their friends.” The kids, it turned out, were not alright.

What actually happened during the COVID-19 crisis is coming into clearer focus with the benefit of hindsight. For the first month,  ministries of education, school districts, and educators scrambled to fill the learning gap with “emergency distance learning,” building upon patchy online infrastructure and cobbled together together curriculum combining e-learning and hastily-assembled ‘learning packages.” With few exceptions, Canadian K-12 education was completely unprepared for the system-wide shutdowns.

Thrown completely off-kilter, educational leadership was left fumbling around in the dark looking for the proverbial light switch. Perpetually optimistic technology-driven educators found ‘silver linings‘ amidst the dark clouds, progressive educators focused on responding to children’s “fears, anxieties and trauma,” and global thought leaders rhapsodized about a “better normal‘ where “Maslow (finally triumphed) over Bloom.“ With little warning, parents were expected to guide “Home Learning” with their housebound children.  It looked ominous, but most educators sounded upbeat, made the best of an unsatisfactory situation, and retained some hope that it would all work out somehow.

Taking a closer look at the May 2020 Angus Reid survey, it’s now clear that, despite everyone’s efforts, the COVID-19 educational experience was decidedly substandard for the vast majority of Canada’s five million K-12 students, and possibly damaging for those from disadvantaged and racialized communities. Here’s a succinct summary of the worrisome findings:

• The biggest worry for over half of all children (ages 10-17) surveyed was “missing out on  work” this school year and next, roughly equal the proportion who feared getting sick themselves.
• A clear majority of children “attending” school online (60 per cent) were bored or  unmotivated, not very busy with the work, but still “keeping up” with the reduced academic expectations.
• Children and teens, outside of homework, spent the vast majority of their time glued to small screens, dominated by watching TV/Netflicks, You Tube (88 per cent), and playing video games (74 per cent).
• Parents may have been doing their best, but it was not good enough, because over half of teens ages 13 to 17 reported needing more help with their work.
• Some 70 per cent of children and teens reported missing seeing friends and participating in extra-curricular activities, but fewer than 1 in 10 (8 per cent) were willing to concede that they missed going to school.

Missing so much regular schooling, after two of the three months, was already having adverse effects. Most of the students reported that they were “missing out” on school work and were struggling to remain positive, mainly because of deteriorating friendships and relationships.  The so-called “home education blues” were real and, for the most part, went unacknowledged and unreported by Canadian school authorities.

Close education observers and inquisitive parents seeking straight talk about the actual impact of the COVID-19 school shutdown invariably come up empty when seeking answers to questions or any evidence to support periodic accounts of heroic individual efforts or hopeful reports of ‘silver linings.” Education reports out of the United States provided us with a much-needed wake-up call when it came to getting the straight goods on what was really happening to students and parents during the school shutdown.

Two key U.S. education stories exposed the harsh realities of COVID-19 education for students, parents and teachers and raised serious questions about the veil of silence shrouding Canadian K-12 education. New York Times education reporter Dana Goldstein blew the lid off the real story on June 5, 2020 with a feature demonstrating the impact in terms of learning loss.  By September 2020, she reported, most students would be “months behind” with “some losing the equivalent of a full year’s worth of academic gains.” Furthermore, “racial and socioeconomic achievement gaps” would “most likely widen because of disparities in access to computers, home internet connections and direct instruction from teachers.”

A Boston Globe feature on May 23 confirmed that the COVID-19 disruption exposed the reality of digital divide. One in five Boston Public Schools children were found to be “unplugged” from Google Classroom and disengaged to the point where they were essentially “virtual dropouts.” Significant education technology challenges and language difficulties were keeping children from continuing school online. That finding was confirmed in a large-scale study of some 800,000 students conducted by a team of Harvard and Brown university researchers. Mining academic research into student use of Zearn, an online math program, they reported that student progress in math between March 15 and April 30 decreased by some 48 per cent in classrooms located in low-income ZIP codes, and by one-third in classrooms in middle-income ZIP codes.

The COVID-19 crisis has demonstrated, in microcosm, the extent of the public disclosure deficit in our provincial public education systems. Without American investigative education reporting, we would probably know little or nothing about the stubborn COVID-19 problems of getting students to engage in distance learning or the incredible proportion of children and teens who skipped out on home learning or lacked proper access to the alternative programs.  Knowing that the kids are not alright should spark some needed public discussion about working together on developing and acting upon a comprehensive, evidence-based learning recovery plan.

What happened to the initial plans for COVID-19 Home Learning in Canadian K-12 education? How did most children and teens fare in terms of “continuous learning” during the COVID-19 school disruption? Why were provincial and district education authorities so tight-lipped about the state of distance learning?  Should ministries of education and school districts be responsible for monitoring, collecting and reporting on alternative distance learning programs?  Does the public have a right to know how many children logged-in, remained engaged, and met the expected curricular standards? 

Read Full Post »

The Knowledge Gap: Who is Challenging the Sugary Elementary School Curriculum Consensus in Canada?

Every school year seems to herald the arrival of a new crop of education books seeking to “fix the education system.”  Some champion the latest educational panacea, others target the supposed causes of decline, and a select few identify a possible pathway for improving teaching and learning or making schools better. Despite significant investments in remedial programs and ‘learning supports,’ a yawning “achievement gap” persists between students from marginalized or low-income families and their more affluent counterparts and, with few exceptions, it has not closed much over the past fifty years.

Two new education reform books, Natalie Wexler’s The Knowledge Gap, and Michael Zwaagstra’s A Sage on the Stage, raise hope that the sources of the problem can be identified and actually addressed in the years ahead. Each of the two books, one American, the other Canadian, offer markedly similar diagnoses and urge policy-makers and educators alike to shore-up the rather emaciated content knowledge-based curriculum. 

Prominent American journalist Wexler demonstrates that elementary school teaching and learning, once considered a bright spot, is so undernourished that most teachers now teach as though it doesn’t matter what students are reading or learning, as long as they are acquiring skills of one kind or another.  Manitoba high school teacher Zwaagstra, in one commentary after another, shows how teaching content knowledge has been downgraded at all levels and overtaken by constructivist experiments embedded in the latest “foolish fads infecting public education.”

Forays into American elementary schools, during Wexler’s field research, produce some alarming lessons.  First graders in a Washington, DC, inner city school are observed, virtually lost, drawing clowns or struggling to fill-in worksheets in a class supposedly based upon a rather dense article about Brazil. Teachers jump wily-nily from topic to topic asking students to read about clouds one day, then zebras the next, completely out of context.  Few elementary teachers seem aware of the science of learning or the vital importance of prior knowledge in reading comprehension. Equally disturbing is the general finding that so many elementary teachers simply assume that children can acquire content knowledge later, after they have a modicum of skills. Such ‘progressive education’ assumptions prevail in most elementary schools, public, private and independent, almost without variation.

Zwaagstra’s book, composed of his best Canadian newspaper commentaries over the past decade, takes dead aim at the prevailing ideology fostered in faculties of education and perpetuated by provincial and school district armies of curriculum consultants and pedagogical coaches. Beginning teachers are trained to resist the temptation to be “a sage on the stage” and instead strive to be “a guide on the side.”  Zwaagstra completely rejects that approach on the grounds that it undermines teacher content knowledge and devalues the expertise of professionals in the classroom. He is, in this respect, speaking the same language as most secondary school teachers who have never really given up the notion that prior knowledge matters and that knowing your subject is critical to higher achievement in colleges and universities.

Zwaagstra speaks up for regular classroom teachers who focus on what works in the classroom and have learned, over the years, to be skeptical of the latest fads. Most regular teachers reading his stinging critiques of ‘discovery math,” whole-language-founded “balanced literacy,” and  incomprehensible “no zero” student evaluation policies will likely be nodding in approval. Not content simply to pick holes in existing theories and practices, he makes a common sense case for strategies that do work, especially in high schools —explicit instruction, knowledge-rich curriculum, and plenty of practice to achieve mastery.

Both Wexler and Zwaagstra go to considerable lengths to spare teachers from the blame for what has gone wrong in the school system. Prevailing pedagogical theories and education professors are identified as the purveyors of teaching approaches and practices floating on uncontested progressive education beliefs. When it comes to teaching reading comprehension, Wexler carefully explains why teachers continue to teach reading comprehension as a set of discrete skills instead of being founded on prior knowledge and expanded vocabulary. It is, in her analysis, “simply the water they’ve been swimming in, so universal and taken for granted they don’t question or even mention it.”  In Zwaagstra’s case, he’s very sympathetic to hard-working teachers in the trenches who cope by carrying-on with what works and developing ‘work-arounds’ when confronted by staunch ideologues or impossible mandates.

What’s really significant about these two education reformers is that both are strong advocates for, and supporters of, the international researchED movement out to challenge and dispel popular myths that have little or no basis in evidence-based research or cognitive science. Zwaagstra is a very popular presenter at researchED Canada conferences and Wexler is one of the headliners at the upcoming American researchED conference, November 16, 2019, in Philadelphia, PA. 

The two authors are very much part of the great awakening made possible by the flourishing of social media conversations, especially on EduTwitter, where independently-minded educators from around the world now go to debate education reform, share the latest research in cognitive science, and discuss ways of grappling with common problems in everyday teaching.

Slowly, but surely, the global edu-gurus are losing their single channel, uncontested platforms and facing more and more teachers equipped to call into question prevailing teaching approaches and fashionable education fads. Moving forward is now less about finding and embracing education evangelists or grabbing hold of,  and riding, the latest fad, and far more about interrogating accepted truths and trusting your teacher colleagues to work out what works in the classroom.

What’s significant about the two books — Natalie Wexler’s The Knowledge Gap and Michael Zwaagstra’s A Sage on the Stage?  Now that the call for content-knowledge curriculum is back in vogue in the United States, will Canadian policy-makers and educators  begin looking more critically at their policies and practices?  With more educators embracing a knowledge-rich curriculum, what would it take to successfully challenge the the sugary progressive education consensus in elementary schools?  

 

Read Full Post »