Measuring What Matters Conundrum: Will Social and Emotional Learning (SEL) Assessments Meet the Reliability Test?

Starting next year, students from Kindergarten to Grade 12 in Canada’s largest province, Ontario, will be bringing home report cards that showcase six “transferable skills”: critical thinking, creativity, self-directed learning, collaboration, communication, and citizenship. It’s the latest example of the growing influence of education policy organizations, consultants and researchers promoting “broader measures of success” formerly known as “non-cognitive” domains of learning.

Portrait of Primary Schoolboys and Schoolgirls Standing in a Line in a Classroom

In announcing the latest provincial report card initiative in September 2017, Education Minister Mitzie Hunter sought to change the channel in the midst of a public outcry over continuing declines in province-wide testing results, particularly in Grade 3 and 6 mathematics. While Minister Hunter assured concerned parents that standardized testing was not threatened with elimination, she attempted to cast the whole reform as a move toward “measuring those things that really matter to how kids learn and how they apply that learning to the real world, after school.”

Her choice of words had a most familiar ring because it echoed the core message promoted assiduously since 2013 by Ontario’s most influential education lobby group, People for Education, and professionally-packaged in its well-funded ‘Measuring What Matters‘ assessment reform initiative. In this respect, it’s remarkably similar in its focus to the Boston-based organization Transforming Education.   Never a supporter of Ontario’s highly-regarded provincial testing system, managed by the Education Quality and Accountability Office (EQAO), the Toronto-based group led by parent activist Annie Kidder has spent much of the past five years seeking to construct an alternative model that, in the usual P4E progressive education lexicon, “moves beyond the 3R’s.”

Kidder and her People for Education organization have always been explicit about their intentions and goals. The proposed framework for broader success appeared, almost fully formed, in its first 2013 policy paper.  After referring, in passing, to the focus of policy-makers on “evidence-based decision making,” the project summary disputed the primacy of “narrow goals” such as “literacy and numeracy” and argued for the construction of (note the choice of words) a “broader set of goals” that would be “measurable so students, parents, educators, and the public can see how Canada is making progress” in education.

Five proposed “dimensions of learning” were proposed, in advance of any research being undertaken to confirm their validity or recognition that certain competing dimensions had been ruled out, including resilience and its attendant personal qualities “grit’/conscientiousness, character, and “growth mindset.” Those five dimensions, physical and mental health, social-emotional development, creativity and innovation, and school climate, reflected the socially-progressive orientation of People for Education rather than any evidence-based analysis of student assessment policy and practice.

Two years into the project, the Measuring What Matters (MWM) student success framework had hardened into what began to sound, more and more, like a ‘new catechism.’  The Research Director, Dr. David Hagen Cameron, a PhD in Education from the University of London, hired from the Ontario Ministry of Education, began to focus on how to implement the model with what he termed “MWM change theory.” His mandate was crystal clear – to take the theory and transform it into Ontario school practice in four years, then take it national in 2017-18. Five friendly education researchers were recruited to write papers making the case for including each of the domains, some 78 educators were appointed to advisory committees, and the proposed measures were “field-tested” in 26 different public and Catholic separate schools (20 elementary, 6 secondary), representing a cross-section of urban and rural Ontario.

As an educational sociologist who cut his research teeth studying the British New Labour educational “interventionist machine,” Dr. Cameron was acutely aware that educational initiatives usually flounder because of poorly executed implementation. Much of his focus, in project briefings and academic papers from 2014 onward was on how to “find congruence” between MWM priorities and Ministry mandates and how to tackle the tricky business of winning the concurrence of teachers, and particularly in overcoming their instinctive resistance to  district “education consultants” who arrive promising support but end up extending more “institutional control over teachers in their classrooms.”

Stumbling blocks emerged when the MWM theory met up with the everyday reality of teaching and learning in the schools. Translating the proposed SEL domains into “a set of student competencies” and ensuring “supportive conditions” posed immediate difficulties. The MWM reform promoters came four square up against achieving “system coherence” with the existing EQAO assessment system and the challenge of bridging gaps between the system and local levels. Dr. Cameron and his MWM team were unable to effectively answer questions voicing concerns about increased teacher workload, the misuse of collected data, the mandate creep of schools, and the public’s desire for simple, easy to understand reports. 

Three years into the project, the research base supporting the whole venture began to erode, as more critical independent academic studies appeared questioning the efficacy of assessing Social and Emotional Learning traits or attributes. Dr. Angela L. Duckworth, the University of Pennsylvania psychologist who championed SEL and introduced “grit” into the educational lexicon, produced a comprehensive 2015 research paper with University of Texas scholar David Scott Yeager that raised significant concerns about the wisdom of proceeding, without effective measures, to assess “personal qualities” other than cognitive ability for educational purposes.

Coming from the leading SEL researcher and author of the best-selling book, GRIT, the Duckworth and Yeager research report in Education Researcher, dealt a blow to all state and provincial initiatives attempting to implement SEL measures of assessment. While Duckworth and Yeager held that personal attributes can be powerful predictors of academic, social and physical “well-being,” they claimed “not that everything that counts can be counted or that that everything that can be counted counts.” The two prominent SEL researchers warned that it was premature to proceed with such school system accountability systems. “Our working title, ” she later revealed, “was all measures suck, and they all suck in their own way.”

The Duckworth-Yeager report provided the most in-depth analysis (to date) of the challenges and pitfalls involved in advancing a project like Ontario’s Measuring What Works.  Assessing for cognitive knowledge was long-established and had proven reasonably reliable in measuring academic achievement, they pointed out, but constructing alternative measures remained in its infancy. They not only identified a number of serious limitations of Student Self-Report and Teacher Questionnaires and Performance Tasks (Table 1), but also provided a prescription for fixing what was wrong with system-wide implementation plans (Table 2).

 

 

 

 

 

 

 

 

 

 

 

Duckworth went public with her concerns in February of 2016.  She revealed to The New York Times that she had resigned from a California advisory board fronting a SEL initiative spearheaded by the California Office to Reform Education (CORE), and no longer supported using such tests to evaluate school performance. University of Chicago researcher Camille A. Farrington found Duckworth’s findings credible, stating: “There are so many ways to do this wrong.” The California initiative, while focused on a different set of measures, including student attendance and expulsions, had much in common philosophically with the Ontario venture.

The wisdom of proceeding to adopt SEL system-wide and to recast student assessment in that mold remains contentious.  Anya Kamenetz‘s recent National Public Radio commentary(August 16, 2017) explained, in some detail, why SEL is problematic because, so far, it’s proven impossible to assess what has yet to be properly defined as student outcomes.  It would also seem unwise to overlook Carol Dweck’s recently expressed concerns about using her “Growth Mindset” research for other purposes, such as proposing a system-wide SEL assessment plan.

The Ontario Measuring What Matters initiative, undeterred by such research findings, continues to plow full steam ahead. The five “dimensions of learning” have now morphed into five “domains and competencies” making no reference whatsoever to the place of the cognitive domain in the overall scheme.  It’s a classic example of three phenomena which bedevil contemporary education policy-making: tautology, bias confirmation and the sunk cost trap.  Repeatedly affirming a concept in theory (as logically irrefutable truth) without much supporting research evidence, gathering evidence to support preconceived criteria and plans, and proceeding because its too late to take a pause, or turn back, may not be the best guarantor of long-term success in implementing a system-wide reform agenda.

The whole Ontario Measuring What Works student assessment initiative raises far more questions than it answers. Here are a few pointed questions to get the discussion started and spark some re-thinking. 

On the Research Base:  Does the whole MWM plan pass the research sniff test?  Where does the cognitive domain and the acquisition of knowledge fit in the MWM scheme?  If the venture focuses on Social and Emotional Learning(SEL), whatever happened to the whole student resilience domain, including grit, character and growth mindset? Is it sound to construct a theory and then commission studies to confirm your choice of SEL domains and competencies?

On Implementation: Will introducing the new Social Learning criteria on Ontario student reports do any real harm? Is it feasible to introduce the full MWM plan on top of the current testing regime without imposing totally unreasonable additional burdens on classroom teachers?  Since the best practice research supports a rather costly “multivariate, multi-instrumental approach,” is any of this affordable or sustainable outside of education jurisdictions with significant and expandable capacity to fund such initiatives? 

 

Read Full Post »

Edutopian Visions: Will Any of the Forecasted “Disruptions” Actually Happen?

American education professor Peter W. Cookson, Jr., currently President of Ideas without Borders, recently set the education world atwitter with a futuristic October 10, 2017 Education Week commentary.  Under the eye-catching title, “Ten Disruptions That Will Revolutionize Education,“ Dr. Cookson proclaimed with declaratory certainty that “Artificial Intelligence (AI) and technology will prove significant for education” in the not-too-distant future.

His Education Week commentary provides a fine example of what Canadian journalist and author Dan Gardner has aptly termed “Future Babble.”  In his 2011 book of the same name, he demonstrated that “experts” in any given field were just slightly better at making predictions than a dart-throwing chimp. In addition, the more certain an expert was of a predicted outcome, and the bigger their media profile, and the less accurate the prediction was likely to be.

Reading Dr. Cookson’s rather ‘edutopian’ musings and mindful of the past record of modern day soothsayers, it’s fair to ask whether any of the ten “disruptions” will ever “revolutionize education.”

Let’s start by summarizing his hypothesis and reviewing his list of “creative disruptions” forecasted to “revolutionize” schooling. The advance of machines, according to Peter Cookson, was to be embraced rather than resisted like the plague. “The development of advanced artificial intelligence, or super-intelligence,” he contended,”opens up doors to discoveries never before imagined. While opinions vary about the speed with which superintelligence will develop, there is little doubt that within the next decade, the cognitive landscape will be very different than it is today.”

Here is the full list of purportedly positive “disruptions”:

1. Digital learners will rebel against intellectual conformity.

2. Learning avatars will become commonplace.

3. Participatory-learning hubs will replace isolated classrooms.

4. Inquiry skills will drive learning.

5. Capacities will matter more than grades.

6. Teachers will become inventors.

7. School leaders will give up their desks.

8. Students and families will become co-learners and co-creators.

9. Formal credentials will no longer be the Holy Grail.

10. Policymakers will form communities of continuous improvement.

His summation amounts to a declaration of faith in the new gospel of “21st century learning.” “If education stays stuck in the past, generations of students will be miseducated,” Cookson claims. “They (students) won’t be equipped to thrive in a world of new ideas and technologies. The current task of educators should be to embrace these changes with an open mind and consider how new disruptions can aid, rather than hinder, learning for all students.”

Cookson’s vision of a digital learning future proved tantalizing to leading education education observers and, whether intended or not, was seen as a provocation.  University of Virginia cognitive psychologist Daniel T. Willingham responded tersely on Twitter: “My bold prediction: none of these 10 will disrupt education. None.”

All but one of the 16 comments on the post on the Education Week website dismissed Cookson’s forecast as either sheer nonsense or a threatening forecast of a dystopian future where teachers were supplanted by robots.  Most of the teacher respondents considered the commentary the hallucinations of a “21st century education” futurist.  Canadian education blogger and Math/Technology teacher David Wees rejected Cookson’s forecast entirely and provided a ‘reality check’ list of his own, pointing out the real obstacles to American educational advance, including the status and salaries accorded to teachers, inequitable funding and resources, and the stark inequalities facing students from marginalized communities.

Cookson’s forecast is so problematic that it is hard to decide where to start and whether there is enough space in a short blog commentary to take issue with each of his prognostications.  Since Cookson provides no research evidence to support his claims, you are expected to accept them as unassailable truths. If one thing is abundantly clear, Cookson exhibits a significant blind spot in his total neglect of the “knowledge domain” in his brief in support of embracing technology-driven. “21st century learning.”

Dr. Willingham is essentially correct in his critique of the education futurists. Since 2008, he has been sounding the alarm that the pursuit of “21st century skills” will prove unwise because the acquisition and application of knowledge still matters and will continue to matter in the future.  Without sound background knowledge, students have more difficulty mastering reading and are susceptible to online hoaxes such as the Pacific Northwest Tree Octopus activity. He goes further in pointing out that mathematics, science, reading, civics, and history are more critical in K-12 education than what are termed “21st century skills.”

Being attuned and open to new research and pedagogical advances is desirable and but so is applying a skeptical eye when confronted with unproven theories. Willingham, for example, is not opposed, per se, to developing critical capacities in students, particularly in new media literacy.  Yet, he and other prominent edutopia skeptics, still worry that futurists are leading us astray and they certainly have past experience on their side.

Where are North American edutopian educators like Peter W. Cookson, Jr. leading us?  Where did he come up with the purported “creative disruptions”?  Is there any evidence, that such changes will improve student achievement or produce better informed, more productive citizens? Without radical changes in the socio-economic conditions of, and schooling provision for, marginalized students, can we expect much of an improvement?  And finally, is it sound thinking to put so much faith in the transformative powers of technology? 

Read Full Post »

Rediscovering the Trivium: What Makes Martin Robinson’s Trivium 21c a Must-Read Book?

With the 2017-18 school year on the horizon, British English teacher and research lead Carl Hendrick produced a feature for The Guardian with the alluring title “Ten books every teacher should read.” Most of the ten books published over the past decade and listed as must-reads for teachers bore mighty familiar names, such as Daniel T. Willingham, John Hattie, Daisy Christodoulou, and Dylan Wiliam. On that list is one wild card offering, Martin Robinson’s highly original and intellectually stimulating 2013 book, Trivium 21c: Preparing Young People for the Future with Lessons from the Past.  It’s a courageous book that tackles the biggest issue of all – what is the true purpose of education and how does contemporary schooling measure up?

The author of Trivium 21c is an unusual fellow, a drama teacher-turned-teacher-philosopher, with an unmistakable independent streak. After struggling at school himself, he turned to teaching and joined the profession in his late twenties. Upon entering the classroom, he thrived as a highly motivational teacher of Drama and the Arts.  His initial Twitter handle was @SurrealAnarchy and that gives some indication of his willingness to engage in creative disruption. He wrote the book as a way of responding to his young daughter’s queries about the meaning of Latin terms and innate curiosity about the real purpose of schooling.

As a classroom teacher, Robinson was troubled by the tide in favour of a utilitarian education to prepare students for assessments and success in the 21st century workplace. “Kids were more focused on exams, grades and learning how to pass, ” he observed, “and were becoming less independent and less creative.”  “The new breed of students were customers demanding a service,” in his view, and increasingly expected to be “fed, some of them force fed” with lessons served up “ready cooked.”

In a field overflowing with inspirational educational leadership guides and magic bullet curriculum reform books, Trivium 21c occupies what headteacher Tom Sherrington described as “different ground altogether.”  It stands out as a manifesto for reforming and revitalizing educational practice, our discourse and our system based on a set of core principles that speak to what education means to individuals, communities, and society.

Robinson’s explorations lead him back to the Trivium, the essential construct of liberal education dating from the time of the Ancients. The Trivium consists of three core components: grammar, dialectic and rhetoric. Here’s a capsule summary of what each element entails:

Grammar: The need for core knowledge — the cultural capital that we accrue through transmission, essentially the things that we all must know to function in the modern world;

Dialectic: The need to question, debate and discuss ideas, to form our own opinions, to engage in authentic experiences, and to grow in our capacities and build character;

Rhetoric: The need to be able to communicate our ideas and knowledge in a variety of forms,  to create and perform with flair and confidence.

 

Moving from Ancient Greece to the present day, Trivium 21c proposes a contemporary trivium (Grammar, Dialectic, and Rhetoric) with the potential to unite progressive and traditionalist pedagogy and approaches among teachers, politicians and parents in the common pursuit of a better education. ‘The three ways of the trivium– knowing, questioning, and communicating — ,” Robinson claims, make for “a great education.” What he wanted for his own daughter was schooling that actually gave her “the grounding” to lead “the good life.”

Education policy and practice in Canada is, as in Britain and the United States, a subterranean battleground. Traditionalists argue for the teaching of a higher order of hard knowledge and deride soft skills. Progressives deride learning about great works of the past preferring ‘21c skills’ (21st century skills) such as creativity and critical thinking.  The bridge, in Robinson’s view, can be found in the trivium because it provides a framework that facilitates “preparing young people for the future with lessons from the past.”

Frustrated by a prevailing educational orthodoxy that seems incapable of  marrying respect for knowledge with creativity, to foster discipline alongside free-thinking, and to value citizenship with independent learning, Robinson favours what might be termed “progressive traditionalism.” Drawing from his work as a creative teacher respectful of the liberal education tradition, he finds inspiration in the Arts and the need to nurture learners with the ability to not only cope but surmount the uncertainties of our contemporary age.  His follow-up 2016 volume, Trivium 21c in Practice, provides a range of exemplars of best practice in a cross-section of U.K. schools.

Author Robinson will soon become better known in Canada for his provocations.  He will be making his first appearance on this side of the pond at researchED Toronto, November 10-11, 2017, at Trinity College, University of Toronto.  It’s not too late to reserve a seat to see him in action with more than two dozen leading educational thinkers and teacher-researchers from Britain and right across Canada. .

Why is Martin Robinson’s Trivium 21c such a refreshing education book?  Can the schism that divides so-called “traditionalists” and “progressives” be bridged through a reinvention of the trivium?  Is it possible to both walk on the shoulders of giants and to make giant creative leaps (from those shoulders) in the pursuit of better education for today’s students?  

Read Full Post »