The Use and Misuse of Education Research: How Does Dr. Avis Glaze’s Raise the Bar Stack Up?

A sweeping Nova Scotia education reform report, Dr. Avis Glaze’s Raise the Bar, is now attracting an incredible amount of scrutiny in the regional media, among academics, and flocks of tweeting ‘parakeets’ on social media.  As one of Canada’s outstanding educators with impeccable Ontario Institute for Studies in Education credentials, the controversy might strike most Canadian education researchers as downright bizarre. In a field – provincial education policy- not known for stellar, evidence-based research, it is also peculiar and unusual enough to warrant some serious investigation.

Two external assessors, Greg Thompson and David Rutkowski, have now weighted-in with a 3 1/2 page typed “third party review” commissioned by the Nova Scotia Teachers’ Union (NSTU).  Its arrival was announced in an NSTU News Release (February 20, 2018) with a proposed headline: “Third party review calls into question the validity of the Glaze report.”  The release date is significant because it was timed to arrive as the province’s teachers were about to vote on whether to take “strike action” to slow down or derail the Nova Scotia government’s plan to proceed with legislation to implement most of Glaze’s recommendations.

The “third party review” was presented by the NSTU as not just a critique of Glaze’s research methodology, but as evidence that the whole initiative was somehow based upon “flawed research” and should be paused or perhaps abandoned. Education research conducted by and for teachers unions is not necessarily suspect or bad for that matter — and much that is conducted by the British Columbia Teachers’ Federation (BCTF) stands up well and contributes to informed public policy discussion.

One of the biggest problems confronting education research everywhere is what is termed “bias confirmation.”  Once you are attuned to the logical fallacy, it’s relatively easy to spot.  In this case, the evidence jumps right out in the first paragraph of the short piece. “When embarking on a radical, systematic restructure, we expect policy makers to use the best information available to inform their policy decisions.,” Thompson and Rutkowski state, before adding a qualifier worth examining: “That said, education has become a “marketplace of ideas’ with policy soothsayers plying their trade in a lucrative international market.”

Clearly the two academics are framing the work of one of Canada’s leading educators as that of one of those “policy soothsayers” engaged in a “lucrative” international business.  They also describe the proposed reforms, several times, as “radical” rather than “transformative” and summarize the contents of the report in a way that highlights its disruptive-ness.  “Disbanding school boards,””setting up a College of Teachers,” and “removing principals and vice principals” from the NSTU  are the only three of the 22 recommendations actually referenced in their review.  They also happen to be the three major sticking points for the union.

The NSTU commissioned “third party review” focuses rather narrowly on Glaze’s survey research methodology rather than the substance of her documentary research based upon more than 70 written submissions and NSSBA research conducted by David MacKinnon of Acadia for the Nova Scotia School Board’s Association (NSSBA). The researchers are, indirectly, slagging all those who submitted briefs informed by research evidence.  What’s most interesting about the short type-script is that it provides an analysis of the methodology in a short piece with no academic references. Most scholarly reviews at least cite sources and provide parenthetic references to supporting documents.

The two researchers are quite effective at picking-apart the survey research methodology and many of their points are well taken and legitimate, even if such quantitative lapses are quite common in public policy research.  The Glaze Report survey was rather simplistic and the results hard to quantify, but — in fairness– the wording was easy to understand and accessible to most Nova Scotians. You can also argue that open-ended questions are more likely to elicit honest, straightforward answers. It was, keep in mind, just one aspect of Glaze’s primarily research-driven project.

Being parachuted into Nova Scotia for such an assignment is not a problem in and of itself, if the researchers demonstrate some grasp of the total context and larger policy environment.  In this case, Thompson and Rutkowski, approach the report as a document in isolation and not part of a continuum of education policy debate and development.

A few examples demonstrate how important it is to properly “read” a policy environment before weighing in to render a judgement on one particular document. If Thompson and Rutkowski  had compared the Glaze Report with the earlier Nova Scotia NDP policy plan, Kids & Learning First (February 2012), they might have reached different conclusions.  That education reform plan came in a glossy, 35-pager with lots of photos and  containing no bibliography. Little or no direct reference was even made to Dr. Ben Levin’s 2011-12 education policy “literature review.”

Looking at the Glaze Report as a continuation of the Myra Freeman Commission of 2013-14 also casts the whole exercise in a different policy context.  The Freeman Report (October 2014) was actually based on a province-wide survey that netted over 19,000 responses and recommended (R 2.6 and R 2.7) that the Government introduce a more robust “performance management system” and “consider” removing “supervisory staff” including principals from the union.  Even though one out of every three teachers (3,167) completed the survey, leading members of the NSTU criticized the Freeman Committee for poor research methodology and the wording of its survey questions.

The recent third party review also makes no reference whatsoever to the most critical piece in situating this particular set of proposed reforms. One would expect that the academics might make some reference to the Nova Scotia Education plan known as The Three R’s, the most recent statement of education policy. If they had consulted that document in their research, they would have discovered that the Department publicly declared its intention to negotiate the key points in contention at the bargaining table.  We understand that the NSTU (behind closed doors) refused to discuss the proposals now featured in the Glaze Report.

All of this does raise the larger question about the state of Canadian education policy research and why organizations such as the NSTU might go further afield in search of researchers. Teaching and learning research lags significantly here in Canada where – with few exceptions – faculties of education are simply not producing ground-breaking, evidence-based research on critical curriculum and pedagogical issues. Compared to Britain and the United States, where the education debate has spawned hundreds of government and independent research institutes, Canada continues to show a dearth of research activity, especially outside the University of Toronto orbit of the Ontario Institute for Studies in Education (OISE). 

The authors might want to do a little more research on Dr. Avis Glaze. If and when they do, they will likely discover that this so-called “policy soothsayer” is revered (outside of Nova Scotia) as an outstanding OISE-trained researcher who, as Ontario Superintendent of Student Achievement, introduced and led the province’s first “What Works” Research-Informed Policy program, producing dozens of research briefs aimed at improving teaching and learning. She will survive a three page type-script note with no supporting references.

Educational research is improving, in part because of Dr. Glaze and a small group of education scholars, but it still has a bad name.  Instead of attacking education issues and problems, conducting evidence-based research, and letting the evidence suggest solutions, many practitioners continually engage in research driven by “bias confirmation.” We all have to guard against it in our work.  One of the most popular topics featured in Educational Leadership is the scourge of “politically-driven” education research.  It’s challenging to rise above it and Dr. Glaze is one education researcher who exemplifies the kind of research that Canadian K-12 education needs more of.

What’s the problem with most Canadian K-12 education policy research?  Should education policy documents be more closely scrutinized and assessed through a research lens?  What constitutes a legitimate “third party peer review”?  Should researchers analyzing documents be well grounded in the evolving education policy world? How can we separate “good” education research from the regular fare of commissioned studies? What needs to be done to clean up the field? 

 

 

Read Full Post »

Education Curriculum Surveys: Why Do School Systems Survey the Public?

Surveying the education public as well as ‘stakeholders’ for their opinion is the latest trend in Canadian K-12 education policy. Two recent Canadian education surveys conducted in Nova Scotia and Alberta provide some recent examples worthy of further discussion.  The recent release of Alberta Education Minister David Eggen’s curriculum survey results (April 13, 2017) also demonstrates that unsuspecting citizens may need help in penetrating through the official spin to get at the actual results.

Facing deep divisions in P-12 education over future directions, and not inclined to follow the research evidence, provincial authorities are going, more and more, to soliciting public opinion utilizing surveys with pre-determined outcomes.  Upon closed scrutiny, the Alberta survey seems wholly designed to confirm intended curriculum directions.

Conducting public surveys is not without its risks. In the case of the 2014 Nova Scotia Education Review survey, a largely unvarnished, no-holds-barred instrument actually backfired on the Education Department. When the N.S. Review Committee headed by Myra Freeman polled 18,500 residents, the results published in October 2014 proved a real jolt and sent the provincial teachers’ union into a tizzy, mostly focused on being excluded from shaping the survey and serving on the commission.

One half of Nova Scotians, the survey found, were “not satisfied with the public school system” and teachers as well as parents identified plenty of reasons why. The report, Disrupting the Status Quo, generated very high expectations — never honoured — that major reform was on the way.  A three-month NSTU teacher work-to-rule in 2016-17 effectively sunk the quality education reform plan and generated a completely new set of teacher-driven demands for improvement in “working conditions.”

Alberta Education had no desire to see that pattern repeated.  Minister Eggen’s curriculum survey looked, and sounded, skewed in the Education Department’s preferred direction – toward more of what is loosely termed “21^st learning.” In Alberta Education futuristic doubletalk, the overarching goal is to produce students who “are agents of change to create the globe that they want to be part of.”

The survey, conducted in October and November 2016 succeeded in attracting some 32,390 respondents, of whom only slightly over half (57%) might be classed as ‘outside the system.’The proposed directions were presented as amorphous curriculum “themes” where respondents are clearly led to certain conclusions. You are, for example, asked whether you agree or disagree with this statement: “Through learning outcomes curriculum should support the development of literacy, numeracy and 21^st century competencies.”  It is impossible to answer if you think basic numeracy and literacy should take precedence over the ill-defined futuristic skills.

Conducting the survey was also further confirmation of the provincial strategy to thwart mathematics education reform. With the Alberta “Back to Basics” petition, initiated by parent Dr. Nhung Tran-Davies of Calmar, AB, piling up 18,332 signatures, the survey attempts, in clumsy fashion, to override that hardened opinion.

The Department’s summary of responses does its best to conceal the extent of resistance to current K-12 Mathematics teaching and curricula.  Sifting through the Mathematics responses, teaching math facts, restoring step-by-step algorithmic thinking, limiting the use of computers, and mastering mental math far outweighed any preference for “21^st century competencies” or its step-child, discovery math.

Instead of addressing these findings, Minister Eggen  ‘cherry-picked’ one example of the desire for ‘relevance’ – support for including financial literacy in grade 4 to 9 classes. That too is a clear sign that parents want their kids to be able to balance a set of sums.

Albertans’ written responses to the open-ended questions are the clearest indication of their true inclinations.  Out of the 15,724 respondents committed enough to do more than tick boxes, the largest segment, again (10 %), favoured refocusing on “math basics” and singled out “discovery math” as a problem. Combined with “learning the basics” (6%) and teaching practical skills (7%), one in four who made comments targeted the lack of rigour in the curriculum.

Judging from the wording of questions, the entire survey also skewed in the direction of student-centred teaching methods. That’s strange because the recent PISA 2015 global results report demonstrated conclusively that “explicit instruction” produced much better student results than “minimally-guided instruction.”

The inherent bias pops up elsewhere. “This survey,” it reported, “was intended to focus on the ‘what’ of current provincial curriculum not ‘how’ teachers teach it.”   Serious curriculum analysts know it’s now virtually impossible to separate the two in assessing program effectiveness.

Provincial education authorities were, at one time, given explicit mandates based upon either firm political policy positions or best practice research. When governments are lost and searching for direction, they may turn to the public to find their bearings. In the case of Alberta, it looks more like surveying for confirmation of the ‘educrats’ own pre-determined direction.

*A condensed version of this Commentary appeared in the Edmonton Journal, April 18, 2017.

Why do school systems survey the public?  Are Canadian provincial governments totally lost on K-12 education and simply looking for direction? Do our Education Department’s  harbour a secret agenda?  Or are they looking for public confirmation of pre-conceived plans for curriculum changes? 

Read Full Post »