Posts Tagged ‘Third Party Peer Reviews’

A sweeping Nova Scotia education reform report, Dr. Avis Glaze’s Raise the Bar, is now attracting an incredible amount of scrutiny in the regional media, among academics, and flocks of tweeting ‘parakeets’ on social media.  As one of Canada’s outstanding educators with impeccable Ontario Institute for Studies in Education credentials, the controversy might strike most Canadian education researchers as downright bizarre. In a field – provincial education policy- not known for stellar, evidence-based research, it is also peculiar and unusual enough to warrant some serious investigation.


Two external assessors, Greg Thompson and David Rutkowski, have now weighted-in with a 3 1/2 page typed “third party review” commissioned by the Nova Scotia Teachers’ Union (NSTU).  Its arrival was announced in an NSTU News Release (February 20, 2018) with a proposed headline: “Third party review calls into question the validity of the Glaze report.”  The release date is significant because it was timed to arrive as the province’s teachers were about to vote on whether to take “strike action” to slow down or derail the Nova Scotia government’s plan to proceed with legislation to implement most of Glaze’s recommendations.

The “third party review” was presented by the NSTU as not just a critique of Glaze’s research methodology, but as evidence that the whole initiative was somehow based upon “flawed research” and should be paused or perhaps abandoned. Education research conducted by and for teachers unions is not necessarily suspect or bad for that matter — and much that is conducted by the British Columbia Teachers’ Federation (BCTF) stands up well and contributes to informed public policy discussion.

One of the biggest problems confronting education research everywhere is what is termed “bias confirmation.”  Once you are attuned to the logical fallacy, it’s relatively easy to spot.  In this case, the evidence jumps right out in the first paragraph of the short piece. “When embarking on a radical, systematic restructure, we expect policy makers to use the best information available to inform their policy decisions.,” Thompson and Rutkowski state, before adding a qualifier worth examining: “That said, education has become a “marketplace of ideas’ with policy soothsayers plying their trade in a lucrative international market.

Clearly the two academics are framing the work of one of Canada’s leading educators as that of one of those “policy soothsayers” engaged in a “lucrative” international business.  They also describe the proposed reforms, several times, as “radical” rather than “transformative” and summarize the contents of the report in a way that highlights its disruptive-ness.  “Disbanding school boards,””setting up a College of Teachers,” and “removing principals and vice principals” from the NSTU  are the only three of the 22 recommendations actually referenced in their review.  They also happen to be the three major sticking points for the union.

The NSTU commissioned “third party review” focuses rather narrowly on Glaze’s survey research methodology rather than the substance of her documentary research based upon more than 70 written submissions and NSSBA research conducted by David MacKinnon of Acadia for the Nova Scotia School Board’s Association (NSSBA). The researchers are, indirectly, slagging all those who submitted briefs informed by research evidence.  What’s most interesting about the short type-script is that it provides an analysis of the methodology in a short piece with no academic references. Most scholarly reviews at least cite sources and provide parenthetic references to supporting documents.

The two researchers are quite effective at picking-apart the survey research methodology and many of their points are well taken and legitimate, even if such quantitative lapses are quite common in public policy research.  The Glaze Report survey was rather simplistic and the results hard to quantify, but — in fairness– the wording was easy to understand and accessible to most Nova Scotians. You can also argue that open-ended questions are more likely to elicit honest, straightforward answers. It was, keep in mind, just one aspect of Glaze’s primarily research-driven project.

Being parachuted into Nova Scotia for such an assignment is not a problem in and of itself, if the researchers demonstrate some grasp of the total context and larger policy environment.  In this case, Thompson and Rutkowski, approach the report as a document in isolation and not part of a continuum of education policy debate and development.

A few examples demonstrate how imKids&LearningFirstportant it is to properly “read” a policy environment before weighing in to render a judgement on one particular document. If Thompson and Rutkowski  had compared the Glaze Report with the earlier Nova Scotia NDP policy plan, Kids & Learning First (February 2012), they might have reached different conclusions.  That education reform plan came in a glossy, 35-pager with lots of photos and  containing no bibliography. Little or no direct reference was even made to Dr. Ben Levin’s 2011-12 education policy “literature review.”

Looking at the Glaze Report as a continuation of the Myra Freeman Commission of 2013-14 also casts the whole exercise in a different policy context.  The Freeman Report (October 2014) was actually based on a province-wide survey that netted over 19,000 responses and recommended (R 2.6 and R 2.7) that the Government introduce a more robust “performance management system” and “consider” removing “supervisory staff” including principals from the union.  Even though one out of every three teachers (3,167) completed the survey, leading members of the NSTU criticized the Freeman Committee for poor research methodology and the wording of its survey questions.

NSTeacherReformThreeRsThe recent third party review also makes no reference whatsoever to the most critical piece in situating this particular set of proposed reforms. One would expect that the academics might make some reference to the Nova Scotia Education plan known as The Three R’s, the most recent statement of education policy. If they had consulted that document in their research, they would have discovered that the Department publicly declared its intention to negotiate the key points in contention at the bargaining table.  We understand that the NSTU (behind closed doors) refused to discuss the proposals now featured in the Glaze Report.

All of this does raise the larger question about the state of Canadian education policy research and why organizations such as the NSTU might go further afield in search of researchers. Teaching and learning research lags significantly here in Canada where – with few exceptions – faculties of education are simply not producing ground-breaking, evidence-based research on critical curriculum and pedagogical issues. Compared to Britain and the United States, where the education debate has spawned hundreds of government and independent research institutes, Canada continues to show a dearth of research activity, especially outside the University of Toronto orbit of the Ontario Institute for Studies in Education (OISE). 

The authors might want to do a little more research on Dr. Avis Glaze. If and when they do, they will likely discover that this so-called “policy soothsayer” is revered (outside of Nova Scotia) as an outstanding OISE-trained researcher who, as Ontario Superintendent of Student Achievement, introduced and led the province’s first “What Works” Research-Informed Policy program, producing dozens of research briefs aimed at improving teaching and learning. She will survive a three page type-script note with no supporting references.

Educational research is improving, in part because of Dr. Glaze and a small group of education scholars, but it still has a bad name.  Instead of attacking education issues and problems, conducting evidence-based research, and letting the evidence suggest solutions, many practitioners continually engage in research driven by “bias confirmation.” We all have to guard against it in our work.  One of the most popular topics featured in Educational Leadership is the scourge of “politically-driven” education research.  It’s challenging to rise above it and Dr. Glaze is one education researcher who exemplifies the kind of research that Canadian K-12 education needs more of.

What’s the problem with most Canadian K-12 education policy research?  Should education policy documents be more closely scrutinized and assessed through a research lens?  What constitutes a legitimate “third party peer review”?  Should researchers analyzing documents be well grounded in the evolving education policy world? How can we separate “good” education research from the regular fare of commissioned studies? What needs to be done to clean up the field? 




Read Full Post »