June 2013
Telling Time with a Broken Clock: The trouble with standardized testing
"The more we learn about standardized testing, particularly in its high stakes incarnation, the more likely we are to be appalled."
– Alfie Kohn
What do standardized test scores really tell us? As with many public policy questions, this is a complex question – yet too many people assume to know its answer. Whole school jurisdictions and entire nations define themselves by their standardized test results, including provincially administered examinations and international instruments such as Program for International Student Assessment (PISA). All of these programs, justified by their so-called impartiality and objectivity, share the assumption that the scores must be the public's “transparent” window into the quality of our schools.
Best-selling author and blogger Seth Godin reminds us that the worst kind of clock is a clock that randomly runs a little fast or a little slow. "If there's no clock," Godin writes, "we go seeking the right time. But a wrong clock? We're going to be tempted to accept what it tells us."[1] Godin's message is that tracking the wrong data or misreading good data can get us into trouble. What if standardized test scores aren't telling us what we think they are telling us? What if the scores are illusions that are giving us false confidence? What if our reliance on standardized testing to judge our schools is like relying on a broken clock for time?
If education policy pundits (many of whom are not teachers) and politicians expect the public to trust the scores as prima facie evidence of the quality of teaching and learning that goes on in our schools, then the public needs to know more about the costs and consequences of standardized testing. It is time to move past our historic reliance on standardized testing programs driven by educational bureaucracies satisfied to measure what is easiest to measure. Canada’s already strong public school systems will not punch past their current levels of performance unless we move to a new approach to public assurance.
Measuring what matters least
Ask any parents what their long-term concerns and goals are for their children, and seldom will you hear about test scores and world rankings. Their concerns are compelling, existential and heartfelt. Parents want their kids to be happy, hard working, motivated, responsible, honest, empathetic, intelligent, collaborative, creative and courageous. Of course we want our children to grow academically, but we also want them to grow emotionally, socially and physically, and this requires a well-rounded education that cannot be evaluated by standardized tests. This is not an argument against academics, after all academics should play a large role in school -- however, even when it comes to numeracy and literacy, standardized tests tend to be limited to measuring forgettable facts while ignoring the higher level creative and critical thinking. It makes a lot of sense to question the scores when you know that the tests are an amazingly contrived and unrealistic form of assessment that measures what matters least. It’s time we shifted from valuing what we measure to measuring what we value.
Campbell’s Law
Well-known (but not well-known enough) in social science, Campbell's Law tells us that “the more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it was intended to monitor.”[2] In the case of standardized testing, corruption and distortion can come in a variety of ways. Here are three examples:
Teaching or testing? Teaching to the test and excessive test preparation invalidates inferences that can be drawn from the scores – yet they are the inevitable response to pressure to produce good test scores. Classroom time is devoured by not only the tests themselves but also practice tests, pre- and post-tests, field tests for the tests, benchmark tests, teacher tests, district tests, and state or provincial tests. Because testing is not teaching, this ultimately leads to a loss of opportunities for students to have a broad range of educational experiences, and the first things to go usually end up being the arts and physical activity – which do not lend themselves to be easily tested.
Learning or cheating? The moment low-stakes test scores are publicized to rank and sort teachers or schools, they become high-stakes tests. Where there is smoke, there is usually fire, and where there is high-stakes standardized testing, there is cheating. We can bemoan this inconvenience or play the blame game, but it won't change anything. When we stop and reflect on why cheaters cheat, systems thinking tells us that cheating has less to do with the characteristics of individual teachers or students and more to do with the priorities of schools and school systems.
Raising children or raising scores? When schools are encouraged to focus on test scores, some come to see children less as individuals of worth regardless of their academic ability, and more as score increasers and score suppressors. Sadly, the more the scores are made to count for teachers and schools, the more the scores count against the children who need the most help. They will be seen as undesirable; after all, they are the students most likely to score low, dragging down the school's ranking. This becomes even truer when (as in the U.S.) merit pay schemes marry teacher pay to the scores and/or when a school's reputation hinges on being publicly ranked. Under these twisted circumstance, schools may come to know more about raising scores than raising children.
We can no more skirt the real-world ramifications of Campbell's Law than Wile E. Coyote could avoid the punishing effects of gravity. Nichols and Berliner explains, "apparently, you can have (a) higher stakes and less certainty about the validity of assessment or (b) lower stakes and greater certainty about validity. But you are not likely to have both high stakes and high validity. Uncertainty about the meaning of test scores increases as the stakes attached to them become more severe."[3]
Socio-economics
Alfie Kohn begins his article “Fighting the Tests: A practical guide to rescuing our schools” by stating, "Don't let anyone tell you that standardized tests are not accurate measures. The truth of the matter is they offer a remarkably precise method for gauging the size of the houses near the school where the test was administered."[4] The inconvenient truth about standardized testing is that socioeconomic status is responsible for an overwhelming proportion (50 to 70 percent) of the variance in test scores. The strongest predictor of student performance on achievement tests is socio-economic status, which is why it is a mistake to believe that the scores tell us about school quality when really they are reflecting affluence or poverty.
No school or school system has ever become great without great teachers, but what can an excellent teacher do about a child who needs glasses, has cavities or is hungry? To say that teacher or school quality is the most important variable in education is at best naive. Education historian Diane Ravitch writes, "Reformers tell us that teachers are the most important influence within the school on student scores, and that is right. But the teacher contribution to scores is dwarfed by the influence of family and other out-of-school factors."[5]
Ultimately, great teachers make great schools, but great teachers can't do it alone – they require the support of an equitable society. If we are not careful, we risk misinterpreting the scores, and instead of waging war on poverty and inequity, we end up waging war on teachers and schools.
Dispelling the Corporate “Reform” Agenda
Finland's Pasi Sahlberg warns us about how the Global Education Reform Movement (GERM) focuses schools on competition, standardization and test-based accountability. With GERM, "education has become a commodity where the efficiency of service delivery ultimately determines performance."[6]
Alongside the rush to introduce unproven technologies into classrooms, standardized testing in the United States has become a political instrument wielded by organizations such as Students First, Democrats for Education Reform and the American Legislative Exchange Council. Linking teacher pay to student test scores, and eliminating tenure and collective bargaining have become popular methods for undermining confidence in public schools so that education entrepreneurs can pour private equity and venture capital into companies that aim to profit from public education.
After a decade of intense standardized testing and sanctions under No Child Left Behind, California Democrats passed a resolution that aims at supporting public education and dispelling the corporate ‘reform’ agenda by stating, “the reform initiatives of Students First, rely on destructive anti-educator policies that do nothing for students but blame educators and their unions for the ills of society, make testing the goal of education, shatter communities by closing their public schools, and see public schools as potential profit centers and children as measurable commodities.”[7]
In the face of tough economic times, some governments might see this as a way of easing pressure on their beleaguered budgets; however, Diane Ravitch warns us that, "Our schools will not improve if we expect them to act like private, profit-seeking enterprises. Schools are not businesses; they are a public good. The goal of education is not to produce higher scores, but to educate children to become responsible people with well-developed minds and good character."[8]
Over the last two decades, the United States has proven to be a cautionary tale for how Canada, and the world, should not reform education, and Canadians would be wise not to think that the 49th Parallel offers any kind of inherent insulation from the corporate education reform agenda.
Building a better clock
The way forward is not to build schools that are a better version of yesterday. It is telling that the demand for standardized testing intensifies the further you get from classrooms, and yet authentic accountability and public assurance needs to happen within schools and communities, not driven by a tired-out model of bureaucratic accountability focused on compliance.
A move away from standardized testing is not a case for the absence of accountability – it is a pathway to supporting innovation and creativity. Ruth Sutton reminds us, "The issue is not whether we need information about the learning and achievement of our children and young people, but what kind of information we need, and how best to gather it."[9] Once we can see that standardized testing is like a broken clock, we can work together to figure out how to build a better clock.
Building a better clock and better schools starts with asking tough questions – which is precisely the spirit behind an Action Canada Task Force report titled Real Accountability or an Illusion of Success?[10] that invites Canadians to take a deeper look at the goals of public education and the role of standardized testing.
Alberta should be congratulated for their recent move away from their grade 3, 6 and 9 Provincial Achievement Tests. Now Alberta requires the courage to see that the replacement for the old tests should not be new tests. From a policy perspective, accountability models that are based on census testing of entire student populations do little to support student progress and are not cost effective. A shift from testing every student to a sample program using performance-based assessments would be less obstructive, cost less and provide more meaningful information.
Given the highly relational nature of the teaching and learning process, the best teachers know that assessment is not a spreadsheet; it's a conversation. They also know that there is no substitute for what teachers observe while their students are actively learning, and this is why the best assessments ask students to actually do something that is in a context and for a purpose. Unlike standardized tests which cannot provide anything more than a limited and incomplete snapshot of a student on a single day, a collection of performance assessments assembled in a learning portfolio can inform the teaching and learning process in a timely fashion, while simultaneously assuring the public that students are receiving a high quality education.
To enhance public assurance, Andy Hargreaves suggests departments of education should shift from a “bureaucratic accountability model to a locally focused, student-driven assurance model based on school-development plans and teachers as leaders in innovation.”[11] The means for accomplishing this can be found in programs like Alberta’s Initiative for School Improvement (AISI). A number of community-based school improvement initiatives such as AISI exist throughout Canada and yet are vulnerable to cuts by government, such as Alberta who recently axed AISI, that see times of fiscal belt-tightening as an excuse to reduce investments in innovation – precisely the wrong time to reduce support for creativity and risk-taking.
It might be argued that standardized testing has allowed us to build good enough schools; after all, even a broken clock is right twice a day. However, business guru Jim Collins reminds us that "good is the enemy of great." If we aspire to create a great school for all children, we need to seek an end to standardized testing and replace it with more sophisticated and more demanding processes for public assurance.
Notes
1 Seth Godin, "The worst kind of clock," Seth Godin (Blog), September 21, 2012,
http://sethgodin.typepad.com/seths_blog/2012/09/the-worst-kind-of-clock.html.
2 Sharon Nichols and David Berliner, Collateral Damage: How high stakes testing corrupts America's schools (Massachusetts: Harvard Education Press, 2008), 26-27.
3 Nichols and Berliner, Collateral Damage, 27.
4 Alfie Kohn, “Fighting the Tests: A practical guide to rescuing our schools,” Phi Delta Kappan (January 2001).
http://www.alfiekohn.org/teaching/ftt.htm.
5 Diane Ravitch, "Why VAM Is Junk Science," Diane Ravitch's Blog, July 16, 2012,
http://dianeravitch.net/2012/07/16/why-vam-is-junk-science/.
6 Pasi Sahlberg, Finnish Lessons: What can the world learn from educational change in Finland? (New York, Teachers College Press, 2011),100.
7 The California Democratic Party, Resolution 13-04.47: Supporting California’s Public School and Dispelling the Corporate “Reform” Agenda (April 14, 2013).
8 Diane Ravitch, The Death and Life of the Great American School System (New York: Basic Books, 2010), 227-28.
9 Alberta Assessment Consortium, A New Look At Public Assurance: Imagining the possibilities for Alberta students.
http://www.aac.ab.ca/resources/pdf/Public%20Assurance%20Doc_final_may31.pdf.
10 Real Accountability or An Illusion of Success? A Call to Review Standardized Testing in Ontario. Action Canada Task Force Report.
http://testingillusion.ca/wp-content/uploads/2013/01/illusion_of_success_EN.pdf.
11 Andy Hargreaves, foreward to A Great School for All: Transforming education in Alberta (Edmonton: Barnett House, 2012).
