Making the Grades by Todd Farley

"Making the Grades: My Misadventures in the Standardized Testing Industry" by Todd Farley

"Making the Grades is an enlightening, entertaining, and eye-opening look at the use and misuse of standardized tests….This book is a humorous yet sobering glimpse at the testing arena--a must-read for policymakers and pundits enchanted by high-stakes tests.”

--Dennis Van Roekel, President, National Education Association


“With so much emphasis these days on standardized testing in education, Todd Farley’s important insights, gathered during his 15 years of working with the people who actually grade the tests, should be read closely. As our nation moves toward improving academic standards and assessments, Farley’s engaging stories make a book about testing interesting reading. This book reminds us that even though an assessment system may look good from the outside, safeguards must be in place to ensure each education decision that is informed by a test score is made with the greatest care.”

--Randi Weingarten, President, American Federation of Teachers


"Todd Farley is the scourge of standardized testing. His book, Making the Grades, is a shocking exposé."

--Diane Ravitch


“This book is dynamite! The nice personal voice (biting anger, conveyed with delicious humor) makes it utterly accessible and enticing—i.e., a good story, wholly apart from the terribly important ammunition it provides to those of us in the ‘testing wars’ at national and local levels.”

--Jonathan Kozol, author of Savage Inequalities


“Mr. Farley is an engaging writer who tells an often hilarious story--hilarious until you remind yourself that these graders, making their hasty and often half-baked judgments, are affecting children's futures. Highly recommended."

--Charles Murray, co-author of The Bell Curve


"Using standardized test scores to evaluate our nation’s teachers and classrooms is a gruesome mistake and Making the Grades shows us why. We see the frightening reality behind the tests and why believing in these“objective” measures is both a comedy and a horror show. Todd Farley’s revealing story shows the price we pay for our false belief in standardized testing: teaching to tests crushes children’s creative ideas, their independent thoughts, and their love of learning."

--Nancy Carlsson-Paige

Ravitch: Should Computers Grade Essays?

This was written by Diane Ravitch who is an advocate for progressive education reform who works tirelessly to call out education deformers and their corporate-style reforms that work to undermine teachers and public education. Diane blogs here and tweets here. This post was originally posted here.

by Diane Ravitch

Todd Farley is the scourge of standardized testing. His book, “Making the Grades,” is a shocking exposé of the industry. Todd spent nearly 15 years scoring tests, and he knows the tricks of the trade.

In this article, he skewers the latest testing craze: machine-scoring of essays.

Having demonstrated the fallibility of humans who score essays, Farley is no more impressed by computer scoring. As he puts it:

“…the study’s major finding states only that “the results demonstrated that overall, automated essay scoring was capable of producing scores similar to human scores for extended-response writing items.” A paragraph on p. 21 reiterates the same thing: “By and large, the scoring engines did a good [job] of replicating the mean scores for all of the data sets.” In other words, all this hoopla about a study Tom Vander Ark calls “groundbreaking” is based on a final conclusion saying only that automated essay scoring engines are able to spew out a number that “by and large” might be “similar” to what a bored, over-worked, under-paid, possibly-underqualified, temporarily-employed human scorer skimming through an essay every two minutes might also spew out. I ask you, has there ever been a lower bar?”

Farley quotes the promoters of automated scoring, who say that the machines are faster, cheaper and more consistent than humans. Also, they make money.

He concludes: “Maybe a technology that purports to be able to assess a piece of writing without having so much as the teensiest inkling as to what has been said is good enough for your country, your city, your school, or your child. I’ll tell you what though: Ain’t good enough for mine.”

One of the responses to Farley’s post came from Tom Vander Ark, who is a tech entrepreneur and a target of Farley’s post.

Vander Ark wrote: “The purpose of the study was to demonstrate that online essay scoring was as accurate as expert human graders and that proved to be the case across a diverse set of performance tasks. The reason that was important is that without online scoring, states would rely solely on inexpensive multiple choice tests. It is silly to suggest that scoring engines need to ‘understand,’ they just need to score at least as well as a trained expert grader and our study did just that.”

A reader of this blog saw this exchange on Huffington Post and sent me this comment:

“Diane–we use an automated essay scorer at my school, and I have seen coherent, well-thought out writing receive scores below proficient, while incoherent, illogical writing (with more and longer words, and a few other tricks that automated scorers like) receive high scores. The students who suffer the most are the highest level students, the verbally gifted writers who write with the goal of actually being understood, “silly” as that may be.”

“In fact, all standardized testing penalizes the brightest students–those who think outside the box. Standardized testing is the box.”