Assessment and measurement are not the same thing

Too many people confuse measurement with assessment as if they were the same thing.

They are not.

Some things are made to be measured. For example, I'm 6'1''. Height is a one-dimensional thing that can be reduced to a measurement in standard units. We need standard units for height or we would have all kinds of mass confusion.

Some things in life, however, are not made to be measured. While my height can be accurately described as 6'1'' without debate, my personality, character, intelligence, athleticism and learning can not be meaningfully reduced to a symbol. When we reduce something as magnificently messy as learning to a number, we always conceal far more than we ever reveal.

The most important things that children learn in school are not easily measured. The most meaningful things in life may, in fact, be immeasurable. The good news, however, is that the most important and meaningful things that we want children to learn and do in school can always be observed and described. This is precisely why it is so important to remember that the root word for assessment is assidere which literally means 'to sit beside.' Assessment is not a spreadsheet -- it's a conversation.

Testsandgrades should be replaced with projects and performances collected in portfolios.

When student learning is made visible to parents through portfolios, blogs, student-led conferences and parent-teacher interviews then they are not nearly so desperate for less meaningful information such as testsandgrades.

This is my 16th year of teaching in public schools. I threw my gradebook away in 2006. For those who are interested in learning more about what school and learning looks like without testsandgrades, you can read my chapter from my book for free here. And you can read all of my blog posts about abolishing grading here.

4 Great Board Games I play with students and family

There are lots of ways to get to know someone -- my two favourite ways are going for a walk & talk and playing a game. Board games are a wonderful way of developing a relationship with someone, and also a great way of assessing a child's skills including their creativity, collaboration, adaptability, resiliency, critical thinking, problem solving, patience, literacy and numeracy. Board games often require a broad range of knowledge, skills and abilities so they are a great way of assessing what someone knows and what they can do with what they know.

Because I have returned to teaching in a Children's Inpatient Psychiatric Assessment Unit, I find it helpful to play a game with a child so that we can start a casual, caring relationship before I help them with their anger, depression, self-harm, addictions and other problems. These kinds of problems are easier to work on after you've worked together to build a train, conquer Tokyo, cure a disease or bury treasure.

Here are four games I like to play with students and my family:

1. Ticket to Ride is a 5 player game where you score points by placing trains that connect cities. This is a wonderful game that is simple enough for those unfamiliar with games but challenging enough to keep everyone involved. The original game features cities in Canada and the United States, but there are many expansions and other stand-alone games that feature Europe, Nordic countries, India and Africa. This game could be played in about 60 minutes. My daughter Kayley learned to play this when she was 6. Watch a review of Ticket to Ride here.

2. King of New Tokyo is a 6 player dice game where your monster scores points by conquering Tokyo and other monsters. This game plays like Yahtzee where you roll and re-roll the dice to achieve special monster abilities. Players must balance short-term gains with long-term objectives. King of New York is another similar stand-alone game with a few more ways for monsters to destroy buildings and score points. Both are excellent games that can be played in about 30 to 45 minutes with children as young as 6. Watch a review of King of Tokyo here.

3. Pandemic is a cooperative game where players work together to fight 5 diseases from spreading across the world. Each player has unique special abilities that require them to collaborate in creative ways. The expansion Pandemic In the Lab makes for an even more enjoyable game as petri dishes are used to manipulate the diseases to find the cure. I spend a majority of my time as a teacher and a parent teaching children to collaborate with others, so I'm excited to see more and more cooperative board games being published. Pandemic is a slightly more complex game that requires more understanding for the rules than the other games on this list. Watch a review of Pandemic here and here.

4. Black Fleet is a four player game where players use their merchant ship to deliver goods around the Caribbean while using their pirate ship to steal and bury treasure to earn doubloons to purchase special abilities and win the game. This is a casual child friendly pirate game that can be played in 40 to 60 minutes. You can watch a review of Black Fleet here.

I'm in Sir Ken Robinson's book Creative Schools!

For the last 10 years, I have worked hard to make my classroom a better place for students to learn. My transformation required that I re-think many features of my teaching such as discipline, homework, lesson planning, accountability, standardization and assessment.

Changing school is no easy task. The last 10 years have been rigorous and vigorous, filled with set-backs and progress. 

My students' success has offered the most powerful proof that I am on to something. I have found validation and support by sharing my work through my blog, twitter and other publications.

In 2011, Alfie Kohn cited my work with rethinking assessment in his article The Case Against Grades.

Most recently, I am excited to see my work cited in Sir Ken Robinson's most recent book Creative School: The Grassroots Revolution That's Transforming Education. 

In chapter seven titled Testing, Testing, Sir Ken Robinson writes:

Some teachers have always used a range of assessment methods in class. The rise of testing has made that more difficult, but some teachers are pushing back in their own classrooms. There are challenges, but there can be enormous benefits too. For example, Joe Bower is a science and language arts teacher in Alberta, Canada, who, six years into his teaching career, decided that he could no longer abide by using grades as his primary form of assessment. 

"I have come to see grades as schools' drug of choice, and we are all addicted... Grades were originally tools used by teachers, but today teachers are tools used by grades." 

What Bower discovered was that the reliance on grading made him less effective as a teacher and had a negative effect on students. He points out that when many students are asked what they got out of a class, they'll respond with something like, "I got an A." While his school insisted that he give grades on report cards, he abolished all other grades in his classroom and delivered the report card grade only after asking his students to assess their own work and recommend the grade they should receive. The students' suggestions usually aligned with his, and there were far more cases where students would have recommended a lower grade than a higher one. The result of doing away with grading was that he eased the pressure on his students and allowed them to focus on the content of their assignments and their classwork rather than on the rubric to score them. 

"When we try to reduce something that is as magnificently messy as real learning, we always conceal far more than we ever reveal. Ultimately, grading gets assessment wrong because assessment is not a spreadsheet -- it is a conversation. I am a very active teacher who assesses students every day, but I threw out my grade book years ago. If we are to find our way and make learning, not grading, the primary focus of school, then we need to abandon our mania for reducing learning and people to numbers."

From Detesting to De-Testing

This post was featured in Cathy Rubin's The Global Search for Education: Our Top 12 Teacher Blogs.

How do you balance preparation for high stakes assessments with teaching and learning in your classroom?


In my classroom, I have replaced tests and grades with projects and performances collected in portfolios. It’s been 10 years since I used a multiple choice test to assess my students, so it’s safe to say that I do not agree with having to administer a standardized multiple choice test for the government at the end of an entire year of making learning visible via blogging.

Teachers are repeatedly told that the best way to prepare students for standardized tests is to teach the curriculum, but this is at best misleading. We know that multiple choice tests require a certain amount of test taking skills, and that students who have a better understanding for the nuances of multiple choice tests can score well without having learned what the tests claim to be measuring.

So how do I live with myself when I have an obligation to administer standardized tests that I don’t support?

In his article Fighting the Tests: A Practical Guide to Rescuing Our Schools, Alfie Kohn writes:

Whenever something in the schools is amiss, it makes sense for us to work on two tracks at once. We must do our best in the short term to protect students from the worst effects of a given policy, but we must also work to change or eliminate that policy. If we overlook the former – the need to minimize the harm of what is currently taking place, to devise effective coping strategies — then we do a disservice to children in the here and now. But (and this is by far the more common error) if we overlook the latter – the need to alter the current reality — then we are condemning our children’s children to having to make the best of the same unacceptable situation because it will still exist.

In the short term, I teach the curriculum the best I can, and I waste as little time as possible preparing students to fill in bubbles. However, as test day approaches we do a practice test in small groups to reduce anxiety and increase familiarity. The best teachers act less like conduits for the tests and more like a buffer that protects students from the harmful effects of testing, so I also assure students and parents that I do not use the standardized test as a part of their report card.

In the long term, I tweet, blog, write articles and talk with anyone and everyone about how and why standardized tests are broken and how and why the alternatives to the tests are far more authentic. I go out of my way to make the alternatives to standardized tests so obviously better that parents and students see the tests as an unfortunate distraction from real learning.

To advocate for authentic alternatives to standardized tests I actively work with my Alberta Teachers’ Association to create local public events with speakers such as Sir Ken Robinson, Alfie Kohn, Pasi Sahlberg, Yong Zhao and Andy Hargreaves. I’ve joined a political party in Alberta and influenced their education policies. I wrote Telling Time with a Broken Clock: the trouble with standardized testing and co-edited De-Testing and De-Grading Schools: Authentic Alternatives to Standardization and Accountability.

Together parents, students and teachers join together to opt-out of testing.

The problem with tests that are not standardized

This was written by Alfie Kohn who writes and speaks on parenting and education. Kohn tweets here and his website is here. This post was originally found here.

by Alfie Kohn

I’m baffled by the number of educators who are adamantly opposed to standardized testing yet raise no objection to other practices that share important features with such testing.

For starters, consider those lists of specific, prescriptive curriculum standards to which the tests are yoked. Here we find the same top-down control and one-size-fits-all mentality that animate standardized testing. Yet from the early days of the “accountability” movement right down to current efforts to impose the Gates-funded Common Core from coast to coast, an awful lot of people give the standards (and the whole idea of uniform standards) a pass while frowning only at the exams used to enforce them.[1]

Example #2: Elaborate rubrics used to judge students’ performance represent another form of standardized assessment that’s rarely recognized as such. The point is to break down something, such as a piece of writing, into its parts so that teachers, and sometimes the students themselves, can rate each of them, the premise being that it’s both possible and desirable for all readers to arrive at the same number for each criterion. Rubrics are borne of a demand to quantify and an impulse to simplify. One result, argues Maja Wilson, is that “the standardization of the rubric produces standardized writers.”[2] But, again, even many teachers who are outraged by standardized tests don’t blink when standardization is smuggled in through the back door. Some insist, against all evidence to the contrary, that there’s no problem as long as one uses a good rubric.

It’s my third example, though, on which I’d like to linger. When teachers test their students, the details of those tests will differ from one classroom to the next, which means these assessments by definition are not standardized and can’t be used to compare students across schools or states. But they’re still tests, and as a result they’re still limited and limiting.

As with rubrics (and grades), there’s a reflexive tendency to insist that we just need better tests, or that we ought to just modify the way they’re administered (for example, by allowing students to retake them). And, yes, it’s certainly true that some are worse than others. Multiple-choice tests are uniquely flawed as assessments for exactly the same reason that multiple-choice standardized tests are: They’re meant to trick students who understand the concepts into picking the wrong answer, and they don’t allow kids to generate, or even explain, their responses. Multiple-choice exams can be clever but, as test designer Roger Farr of Indiana University ultimately concluded, there is no way “to build a multiple choice question that allows students to show what they can do with what they know.”

We can also concede that some reasons for giving tests are more problematic than others. There’s a difference between using them to figure out who needs help — or, for more thoughtful teachers, what aspects of their own instruction may have been ineffective — and using them to compel students to pay attention and complete their assignments. In the latter case, a test is employed to pressure kids to do what they have little interest in doing. Rather than address possible deficiencies in one’s curriculum or pedagogy (say, the exclusion of students from any role in making decisions about what they’ll learn), one need only sound a warning about an upcoming test — or, in an even more blatant exercise of power, surprise students with a pop quiz — to elicit compliance.

Even allowing for variation in the design of the tests and the motives of the testers, however, the bottom line is that these instruments are typically more about measuring the number of facts that have been crammed into students’ short-term memories than they are about assessing understanding.[3] Tests, including those that involve essays, are part of a traditional model of instruction in which information is transmitted tostudents (by means of lectures and textbooks) so that it can be disgorged later on command. That’s why it’s so disconcerting to find teachers who are proud of their student-centered approach to instruction, who embrace active and interactive forms of learning, yet continue to rely on tests as the primary, or even sole, form of assessment in their classrooms.

While some of their questions may require problem-solving skills, tests, per se, are artificial pencil-and-paper exercises that measure how much students remember and how good they are at the discrete skill of taking tests. That’s how it’s possible for a student to be a talented thinker and yet score poorly. Most teachers can, without hesitation, name several such students in their classes when the exams are designed by Pearson or ETS, but may fail to see that the same thing applies in the case of performance on tests they design themselves.

Not only do tests assess the intellectual proficiencies that matter least, however — they also have the potential to alter students’ goals and the way they approach learning. The more you’re led to focus on what you’re going to have to know for a test, the less likely you are to plunge into a story or engage fully with the design of a project or experiment. And intellectual immersion can be all but smothered if those tests are given, or even talked about, frequently. Learning in order to pass a test is qualitatively different from learning for its own sake.[4]

***

Many years ago, the eminent University of Chicago educator Philip Jackson interviewed 50 teachers who had been identified as exceptional at their craft. Among his findings was a consistent lack of emphasis on testing, if not a deliberate decision to minimize the practice, on the part of these teachers.[5]

The first reason for this, I think, is that exemplary educators understand that tests are not a particularly useful form of assessment. Second, though, these teachers learned at some point that they didn’t need tests. The most impressive classrooms and curricula are designed to help the teacher know as much as possible about how students are making sense of things. When kids are engaged in meaningful, active learning — for example, designing extended, interdisciplinary projects — teachers who watch and listen as those projects are being planned and carried out have access to, and actively interpret, a continuous stream of information about what each student is able to do and where he or she requires help. It would be superfluous to give students a test after the learning is done. We might even say that the more a teacher is inclined to use a test to gauge student progress, the more that tells us something is wrong — perhaps with the extent of the teacher’s informal and informed observation, perhaps with the quality of the tasks, perhaps with the whole model of learning. If, for example, the teacher favors direct instruction, he or she probably won’t have much idea what’s going on in the students’ minds. That will lead naturally to the conclusion that a test is “necessary” to gauge how they’re doing.[6]

Assessment literally means to sit beside, and that’s just what our most thoughtful educators urge us to do. Yetta Goodman coined the compound noun “kidwatching” to describe reading with each child to gauge his or her proficiency. Marilyn Burns insists that one-on-one conversations tell us far more about students’ mathematical understanding than a test ever could — since all wrong answers aren’t alike. Of course this assumes that we’re really interested in kids’ understanding, not merely their level of phonemic awareness or ability to apply an algorithm. The less ambitious one’s educational goals, the more likely that a test will suffice — and that the wordstesting and assessing will be used interchangeably.

One can fill a bookshelf with accounts of other forms of authentic assessment: portfolios, culminating projects, performance assessments, and what the late Ted Sizer called “exhibitions of mastery”: opportunities for students to demonstrate their proficiency not by recalling facts on demand but by doing something: constructing and conducting (and explaining the results of) an experiment, creating a restaurant menu in a foreign language, turning a story into a play. In other words, when some form of evaluation is desired after, rather than during, the learning, tests stillaren’t necessary or even particularly helpful. They needn’t be used for “summative,” let alone for “formative,” assessment.

Many of us rail against standardized tests not only because of the harmful uses to which they’re put but because they’re imposed on us. It’s more unsettling to acknowledge that the tests we come up with ourselves can also be damaging. The good news is that far superior alternatives are available.

NOTES

1. See my essay “Beware of the Standards, Not Just the Tests,”Education Week, September 26, 2001. This phenomenon is even more pronounced in Canada. Its education system is completely decentralized; each province controls its own policies. Despite the considerable variation in the amount of testing from one to the next, however, all of the provinces have very specific grade-by-grade curricula that every teacher is expected to teach. Objections to this level of control, with the concomitant diminution of autonomy for teachers, are rarely heard — even in provinces where there is outspoken resistance to testing.

2. Maja Wilson, Rethinking Rubrics in Writing Assessment(Heinemann, 2006), p. 39.

3. A spate of recent studies that attracted considerable attention in the popular press argues that frequent tests (including self-tests) are more effective than other forms of studying. But the outcome measure in these studies is almost always limited to the number of facts that are correctly recalled on later tests. Rather than offering an argument in favor of conventional assessment, these experiments actually illuminate how words like “learning” and “achievement” — as used by researchers and journalists alike — often mean little more than the successful, and presumably temporary, process of memorizing facts. For a close look at one such study, see this essay.

4. I recently made this point — about how the anticipation of being tested can distract students from engaging with ideas — in a Twitter post that was retweeted more than 400 times. This degree of popularity led me to suspect I had been misunderstood. I followed up with a clarification that all tests have this effect, not just standardized tests. The retweet rate dropped off by 90 percent.


5. Philip W. Jackson, Life in Classrooms (Teachers College Press, 1968/1990).

6. Frank Smith once wrote, “A teacher who cannot tell without a test whether a student is learning should not be in the classroom.” I see what he means, but his formulation strikes me as a bit harsh. Teachers need help to learn how to assess without tests, and they need support and encouragement to eliminate a practice that is still used by most of their colleagues and widely expected by administrators, parents, and the students themselves. Moreover, the barrier to gauging how successfully students are learning often lies not with the teacher but with features of the school structure, such as classes that are too large or periods that are too short. That’s an argument for organizing to change these problematic policies, not for continuing to test.

Teacher reflects on Alberta's new Student Learner Assessments

This was written by Catherine Dohn who is a grade 3 teacher in Edmonton, Alberta. Catherine tweets here and blogs here. This post was originally found here.

by Catherine Dohn

For those of you not in Alberta, you might not have heard about the new government Student Learning Assessments otherwise known as SLA’s. This is the pilot year where they rolled them out for all grade 3 students in Alberta and being that I moved to grade three this year, I was able to experience them firsthand.

Now Alberta Education went into these SLA’s with some pretty lofty goals. They were going to replace the dreaded Provincial Achievement Exams of old with something new and better. They would…..”better enabling parents and teachers to be aware of a child’s strengths or areas needing improvement. The SLAs are essentially “readiness” assessments that can be used to determine the programming needs for students for the school year, and support more personalized learning.” (Alberta Education Information Bulletin)

However, after being immersed in them for these last 2 weeks, I am now going to share what I have learned.

1. Awareness of Child’s Strengths/Areas of Improvement: well, being that I spend 5 days a week and almost 8 hours a day with my gang, I already had a pretty good idea of where they were at, both with their strengths and weaknesses. Funny enough, the first few days, I had a few tell me point blank “I am not really that good at reading, Mrs. D.” or “I don’t really like writing, it’s hard for me.” So those that were already walking in the door with challenges, both they and I knew this. The thing is that they and I spent quite a bit of time those first few weeks building up their confidence in these areas – finding strategies to help them tackle the areas they found hard. I worked hard helping them realize that while it might not be something they were not that great at, I constantly reinforced the idea that they might not be good “yet.” If they and I worked together, maybe we might find ways to help them get better so they could change that statement of “I am not good at reading” to “I am starting to find strategies to be a better reader.”

So then these SLA’s came with 4 different pieces of assessment that involved a lot of reading and a lot of writing and I saw those students who had started to see themselves in a different light, start to shut down. The multitude of questions, with these expectations of higher level thinking was something my struggling students found challenging and I could not help them. I had to say things like “try your best” and “it’s okay, I know you are trying hard” So now I am back to square one with some of them and I will have to try to again build up their confidence in themselves after feeling like failures. Some of my shy, quiet students who would get quite anxious about things actually wrote things like “SLA’s I am done with you.” “If I have to write this again, I will freak out.” And yes, this would be even though I did not make a big deal out of all this, I kept saying it was just an experiment to see how we would all do and not to worry.

The other sad thing was that my more bright students, the ones I knew had a lot to offer, even they did not get the point of doing a task multiple times and then evaluating themselves on it again and again. I heard a lot of “didn’t we do this already with the last question?” and “why do I have to do this?” So yet again, these students did not really show what they are truly capable of because even they thought “what’s the point of all this?” and did what they could to just get done.

2. An Information, Reflective Piece: so the point of all these different literacy and numeracy pieces was this going to inform and reflect my students to better guide my instruction. As well, I would be able to take these out and use them as something to talk to parents about. I really look forward to these conversations in that if I show parents some of the numeracy questions with the self reflective pieces, I am sure I will spend more time explaining what the questions were actually asking and validating the point of this type of questioning. With so many parents already questioning the “new math” they don’t understand, this is probably not going to help me much.

The reflection I might have, if I was a new teacher to grade 3, is that I better prepare my students better for this type of assessment. So instead of spending time those first few important weeks building a relationship with my student, I start practicing with 7 and 8 year olds how to self assess, how to reflect because gosh knows those skills come built in at the beginning of a school year in grade three. (yes that was a bit of sarcasm, my apologies)

3. My Frustration – my biggest frustration was not the hours of extra work for myself – inputting students onto the system so they could actually do the test, figuring out the pieces and parts to administering these the best way possible, marking and going over each assessment piece. It was not how I put the rest of the school out at times – SLA’s meant no one could use the internet because doing the digital pieces that took every bit of wifi bandwidth our school had. No one could use laptops or computer labs because they were all booked up for us. Don’t get me started on the amount of trees I killed photocopying everything needed. My biggest frustration was seeing one of my students lay his head down on his desk and cry because he just couldn’t get what the question was asking, it was seeing one of my girls go “Mrs. D. I just don’t get what to do” It was seeing some of my students take close to 2 hours to get done on the computer because of glitches, freezing, having to shut down and hope it would not make them start from the beginning.

So to the powers that be, I have the following to say about what I learned… I honestly think this could have been done so much better. I am not sure why you couldn’t trust that I am a professional, that I am able to find and use assessment pieces that will truly reflect my students’ abilities. But I guess, like my students, I have areas of improvement too….

Essay-marking software gives high marks for gibberish, U.S. expert warns Alberta

This was written by Andrea Sands who is a journalist with the Edmonton Journal. Sands tweets here. This post was originally found here.

by Andrea Sands

Alberta Education should not allow high-stakes Grade 12 diploma exams to be marked by computer because today’s essay-marking software is terribly flawed, says a retired Massachusetts Institute of Technology (MIT) professor and writing-assessment expert.

Les Perelman has been an outspoken critic of automated essay marking and worked with MIT students to invent computer software called BABEL (Basic Automatic B.S. Essay Language Generator), designed to trick today’s essay-scoring programs. BABEL generates gibberish essays which, nonetheless, get high marks from so-called “robo graders.”

“I realized very early on that machine grading of writing really is, essentially, impossible,” Perelman said from Massachusetts.

“I can guarantee you, smart 18-year-olds will try to trick it. If they find out it favours certain things like word length, obscure words and pretentious language, then that is exactly what is going to be probably taught in schools.”

Last fall, Alberta Education paid U.S. company LightSide $5,000 to do a feasibility assessment on whether computers could accurately mark Grade 12 diploma-exam essay questions. Every year, about 190,000 Alberta students write the exams, worth 50 per cent of a student’s final mark.

The LightSide report concluded its software could reliably grade student essays and, in the test analysis, did so more accurately than teachers.

Perelman said the report is “fudging” because of the way data was analyzed and because there’s not enough information and explanation in the report. “I’m a pro at this and there were tables I could not understand.”

Computers can’t analyze meaning and essay-grading software usually inflates grades for essays that are longer and use uncommon words — “egregious” instead of “bad,” or “plethora” instead of many, Perelman said.

“They’re counting word length, or they’re counting types of words, or they’re counting sentence length, or they’re counting connective words like ‘consequently,’ ‘moreover,’ things like that, that show cohesion. But all those things are very mechanical.”

About nine big companies sell essay-marking programs, including Pearson, ETS, CTB/McGraw-Hill, Measurement Inc. and LightSide, which is a fairly new company.

They are lining up to win “the big prize,” said Perelman — contracts to machine-score the increasing number of tests that will result from the new Common Core State Standards Initiative. Almost all U.S. states are now implementing new standards for kindergarten through Grade 12 in English language arts/literacy and mathematics that will require annual testing.

According to LightSide’s website, the Common Core “will require more writing in every classroom, meaning more time grading and even less time for teachers to work directly with students.”

Perelman does, however, credit LightSide as being one of the more reputable companies. Company founder Elijah Mayfield, who led the Alberta study, has responded to Perelman’s criticisms by trying to improve LightSide’s products and acknowledges the limitations of computerized grading, Perelman said.

“He’s an engineer. Most of the other companies are salesmen. I still think he’s wrong.”

Mayfield, who headed up the Alberta feasibility assessment, said a confidentiality agreement prohibits him from speaking to the Journal about the report.

The U.S. National Council of Teachers of English argues the Common Core initiative is pushing companies, testing agencies and education organizations to use automated essay grading because it’s cheaper than paying people to mark tests.


Alberta Teachers’ Association president Mark Ramsankar said it’s unfair to have students spend hours writing an exam that’s marked by a machine.

“How does a machine look at the symbolism contained in a piece of writing and interpret that as symbolism?” Ramsankar said. “I’d like to see how Shakespeare stacks up in a computer-generated mark.”

Both the ATA and the Canadian Teachers’ Federation oppose machine marking of essays, particularly on diploma examinations that are the culmination of a year’s worth of students’ learning.

“Again, we’re seeing practices that we’re looking at importing from the United States,” said federation president Dianne Woloschuk. “Their education system is in a crisis. Their students are not doing well.”

Alberta Education continues to evaluate whether machine-scoring of essays could be useful here. However, Premier Dave Hancock and Education Minister Jeff Johnson have said there are no plans to pursue it at this time.

asands@edmontonjournal.com

Twitter.com/Ansands

‘Mankind will always conduct prejudice’


English 30-1 students in Alberta study the novel Pride and Prejudice, by Jane Austen, so retired MIT professor Les Perelman entered the key words “pride, prejudice” and “father” into his essay generator, BABEL. The program creates mechanically correct but “completely incoherent” essays that have fooled automated-marking programs.

Perelman’s sample was declared “off-topic,” but scored 88 per cent when Perelman ran it through the home-schooling version of IntelliMetric, under the software’s category “challenges of parenthood.”

IntelliMetric technology is used in the United States to score the Graduate Management Admission Test (GMAT), which graduate students take to get into management programs in business schools.

Here is the text from the BABEL-generated essay:

Keywords:

pride: [‘pridefulness’, u’pride’]

prejudice: [‘preconception’, ‘bias’, u’prejudice’]

father: [‘begetter’, u’father’, ‘male parent’]

Essay:

Pridefulness with decency has not, and in all likelihood never will be malevolent, humane, and considerate. Mankind will always conduct prejudice; many for an advance but a few on pulchritude. a quantity of pride lies in the study of reality as well as the area of semantics. Why is pride so efficacious to depreciation? The reply to this query is that male parent is rivetingly and gregariously febrile.

Rationale, usually by appreciation, might feign prejudice. If nearly all of the appendages adjure an explanation of the erratically or idolatrously pagan disparagement, the haphazard preconception can be more falteringly sublimated. Additionally, an orbital is not the only thing simulation reacts; it also spins at male parent. Our personal altruist on the exposure we arrange can surprisingly be an interloper. Be that as it may, knowing that executioner can be the assassination, most of the comments to my accusation civilize irrelevant scrutinizations. In my philosophy class, all of the advancements by our personal allusion of the demarcation we decry accede amplifications which deliberate with analyses but masticate veracity that should inconsistently be a contradiction and occlude circumspections for expositions. Begetter which is mimicking in how much we cavort ousts whiner of our personal inquiry to the apprentice we propagate as well. an accession will enthusiastically be a concurrence on the insinuation, not an intercession. In my experience, none of the reprimands by our personal advocate at the sophist we substantiate contemplate postulation that blusters but append. a abundance of father changes culmination for bias.


As I have learned in my literature class, humanity will always depreciate father. Even though the brain counteracts a gamma ray to contentment, the same pendulum may catalyze two different neutrinoes with the remarkably accumulated culpability. Although the same neuron may receive two different brains, radiation processes orbitals of disenfranchisements on a taunt. The plasma is not the only thing a gamma ray oscillates; it also transmits neutrinoes for abandonment at the trope by father. The diagnosis of begetter changes a plethora of preconception. The less eventual allocutions pledge thermostats, the more an organism inaugurates those in question.

Malcontent, normally on the assumption, demolishes father. As a result of scintillating, all of the adjurations hobble equally with prejudice. Also, male parent to speculations will always be an experience of humankind. In my theory of knowledge class, some of the juggernauts of my scenario assimilate probes by the search for semiotics. Still yet, armed with the knowledge that propaganda can be a demonstration or homogenizes, many of the lamentations for my dictum abandon periodicity and voyage. In my philosophy class, almost all of the domains at our personal denouncement by the comment we admonish explain demolishers which declare the demolisher with the quip on gluttony that enthrals speculations or disparage agriculturalists. Pride which utters substantiation may boastfully be propinquity or is avowed but not impartial of my advancement also. a fetishistic fulmination belittles the people involved, not assemblage. Our personal conveyance to the reprobate we implore should be the analysis. The tendentiously vast prejudice changes a quantity of pridefulness.

Bias has not, and undoubtedly never will be reticent yet somehow gluttonous. However, armed with the knowledge that a report with assemblies accounts, all of the tyroes for my amplification shriek. By the fact that gratuitous dictators are articulated at pride, most of the amplifications confide too by pride. Prejudice will always be a part of human society. Pridefulness is the most feckless proclamation of human life.

Algorithms marking essays? That diploma idea deserves a failing grade

This was written by Paula Simons who is a columnist with the Edmonton Journal. Simons tweets here. This post was originally found here.

by Paula Simons

The word “essay” comes from a Latin root, meaning to put something on trial, to put something to the test. That’s why we ask students sitting Grade 12 diploma exams in Language Arts and Social Studies to write timed essays as part of their test. Composing an essay doesn’t just test your ability to use correct English. It tests your ability to think critically. It tests your ability to make an argument, supported with facts. It tests your ability to critique conventional wisdom and articulate original insights.

Learning to write a cogent essay in an hour isn’t just excellent training for would-be newspaper columnists. Not every graduate will need to do trigonometry or balance a chemical equation in adult life. But we all need to know how to marshal facts to advance a convincing argument, whether we’re fighting a traffic ticket, negotiating a raise, or convincing skeptical friends to try a new restaurant.

Nonetheless, Alberta Education is apparently giving serious consideration to contracting out the marking of diploma essays to an American computer program that uses complex algorithms to predict and assign student grades. A number of American states are already using such programs to assess “high stakes” essay tests.

In January, LightSide, a company founded by graduate students from Carnegie Mellon University in Pittsburgh, presented a research report to Alberta Education. The confidential report, obtained by the Journal’s Andrea Sands, claims its software was 20 per cent more reliable than Alberta’s human markers.

LightSide’s system doesn’t detect grammatical errors. It has no capacity to fact-check, nor to analyze critical thinking.

“We cannot evaluate whether the points made in a series of claims lead naturally from one to another,” reads the study, written by LightSide CEO Elijah Mayfield. As well, says Mayfield, “on-topic responses that fail to address subtle factual nuances, misinterpret a particular relationship between ideas, or other factual errors will likely be scored highly if they are otherwise well-written.”

How does the computer know if something is well-written? It compares the essay it's evaluating to the hundreds of “training samples” in its memory.

“Computers can’t read a student’s essay, but they’re excellent at making lists — compiling, for a given essay, all of the words, phrases, parts of speech, and other features that characterize a student’s work,” says LightSide’s website. “Our software compares the differences in the features of a weak essay and a strong essay — as evaluated by a human reader. It then identifies the small things that might only appear in the strongest essays — vocabulary keywords, structural patterns of sentences, use of transition sentences ... If the writer has all of the little things right, just like the previously high-scoring example essays, they probably should receive the same score.”

In other words, if the essay’s style, vocabulary and syntax pattern match those of sample essays that earned high grades from human markers, the software will award similar grades, even if an essay is ungrammatical, illogical, or full of factual errors.

It’s quite fascinating, linguistically speaking. Certainly, we shouldn’t romanticize human markers, who bring their own biases, incompetencies, and idiosyncrasies to the grading process. A tired or frustrated or overwhelmed marker may not give consistent scores. Over thousands and thousands of student essays, machine marking may indeed offer statistical superiority.

But writing for a human marker, however flawed, is different than writing for a soulless algorithm. Writing is an intimate act of communication. When you write an essay, you write for an audience. You write to be understood, to entertain, to provoke, to connect, to share your insights and passions. Young essayists deserve the dignity of writing for sentient readers. Even if we accept the premise that machine marking yields more “accurate” results, it robs students of the relationship between writer and reader, of the right to be heard and understood.

How serious is Alberta Education about this? It depends on whom you ask — and when. Premier Dave Hancock praised the notion in the legislature last month, lambasted it at an Edmonton Journal editorial board Wednesday, then gave it guarded approval in an interview Thursday. Given the current Sturm und Drang in the Education portfolio it would seem madness for Hancock and beleaguered Education Minister Jeff Johnston to start another fight with Alberta teachers right now. But who knows what a new premier (or new education minister) might decide next fall?

If diploma exams need more consistent scoring, perhaps Alberta Education should invest in better marker recruitment, training and compensation. If we expect Alberta students to take diploma exams seriously, we should all do the same.

Computers 'dramatically more reliable' than teachers in marking Alberta diploma-exam essays: study

This was written by Andrea Sands who is a journalist with the Edmonton Journal. Sands tweets here. This post was originally found here.

by Andrea Sands

Phil McRae with the ATA.
Photo by John Lucas, Edmonton Journal

A computer could do a better job than a teacher in marking Grade 12 diploma exam essays, a government-commissioned study says.

Last fall, Alberta Education sent two 2013 diploma-exam questions along with nearly 1,900 student essay answers that had been graded by teachers to LightSide, a Pennsylvania company that develops computer software to score student essays.

LightSide’s automated algorithms outperformed human reliability in the Alberta study by about 20 per cent, said the company’s January 2014 report to the government.

“We are certain that LightSide is able to reproduce scoring behaviour at least as reliably as human graders, and in many cases we believe that our automated performance would be dramatically more reliable than human grading,” the report said.

The study indicated Alberta Education’s human scoring was quite unreliable, below the threshold LightSide recommends for high-stakes testing.

Alberta should consider investing in “a more stringent training process for human graders,” the report said. “It is somewhat alarming to see human reliability so low.”

The $5,000 study suggests LightSide’s marking program is more reliable than a single human marker, but Alberta uses a double-marking system, said Neil Fenske, Alberta Education’s executive director for assessment.

At least two teachers grade each diploma-exam essay and, if the grades differ, it goes to a third marker.

“So we’ve built a system that is highly reliable ... but it’s very labour-intensive as well,” Fenske said.

Alberta Education commissioned two previous studies — one three years ago and one 15 years ago — to see if computerized essay marking could work, Fenske said.

Further study is needed because LightSide examined a very small sample, Fenske said.

However, the report does show automated technology has evolved enough that it could be useful, if Alberta combined the marking power of people with the speed and reliability of machines, he said. That could mean one person marks an essay, then it’s run through a computer for grading, and sent to another person if there’s a discrepancy.

The province will also soon need diploma exams marked more often than before. Last year, the department announced Grade 12 diploma exams will be offered more often and digitally, part of efforts under Inspiring Education to make the school system more flexible.

“Because marking means teachers out of the classroom, one of the things we have to take a look at is, is there a way that we can build a better marking system that’s better for students but maybe also keeps more teachers in the classrooms?” Fenske said.

Alberta Education has had trouble this year recruiting enough teachers to grade diploma exams, which are worth 50 per cent of a student’s final mark.

Around the same time the LightSide report was commissioned, Education Minister Jeff Johnson cut a grading honorarium in half — from $200 to $100 — for teachers who volunteer to mark diploma exams on a regular workday. Fewer teachers volunteered to do the marking this year, and it’s taking longer as a result.

Johnson and Premier Dave Hancock said this week the honorarium cut should be re-examined.

Asked about the LightSide report at a Journal editorial board meeting this week, Hancock said he hasn’t yet read the study, but is not interested in having machines grade diploma-exam essays.

“I think that’s an absolute disastrous way to go,” Hancock said. “There are things that teachers bring to the process that are very important.”

Teachers also benefit from professional development when they mark the exams, meeting with colleagues from across the province and discussing education standards, Hancock said.

“At this time, there are no plans to institute digital scoring systems in provincially graded essays like diploma exams,” Johnson said in a statement.

Last weekend, Alberta Teachers’ Association delegates voted unanimously at their annual meeting in Calgary to opposed machine scoring of essay questions.

The ATA was never told about the LightSide study, but machine-marking has sparked heated debate in the United States, said Phil McRae, ATA executive staff officer and adjunct education professor with the U of A.

Standardized testing in the U.S. is growing, prompting governments and school districts to look to computer scoring to keep grading costs down, said McRae, who researches technology in education.

“It’s about reducing costs, whether it’s development of the items for the tests, administration or scoring.”

Testsandgrades and teachers

If a child learns but isn't tested, does anyone care?

Why some schools are giving letter grades a fail

This was written by Erin Millar who is a journalist and author with a lifelong interest in education, innovation and creativity. For nearly a decade she has written for leading Canadian and international publications including Reader’s Digest International, Maclean’s, The Globe and Mail, The Walrus, The Times of London and others. This piece first appeared here.

by Erin Millar

Krista Wolfram credits an innovative new assessment program with helping catch her daughter’s difficulties in school early. When Antonio Vendramin, principal of Georges Vanier Elementary school in Surrey, B.C., announced to parents in November that the school would no longer award letter grades, Ms. Wolfram was skeptical. “Some of us were scared of change,” Ms. Wolfram, whose daughter is in Grade 2, recalls. “I grew up with As, Bs and Cs.”

Instead of reporting to parents only two or three times a year, teachers began regularly communicating using an online student portfolio system called Fresh Grade. Ms. Wolfram quickly discovered that her daughter was having difficulty with her writing. “Every day her teacher would snap a photo of her journal or a video of her writing with her phone or iPad,” Ms. Wolfram explains. “I could see exactly where she was struggling, and I could work with my daughter and her teacher to help.”

Because Ms. Wolfram was able to intervene early, her daughter was writing at her grade level by the time she received her report card just before spring break. “If we had to wait until her report card to find out, she would have failed writing,” she says. “I wouldn’t have known she was struggling.”

The Surrey Board of Education pioneered a pilot program eliminating letter grades in several elementary schools in September and now more than 40 classes at 13 elementary and six secondary schools have joined the experiment. Nine more schools are set to join soon, and the results will be reviewed this summer.

The Surrey school district is not alone. Schools around the world are experimenting with new ways to assess student achievement that do not rely solely on high-stakes reports that use numerical marks and letter grades. The movement is in part a response to calls from employers for the school system to emphasize skills such as creativity and communication, not just knowledge of traditional subjects. But even as recent research suggests that descriptive feedback better supports students in developing these soft skills than traditional grades, parents, educators, and higher education institutions are struggling to adapt. If there are no grades, how we will know whether students are prepared for jobs or further study?

The idea that traditional measures of academic achievement don’t support learning isn’t new; in 1998, the British education researchers Paul Black and Dylan Wiliam published a widely-cited article demonstrating how increasing descriptive feedback raises student academic achievement. More recently, the Nobel Prize-winning economist James Heckman has published numerous studies showing how grades and standardized tests don’t predict later life success such as employment and income level.

In response, school boards in Canada are implementing new approaches to assessment. Elementary students in Ontario no longer receive letter grades on their fall report card. Quebec has reduced the number of formal reports to parents, opting for more descriptive feedback.

In June, the Calgary Board of Education announced a plan to eliminate letter grades up to Grade 9, a measure that was hotly debated. The new report cards would have used four phrases − exemplary, evident, emerging or support required − and not included written comments from teachers. Many parents were concerned about the proposal. Cathy Ward, a spokeswoman for the Calgary Board of Education, says that while some schools have begun transitioning to new ways of assessment, the board is reviewing the plan and no firm decisions will be made until they consult further with parents and teachers.

At Fraser Heights Secondary School in Surrey, English teacher Leah Samson no longer uses numerical marks to give students feedback on assignments. She devotes considerable class time to teaching students how to assess their own learning and give effective peer feedback. She also has regular one-on-one meetings with students to discuss their goals and progress. Instead of Ms. Samson telling students how they are doing, they are expected to articulate to her what they learned in class, how it relates to their learning goals and where they’re struggling.

Ms. Samson is still required to give students a letter grade at the end of each semester, but the letters have taken on a whole new meaning to her and her students. Struggling students have a much clearer idea of how they can move from a C to a B or an A. “Once grades are removed, students are learning for themselves rather than learning for their teacher.”

The move away from grades matches a growing belief among employers that traditional assessment is not the best way to help students develop the skills they need to succeed in today’s world. In national and global surveys, employers don’t complain about applicants lacking specific knowledge or technical skills, which are easy to test and express in a letter grade; they want employees who can analyze critically, collaborate, communicate, solve problems and think creatively. A 2012 McKinsey & Co. survey of 8,000 students, educators and employers in nine countries found that there was a gap between what educators thought students needed to succeed and what employers really wanted. “Education providers will say that all skills are important, whereas employers will place much clearer prioritization on soft skills – where the likes of team work and work ethic come out quite strongly,” Mona Mourshed, director of education at McKinsey, said.

Traditional assessment largely focuses on measuring students’ ability to regurgitate information and was designed for certification and accountability purposes rather than to support student learning, argues Maria Langworthy, chief research officer with Ontario education expert Michael Fullan’s research group. “Think about working in a knowledge-based economy where the sort of end products we produce are intellectual, things like software, design, social policy,” she explains “There’s no multiple-choice test that can capture the value and complexity of those sorts of products.”

David Staples, the Wildrose and their war on teachers and learning

Here is Bruce McAllister and David Staples
 talking with Alberta teachers.

David Staples is a columnist who has an interest in education.

Bruce McAllister is a Wildrose MLA and education critic in the Alberta Legislature.

Together, they are waging war on teachers and learning by demanding that teachers teach in a way that mandates children play a passive role in school. Together, they argue there simply is not enough memorization and tests in school.

Standardized Testing

When the Wildrose and David Staples cite a real world need for annual standardized testing, I ask some questions:

1. As a columnist, can you share the standardized multiple choice test that the Edmonton Journal makes you do to keep you accountable and transparent? As a politician, would you be willing to take Alberta's Diplomas exams and have your results published for all to see?

2. As a columnist, can you share the standardized rubric that the Edmonton Journal uses to score and judge your columns? As a politician can you share the scoring guide that citizens use to score and judge your work?

4. As a columnist or a politician, can you show me the column you wrote or the bill you voted on where you are not allowed access to the Internet, fact-check or talk to anyone? 

5. As a columnist or a politician, if there were no standardized test scores, what would you know about education?

We need to stop thinking we can meet all
children's needs by pretending all children
have the same needs.

It is hypocritical for adults to demand students and teachers be held accountable in ways that they would not hold themselves to.

Standardized testing is what constitutes an amazingly contrived and unrealistic form of assessment that is used by people outside the classroom to judge and control what happens inside the classroom without ever visiting the schools.

Teachers are not afraid of accountability -- but they do oppose being held accountable for things out of their control. Teachers also know that there is nothing transparent about having children fill in bubble-tests.

The best feedback parents can receive about their children's learning is to see their children learning. The best teachers don't need tests because they make learning visible via projects and performances collected in portfolios.

This is a shift from test and punish accountability to more authentic public assurance. The Alberta Teachers' Association also outlines a vision for A Great School for All, and the Alberta Assessment Consortium offers A New Look at Public Assurance.

And here's my story about how I teach and my students learn without grades.

"Old" and "New" Math

Staples continued his war on learning with a column that featured Ken Porteous who is a retired chemical engineering professor from the University of Alberta. Porteous writes: 

The discovery approach has no place in arithmetic at the junior elementary level. There is nothing to discover.

If there was ever a need for a single statement that one could show people such that their response would predict whether they knew anything about how children learn -- this is it. 

To carry this mindset out to its (il)logical conclusion, I guess there is nothing left to discover in this world...

Teachers and other early childhood development experts who understand how children learn define their careers by children's Aha! moments. These are the moments when metaphorical lightbulbs illuminate on top of children's heads. Anyone with a clue about how children learn knows that these Aha! moments rarely, if ever, happen because kids were simply told to have them. Aha! moments are not passively absorbed or memorized -- they are actively constructed by the student with the artful guidance of a teacher.

The best teachers have teeth marks on their tongues because they know that when kids are simply told the most efficient way of getting the answer, they get in the habit of looking to adults instead of thinking things through for themselves. They understand that learning happens when the child is ready to learn, not necessarily when someone is ready to teach -- teachers call these teachable moments.

I am a huge supporter of teacher professional development where teachers learn how to be better teachers, but let's not delude ourselves into thinking that a back to basics approach that romanticizes the past will make things better for our children.

Let's not pretend that traditional math instruction didn't confuse and turn a lot of students off of math. When adults think back on their schooling, it's easy to succumb to something called Nostesia which is a hallucinogenic mixture of 50% nostalgia and 50% amnesia which distorts rational thinking.

Wishing tomorrow to be just like yesterday won't make today a better place. We aren't going to get more children to love math by pretending that school already doesn't have enough lectures, direct instruction, worksheets, textbooks, tests and memorization.

Staples and the Wildrose would like Albertans to believe that they are waging war against the government and education consultants but the truth is they are also attacking teachers who work hard to engage students in a way that has them play a more active role in constructing their own understanding with the artful guidance of their teacher.

While some teachers and parents may agree with Staples and the Wildrose, it's important to note that many teachers in Alberta feel that they are doing more harm than good. When Staples and the Wildrose mislead the public by telling teachers how they have to teach, they make it harder for great teachers to do their job.

Here's my take on the math wars, and Alfie Kohn's article answers the question: What works better than traditional math instruction?

Columnists are not Journalists and (most) Politicians are not teachers

Staples is a columnist -- which is not the same as a journalist, and I fear that too many people don't understand the difference.

He is not required to check his biases or opinions at the door -- in fact, as a columnist,  he has a better chance of selling newspapers and collecting page-views online with his biases and opinions fully intact. Staples is biased because that is his job.

Research isn't sexy and it doesn't sell unless it's accompanied by sensationalism, and when it comes to sensationalism, Staples sells the Wildrose. Making claims that teachers are no longer teaching children basic arithmetic may make for a snappy headline and a wedge issue to gain cheap political points for the next election but it couldn't be further from the truth.

As a side note, when I tried to share my math post with Bruce McAllister on his Facebook page, he deleted it and blocked me. You'd think that the opposition party would have a keen sense of appreciation for opposition, but I guess not.

"I wish a columnist and politician with no teaching experience would just
 come in and tell me how to teach," said no teacher ever.

And yet Staples isn't always wrong -- he knows just enough about education to get in trouble. His columns are filled with half-truths that are supported by cherry picked research, revisionist history and preconceived notions. He props up math PhDs, engineers, testing consultants, bureaucrats and others who have expertise in areas other than teaching young children math.

Canadians love their Olympians, but nobody confuses a hockey players' expertise for a rhythmic gymnastics coach. Similarly, a PhD in mathematics or engineering is not a PhD in early childhood development, psychology or math education.

Mathematicians are not (necessarily) Math Teachers

The best math teachers understand math and how children learn math -- these are two different skills. It is irresponsible to simply assume that someone who is good at math knows anything about how to teach it.
Just because you know how to skate or shoot a puck doesn't mean you have a clue how to properly teach young children how to skate or shoot. If you want to coach organized hockey in Canada, you are required to be educated through a certification process. One expectation is for coaches to learn the content of hockey, and another expectation is to learn how to teach children to skate and shoot.

The teaching part is so important that even if you played hockey at a high level, you would still be required to take the certification program. Knowing how to play hockey or how to do math is necessary but not sufficient for coaching or teaching -- this is why we have coaching and teaching certification programs.

Getting advice on how to teach or play hockey from someone who has never taught or played hockey is kind of like getting advice from a virgin on how to get laid. Opinion needs to be based on experience and expertise -- Staples and the Wildrose have neither.

I'm not saying that there isn't a place for columnists and politicians -- what I'm saying is that columnists and politicians need to be kept in their place, because when David Staples and the Wildrose confuse having an interest in education with being experts, they mislead people.

Success -- it's not always what you see

Four years ago I wrote a post titled What leads to success? I took an old TEDtalk and explained how testsandgrades do not lead to success.

When I came across this graphic on Twitter via Greg Miller, I have a couple questions:

Students. Have you ever worked really hard and learned a lot about something and received a low grade? Have you ever slacked off and learned almost nothing but received a high grade?

Parents. Can you think of someone you went to school with, and you knew they were really smart, but they always received low grades? Can you think of someone who received really high grades but you knew they were a dolt and that they had, at best, a superficial understanding?

Teachers. Can you think of a student who you knew to be a critical and creative thinker but often scored low on standardized tests? Can you think of a student who you knew to be quite a shallow and superficial thinker, but often scored high?

When I ask these questions to students, parents and teachers, I often get a lot of head nodding. People seem to understand the point.

Testsandgrades are broken. 

They don't tell us what we think they tell to us and they distract us from learning.

For an authentic alternative to testsandgrades, check out my chapter Reduced to Numbers: from concealing to revealing learning.

Here's my chapter from the book De-Testing and De-Grading Schools

In June of 2013 I co-edited a book with Paul Thomas called De-Testing and De-Grading Schools: Authentic Alternatives to Standardization and Accountability. 

Here's my chapter: Reduced to Numbers: From Concealing to Revealing Learning.

Feel free, to read and share widely.

Consider joining the Grading Moratorium here. Email me if you are interested: joe.bower.teacher@gmail.com

  Reduced to Numbers by Joe Bower by bowerjj

One of the best videos on education reform

Take less than 15 minutes and watch this video on education reform and the landscape of assessment in schools.

It's that good.

12 characteristics of progressive schools

This was written by Alfie Kohn who writes and speaks about parenting and education. His website is here and he tweets here. This post was originally found here.

by Alfie Kohn

To create the schools our children deserve, it’s probably not necessary to devise specific policies and practices for every occasion. Rather, these will follow logically from a few core principles that we devise together. Here’s a sample list of such principles, intended to start a conversation among educators, parents, and (let’s not forget) the students themselves:

1. Learning should be organized around problems, projects, and (students’)questions -- not around lists of facts or skills, or separate disciplines.

2. Thinking is messy; deep thinking is really messy. Therefore beware prescriptive standards and outcomes that are too specific and orderly.

3. The primary criterion for what we do in schools: How will this affect kids’interest in the topic (and their excitement about learning more generally)?

4. If students are “off task,” the problem may be with the task, not with the kids.

5. In outstanding classrooms, teachers do more listening than talking, and students do more talking than listening. Terrific teachers often have teeth marks on their tongues.

6. Children learn how to make good decisions by making decisions, not by following directions.

7. When we aren’t sure how to solve a problem relating to curriculum, pedagogy, or classroom conflict, the best response is often to ask the kids.

8. The more focused we are on kids’ “behaviors,” the more we end up missing the kids themselves -- along with the needs, motives, and reasons that underlie their actions.

9. If students are rewarded or praised for doing something (e.g., reading, solving problems, being kind), they’ll likely lose interest in whatever they had to do to get the reward.

10. The more that students are led to focus on how well they’re doing in school, the less engaged they’ll tend to be with what they’re doing in school.

11. All learning can be assessed, but the most important kinds of learning are very difficult to measure -- and the quality of that learning may diminish if we try to reduce it to numbers.

12. Standardized tests assess the proficiencies that matter least. Such tests serve mostly to make unimpressive forms of instruction appear successful.

What people think success looks like

Most people understand that success is messy.

And so is learning.

Yet, adults can be pit against one another when they try and figure out how to best teach children that learning and success are messy.

While it may seem counterintuitive to suggest that we should not evaluate and judge students while they are still learning, this is precisely what the research has been showing us.

The real challenge isn't to just agree that success is messy. 

That's too easy. 

The real challenge is to see how some of the things we have always done in school are actually working against our best intentions.

Mr. Bower, I can't do my project anymore

I sometimes think about a conversation I had with one of my grade 8 students while we were learning about World War II and The Holocaust. It went something like this:

Reagan came up to me in the middle of class and said, "Mr. Bower, I can't do my project anymore."

I was a little taken aback. Reagan had been so eager to start her research on Dr. Mengele and initiated a majority of her project. At first glance such a pronouncement from a student might easily be labelled as defiant, but I was pretty sure there was something else going on here. So I asked, "Really, Reagan? Why's that? What's up?"

"At first I thought I wanted to learn more about what happened during The Holocaust, and then I started researching Dr. Mengele in that book you suggested, The Holocaust Chronicle. But now I'm just saddened by it all. It makes me so sad to read about the awful things that these people did to others. I just don't think I want to do this anymore. I don't want to be sad."

"That's fair. I know that sometimes I have a hard time reading books about The Holocaust. Sometimes it's hard to spend a lot of time focusing on such an uncomfortable topic." 

"Yeah. Totally."

"Before you quit your project, Reagan, I have a question for you. Would it be worse if some people like you and me got a little sad from spending time learning about The Holocaust or would it be worse if we avoided being sad and just forgot about The Holocaust?"

Reagan stood there looking at me.

She didn't say a word.

I knew she was thinking.

In short order she went from looking perplexed to certain. She said, "It would be way worse if we forgot."

"Why is that?"

"Because if we forget, we might avoid being sad, but we would risk allowing it to happen again. And we can't do that." She turned and went back to her project. 

I was proud of Reagan.

***

A couple thoughts:

• Can you see how asking Reagan about what was up was more productive than just assuming she was being lazy or defiant? People like it when you seek to understand them before you seek to be understood -- and children are people, too.

• Can you see how lecturing Reagan about why we need to learn about The Holocaust would have missed the point? Can you see how asking provocative questions that inspire thought are the real work of teachers? 

• Can you see how a text-book, computer software or app can't do this? Reagan needed "just-in-time" feedback and guidance that only a real life teacher that she has a relationship with could provide.

• Can you see how this conversation with Reagan would be very difficult to quantify or symbolize on the report card and yet witnessing her new-found realization is what might matter most? The most important things that happen in school may be difficult, if not impossible, to measure but they can always be observed and described -- this is why assessment is not a spreadsheet, it's a conversation. And when we try to reduce learning to a number, we always conceal far more than we ever reveal.

A Q&A on computer grading

I've written about robo-graders here. 

Here's a 4 question Q & A where I'll ask the questions and the answers come from a must-read post titled Computer Grading Will Destroy our Schools.

1. Why is there a push to have student's multiple choice tests and writing scored by a computer?

The reason for the push is both grim and obvious: money. Now that our schools are going to have more standardized tests, there are going to be more student paragraphs that need grading. Grading written work is laborious and time-consuming and, from a school board’s point of view, expensive. What computers offer is the ability to do this task faster and – once they are up and running – more cheaply. To be fair to the school boards, assuming the computer programs work, doing this task more efficiently could yield some benefits. It would lift a significant burden from the harried, overworked and underappreciated group of teachers and grad students we currently pay to grade standardized tests. But I suspect that, for most people, the thought of a computer “reading” essays is reflexively anxiety-provoking. It brings out the inner Luddite. Are we really supposed to believe that a machine can do just as good a job as a human being at a task like reading?

2. Does computer grading eliminate teacher bias?

The way supervised machine learning basically works is this: The computer treats the student’s essay as though it were just some random assemblage of words. Indeed, the jargon term for the main analytical technique here is actually “Bag of Words” (the resonance of which is simultaneously kind of insulting and weirdly reassuring). The program then measures and counts some things about the words that the programmer thinks are likely to be correlated with good writing. For example, how long is the average word? How many words are in the average sentence? How accurate are the quotations from the source text, if any? Did the writer remember to put a punctuation mark at the end of every sentence? How long is the essay? 

The programmer then “trains” the computer by telling it the grades assigned by human graders to a “training set” of essays. The computer compares – mathematically – the various things it has measured to the grades assigned to the essays in the training set. “Johnny’s essay had an average word length of 5 letters and an average sentence length of 20 words. The human told me that Johnny gets a B+. Now I know that much about essays that get a B+.” From there, given an average word length, sentence length, word frequency and so on, the computer is able to calculate the probability that a given student essay would receive a particular grade. When it encounters a new essay, it can take the things it knows how to measure, and – based on what it learned from the grades assigned to the training essays – simply assign the most probable grade. 

It turns out that this works surprisingly well. Shockingly well. What we might think of as totally surface-level or accidental features – like having more words per sentence – are actually correlated very strongly with earning better grades. Statistical analyses, at least, tell us that machine-learning techniques perform just as well as human beings – that is, their grades for new essays are the same or similar a huge majority of the time. Plus there are some ways in which the computers might actually be better. The people who presently do essay grading for tests like the AP English exam work long hours and have to grade essay after essay. Unlike computers, they get tired and irritable and bored. Also unlike computers, they come pre-equipped with a whole bunch of biases. These little nuisances and irritations, we might hope, will wash away if we instead let a computer sort unfeelingly through the bag of words. 

But, alas, it isn’t so. Since the process by which the computer “learns” is anchored to grades assigned by human beings – the training set teaches the computer what kinds of grades we tend to give to what kinds of essays – the tiresome, unsexy little things that make us imperfect are built right into the system. For instance, if the graders who grade the training set tend to strongly penalize nonstandard uses of English – including nonstandard uses more common among racial minorities – so too will the machine. The computer will operationalize, and then perpetually reinstate, the botherations and biases we feed it. The best strategy, thus, will be to use mechanized, look-alike writing, which will be tautologically defined as good writing because it is associated with receipt of a good grade.

 3. Can students "game" these computer graders?

One obvious problem is that if you know what the machine is measuring, it is easy to trick it. You can feed in an “essay” that it is actually a bag of words (or very nearly so), and if those words are SAT-vocab-builders arranged in long sentences with punctuation marks at the end, the computer will give you a good grade. The standard automated-essay-scoring-industry response to this criticism is that anyone smart enough to figure out how to trick the algorithm probably deserves a good grade anyway. But this reply smacks of disingenuity, since it’s obvious that the grade doesn’t reflect what it’s “supposed to” – namely, the ability to write a reasonably high-quality essay on some more or less arbitrary topic. 

Another slightly less obvious problem is that, since the computer is just measuring and counting, it can’t actually give you meaningful feedback or criticism. It has no idea what big-picture themes you were exploring, what your tone was, or even what you actually said. It just tries to approximate the score you should get – that is, it tries to put you into a little box. While this kind of box-sorting is fine for literal grading, it doesn’t really help with teaching you to be a better writer. 

There are other problems, too. A former professor at MIT named Les Perelman has pointed out that the way the automated-essay-grading companies are analyzing their software’s performance is unfairly biased toward the machine. Perelman’s paper, although eye-glazingly dense in data analysis, notes that while a human grader’s reliability is checked by comparing his or her grades to someone else’s, the machine’s reliability is checked against a resolved grade, which reflects the judgments of multiple human readers. But the standard statistical measure of agreement, called Cohen’s Kappa, is – as Perelman puts it – “meant to compare the scores of two autonomous readers, not a reader score and an artificially resolved score.”

 4. What's the worst that can happen with computer graders?

Our culture will stop engaging with students on those very aspects of the humanities that make them worth studying in the first place. We are going to end up with a system that dispenses rewards in a way that is indifferent to – and divorced from – the most alluring parts of the humanities, those creative capacities that they let us engage. If our instruction in the humanities necessitates ignoring these abilities, then it is my opinion that there no longer is much point to teaching the humanities at all, and we should end the charade. In other words, if this kind of mechanized, standardized-test-friendly drivel is all we can offer our children as “the humanities,” then who cares about the humanities? 

Once the use of automated essay grading becomes common knowledge, the implicit message will be hard to miss. For any self-aware, warm-blooded American teenager, the conclusion will be all but inescapable: Nobody cares what you have to say. It could be brilliant and moving; it could be word-salad or utter balderdash; it really doesn’t matter. Content, feeling, creativity, thematic depth – none of it matters. Today’s students will recognize this; they will react to it; and it will inform who they grow up to be. Indeed, I confess that if I were a teenager, my response would be the same as theirs – the selfsame response that we tend to associate with (and dismiss as just) teen angst. What is the point, after all, in being rewarded by a system that doesn’t care who you are? If no one is going to read the essays, we might as well rip them up.