This was written by Andrea Sands who is a journalist with the Edmonton Journal. Sands tweets here. This post was originally found here.
by Andrea Sands
 |
| Phil McRae with the ATA. Photo by John Lucas, Edmonton Journal |
A computer could do a better job than a teacher in marking Grade 12 diploma exam essays, a government-commissioned study says.
Last fall, Alberta Education sent two 2013 diploma-exam questions along with nearly 1,900 student essay answers that had been graded by teachers to LightSide, a Pennsylvania company that develops computer software to score student essays.
LightSide’s automated algorithms outperformed human reliability in the Alberta study by about 20 per cent, said the company’s January 2014 report to the government.
“We are certain that LightSide is able to reproduce scoring behaviour at least as reliably as human graders, and in many cases we believe that our automated performance would be dramatically more reliable than human grading,” the report said.
The study indicated Alberta Education’s human scoring was quite unreliable, below the threshold LightSide recommends for high-stakes testing.
Alberta should consider investing in “a more stringent training process for human graders,” the report said. “It is somewhat alarming to see human reliability so low.”
The $5,000 study suggests LightSide’s marking program is more reliable than a single human marker, but Alberta uses a double-marking system, said Neil Fenske, Alberta Education’s executive director for assessment.
At least two teachers grade each diploma-exam essay and, if the grades differ, it goes to a third marker.
“So we’ve built a system that is highly reliable ... but it’s very labour-intensive as well,” Fenske said.
Alberta Education commissioned two previous studies — one three years ago and one 15 years ago — to see if computerized essay marking could work, Fenske said.
Further study is needed because LightSide examined a very small sample, Fenske said.
However, the report does show automated technology has evolved enough that it could be useful, if Alberta combined the marking power of people with the speed and reliability of machines, he said. That could mean one person marks an essay, then it’s run through a computer for grading, and sent to another person if there’s a discrepancy.
The province will also soon need diploma exams marked more often than before. Last year, the department announced Grade 12 diploma exams will be offered more often and digitally, part of efforts under Inspiring Education to make the school system more flexible.
“Because marking means teachers out of the classroom, one of the things we have to take a look at is, is there a way that we can build a better marking system that’s better for students but maybe also keeps more teachers in the classrooms?” Fenske said.
Alberta Education has had trouble this year recruiting enough teachers to grade diploma exams, which are worth 50 per cent of a student’s final mark.
Around the same time the LightSide report was commissioned, Education Minister Jeff Johnson cut a grading honorarium in half — from $200 to $100 — for teachers who volunteer to mark diploma exams on a regular workday. Fewer teachers volunteered to do the marking this year, and it’s taking longer as a result.
Johnson and Premier Dave Hancock said this week the honorarium cut should be re-examined.
Asked about the LightSide report at a Journal editorial board meeting this week, Hancock said he hasn’t yet read the study, but is not interested in having machines grade diploma-exam essays.
“I think that’s an absolute disastrous way to go,” Hancock said. “There are things that teachers bring to the process that are very important.”
Teachers also benefit from professional development when they mark the exams, meeting with colleagues from across the province and discussing education standards, Hancock said.
“At this time, there are no plans to institute digital scoring systems in provincially graded essays like diploma exams,” Johnson said in a statement.
Last weekend, Alberta Teachers’ Association delegates voted unanimously at their annual meeting in Calgary to opposed machine scoring of essay questions.
The ATA was never told about the LightSide study, but machine-marking has sparked heated debate in the United States, said Phil McRae, ATA executive staff officer and adjunct education professor with the U of A.
Standardized testing in the U.S. is growing, prompting governments and school districts to look to computer scoring to keep grading costs down, said McRae, who researches technology in education.
“It’s about reducing costs, whether it’s development of the items for the tests, administration or scoring.”
No comments:
Post a Comment