This was written by Phil McRae who is an executive staff officer with the Alberta Teachers` Association. Dr. Phil McRae’s Biography, Research, Writing, Scholarship and Presentations can be found at www.philmcrae.com, and you can follow him on Twitter here. This post first appeared here.
by Phil McRae
When Alberta Education announced it was moving away from provincial achievement tests (PATs) and toward digital student learning assessments (SLAs), many educators, parents and students cheered.
PATs did little to help teachers diagnose and respond to student learning needs, but they did much to create stress for students and to encourage school ranking. But will the new digital SLAs—to be administered at the beginning of each school year in Grades 3, 6 and 9—provide teachers with useful information? While the Association remains committed to working with government on the new grade 3 SLAs, important long-term questions need to be addressed.
PATs will be phased out over the next three years as the new digital SLAs are phased in, by 2016/17. Grade 3 PATs will be phased out first, with the new digital SLAs being administered to incoming Grade 3 students as early as September 2014. The aim of the digital SLAs is to support teacher assessments in literacy and numeracy benchmarks through the digital platform offered by Alberta Education. The proposed SLAs in Grade 3 will include both machine-scored short-response digital items and performance assessments marked by the teacher.
Although this form of assessment sounds promising, a few things should be considered. The current focus is on objectively scored digital assessment items, but examples are emerging of automated essay scoring of student-produced writing tasks. Alberta Education is piloting the machine scoring of student essays. Although details of the pilot have not been articulated, Alberta Education has contracted with LightSide Labs (www.lightsidelabs.com), based in Pittsburgh, to provide an “exploratory” pilot using student data. LightSide Labs claims its educational writing assessments “matched human reliability faster and at a fraction of the cost.”
The use of computer technologies (from word processors to on-screen testing programs) to assess student work is called e-assessment and includes computer-based testing, computerized adaptive testing, computer-based assessment and digital assessment.
Computer-based testing has three essential elements:
1. Test item development: Hundreds or thousands of digital items can be generated in seconds by a single computer program.
2. Test administration: Tests are administered online, thus eliminating or reducing the costs associated with exam delivery and security. However, the final access costs to e-assessments are borne by the end users (personal device, institution bandwidth or school computers).
3. Test scoring, analytics and reporting: Test reporting is fully automated and instantly reported.
The world of e-assessments is growing rapidly, as evidenced by a $1.4 million Canada Research Chair award in educational measurement to Professor Mark Gierl at the University of Alberta. Gierl, an international leader in the field, will research approaches to producing a large number of test items that university educators will require for the transition to computerized educational testing, also known as automatic item generation.
Gierl argues that the following four principles should account for adopting e-assessments:
1. There should be a shift from infrequent summative assessments (for example, two midterms and one final) to more frequent formative assessment (for example, 8–10 exams or more per term).
2. Testing on demand is required where students can write exams at any time and at any location.
3. Assessments should be scored immediately and students should receive both instant and detailed feedback on their overall performance as well as their problem-solving strengths and weaknesses.
4. There should be much less time and effort spent implementing these principles in large classes compared to the amount of time currently spent on assessment-related activities.
Proponents of e-assessments point to benefits such as cost-cutting, expediency of data transfer and an efficient and effective 21st-century learning system.
The move toward e-assessments is fundamentally about reducing costs associated with humans involved in the testing process, with a view to increasing efficiencies within the system. The e-assessment movement argues that paper-based testing is dead and it claims that computer-based testing will either eliminate or automate two-thirds of the testing activities that teachers currently do manually (for example, item generation, administration, scoring, analyzing and reporting).
E-assessment advocates assert that 16,000 essays can be graded every 20–40 seconds, as compared to the current six-week window for marking and returning tests to students. But additional challenges arise when writing tasks are coupled with machine scoring. Machine scoring is currently limited in its ability to handle the semantics of complex written responses. For example, where does the student’s writing in the margins or brainstorming work get accounted for in the e-assessment? Is process lost, while only the final product is assessed? How can a machine assess a student on critical thinking and effective communication in a personal essay? Although e-assessment can detect spelling and grammar errors, will it detect irony, subtlety, truth, emotion and depth in the writing? Will clichés and witty barbs go unnoticed (or misinterpreted)? In short, a machine cannot engage meaningfully with a person on an intellectual, creative or emotional level.
While Alberta Education maintains that the rollout of the SLAs in Grades 3, 6 and 9 will include classroom-based, teacher-driven assessments, there are indications that the government is committing resources to digital testing platforms with very limited resources to support comprehensive professional development for teachers. With the shift of the delivery of diploma examinations to a digital platform, the same problems persist: the excessively high weighting of the examinations and the refusal to give teachers access to the examinations following their administration.
In the end, it is important to remember that while technology has a place in education assessment, its mechanized and standardized valuations are no replacement for the sound judgment and ability to interpret context and meaning that teachers bring to the equation. If the new provincial assessment initiatives are to succeed, the government needs to invest in building the assessment capacity of teachers rather than what sometimes appears to be an almost single-minded focus on investing in digital technologies. ❚
More about e-assessment
“Lies, Damn Lies, and Statistics, or What’s Really Up With Automated Essay Scoring,” by Todd Farley, author of Making the Grades: My Misadventures in the Standardized Testing Industry
—www.huffingtonpost.com/todd-farley/lies-damn-lies-and-statis_b_1574711.html
“Computer Grading Will Destroy Our Schools,” by Benjamin Winterhalter
—www.salon.com/2013/09/30/computer_grading_will
_destroy_our _schools
Alberta Education
A presentation on the government’s move to digital assessments is available from Alberta Education
—http://prezi.com/dgr4tn_gn9g7/jtc-nov-2013/?utm_campaign=share&utm_medium=copy
by Phil McRae
When Alberta Education announced it was moving away from provincial achievement tests (PATs) and toward digital student learning assessments (SLAs), many educators, parents and students cheered.
PATs did little to help teachers diagnose and respond to student learning needs, but they did much to create stress for students and to encourage school ranking. But will the new digital SLAs—to be administered at the beginning of each school year in Grades 3, 6 and 9—provide teachers with useful information? While the Association remains committed to working with government on the new grade 3 SLAs, important long-term questions need to be addressed.
PATs will be phased out over the next three years as the new digital SLAs are phased in, by 2016/17. Grade 3 PATs will be phased out first, with the new digital SLAs being administered to incoming Grade 3 students as early as September 2014. The aim of the digital SLAs is to support teacher assessments in literacy and numeracy benchmarks through the digital platform offered by Alberta Education. The proposed SLAs in Grade 3 will include both machine-scored short-response digital items and performance assessments marked by the teacher.
Although this form of assessment sounds promising, a few things should be considered. The current focus is on objectively scored digital assessment items, but examples are emerging of automated essay scoring of student-produced writing tasks. Alberta Education is piloting the machine scoring of student essays. Although details of the pilot have not been articulated, Alberta Education has contracted with LightSide Labs (www.lightsidelabs.com), based in Pittsburgh, to provide an “exploratory” pilot using student data. LightSide Labs claims its educational writing assessments “matched human reliability faster and at a fraction of the cost.”
The use of computer technologies (from word processors to on-screen testing programs) to assess student work is called e-assessment and includes computer-based testing, computerized adaptive testing, computer-based assessment and digital assessment.
Computer-based testing has three essential elements:
1. Test item development: Hundreds or thousands of digital items can be generated in seconds by a single computer program.
2. Test administration: Tests are administered online, thus eliminating or reducing the costs associated with exam delivery and security. However, the final access costs to e-assessments are borne by the end users (personal device, institution bandwidth or school computers).
3. Test scoring, analytics and reporting: Test reporting is fully automated and instantly reported.
The world of e-assessments is growing rapidly, as evidenced by a $1.4 million Canada Research Chair award in educational measurement to Professor Mark Gierl at the University of Alberta. Gierl, an international leader in the field, will research approaches to producing a large number of test items that university educators will require for the transition to computerized educational testing, also known as automatic item generation.
Gierl argues that the following four principles should account for adopting e-assessments:
1. There should be a shift from infrequent summative assessments (for example, two midterms and one final) to more frequent formative assessment (for example, 8–10 exams or more per term).
2. Testing on demand is required where students can write exams at any time and at any location.
3. Assessments should be scored immediately and students should receive both instant and detailed feedback on their overall performance as well as their problem-solving strengths and weaknesses.
4. There should be much less time and effort spent implementing these principles in large classes compared to the amount of time currently spent on assessment-related activities.
Proponents of e-assessments point to benefits such as cost-cutting, expediency of data transfer and an efficient and effective 21st-century learning system.
The move toward e-assessments is fundamentally about reducing costs associated with humans involved in the testing process, with a view to increasing efficiencies within the system. The e-assessment movement argues that paper-based testing is dead and it claims that computer-based testing will either eliminate or automate two-thirds of the testing activities that teachers currently do manually (for example, item generation, administration, scoring, analyzing and reporting).
E-assessment advocates assert that 16,000 essays can be graded every 20–40 seconds, as compared to the current six-week window for marking and returning tests to students. But additional challenges arise when writing tasks are coupled with machine scoring. Machine scoring is currently limited in its ability to handle the semantics of complex written responses. For example, where does the student’s writing in the margins or brainstorming work get accounted for in the e-assessment? Is process lost, while only the final product is assessed? How can a machine assess a student on critical thinking and effective communication in a personal essay? Although e-assessment can detect spelling and grammar errors, will it detect irony, subtlety, truth, emotion and depth in the writing? Will clichés and witty barbs go unnoticed (or misinterpreted)? In short, a machine cannot engage meaningfully with a person on an intellectual, creative or emotional level.
While Alberta Education maintains that the rollout of the SLAs in Grades 3, 6 and 9 will include classroom-based, teacher-driven assessments, there are indications that the government is committing resources to digital testing platforms with very limited resources to support comprehensive professional development for teachers. With the shift of the delivery of diploma examinations to a digital platform, the same problems persist: the excessively high weighting of the examinations and the refusal to give teachers access to the examinations following their administration.
In the end, it is important to remember that while technology has a place in education assessment, its mechanized and standardized valuations are no replacement for the sound judgment and ability to interpret context and meaning that teachers bring to the equation. If the new provincial assessment initiatives are to succeed, the government needs to invest in building the assessment capacity of teachers rather than what sometimes appears to be an almost single-minded focus on investing in digital technologies. ❚
More about e-assessment
“Lies, Damn Lies, and Statistics, or What’s Really Up With Automated Essay Scoring,” by Todd Farley, author of Making the Grades: My Misadventures in the Standardized Testing Industry
—www.huffingtonpost.com/todd-farley/lies-damn-lies-and-statis_b_1574711.html
“Computer Grading Will Destroy Our Schools,” by Benjamin Winterhalter
—www.salon.com/2013/09/30/computer_grading_will
_destroy_our _schools
Alberta Education
A presentation on the government’s move to digital assessments is available from Alberta Education
—http://prezi.com/dgr4tn_gn9g7/jtc-nov-2013/?utm_campaign=share&utm_medium=copy
No comments:
Post a Comment