Essay-marking software gives high marks for gibberish, U.S. expert warns Alberta

This was written by Andrea Sands who is a journalist with the Edmonton Journal. Sands tweets here. This post was originally found here.

by Andrea Sands

Alberta Education should not allow high-stakes Grade 12 diploma exams to be marked by computer because today’s essay-marking software is terribly flawed, says a retired Massachusetts Institute of Technology (MIT) professor and writing-assessment expert.

Les Perelman has been an outspoken critic of automated essay marking and worked with MIT students to invent computer software called BABEL (Basic Automatic B.S. Essay Language Generator), designed to trick today’s essay-scoring programs. BABEL generates gibberish essays which, nonetheless, get high marks from so-called “robo graders.”

“I realized very early on that machine grading of writing really is, essentially, impossible,” Perelman said from Massachusetts.

“I can guarantee you, smart 18-year-olds will try to trick it. If they find out it favours certain things like word length, obscure words and pretentious language, then that is exactly what is going to be probably taught in schools.”

Last fall, Alberta Education paid U.S. company LightSide $5,000 to do a feasibility assessment on whether computers could accurately mark Grade 12 diploma-exam essay questions. Every year, about 190,000 Alberta students write the exams, worth 50 per cent of a student’s final mark.

The LightSide report concluded its software could reliably grade student essays and, in the test analysis, did so more accurately than teachers.

Perelman said the report is “fudging” because of the way data was analyzed and because there’s not enough information and explanation in the report. “I’m a pro at this and there were tables I could not understand.”

Computers can’t analyze meaning and essay-grading software usually inflates grades for essays that are longer and use uncommon words — “egregious” instead of “bad,” or “plethora” instead of many, Perelman said.

“They’re counting word length, or they’re counting types of words, or they’re counting sentence length, or they’re counting connective words like ‘consequently,’ ‘moreover,’ things like that, that show cohesion. But all those things are very mechanical.”

About nine big companies sell essay-marking programs, including Pearson, ETS, CTB/McGraw-Hill, Measurement Inc. and LightSide, which is a fairly new company.

They are lining up to win “the big prize,” said Perelman — contracts to machine-score the increasing number of tests that will result from the new Common Core State Standards Initiative. Almost all U.S. states are now implementing new standards for kindergarten through Grade 12 in English language arts/literacy and mathematics that will require annual testing.

According to LightSide’s website, the Common Core “will require more writing in every classroom, meaning more time grading and even less time for teachers to work directly with students.”

Perelman does, however, credit LightSide as being one of the more reputable companies. Company founder Elijah Mayfield, who led the Alberta study, has responded to Perelman’s criticisms by trying to improve LightSide’s products and acknowledges the limitations of computerized grading, Perelman said.

“He’s an engineer. Most of the other companies are salesmen. I still think he’s wrong.”

Mayfield, who headed up the Alberta feasibility assessment, said a confidentiality agreement prohibits him from speaking to the Journal about the report.

The U.S. National Council of Teachers of English argues the Common Core initiative is pushing companies, testing agencies and education organizations to use automated essay grading because it’s cheaper than paying people to mark tests.


Alberta Teachers’ Association president Mark Ramsankar said it’s unfair to have students spend hours writing an exam that’s marked by a machine.

“How does a machine look at the symbolism contained in a piece of writing and interpret that as symbolism?” Ramsankar said. “I’d like to see how Shakespeare stacks up in a computer-generated mark.”

Both the ATA and the Canadian Teachers’ Federation oppose machine marking of essays, particularly on diploma examinations that are the culmination of a year’s worth of students’ learning.

“Again, we’re seeing practices that we’re looking at importing from the United States,” said federation president Dianne Woloschuk. “Their education system is in a crisis. Their students are not doing well.”

Alberta Education continues to evaluate whether machine-scoring of essays could be useful here. However, Premier Dave Hancock and Education Minister Jeff Johnson have said there are no plans to pursue it at this time.

asands@edmontonjournal.com

Twitter.com/Ansands

‘Mankind will always conduct prejudice’


English 30-1 students in Alberta study the novel Pride and Prejudice, by Jane Austen, so retired MIT professor Les Perelman entered the key words “pride, prejudice” and “father” into his essay generator, BABEL. The program creates mechanically correct but “completely incoherent” essays that have fooled automated-marking programs.

Perelman’s sample was declared “off-topic,” but scored 88 per cent when Perelman ran it through the home-schooling version of IntelliMetric, under the software’s category “challenges of parenthood.”

IntelliMetric technology is used in the United States to score the Graduate Management Admission Test (GMAT), which graduate students take to get into management programs in business schools.

Here is the text from the BABEL-generated essay:

Keywords:

pride: [‘pridefulness’, u’pride’]

prejudice: [‘preconception’, ‘bias’, u’prejudice’]

father: [‘begetter’, u’father’, ‘male parent’]

Essay:

Pridefulness with decency has not, and in all likelihood never will be malevolent, humane, and considerate. Mankind will always conduct prejudice; many for an advance but a few on pulchritude. a quantity of pride lies in the study of reality as well as the area of semantics. Why is pride so efficacious to depreciation? The reply to this query is that male parent is rivetingly and gregariously febrile.

Rationale, usually by appreciation, might feign prejudice. If nearly all of the appendages adjure an explanation of the erratically or idolatrously pagan disparagement, the haphazard preconception can be more falteringly sublimated. Additionally, an orbital is not the only thing simulation reacts; it also spins at male parent. Our personal altruist on the exposure we arrange can surprisingly be an interloper. Be that as it may, knowing that executioner can be the assassination, most of the comments to my accusation civilize irrelevant scrutinizations. In my philosophy class, all of the advancements by our personal allusion of the demarcation we decry accede amplifications which deliberate with analyses but masticate veracity that should inconsistently be a contradiction and occlude circumspections for expositions. Begetter which is mimicking in how much we cavort ousts whiner of our personal inquiry to the apprentice we propagate as well. an accession will enthusiastically be a concurrence on the insinuation, not an intercession. In my experience, none of the reprimands by our personal advocate at the sophist we substantiate contemplate postulation that blusters but append. a abundance of father changes culmination for bias.


As I have learned in my literature class, humanity will always depreciate father. Even though the brain counteracts a gamma ray to contentment, the same pendulum may catalyze two different neutrinoes with the remarkably accumulated culpability. Although the same neuron may receive two different brains, radiation processes orbitals of disenfranchisements on a taunt. The plasma is not the only thing a gamma ray oscillates; it also transmits neutrinoes for abandonment at the trope by father. The diagnosis of begetter changes a plethora of preconception. The less eventual allocutions pledge thermostats, the more an organism inaugurates those in question.

Malcontent, normally on the assumption, demolishes father. As a result of scintillating, all of the adjurations hobble equally with prejudice. Also, male parent to speculations will always be an experience of humankind. In my theory of knowledge class, some of the juggernauts of my scenario assimilate probes by the search for semiotics. Still yet, armed with the knowledge that propaganda can be a demonstration or homogenizes, many of the lamentations for my dictum abandon periodicity and voyage. In my philosophy class, almost all of the domains at our personal denouncement by the comment we admonish explain demolishers which declare the demolisher with the quip on gluttony that enthrals speculations or disparage agriculturalists. Pride which utters substantiation may boastfully be propinquity or is avowed but not impartial of my advancement also. a fetishistic fulmination belittles the people involved, not assemblage. Our personal conveyance to the reprobate we implore should be the analysis. The tendentiously vast prejudice changes a quantity of pridefulness.

Bias has not, and undoubtedly never will be reticent yet somehow gluttonous. However, armed with the knowledge that a report with assemblies accounts, all of the tyroes for my amplification shriek. By the fact that gratuitous dictators are articulated at pride, most of the amplifications confide too by pride. Prejudice will always be a part of human society. Pridefulness is the most feckless proclamation of human life.

Boycott Pearson

Share Widely: From United Opt Out.

Pearson, ALEC, and the Brave New (Corporate) World:  Stand Up to Pearson Now!

(Note: We have revised our Boycott Pearson information for clarity and in order to add additional research – please use this version, posted April 29, 2012, when sharing.)

Supporters of Public Education,

The curtain has been pulled aside recently from the American Legislative Exchange Council (ALEC), exposing the seedy underbelly of our democracy. Organizations like ALEC circumvent the democratic process in favor of corporations. Financial resources are used to influence public officials and provide model legislation meant to easily pass through state houses of governance. Recent examples include infamous “Stand Your Ground” laws and others that seek to limit the voting rights of marginalized populations. Education reform legislation is also part of ALEC’s agenda, with substantial sponsorship from corporate funds to divert the flow of valuable taxpayer dollars away from public schools.

ALEC-inspired advocacy for public education reform typically follows a path to privatization; that is, viewing educational practices vis-à-vis economic and capitalist principles. Strict school choice models, vouchers, private charter management organizations, and the erosion of collective bargaining rights are all examples of the economic management of public education. As opposed to a valuable public good, certain entities prevalent in the education reform debate are forcing schools to motivate themselves by profit and competition. What it means to be an educated person (e.g., college and career ready), what is important to teach (e.g., common standards), and how success is measured (e.g., standardized tests) are currently under significant transformation without the thorough vetting via democratic processes. And with the frustration and confusion ensuing from rapid developments occurring behind closed doors, outside the public spotlight of democracy, there are large corporations conveniently present to sell us products that will solve all of our problems.

Pearson is one such entity that as of late always seems to be at the right place and precisely at the right time. In other words, just as new legislation is passed, as new educational mandates are set, Pearson is suddenly able to provide the legions of educators and school systems clamoring for some kind of answer with just the right product. How can this be? In recent years, this once relatively small publishing house turned itself into a massive provider of a range of educational products, from traditional print materials for the K-12 sector, higher education resources and technology solutions for public school systems. It is one thing to have various products to sell and to allow the marketplace to judge their success or failure. It is another matter to reorganize the rules so that Pearson products are all one needs to buy to satisfy a range of emerging Federal and State education mandates.

For better or for worse, education reform in the United States is largely controlled by legislation. It appears then that Pearson is successfully implementing a two-pronged approach: grease the democratic process in their favor so that certain rules must be followed and from the other side perfectly match their own products so they have exactly what can be bought to satisfy those requirements. Pearson, through connections to ALEC, has become the dominant provider of education resources and services in the K-12 and post-secondary markets. The following are some of the affiliations that made this perfect alignment possible:

• Pearson acquired the Connections Academy, whose co-founder and executive VP is Mickey Revenaugh, also the co-chair of the ALEC Education Task Force. Both Connections and the for-profit University of Phoenix have been or are currently subsidiaries of the Apollo Management Group. The CEO of AMG, Charles (Chaz) Edelstein, was Managing Director of Credit Suisse and Head of the Global Services group within the Investment Banking division, based in Chicago. He is also on the Board of Directors for Teach for America, which is a provider of temporary and inexperienced teachers and also frequently associated with corporate education reform. One prominent name in this regard is TFA alum Michelle Rhee, the failed former Chancellor of DC public schools.

• According to Pearson’s website: “Pearson Education and the University of Phoenix, the largest private (for-profit) university in the United States announced a partnership which will accelerate the University’s move to convert its course materials to electronic delivery.” [emphasis added]. As such, Pearson will certainly provide the materials and the mode of transmission. It must also be stated here that many for-profit universities have been under investigation for student loan fraud and unethical recruitment practices.

• America’s Choice was also recently acquired by Pearson. This organization is directly associated with the Lumina, Broad, and Walton Foundations, all active members of ALEC. They each promote so-called “innovations” that appeal to the corporate and for-profit mindset.

• Bryan Cave, LLP is the lobbying firm for Pearson. Edward Koch is currently one of the partners at Bryan Cave. Edward Koch sits conveniently and comfortably on the board for StudentsFirst NY, a branch of the national initiative StudentsFirst, which is the brainchild of failed former Chancellor of DC public schools Michelle Rhee. It must also be stated that Rhee’s tenure is under a dark cloud of investigation for rampant test cheating and tampering in the district.

• Pearson is contracted with Stanford University to deliver the Teacher Performance Assessment (TPA) to more than 25 participating states. According to Pearson’s website, “TPA is led by Stanford University, American Association of Colleges for Teacher Education, and Pearson.”Furthermore, “Pearson’s electronic portfolio management system will support candidates, institutions of higher education, and state educational agencies by providing registration and account management services,submission of the portfolio for scoring and results reporting.”[emphasis added]. Pearson provides the administrative management skills and broad-based technology and delivery systems that will support the Teacher Performance Assessment (TPA) and bring it to a national scale. Stanford University’s Office of Technology Licensing (OTL) selected Pearson to provide these needed services for the TPA. Let it be known that the U.S. Dept. of Ed. is currently considering teacher preparation programs to be evaluated based on accountability measures similar to public schools.

• Sir Michael Barber is the current Chief Education Advisor for Pearson. It is no secret that Mr. Barber is a powerful advocate for the free-market approach to education, including union busting, merit pay, and turning public schools into privately run charters.

• Pearson contracts with Achieve to manage the PARCC assessments. Achieve is funded by Lumina, State Farm (both members of ALEC) andThe Alliance for Excellence in Education (AEE). AEE chairman Bob Wise is a regular contributor to and participant with the ALEC educational agenda. Moreover, PARCC awarded Pearson a contract in January to develop a new Technology Readiness Tool, which will support state education agencies to evaluate and determine needed technology and infrastructure upgrades for the new online assessments. Pray tell, who will sell those upgrades?

• The Tucker Capital Corporation acted as exclusive advisor to The American Council on Education (ACE) and Pearson on the creation of a groundbreaking new business that will drive the future direction, design, and delivery of the GED testing program.

• The Council of Chief State School Officers (CCSSO) partners with a whole cast of other organizations that promote a corporate, anti-public education reform agenda. CCSSO Central “partners” include (among others) McGraw-Hill and Pearson. CCSSO Director Tom Luna works closely withJeb Bush, whose associations with ALEC and corporate-reform are too numerous to mention.

• GradNation is a special project of America’s Promise Alliance, sponsored by Alma and Gen. Colin Powell. Grad Nation sponsors include State Farm (ALEC), the Walton Foundation (ALEC), AT&T (on the corporate board of ALEC), The Boeing Company (ALEC), the Pearson Foundation and Philip Morris USA (ALEC). The GradNation Summit list of presenters reads like an ALEC yearbook.

• Gen. Colin Powell sits on the Board of Directors for The Council for Foreign Relations, which issued an “Education Reform and National Security” report (co-chaired by Joel Klein and Condoleeza Rice, directed by Julia Levy). The report states, among other things, that: “The Task Force believes that though revamping expectations for students should be a state-led effort, a broader coalition … including the defense community, businesses leaders, the U.S. Department of Education, and others … also has a meaningful role to play in monitoring and supporting implementation and creating incentives to motivate states to adopt high expectations. The Defense Policy Board, which advises the secretary of defense, and other leaders from the public and private sectors should evaluate the learning standards of education in America and periodically assess whether what and how students are learning is sufficiently rigorous to protect the country’s national security interests.”[emphasis added].

• According to Susan Ohanian: “In the introduction to the Education Reform and National Security report, Julia Levy, Project Director, thanks ‘the several people who met with and briefed the Task Force group including the U.S. Secretary of Education Arne Duncan, Mary Cullinane formerly of Microsoft [Philadelphia School of the Future] [now Vice President of Corporate and Social Responsibility for Houghton Mifflin Harcourt], Sir Michael Barber of Pearson and David Coleman of Student Achievement Partners …’ They were briefed by Houghton Mifflin Harcourt, andPearson.”

• Pearson has partnered with the Bill and Melinda Gates Foundation to create a series of digital instructional resources. In November 2011, the Bill and Melinda Gates Foundation gave ALEC $376,635 to educate and engage its membership on more efficient state budget approaches to drive greater student outcomes, as well as educate them on beneficial ways to recruit, retain, evaluate and compensate effective teaching based upon merit and achievement (the Gates Foundation recently withdrew its support for ALEC under the heat of public pressure). However, their billions of dollars still flow to other far-reaching organizations dedicated to dismantling public education.

• The National Board of Professional Teaching Standards is a private-sector member of ALEC. Bob Wise (Chairman, of NBPTS) and Alliance for Excellent Education presented on “National Board’s Fund Initiative to Grow Great Schools” at the Education Task Force Meeting at the 2011 ALEC annual picnic. According to the NBPTS website, they “announced that it has awarded Pearson a five-year contract for the period 2009-2013 to develop, administer and score its National Board Certification program for accomplished teachers. Pearson will collaborate with NBPTS to manage its advanced teacher certification program in 25 certificate areas that span 16 subject areas.”

• Pearson has also acquired partnerships with companies to deliver PARCC, SAT testing, GED testing, and was the central player (through Achieve) in the design of the National Common Core Standards. The GED Testing Service, while wholly owned by the American Council for Education, entered into a joint venture with Pearson to transform the GED for some 40 million adult Americans (one in five adults) lacking a high school diploma. This is an entirely new market.

Even with all of Pearson’s efforts, they are not the only game in town. McGraw-Hill is another publisher forging similar connections and making money hand over fist due to NCLB-mandated reading programs like Open Court and SRA Reading Mastery. Of course, after billions spent on Reading First and the McGraw-Hill materials, the federally funded evaluation of the program showed no increase in reading comprehension by third grade. McGraw-Hill is also one of the biggest test publishers in the U.S. and publishes the CTBS, the central competitor to Pearson’s illustrious SAT-10.

The legislation forced upon states to adopt the curriculum (i.e., the Common Core) and its required testing measures (i.e., PARCC) essentially eliminates the possibility of consumer choice (supposedly a key concept in free market ideology) and requires that taxpayer dollars for education be handed over to Pearson and McGraw-Hill as the sole providers of nearly all educational resources available to the schools. It is frightening that Pearson, profiting billions from public education, is simultaneously operated by and sponsors organizations that promote the destruction of public education. It is essentially forcing the public to pay for the demise of its own education system.

It is possible that Pearson and its allies will deny and attempt to refute the information bulleted above. Perhaps the magnitude of their efforts will project the magnitude of their guilt. Whatever the semantics here, if a connection is really an association, if ownership is actually sponsorship, or if partnership actually means membership, it is interesting and coincidental that the above cast of characters constantly find themselves associated with each other. Additionally, the common friend to all seems to be Pearson.

If Pearson is truly interested in profit, as all corporations typically are, then consumer pressure is the best way to be heard. We at United Opt Out National are calling on everyone to take a stand against Pearson by doing any or all of the following:

• Refuse to buy their materials or adopt them in your courses or for personal use.
• Bring these concerns to local PTAs, school boards and libraries.
• If required to use Pearson products due to professional obligations, do so under public protest.
• Promote the use of ACT rather than SAT, as SAT is a Pearson product.
• Inform Pearson of your actions.
• If you are in higher education, discuss your concerns with your local Pearson representative, informing them that for these purposes you are not going to adopt their materials in any of your courses.

Raise public awareness so the brakes can be put on this madness. Please see our sample letter at the end of this research document, which you are encouraged to share so that others may refuse Pearson products.

Sincerely,

United Opt Out National