Medium 9780981709154

Making the Grades

Views: 1297
Ratings: (0)
In this alternately amusing and appalling exposé of the standardized test industry, fifteen-year veteran Todd Farley describes statisticians who make decisions about students without even looking at their test answers; state education officials willing to change the way tests are scored whenever they don't like the results; and massive, multi-national, for-profit testing companies who regularly opt for expediency and profit over the altruistic educational goals of teaching and learning. Although there are absurd moments--as when Farley and coworkers had to grade students based on how they described the taste of their favorite food-- the enormous importance of standardized tests in the post “No Child Left Behind” era make this no laughing matter.

“This book is dynamite! The nice personal voice makes it utterly accessible and enticing, wholly apart from the terribly important ammunition it provides to those of us in the `testing wars' at national and local levels.”—Jonathan Kozol, author of Savage Inequities

List price: $19.95

Your Price: $14.96

You Save: 25%

Remix
Remove
 

12 Slices

Format Buy Remix

Contents

PDF

 

Chapter 1. SCORING MONKEY

PDF

CHAPTER

1

Scoring Monkey

I

BEGAN TO DOUBT the efficacy of standardized testing in

1994, about four hours into my first day scoring student responses to a state test. At the time I was a 27-year-old slacker/part-time grad student at the University of Iowa, and my friend Greg had referred me to NCS (National Computer

Systems, a test-scoring company in Iowa City) as a good place to get decent-paying and easy work. Soon thereafter, after a perfunctory group interview that entailed no more than flashing my college diploma at an HR rep and penning a short essay about “teamwork” (an essay I’m pretty sure no one read), I had myself a career in “education.”

On my first day, we new employees, as well as dozens of more experienced scorers, met at the company’s rented property on the north side of Iowa City, a warren of tiny rooms filled with computers in the dank downstairs of an abandoned shopping mall. Within 10 minutes of sitting down, the gent sitting next to me—named Hank, a floppy leather hat perched on his head, a pair of leather saddlebags slung across his shoulder—

 

Chapter 2. NUMBERS

PDF

CHAPTER

2

Numbers

H

AVING DISCOVERED my 1994 price—ethics included!— was just $7.75 an hour, I reported back for my second day at NCS and began to see what the gig was really all about. After listening to a cursory, 10-minute review of the bicycle safety item and scoring rules, we began to read and score student responses again. I never again saw or heard of that problematic poster with the busted-up bike “stopped” at the stop sign that had so annoyed me the day before. It had been directed to one of my fellow scorers, and one of them— apparently without much thought, because no one ever mentioned it—scored it, either giving it full credit for so obviously understanding the value of stop signs, or maybe giving it no credit because he or she believed (as I did) the poster provided absolutely no clear understanding of anything. While I was curious what score my unknown coworker might have given the response, I was even more intrigued to think what some test developer/education expert imagined that score meant. To me the poster was absolutely unfathomable, but someone out there believed it was an indicator of student learning.

 

Chapter 3. THE WHEAT FROM THE CHAFF

PDF

CHAPTER

3

The Wheat from the Chaff

M

Y INTENTION in moving to Iowa City had been to establish residency so if I were eventually granted admission into the Iowa Writers’ Workshop, I’d only have to pay in-state tuition. Additionally, given my spotty academic record as an undergraduate (2.8 GPA—I cared a lot about soccer during my first collegiate go round), I figured I would take classes at the university to get good grades, do some writing, and convince a professor or two to write me a recommendation for the Workshop.

It was an excellent plan, and it actually bore fruit. I earned

A’s in undergrad courses in nonfiction writing, and a couple of my submissions were extremely well received. I wrote a story about returning to small-town Maine, after college in big-time

Montreal, to help my parents run their podunk general store, and it led the instructor to comment during a classroom discussion that the tale had “deeply moved” him. Another story about my last college soccer game—an account that implied

 

Chapter 4. OFF TASK

PDF

CHAPTER

4

Off Task

B

IG DUMMY that I am, I did fail both qualifying tests on that first writing project, although I was also lucky enough to discover that didn’t necessarily preclude me from actually working. As noted, NCS was kind enough to rehire me, post–qualifying test failures, as soon as it discovered it was short on personnel. Still, while it may have un-fired me, the company did not do so carelessly. It did so only by putting me on “probation,” informing me that although I could come back to work, I could only do so conditionally. When rehiring me, the HR department was very clear a close eye would be kept on me and my kind.

While I wasn’t exactly thrilled to return to work bearing the stigma of probation, I did prefer it to unemployment.

When I asked Greg, however, exactly what “on probation” meant, he had a good laugh.

“Please,” he said. “How many unemployed people with college degrees do you think there are in this town? How many people willing to do this work? Trust me—they’re not getting rid of you now.”

 

Chapter 5. TABLE LEADER

PDF

CHAPTER

5

Table Leader

M

Y PRINCIPLED “never again” stance to give up standardized test scoring was short-lived. Really, my antitesting position hadn’t resulted from any sense of moral outrage, just the fact I found the whole thing unseemly.

I’d always had a pretty healthy ego, and glancing at student responses and lumping them into piles of 1, 2, 3, 4, 5, or 6 wasn’t exactly the way I’d planned to make a living. It was a job I felt I could have done starting about the eighth grade, and frankly I felt the whole thing was beneath me.

My parents convinced me otherwise. When I explained to them over the course of the summer how foolish I believed test scoring was, my folks didn’t seem particularly put off. Admittedly, my summary of the job didn’t include my own tendency toward daydreaming and/or cheating, so my attempt to explain the ambiguity and inconsistency of the work fell on deaf ears.

My mother and father heard only that I had a “professional” job at a big corporation, so they were pleased.

 

Chapter 6. THE ORACLE OF PRINCETON

PDF

98

MAKING THE GRADES

“They’re too kind,” I said, “too kind.” I pretended I couldn’t imagine getting such high praise, but in my head I was beaming, those fawning reviews of my managerial style—nay, of my very essence—confirming my long-held belief that I was extraordinary.

Maria continued reading to herself, shaking her head with disbelief. “I’ve never seen such evaluations,” she said, and before she walked away, she patted my arm. “Great job, Todd.”

I was proud I’d had such an effect on my scorers and happy to have made such a favorable impression on Maria.

Those flattering reviews occurred at just the right time, because by the spring of ’98, the testing business was booming, and NCS was beginning to open scoring centers all across

America. They would need experienced supervisors to travel to other states to oversee projects, and when Maria thought about whom to send, my name was near the top of her list.

Finally I was getting somewhere, because even if my efforts to get into the Writers’ Workshop hadn’t exactly panned out (a tactful rejection letter having landed in my mailbox that very spring), traveling for NCS meant two things: four more dollars an hour, and living on an expense account. Ka-ching!

 

Chapter 7. THE KING OF SCORING

PDF

CHAPTER

7

The King of Scoring

B

ACK AT THE first range-finding meeting I attended, Harlan and I hadn’t pulled the wool over the eyes of those teachers for the fun of it. We pulled the wool over their eyes because it was the only way to get the work done. While those teachers may have been experts in the classroom, Harlan and I were experts at the scoring center, and the two weren’t necessarily related. Expecting classroom teachers to know how to differentiate between score points on a standardized test was as foolish an idea as leaving me in charge of a room full of fourth graders. It was the difference between teaching 20 students and testing 60,000.

Once I sat through a range-finding session with a bunch of classroom teachers who were writing the rubric for a dichotomous item (a correct answer would earn a 1, an incorrect answer a 0). The teachers asserted quite confidently that

“Holly” was the correct answer to the question “Who is telling this story?” (a first-person narrative about a girl’s family). The teachers told me quite assuredly that not only was “Holly” the correct answer, but it was the only correct answer. When I asked

 

Chapter 8. A REAL JOB

PDF

CHAPTER

8

A Real Job

A

FTER GETTING rejected by the Writer’s Workshop, I had no better plan than to slog away in testing. I may not have wanted to, but I faced a number of obstacles in advancing into any other line of work. First, I was living in a place where the percentages were against me, because Iowa

City was a burg full of well-educated townies and very few jobs.

Second, my grandiose sense of self continued to flourish unabated, and because I still imagined myself a writer and possible genius, I wasn’t ecstatic about the idea of some career in

Iowa City at a shipping company, meat-packing plant, or the university’s administrative offices.

Instead, I went on the dole. After completing a four-month project for Maria in January 1999, I stopped working. There were no scoring projects happening and hence nothing for me to do. Per Greg’s suggestion, I dropped in to the state unemployment office, where I happily discovered I could get checks every week for not doing anything. While I’d always been fundamentally opposed to receiving such government largesse, I was even more opposed to demeaning myself by actually, you

 

Chapter 9. MY OWN PRIVATE HALLIBURTON

PDF

CHAPTER

9

My Own Private

Halliburton

B

Y MARCH 2005, I’d been “retired” from ETS for a year and a half, but it had been a lucrative year and a half indeed. As soon as I’d hung up my ETS pencil, the offers for consulting work started flooding in, and those offers were not small. I accepted because I had to pay the bills (damn, that was some Cobra health payment!) and because I was unqualified to do anything else. I took gigs with former colleagues at ETS to develop training materials, with Maria (who was in business for herself in Iowa City) to write test items, and with Riverside

Publishing in Chicago, which had mysteriously gotten my name and quickly signed me to a contract as a “test-scoring expert.” A friend from NCS had also called to ask if I could lend a hand on a scoring project, but the pay was so small (17 bucks an hour) I had to make a serious effort not to laugh in my old pal’s face.

Being a consultant was like running my own private Halliburton: I did what I felt like and charged what I wanted, the

 

Chapter 10. WORKING IN THEORY

PDF

186

MAKING THE GRADES

write questions only a certain population can answer. They don’t write questions in German, because then only speakers of that

Teutonic language could answer; they don’t write questions anymore about “regattas,” because there are only so many scions of aristocratic, sailing families in the test-taking population these days; and they don’t write questions that are racist or sexist, or are disparaging to any particular religion/nation/culture/ subculture/sub-subculture (we can’t be hurting anyone’s feelings now), or are focused on any topic that might be disturbing or upsetting or even the slightest bit unsettling to the delicate student population (because having to face up to the realities of the world might be so discombobulating, little Billy or Mary— or Jose or Paulita, DeShawn or LaKeesha, Hideki or Chidori,

Jagdeep or Amita—might not be able to concentrate enough on the vitally important test in front of them).

But it seemed to me obsessing over those guidelines could be something of a slippery slope, and it was a slope that I thought we slid down too often. While conceding some topics definitely needed to be avoided on tests (mature topics like sex and drugs, controversial topics like abortion or the war in Iraq),

 

Chapter 11. WARM BODIES

PDF

CHAPTER

11

Warm Bodies

T

HE ONLY REASON I accepted another job scoring tests in the spring of 2007 is because I was made an offer I couldn’t refuse. At the time, I was happily ensconced in the East Village of New York City, trying to make it as a writer:

I’d already gotten a couple of magazine stories published, and

I never would have returned to testing except an ex-colleague of mine wanted to ship me to Iowa City for three weeks to train the scoring of the NAEP reading test. As that sounded like nothing to me but a paid vacation to my beloved former

Shangri-La, I stopped writing long enough to undertake the task.

The fun began for me for NAEP reading 2007 even before

I left my apartment, when I sat through yet another ridiculous conference call, this one barely audible due to a bad phone connection. On the line were more than 30 people, including me in New York and representatives of ETS, Pearson Education, various federal government agencies, and the NAEP

Reading Committee spread all over the United States. We

 

Details

Print Book
E-Books
Slices

Format name
PDF
Encrypted
No
Sku
B000000023338
Isbn
9781609944735
File size
2.02 MB
Printing
Allowed
Copying
Allowed
Read aloud
Allowed
Format name
PDF
Encrypted
No
Printing
Allowed
Copying
Allowed
Read aloud
Allowed
Sku
In metadata
Isbn
In metadata
File size
In metadata