Believe you me, being the assessment equivalent of a vegetarian butcher means being in perpetual state of cognitive dissonance. On one hand, I’m Team Rubrics! Team Performance-Based Assessments! Team Portfolio! on the other, Let’s talk p-values! Let’s discuss passage selection criteria! Let’s talk large-scale test design!
So here I am yelling, “let’s work to make sure schools are chock full of authentic, meaningful and engaging tasks and assessments!” in one breath and then trying to politely ask, “May I pushback against your criticism of this multiple choice item?” in the next.
I try to straddle that line. I fail a lot but I’m going to keep trying for one main reason – if we can’t agree on the the easy stuff, what’s going to happen with the messy stuff?
With the PARCC test, consider that:
- it’s likely that more adult eyeballs saw PARCC items before students than any other test in history
- teachers reviewed items (at every step)
- teachers sorted through texts in order to find passages that would work
- students across the country piloted the test and their data were analyzed
- the foundations of psychometrics were used in every step of the process
You don’t have to take my word for it. You can read the words of a teacher who was in the room for the work. Or this one.
PARCC had a process to follow because people have been studying multiple choice since their appearance on the scene. Stepping away from PARCC itself, there’s the challenge that:
- a passage from a text may have a different readability level that the entire text
- adults are terrible at eyeballing item difficulty
- multiple choice item design has a ton of research behind it
- States are still required to do annual testing
At some point, we’ll get around to better assessments like what’s happening in the NYS Performance Consortium, New Hampshire, or on the local-level across the country. At some point, we’ll move past multiple choice and get to what’s next.
The problem with what’s next is that the performance-based assessments pool is shallower. To be sure, there are structures, guidelines, protocols, and research around ensuring reliability, but the pool is not nearly as deep as it is for multiple choice.
In effect, what’s next is going to be messy as all get up. If the field doesn’t trust the subset of field that designed the easier stuff using research-based practices, what’s going to happen in the next evolution?
Although I’m a NY-er and most familiar with our in-state assessments, I’m always happy to discuss assessment design, be PARCC or PBA. Hit me up on Twitter or in the comments!
