The Tension of Time Limits

Spoiler: I have no idea why NYSED made the decision to lift time limits on the state tests. I was not in the room. I’m a wild speculator, just like anyone else outside NYSED who is writing or talking about the changes in time limits.

Caveat: I live in the land of authentic, performance-based, portfolio assessment. I’m a large-scale testing tourist.

Bias: I believe that those who go to work for any state education department keep their humanity cards. I believe, like all of us, they are trying to make the best decisions possible with the information they have, within the constraints they see. So, when it comes to this decision, I believe they are trying to attend to Opt Out members’ concerns in ways that make sense and are doable given policy constraints.

So let’s look at testing time.

The first given we have to accept is that the goal of testing time limits is to provide students enough time to do their best but not so much time that the test takes over a student’s entire day. Most tests students take have a built time limit such as one period or one class block. Since large-scale tests like NYS’ are given in 700 school districts, there is no pre-existing limit so they have to figure it out. Unlike the Regents exams, for which classes are paused for a week, the 3-8 tests happen during a school day. Thanks to the intertubes, there’s a paper trail we can follow to see the thinking behind the current limits for the 3-8 tests.

From the 2015 Test Administrator’s Guidetime1

This is what 5th graders are expected to do with that time. From the Teacher’s Guide to the 5th Grade ELA Test:


So how is NYSED, or any test designer, supposed to know how much time is the right amount of time? There are few things we can look to.

First, they estimate how long it should take students to take the test. From the 2014 NYS Testing Program Technical Report.


To review the math for 5th graders:
42 questions that are estimated to take one minute each: 42 minutes
6 passages that are estimated to take 5 minutes each: 20 minutes

42 + 20 = 62 minutes
(which leaves 28 minutes of “extra” time given 90 minute limit for students *without* extended time)

These rules of thumb are referenced in several texts and guidance documents around test design. I found references going back to studies in 1973 that speak to “one-minute per multiple choice item.” All of that said, test designers need to find out if the “rule of thumb” actually holds. So they look at a statistic called speededness (Tech report, page 46):


The technical report concludes:

The industry standard general rule of thumb is that omit rates for multiple-choice items should be less than 5.0%. Omit rates across multiple-choice and constructed-response items on the Grades 3–8 Common Core ELA and Mathematics Tests typically ranged from 0% to 3%. As may be expected, omit rates tended to increase for items at the end of the test booklets, and only for ELA Grade 3 in Books 1 and 3 did items initially exceed that 5% threshold. It was for that reason that the last two operational items in Book 1 (both MC) and the last operational item in Book 3 (a 4-point CR item) were dropped from scoring and all analyses presented herein. In general, omit rates rarely exceeded 3%, even for the last items within a booklet. That is, these omit rates remained within the acceptable range for large-scale achievement tests. In summary, the low omit rates observed across entire forms are consistent with tests that are not speeded. [emphasis mine]

In other words: generally speaking, students weren’t rushed. They had enough time. When they didn’t, items were removed from students’ scores if there was a slight indication they were rushed. The statistics, though, present a different picture than the stories of stressed students told by those who presented at forums, wrote letters, or post on social media. It’s these stories that I suspect informed SED’s decision. From the memo to the field: this change in policy may help alleviate the pressures that some students may experience as a result of taking an assessment they must complete during a limited amount of time.

There’s a great text on test blueprints that gets into a lot of this and more importantly, it speaks to how issues such as time limits are things that teachers need to consider when designing their own tests. So while SED’s announcement will likely generate conversation about large-scale tests and time limits, here’s hoping a little bit of that seeps over into conversations about teacher-created tests.

Postscript: I can’t find a source to cite at the moment, but I know I’ve read texts about the unintended consequences of untimed tests. I’m going to try and dig them up over the weekend for another post on this scintillating topic.

Second Postscript: The question came up around why 42 items – or why so many items in Book 1? That’s an issue of validity. Which could make for another post or textbook…

Third Postscript: So what about students with extended time? I’m trying to figure that out based on how our neighbor-ish over in Massachusetts handle it. Some tweets on it.






Adventures in Data Collection

I took a lot of shit when I had the handle “DataDiva” on Twitter. A fair amount of it, I brought on myself by not considering the impact of 140 on a fellow human or by being clueless about a particular topic. The rest of it came from the first word in my handle.  More than once, someone tweeted at me out of the blue about how data was killing public education. I never quite figured out how to respond to those tweets so … I changed my handle. I picked the name a decade ago for a project because it was funny and I like alliteration and was mostly fine with letting it go.

I’ve seen the light that “data” is troublesome. I remain fully team “evidence of student learning”, on the side of documentation for learning, and will wave whatever flag I need to wave to get student work to the table without using that word. Regardless, I’m still about doing data collection right. I’m not talking here about corraling numbers into columns in Excel, which is a noble pursuit unto itself, but the lovely, messy noisy stuff that is people’s opinions.

If you want to collect people’s opinions about a topic or issue, you can design a survey. So far, so easy. If your area of interest is people’s voting plans, you ask a straightforward question such as “Which of the following candidates do you plan on voting for?” Anything beyond that forced-choice, neutral question and you get into the art and science of survey design.

Consider these two simple demographic questions.

The one on the right comes from a group of researchers based at a large university looking to collect national evidence about a particular movement in education. Their project likely went through the Institutional Review Board (IRB) at their university. The first page of their survey explains their purpose, goals, and intent. The demographic question comes at the end of the survey, after the important questions have been asked.

The one of the left comes from a grass-roots organization seeking to collect information from members. There is no research statement. There is no information about purpose, goals, or intent. The survey is framed in an email to members but once the survey is opened, there’s no context, just a question about the participant’s age and then Q2.

The demographic question on the left should raise red flags*. And to be clear, this isn’t about the nature of that survey, the project’s goals, or the organization. Rather, it’s about the tension of trusting the results of a survey when it’s clear the survey is lacking traits of quality found on the right. So, as a reformed DataDiva, I’d gently ask that before providing a response to a survey, look for hallmarks of quality.

  • Is there a statement of purpose, research, or intent?
  • Are the questions as neutral as possible?
  • Can you ascertain how your response will be used?
  • Do you detect a bias in the survey? Are the desired results telegraphed in the questions?
  • Will they use your data to support their claim or study the issue?
  • If a question strikes you as unfair or biased or if you have a question, do you know who the author is or who to contact?

Your opinion has weight. If you’re going to give your voice to someone else to use, there’s no harm in taking a beat to be an informed gifter. On the flip side, it’s helpful to understand why professional survey designers and academic researchers do what they do. In many cases, the respondent is forced into a choice without a chance to explain context. This design feature isn’t a flaw in the survey, rather it’s a function of the purpose, goals, and data collection needs. Often, research surveys will end with a “can we contact you for more information?” If you want to explain more, make sure you provide accurate and current contact information.

* The red flags?

  1. Male/female are sex-related words, not gender.
  2. Transgender isn’t a separate gender. A woman may identify as transgender but she’s still a woman. Note that the survey on the right leaves the choice up to the user.
  3. “Identify” is an extraneous word. In survey design, every word should be there because its inclusion helps the researcher.
  4. It’s the second question of the survey. There are different opinions on if demographic questions should go at the beginning or the end. What comes first in a survey is incredibly important in terms of getting a complete responses from participants. In other words, why did the designers put them first?




No True Scotswoman

A kindergarten teacher wrote a thing.

There are a couple of ways of reading the thing she wrote.

A. She’s a new teacher*, trying to make the best of a situation.
B. She’s been bought off to so the website can claim a teacher byline.
C. She’s a disgrace to the profession. Get her away from children.
D. She believes what she wrote.

For reasons, a not small number of people wrote responses to her thing built around B or C. A small number went with A and then the C group used B as a reason why the author of A pieces rose to her defense. In their responses, busy professionals, many of them teachers, took time out their day to write things to call her “a reformster”, “a child abuser”, and “not a real teacher.” She’s “stupid.” She ruined someone’s day. It must be satire as no one could be that …. that… so NOT a teacher.

I’m thinking D. She appears young. So incredibly, delightfully young. She’s likely grown up in the world of NCLB. Testing has likely been a part of her world since at least middle school. Despite that, she became a teacher. And then she wrote a thing. And people in her profession appear to be, I dare say, gleefully joining a tribe of pointing and scolding – You’re not a real teacher. You should have known better. 

She didn’t.  She found a way through and then she wrote a thing. And members of her profession are condemning her for it. There are no questions to her in the comments section. There are no statements of empathy, no inquiries into what else happens in her classroom. There is some professional empathy (hey kid, let’s have coffee and talk or it’s not her fault) but it’s framed around how very wrong she is.

20% of teachers leave the profession within the first five years. If, in 5 years, we find that Bailey is among that group, anyone want to take a guess on what she’ll say in her exit interview?

PostScript: I completely and totally get where the concerns come from. No doubt. My grrfuffle isn’t about the content of her post or the legitimacy of a response. It’s that when she googles her name, posts *about* her will come up. It’s that her profession turned against her and used the fallacy of “No True Scots[woman]” to do it. That irks me.

*Peter Greene (who commented below) researched Bailey via LinkedIn and has concluded she’s only passing through a classroom on her way to something else. There’s a lot to unpack there about many things. Perhaps another time. Meanwhile, I’m going to link to the fallacy again and have an extra glass of wine with dinner.


On 2016 and Hypocrisy

Two quotes. Same author. Different texts. Different days.
“Even so, their test scores were nothing to brag about.”
“Protect your children… Opt out [of tests] in 2016.”

To be human is to be a hypocrite. We forgive ourselves for actions that we punish others for. We frame our words within the gorgeous complexity that is our internal dialogue but judge others’ based on a context of our choosing. When we contradict ourselves, we can re-frame the dissonance in a way that soothes our feathers. When others do it, we snipe, parse, point, and snark.

To be human is to want it both ways. Volunteers who work at women’s health clinics tell of parents who protest in front of the clinic on Monday and bring a daughter in on Tuesday. Public school advocates speak about their love and adoration for the system and then enroll their children in private schools. Self-proclaimed male progressives write blog posts in which they explain to a group of female professionals how they should do their job. We want nuance. We use hyperbole. We want something new, something better. We rally around “taking back” and “restoring.”

To be a human in education is to be a time-traveler. Teachers exist in the present with the child and the future with the adult that child will become. Those of us who support teachers, who sit on the edges trying to help them make sense of a profession that seems to lurch from firestorm to firestorm, work to use the patterns from the past to inform the present. For parents, I suspect it’s a constant dance between all three – their baby and their great love, their child, and their child’s future.

Given the number of fires that are currently burning in the world of education, it’s a pretty safe bet that 2016 is going to a tumultuous year. It’s my hope we humans who are educators get better at traveling through time. That we get faster at learning from our profession’s yesterdays. We attend more thoughtfully to the biases we carry from our pasts into our presents. That we actively work to ensure our interactions with others don’t leave painful hits they’ll carry forward. It’s my hunch students are looking to see how their teachers handle discourse. They are watching how their parents engage with those who disagree with them and learning. If we want to ensure that the future debaters, discourse-havers, and opinion-makers can do it well, we’re going to have to get better at how we human.

Here’s hoping 2016 is a year in which representatives from various groups in education model that adults can disagree and do it fiercely, gently, thoughtfully, spontaneously, and with as little hypocrisy as possible.


A response to Michael Petrilli

So Michael Petrilli wrote about IUDs. I’m a big fan of  IUDs. They’re the reason I’m a non-parental taxpaying educator. And after reading Petrilli’s post, I have a request: Sir. Please stop talking about birth control and teenage pregnancy. I understand you have good intentions. I can see how your idea makes great sense to you: Want to reduce pregnancy? Teach high school students about the most effective way to reduce pregnancy. I get it. It’s a great idea.  But I’m asking you to stop talking. Demur, defer, elevate, and support instead.

Your staff was almost right – “nobody wants to hear a middle-aged man, much less a Republican, talk about birth control.” The problem is, though, that’s exactly who we do hear talking about birth control. From Congress to the courts, middle-aged men have always been talking about the right way for girls and women to handle their bodies.  Your suggestion, and your chart, adds “the classroom” to that list. 

Purity Pledges, pregnancy shaming campaigns, abstinence-only sex ed – all ideas that originated with and were advocated by middle-aged men; men who spoke from a position of authority and privilege, such as yourself. This isn’t to say middle-aged men can’t have great ideas. After all, hormonal birth control was invented by a middle-aged man (although funded by a woman.) However, when it comes to this issue, it’s worth re-considering how you add your voice. It’s likely there are a few challenges to your plan that may not have occurred to you. 

  • Schools are just now starting to realize the sexism inherent in the dress codes that tell girls what to put *on* their bodies and you’re advocating they tell them what to put *in* their bodies? Consider the long history of middle-aged men policing young girl’s bodies and appearances, especially for girls of color. Does your post add to that policing or help end it? Additionally, does your post help end stereotypes about who becomes a teenage mother or support them?
  • Parents are afraid to give their children shots that will LITERALLY prevent cancer because of a perceived connection to sex. How would the IUD message succeed when pediatricians can’t get parents over HPV vaccination fears?
  • In order to work, an intrauterine device (I noticed you didn’t use its full name) has to be inserted inside the uterus. Waaay up inside. To be sure, gynecologists go out of their way to reduce discomfort, but getting one inserted isn’t as simple as a shot in the arm. It’s invasive. Consider the impact this has on girls who have been abused: you, a middle-aged guy suggesting they get an IUD because you think it’s the best form of birth control.
  • IUDs are hormonal. The act of introducing hormones for most girls and women is a non-issue and can result in a reduction or elimination of menstruation. For a few, it can have the opposite effect and result in multiple visits to the doctor to resolve. Consider the impact this has on girls who can barely talk to their parents about menstruation, much less changes to it based on the introduction of an IUD.

And please do not mistake me. This is not about IUDs themselves. Like I said, I’m a fan. They’re fantastic. They’re safe. They’re great for all of the reasons listed in your links. Nor is this about sex ed. I, like you, am an advocate of providing all public school students with comprehensive, accurate, and thoughtful information about reproduction and human sexuality.

This is about inviting you, a white, male, middle-aged “Republican who supports preventative sex ed in schools,” to defer. You said it’s “an issue which nobody is talking about.” I say: You’re not listening. Defer to the women and girls who know this issue better and see the tension in you telling them how to handle this decision.

Defer to women like Gloria Malone, who is a young mother and blogs about her experiences.

Demur to groups like Futures without Violence who do important work around the issue of reproductive coercion by teenagers in abusive relationships.  

Elevate The National Campaign to Prevent Teen and Unplanned Pregnancy because, as their title implies, they’ve figured out what works and need help getting the word out. (And in case you missed it, they did a really interesting study on how young Republicans feel about contraception.)

Support and call for others to support Planned Parenthood who has extensive information about IUDs and teenagers and is doing the hard work of getting safe, medical care to those who need it.

Defer, demur, elevate, and support instead. There are plenty of problems in education, Michael, and many of your ideas are interesting and compelling. This one, though …. any solution you come up with will be limited by your experiences. As a middle-class, white, male Republican, show the young women of America you’re different. You’ll be the kind of Republican that elevates their voices and solutions to the challenges they face, not the kind that lectures down at them.



The Backfire Effect

Last week, I was in a Twitter conversation around large-scale testing that went basically like this:


Tweeter-er: Those leaves are bad because they’re poison ivy.
Me: Hey now, those are oak leaves.
T: No. They’re poison ivy. See the veins?
Me: Both have veins. *points to label*  Oak leaves.
T: They’re labeled wrong.
Me: *toddles under the weight of evidence showing they’re oak*
T: Poison ivy being fed to children! (which is then RTed by lurkers)

By my count, this Tweeter-er and I have danced this particular dance at least four times around the floor. Each time, the Tweeter-er thanks me politely and then goes off to join conversations about the danger of the poison ivy oak leaves. Each time, I rant a bit, I rail a bit, I proclaim my dislike of oak leaves in the first place, I triple-check to make sure they really are oak leaves, and then I realize that I’ve made it worse.

One of my favorite essential questions is what’s the point of writing an argumentative essay if we rarely change our minds? It’s a humdinger of epic proportion. Just think about it. What’s the point of opinion pieces or editorials? Yes, we rally the troops on the other “side.” We preach to the choir on our “side.” Sometimes we change minds but ever so rarely.

We do sometimes change our minds through writing and reflection, but more often, our words can cause others who hold the opposite or a different claim to more firmly believe they’re right.

  • Salon explored this phenomenon, known as the backfire effect, among Trump voters and parents who do not vaccinate their children here.
  • Big Think gets at it here and explores the role of “omniscient sources.”
  • Brain Pickings and the fantastic Maria Popova got at it and some solutions here.

What does this look like in practice? Peep the increase in declarations and re-commitment to the Opt Out movement right after the US Department of Education threatened to take away money if 95% of students weren’t tested. In other words, the USDOE laid out the facts to say, “you don’t have the right to opt your children out.” The response across blogs and Twitter from the Opt Out movement was basically, “oh yeah? Watch this.”

This isn’t about causation versus correlation, rather it’s that when we hear facts and information that contradict what we believe to be true, we’re likely to feel that we are MORE right, more on the side of good, and that the other side is more wrong and more bad.

I’ve read that the solution is to present emotional appeals instead of facts and logic. Or the approach that claims are better handled than sides or personalities. (My personal favorite.) Yet, it’s unclear if either actually makes a difference. In the meantime, I wonder what I’d lose by saying, “Dude. It’s an effin oak leaf already. Respect my authoritah.”

My hunch is that there is nothing I can do or say as an individual to handle this the right way or mitigate the backfire effect.  Instead, I suspect it’s going to require that anyone engage in educational discussions and discourse accept the possibility, however slight, that we might be wrong. That our claims may not be supported by the evidence or that a claim is standing on shaking ground. So we accept that anything we say might be wrong and live with that discomfort.

The challenge is, of course, there are non-negotiables. Race, equality, equity. People can be unequivocally wrong about these issues and there are times when there is no counter-claim. We can say the wrong thing and do the wrong thing and absolutely need to account for using the wrong word, implying the wrong thing, or not attending to what really matters in the big picture. So how do we reconcile that in 140? How do we talk to each other and through these issues, and poke at the edge of what matters while allowing each other the space to fail and for words to fail us?

All of that said…. dude. It’s an oak leaf.




We Kept Talking

Like many, the news of Joe Bower’s passing stopped me short. On one level, it was the dead-center heart hit and reminder of how short life can be, even for those who are seemingly young and healthy.

On the other, it was of loss for a type of discourse that used to happen in the edu-blog-o-sphere. When Grant Wiggins passed, I found myself wondering whose voice would act as the calming balm of reason in a discussion about teaching and learning. With Joe’s, it was a more visceral feeling about the old days of Twitter.

There are threads going back to 2011 in which, Joe, Diane Ravitch, Grant Wiggins, Michelle Rhee and myself talked through what teachers can control. For real. That happened. It was before Ravitch blocked me and when Rhee stayed with threads as personal insults thrown her way weren’t really yet a thing. Yet.  We kept talking.

Joe and I went around regarding multiple choice tests. He wanted them all gone. Now. I agreed but didn’t think it was reasonable to ask teachers with 120 students and a mandated grading system to walk away from a tool in their assessment toolbox because he said so. I felt we could leverage the tool to support students and teachers. He didn’t. We kept talking.

We went back and forth around rubrics like it was a cage match. He was the reason I created my rubric wiki; his questions got me to see that I couldn’t anoint myself as THE expert and expect my word on rubrics to be accepted. We kept talking.


We kept talking. We exchanged emails, boosted each others’ writing, and then our words faded away until we waved at each other and occasionally passed each other in comments sections.

I wasn’t able to find a single instance where either of us started a tweet with “No.” or “You’re wrong.” There were questions and wonderings. No insults beyond the occasional friendly jab in a thread that had been going on for days. We kept talking.

It seems odd to be nostalgic for five years ago but there you go. It seemed that during those exchanges way back when, it really was about seeking to understand. The mindset of “teach me” trumped “let me learn you.” Reflection is casting my memories of those chats in a favorable light: there was give and take. No shouting down. Plenty of peaceful prodding. We didn’t exchange talking points. We talked. Educator to educator. Human being to human being. Not avatar to avatar.

Will we get that back? Should we? What did we lose or gain when the shift to “sides”, “pro”, “anti” became so explicit? When words like “reformers” are thrown like insults and Twitter users tag each other to get RTs, rather than engage in discourse? Meanwhile, in so many cases, the differences between the so-called sides is equal parts gaping maw and pencil-thin sliver.

I don’t know. But I do know I’m gonna miss the hell out of Joe’s voice.