In 2000, researchers at Princeton found there may be something to the storybook stereotype of the wicked stepmother — at least from a public health perspective. Children who lived with stepmothers, they reported, were taken to the doctor and dentist less often; were less likely to wear seat belts; and were more likely to live with someone who smoked. Risk factor upon risk factor stacked up against those kids. The only mitigator: If the child was in regular contact with his biological mother.
Think about all the things a scientist would have to know in order to come up with that finding. Who is in the family? When does each family member go to the doctor? When each member travels, where does he or she go? How frequently is each member in a car? Who in the household smokes? If there are two kids, half siblings, are they being treated differently?
If a stranger came up and started asking you this stuff, you might be tempted to tell them to get lost. But every year since 1957, tens of thousands of Americans have opened their homes to government survey takers who poke and prod their way through a list of intimate and occasionally uncomfortable questions. That process is part of the National Health Interview Survey, the gold standard of health data in the United States.
Experts say it is unique in its sample size, the scope of its questions, and how long it has existed. The NHIS is crucial to our ability to track the prevalence of diseases and health-related behaviors, and to answer complex questions involving health, income and family demographics. And now it’s about to change. The National Center for Health Statistics, the branch of the Centers for Disease Control and Prevention that conducts the survey, is planning a big shift in how the NHIS works, one that some scientists fear will impair their ability to learn about things like how stepmothers can subtly disadvantage children — investigations that end up shaping everything from the way your money gets spent to the policies your legislators vote on.
At issue is the question of how long anybody can reasonably expect Americans to put up with sharing intimate details of their lives with taxpayer-funded bureaucrats. The 35,000 households — more than 87,000 individuals — who will be interviewed for the NHIS this year will have to devote an average of 90 minutes of their time to the survey. They aren’t paid. There’s no prize at the end. “That’s a lot to ask of people,” said Stephen Blumberg, associate director for science at the National Center for Health Statistics.
The thing about a gold standard is that every researcher wants to be a part of it. Twenty years ago, the NHIS lasted about an hour. But over time, more government agencies and scientists wanted to add more and more questions to make it more and more useful. Meanwhile, participation has been dropping. In 1997, the survey had a 91.8 percent response rate. “The response rate today is about 70-73 percent,” Blumberg said.
So scientists and the government are caught in a tug of war over the survey. Pull too hard one way, and they risk losing access to valuable information. Yank it back the other, and Blumberg worries that the participation rate could fall below 60 percent. According to The Journal of the American Medical Association, a rate below that threshold risks introducing bias based on who chooses to participate and who doesn’t.
Blumberg and his staff want to make the survey 30 percent shorter. To do that, they’ve proposed a change in the methodology — one that has independent researchers like Steven Ruggles, director of the Minnesota Population Center, worried. Now, survey methodology may not keep you up at night, but for people like Ruggles and Blumberg, it’s a big deal. And we here at FiveThirtyEight tend to agree. Facts are only as fact-y as the methods used to obtain them. We know what we know only if we understand how the surveys got made.
Originally, the NHIS was conducted “family style,” with as many family members as possible, all in one place and answering questions about themselves. That changed in 1997. Today, the survey is done in two parts. First, interviewers ask questions about the family as a unit and about all of the individuals in the family. Then they randomly choose one adult and one child, who answer much more detailed questions about themselves. The proposed redesign would drop that first part altogether. Some broad information — questions about family income, for instance, and the financial burden of medical care — would shift to the random adult portion. Other information once collected for everyone in the household — health, habits, how they live — would be collected only from the randomly selected adult and child.
Now think back to the stepmother study. What if the randomly chosen child wasn’t the stepkid? What if it was, but we knew nothing about what the stepmom reported about her children? All of a sudden, a finding that could change policy, affect interpersonal relationships, or make some people so mad that they have to go out and try to refute it — just vanishes in a puff of missing data.
Ruggles didn’t write the stepmother study, but he is in charge of the organization that helped compile information for the researchers who did. The Minnesota Population Center collects, compiles and redistributes data from the NHIS and other surveys. He told me that the feds want to cut the survey back because they don’t use it the same way as the 3,500 researchers who subscribe to the Minnesota Population Center database. “The real issue here is that there are statisticians in the federal government that think the reason why we have these surveys is to produce printed reports about tobacco use trends over time,” he said. For people who are focused on tracking health trends, losing the detailed data on whole families won’t make much of a difference.
By contrast, researchers like Ruggles are interested in microdata — the information about individuals that allows scientists to make connections between variables and health outcomes. “It produces a couple thousand academic articles every year,” Ruggles told me. “Nobody is writing articles about tobacco use dropping 2 percent.”
He also faulted the government’s rationale for cutting back the survey: Nobody knows whether making it shorter will actually increase the response rate.
The NHIS is not the only survey that’s losing respondents over time. Across the board, fewer and fewer Americans are choosing to participate in surveys of all kinds. This trend has been documented by the National Research Council and the Pew Research Center. Scientists I spoke to repeatedly referred to this problem as “the elephant in the room.”
It’s not clear why this is, but the length of the survey is probably not the primary driver, said James Lepkowski, director of the University of Michigan’s program in survey methodology. “If length is an issue, what you should see is that people break off in the middle. That’s actually very rare.”
Blumberg acknowledged that “we don’t have direct experimental evidence that shortening the survey will improve response rates.” Instead, he cited the results of a natural experiment: When the 2009 and 2011 surveys happened to take about 15 minutes less than those conducted in 2010 and 2012, he said, the response rates for the shorter surveys were higher. But the difference is tiny.
In any event, the response rate isn’t the only reason to change the NHIS, Blumberg said: The survey has become unwieldy, and its length makes it too expensive to administer. Making it shorter would save money and is a responsible use of taxpayer investment, he told me. Even Ruggles thinks the survey needs to be trimmed; he just disagrees about what should be cut. “The instrument is a mess,” he said, referring to the survey. “Every little program has its own questions.” For example, he suggested reducing the number of questions about health insurance and what kinds of diseases people have. But he also admitted that if those things were cut, researchers who used that information would probably be as unhappy as he is about the microdata.
And that, Lepkowski said, is the government’s real problem with the NHIS. “When you make a policy decision like this, you can’t win,” he said. “That just goes with the job, and frankly they don’t get paid enough for this.”