If you followed the news this week, you might think that teens who try electronic cigarettes are bound to take up Marlboros too. “Yep, e-cigarettes are a gateway to smoking,” read a news story published by the Fred Hutchinson Cancer Research Center. “Teens who vape appear more likely to smoke” was the headline at Reuters, and CBS Boston ran a story titled “E-Cigarette Smoking Gateway To The Real Thing, Study Finds.”
This is what happens when 16 people are made to represent an entire population.
Those headlines were reporting on a study published Tuesday in the journal JAMA Pediatrics in which researchers asked volunteers ages 16 to 26 a series of questions on two occasions, a year apart.
On the first survey, 694 people answered “definitely no” when asked the following: “If one of your friends offered you a cigarette, would you try it?” and “Do you think you will smoke a cigarette sometime in the next year?” The researchers deemed these respondents “nonsusceptible.” Importantly, some of these respondents said they had used e-cigarettes.
A year later, those same 694 participants were surveyed again, and this time, 37.5 percent of the original e-cigarette users said that they’d gone on to smoke traditional cigarettes. That’s a big percentage considering they weren’t supposed to be susceptible, especially when you consider that only 9.6 percent of the respondents who hadn’t tried e-cigs before that first survey had taken up smoking during the same time period.
The buzziest finding: Compared with people who hadn’t used e-cigarettes before the first survey, those who had were about eight times1 as likely to progress to trying a tobacco cigarette by the time of the second survey.
Those startling numbers — an 8x multiplier and 37.5 percent conversion rate — were the kind that made their way into the journal’s press release and the news stories. And as press releases go so goes overhyped journalism. If only the numbers were worthy of the headlines.
To understand why they’re not, let’s look at where that big 37.5 percent number comes from. All those “nonsusceptibles” who said they had tried e-cigarettes on the first survey? There were only 16 of them (2.3 percent of 694). And a grand total of six of those 16 people started smoking during the one-year period between the first and second surveys. Voila, six out of 16 makes 37.5 percent — it’s a big number that comes from a small number, which makes it a dubious one.
So because six people started smoking, news reports alleged that e-cigs were a gateway to analog cigs. The study could have just as easily been framed another way: Ten times as many people who hadn’t vaped became smokers as those who’d used e-cigarettes. (Sixty-five of the 678 “nonsusceptibles” who had never vaped eventually took a puff of a traditional cigarette.)
The study’s lead author, Brian Primack, a professor at the University of Pittsburgh School of Medicine, told me that given the “statistical significance” of the results, it’s OK to draw conclusions. In medicine, he said, scientists often base an entire finding on a small group. “We’ll find that seven people had a heart attack in this group and only four had a heart attack in this group, and based on that, we will forever say that you should take Lipitor,” he said. He wasn’t expecting the small sample in this study to yield statistically significant results, but after analyzing the data in numerous ways, “it was all just very consistent,” Primack said. “We think we really are getting a signal here.” If the peer reviewers had decided they couldn’t base their conclusions on 16 people, “then that’s their prerogative,” Primack said, but the paper was accepted by the journal.
I asked Andrew Vickers, a statistician at Memorial Sloan Kettering Cancer Center, to take a look at the analysis. He said it wasn’t surprising that there was an association between e-cigarettes and smoking. The real question is how big it is. While this study suggests that the effect is large, Vickers found reason for caution in what statisticians call the “confidence interval,” a plausible range of values for the study results. The confidence intervals here are “ridiculously wide,” he said, which means that the estimate that e-cigarette users are about eight times as likely to take up smoking as non-users is just a rough approximation — the true increase in risk could be anywhere from 30 percent to 5,700 percent.
Whether vaping provokes kids to start smoking or is just another novel thing for would-be smokers to try remains unclear. The number of kids and teens using e-cigarettes is still small — less than 4 percent of middle-school and 14 percent of high-school students have tried the devices, according to the latest report from the Centers for Disease Control and Prevention.
Primack’s small study is just one sliver of information in a complicated body of evidence on the effect of e-cigarettes. Numerous studies have looked at the relationship between e-cigarette use and smoking, and several have found that smoking rates were higher in e-cigarette users than in non-users. Yet other studies have suggested that e-cigarettes may help smokers quit, and a recent U.K. government report concluded that e-cigarettes may reduce smoking, even in those not intending to quit.
The media does the public no favors when it presents a single study (especially a small one like this) as gospel, rather than just a small addition to the amassing evidence. After spending more than 40 minutes on the phone with Primack, I’m convinced of two things — his intentions were noble, but his study does little to answer the question at hand: Are e-cigarettes a gateway to smoking among kids and teens? Even if the numbers in the study were larger, statistical analysis can’t tell you whether the data you collected is the right data for answering your question. Here the answer is clearly no. The survey’s respondents weren’t all fresh-faced adolescents, many were older teens and 20-somethings (the average age of those who’d tried e-cigarettes was 19.5), and it’s not possible to verify the accuracy of their self-reported smoking histories.
When I asked Primack whether he agreed with the headlines touting his study as proof that e-cigarettes are a gateway to smoking, he hedged. The press tends to get dramatic, he said. “Yeah, it might be, I guess, a little bit overblown, but on the other hand, not that overblown because we’re starting to get a few different studies showing the same thing,” he said. For instance, he pointed to a study published last month showing that high school students in Los Angeles who used e-cigarettes had higher smoking rates than non-users.
It’s plausible that young people who try e-cigarettes might also take up smoking (especially after becoming habituated to nicotine). But as Primack himself pointed out to me: The respondents in his study are not a nationally representative sample. Are the experiences of those 16 e-cigarette users typical? We don’t know. But uncertainty doesn’t make for sexy headlines.
Read more: Science Isn’t Broken: It’s just a hell of a lot harder than we give it credit for