In the classic Mel Brooks comedy “Blazing Saddles,” a crooked political flunky tells a Mexican bandit, “Be ready to attack Rock Ridge at noon tomorrow. Here’s your badge.” The bandit tosses the badge aside and replies, “Badges? We don’t need no stinking badges!”1 His message is clear: Authority and respect come from actions, not accolades. Badges are for Boy Scouts, not badasses.
When psychologist Brian Nosek and his colleagues proposed awarding badges to scientific papers that use certain open and transparent methods, it was easy to imagine researchers reacting like the bandits. Scientists, after all, are supposed to work for truth, not some patch on their lab jacket or paper. But Nosek, co-founder of the Center for Open Science and a leader in the movement to make science more rigorous and transparent, viewed the idea as a low-cost way to encourage and reward good research practices. He was confident in the approach: It exploits a psychological phenomenon — signaling good behavior — that’s familiar, he says, to anyone who’s taken Psych 101. And it looks as if it worked.
In January 2014, the journal Psychological Science adopted three Center for Open Science badges to signal papers that use transparent practices. The “Open Data” badge recognizes papers whose data is available in a public, open-access repository. An “Open Materials” badge indicates that the researchers have made public the methodology needed to reproduce the reported procedure and analysis. A third badge is awarded for “preregistration” — committing to a study design and methodology before the research begins. These steps are part of a larger effort by Nosek’s group and others to address the replication crisis in science and discourage questionable research practices like p-hacking.
The badges, as illustrated by the Center for Open Science.
As silly as they might seem, Nosek said the badges served a well-established purpose, by giving researchers a visible means to communicate information about their identities, beliefs, values and behaviors. People use such signaling all the time — think bumper stickers and hipster beards. Badges give scientists a way to signal that they care about research transparency, Nosek said.
And it appears that psychologists are eager to engage in such signaling. In an analysis published in PLOS Biology on Thursday, Nosek’s team reports that since Psychological Science adopted the badges, data sharing has risen nearly tenfold in papers it publishes, reaching nearly 40 percent of all papers published in the first half of 2015. (The team assessed a total of 838 papers from the journal.)
They didn’t just take authors’ words for it. The new study’s first author, Mallory Kidwell, a project coordinator at the Center for Open Science, said she and her colleagues checked to see whether data was in fact available — and, if so, whether it was complete, correct and usable. The results showed that before badges, most reports of data sharing were greatly exaggerated.
| SHARE OF DATA REPORTEDLY AVAILABLE THAT WAS… | ||||
|---|---|---|---|---|
| JOURNAL | ACTUALLY AVAILABLE | COMPLETE | CORRECT | USABLE |
| PSCI* with badges | 100.0% | 82.6% | 91.3% | 76.1% |
| PSCI pre-badges | 90.0 | 20.0 | 60.0 | 50.0 |
| Journals without badges** | 40.5 | 16.2 | 27.0 | 21.6 |
The researchers also looked at the sharing of research materials, which also increased with the advent of badges, though not as much — rising from 13 to 30 percent of papers. The difference may come down to differences in baseline rates, Kidwell said. (Before badges, about 13 percent of papers in Psychological Science shared materials, versus only 2.5 percent that shared data.)
Lindsey Hicks, a doctoral student in psychology at Florida State University and a lead author of a study in the April 15 issue of Psychological Science that earned an open materials badge, said the badge was a nice bonus, but it didn’t change her approach. “Open practices are important to me, so I was happy to do extra work for the open materials badge we were awarded,” she said. But because her group had started data collection before the badge system was in place, she said, “the availability of badges did not really influence the way we conducted our work or how we chose to publish it.”
I wondered whether the new study really showed that badges helped spread open science practices or were simply marking a change in the landscape that was already underway. When I posed that question to Eric Eich, a University of British Columbia psychologist who was Psychological Science’s editor when it adopted the badges, he seemed convinced they had created some change. Kidwell’s data suggests that badges can make “a meaningful difference in authors’ willingness to share relevant materials and data,” he said — at least for the psychology journals they studied.
He also said the tenfold rise in data sharing surprised him. The badges weren’t a tough sell, “though I doubt anyone thought it would be so effective so quickly,” Eich told me. “I certainly didn’t.”
The badges are tools for nudging good behavior, not ends in themselves. The ultimate aim is for researchers to routinely provide public access to their data and the materials underlying their work. “Once such sharing becomes normative, the badges program can be retired, having served its purpose,” Eich said. Psychological Science is just one of six journals that have adopted the badges, including the American Journal of Political Science, which recently came on board. “A big barrier to data sharing is that everyone thinks that it’s hard and no one does it,” Nosek said, adding that the new analysis undermines those claims. “A lot of people said it’s not going to work. Maybe the data will convince them.”