With fall sports season underway, concussions are all over the news. In the past two weeks, at least three NFL players — New York Jets cornerback Buster Skrine, Kansas City Chiefs wide receiver Jeremy Maclin and Jacksonville Jaguars running back Bernard Pierce — have suffered concussions during a game. Such incidents underscore the risks of a sport where crashing into opponents is part of normal play, and the long-term consequences of hits to the head during athletic competition have sparked lots of worry both inside and outside the NFL, much of it focused on youth athletes. Parents worry about their kids, sports programs worry about liability issues, and coaches worry about what to do when athletes take a blow to the head.
Concussion tests have become the favored response to these concerns for professional and amateur athletes, promising to bring a sense of reassurance in the face of a scary threat. But these tests aren’t offering answers; they’re merely offering numbers. And numbers do not automatically translate to useful information. Here, they may instead serve as a distraction by making decisions about letting athletes return to play seem more objective and certain than they really are.
At least five companies sell computer-based concussion tests, which measure various neurocognitive functions, but ImPACT has become the most common, with more than 7,000 high schools, 700 colleges and universities and 200 professional sports teams using the test, the company says. The computerized ImPACT test takes about 25 minutes to complete. It assesses memory and reaction times by asking participants to recall words and shapes and respond to stimuli. At the end, it produces scores in five categories: verbal memory, visual memory, processing speed, impulse control and reaction time. The cost starts at $400 for 100 baseline tests and 15 post-injury ones, but the price can rise steeply from there, if the test is given by a doctor or clinic that charges fees for the service.
The rationale for baseline concussion testing makes intuitive sense — you measure people’s neurocognitive skills at baseline, when they’re healthy. If they hit their heads, you test them again to measure whether their memory, reaction time or other cognitive skills have changed. A decreased score suggests that something is wrong, and you take a closer look at the athlete (and perhaps do more medical exams) before you give her the OK to return to activity. The main worry is that the athlete will get another concussion before the first is healed. Another much rarer but far scarier concern is a potentially fatal brain swelling from a condition called second impact syndrome.
The problem is that the tests can’t tell us whether it’s safe for an athlete to get back in the game; they just offer something for teams and programs to show they’re serious about concussions. That something produces numbers, which makes it seem scientific. But the numbers may be giving a subjective decision — whether an athlete should return to play — a false veneer of objectivity.
“The problem is that there’s just no evidence that just because your score is back to baseline, your risk is, too,” said Robert Sallis, team doctor at Pomona College and a past president of the American College of Sports Medicine.
And it’s not clear that tests such as ImPACT provide a reliable measure of someone’s cognitive status. The tests are supposed to produce similar scores if a healthy person takes them at different points in time, but in reality, many variables can affect performance, said Steven Broglio, director of the NeuroTrauma Research Laboratory at the University of Michigan. Fatigue, sleep deprivation, motivation, and even the test-taking environment can affect performance from one trial to another.
In one case Sallis saw, the baseline was recorded in a learning lab, and the post-injury test was conducted elsewhere. When he had the player repeat the test in the original location, the score returned to baseline. “It turned out the mouse wasn’t as fast on the other computer. When we cleaned out the track ball, the scores improved,” Sallis said.
Broglio and his colleagues published a systematic review this year that found methodological inconsistencies in many studies used to estimate ImPACT’s reliability. It concluded that with the exception of cognitive processing speed, the scores generated by ImPACT had “poor to moderate reliability.” Depending on the study, 22 percent to 46 percent of healthy participants who were given the test again produced scores that would be marked as a “reliable” change from baseline, suggesting a concussion.
The issue of reliability is especially important in adolescent athletes whose brains are still developing. For younger players, “a baseline may not be valid eight months from now,” said Kevin Guskiewicz, co-director of the Matthew Gfeller Sport-Related Traumatic Brain Injury Research Center at the University of North Carolina. And test scores could improve naturally the second or third time, he said, because people get better at taking the test.
ImPACT’s creator, neuropsychologist Mark Lovell, acknowledged that “no test is perfectly reliable” but said that doctors use many measures to gauge health that can vary from day to day, such as blood pressure. Multiple studies have shown that ImPACT is reliable from “week to week and month to month,” Lovell said. Broglio countered that many of the positive studies were done by people with a financial interest in the tests. Independent researchers have studied the test, too, and their results have been more mixed, said Broglio, who examined all the research for his recent review.
Despite its limitations, ImPACT testing has been adopted by thousands of sports programs across the country. A survey of nearly 400 certified athletic trainers who used ImPACT found that almost 95 percent of them used baseline testing with their athletes, but only slightly more than half ever examined these tests for validity by checking scores against standards included in the test’s instruction manual.
Even though trainers in the survey had taken the time and effort (not to mention expense) to conduct baseline testing, most of them said they weren’t relying much on the tests to decide whether to send a player back to play. Almost none of the trainers surveyed said they would recommend that athletes with concussion symptoms return to play just because their neurocognitive test scores were similar to their baselines, but some would give the go-ahead for a symptom-free athlete to get back in the game even if the scores remained below baseline.
In most cases, there isn’t a scientific point to the tests, Sallis said. There’s no role for ImPACT if a player shows symptoms. “If they’re still having headaches and balance issues, I’m not letting them go back to play no matter what their score is,” Sallis said.
In cases where a player no longer has symptoms, the test might simply serve as a way for decision-makers to show that they’ve done some due diligence, Sallis said, even if it’s mostly for show.
Lovell points to ImPACT as a way to detect instances where an injured athlete hides symptoms in order to keep playing. Athletes at all levels have an incentive not to report symptoms so they don’t miss a game. “We learned a long time ago you can’t just ask them how they feel,” Lovell said. “We’ve designed the test to pick up on things that they may not be willing to tell us.”
If an ImPACT test comes back abnormal, it might prod a trainer or doctor to re-examine the athlete. Of course it’s also possible that the athlete performed especially well on the baseline test, a scenario some athletes have tried to avoid by lowering their efforts on the baseline. (Lovell claims the test can detect when someone is intentionally blowing it, and at least one study does show that some types of cheating can be detected.)
No one, including Lovell, recommends that an ImPACT test serve as the sole basis for a return-to-play decision. “It’s not a magic test you give to make a decision — it’s never been touted that way,” Lovell said. “It’s a piece of the puzzle.”
That may be so, but it’s a useless piece, and one that takes resources away from things that might make a difference, said Christopher Randolph, a professor at Loyola University Medical Center and former team neuropsychologist for the Chicago Bears. “There basically are no data that would suggest that once someone has recovered from objective symptoms that you can detect anything with these tests,” he said. “They provide nothing more than a false sense of security.”
The tests also provide a steady and profitable business for those who sell and administer them. “This is a cottage industry that has sprung up around fears of liability,” Randolph said. ImPACT sells its tests to schools and teams, and it also sells courses and credentials that certify providers to administer the test and charge fees for doing so. Concussion clinics have popped up around the country to provide these tests.
Despite all the infrastructure that’s sprung up to ensure that a player doesn’t return to the game before it’s safe to do so, there is no good scientific answer to the question of how soon is too soon. Randolph and his colleagues published a study in 2009 comparing what happened to athletes who returned to play before their concussion symptoms had completely resolved versus those who’d sat out until they were asymptomatic. (The data was collected before holding players out had become standard practice.) The results showed that athletes who resumed competition before they were entirely free of symptoms had lower rates of repeat concussion than those who sat out until they were symptom-free. The study was observational, rather than a randomized trial, so it’s not definitive evidence, but it’s one of the only studies that has tried to quantify the risk of playing after a concussion.
A randomized trial published early this year found no benefit from putting concussed kids and adolescents on a regimen of strict rest for five days, compared with one or two days of rest followed by a gradual return to normal activities. “I don’t know why we treat concussion any differently from any other injury where we use our clinical judgment about when someone is ready to go back to play,” Randolph said.
No one is saying that concussions don’t warrant ample caution, but this caution would make a bigger difference if it were directed toward prevention. Schools and sports programs want to protect their athletes’ health, and they also want to protect themselves from liability. As alluring as baseline tests may be, they’re no substitute for clinical judgment. Incorporating uninformative data into the process doesn’t make for better decisions; it only adds confusion and, in some cases, a false sense of certainty. If sports programs want to show that they’re serious about concussions, they could save the resources they’re using on ImPACT tests and focus them, instead, on preventing concussions in the first place.