There’s a good argument to be made that MLB launched the new sports–data revolution in 2006, when it introduced PITCHf/x. The technology used cameras to measure the velocity, position, and break of every pitch in real time, transforming how sabermetricians analyzed the sport. But this season, PITCHf/x was phased out in favor of Statcast, a newer and more advanced system that tracks the ball (and players) using a combination of radar and cameras.
On paper, Statcast is an incredible leap forward — and when it works, it’s amazing. But so far, it has struggled to measure the basic elements of pitching that PITCHf/x had down cold, causing confusion among sabermetricians and fans alike.
It all started the first weekend of the season, when observers noted some unusual pitch velocity readings from San Francisco Giants hurler Madison Bumgarner. Bumgarner’s fastball was up almost two full miles per hour compared with last year; in a league where every tick matters, that reading could have meant a much better season for the Giants’ ace than expected. It wasn’t just Bumgarner: FanGraphs writer Dave Cameron quickly noticed that velocity numbers had jumped across the league. Days after the changes were noted, MLB data guru Tom Tango clarified in a blog post that the changeover from PITCHf/x to Statcast had altered the way pitch speed was recorded, making it appear that velocity had increased. An MLB Advanced Media spokesperson who requested not to be identified said “the transition saw unexpected issues that have been resolved,” but declined to comment further.
And tracking velocity was only the beginning of Statcast’s troubles. Real-time data from MLB’s Gameday app has been inconsistent or obviously erroneous in the season’s first month. Some days, it has gone missing altogether, only to reappear later without explanation. Statcast has always had gaps in its data, but in previous years, that missing information was limited to batted-ball velocity and launch angle. The pitch-tracking issues that cropped up this year are in dramatic contrast with those we saw from PITCHf/x, which tended to miss only a handful of throws a season.
Even if you focus solely on the pitches that Statcast successfully tracks, its measurement error is much higher than PITCHf/x’s was. We can tell whether a park is systematically measuring pitches incorrectly by looking at the average vertical and horizontal coordinates of pitches there. If the data from a particular park tends to always be a bit high or a bit outside compared to when the same pitchers throw at other parks, it’s likely that the measurements are off. And according to models I built to measure the systematic error in each ballpark,1 the new system is struggling to determine where the ball crosses the plate. Here’s what those errors look like when averaged across the league.
Errors in both horizontal and vertical movement have never been higher in the four years that Statcast has made some of its data publicly available.2 So it’s not just your imagination as you watch the game on TV: In-broadcast representations of the strike zone (like FoxTrax) take their data from Statcast, and Statcast’s errors, in turn, have bred anger with umpires and confusion over how pitches are being called.
Statcast runs into the most trouble when it’s quantifying pitch break, or the degree to which pitches move up and down or side to side as they travel between the mound and the plate. Third-party observers have catalogued numerous inaccuracies with Statcast’s break numbers. “It appears that the current Statcast/TrackMan h[orizontal]/v[ertical] break can be up to 3 inches divergent from the truth, simply comparing it to 2016 PITCHf/x data,” said Kyle Boddy, a data-driven trainer with multiple MLB clients.
Even the average Statcast-reported break number is about an inch off. Some readings are especially egregious: One pitch was originally reported to have arced upward more than 20 inches on its trip from the mound to the plate. The combination of errors in velocity and break have rendered some pitches impossible to classify, further confusing sabermetricians.
Making matters worse, some ballparks show much larger errors than others. So far this season, Atlanta’s brand-new SunTrust Park appears to have the most accurate vertical break numbers, only off by two-tenths of an inch on average. Meanwhile, Cincinnati’s Great American Ball Park shows the worst errors, missing by an average of 2.4 inches per pitch. So not only are the errors bigger than in the days of PITCHf/x, they’re also more inconsistent: Last year, every park’s errors ranged from 0.04 to 1.4 inches.
Park-specific calibration errors such as these may explain other aberrant MLB trends. Despite the aforementioned league-wide hike in measured velocity, Chicago Cubs starters have registered lower fastball velocities than last year, sparking concern among Cubs fans. Writers have pointed to poor starts by Chicago pitchers as evidence that the velocity drop-offs are real, and even suggested that it could be part of a conscious effort by Cubs pitchers to decrease fatigue. But the far simpler explanation is bad data: If the club’s pitch tracker is poorly calibrated, it could make it look like pitchers are losing velocity when in fact the readings are just wrong. Supporting this idea is the fact that opposing teams’ pitchers in Wrigley have also registered a lower raw velocity than average. Unless the Cubs’ velocity woes are contagious, it seems likely that Statcast errors are driving some of their low numbers.
The root cause of Statcast’s troubles is unknown. The problems could originate in the hardware, the computer code processing the resulting data, or any other part of a complex system. The hardware part of Statcast — the part that actually tracks pitches — is a radar system sold by a company called TrackMan. Boddy’s company, Driveline Baseball, maintains their own TrackMan machine and has previously characterized its performance. “It is well-known in the industry that TrackMan has a lot of calibration issues, especially in nonstandard deployments.” Boddy said. For a radar system that works best in empty environments, it hardly gets less standard than trying to take measurements in a crowded MLB stadium on game day.
The good news is that MLB could learn from the last major technological innovation it deployed. When PITCHf/x first came out in the postseason of 2006, there were major issues with its initial calibration. “The data was open sourced and required tons of work from the public sphere to massage and get right,” Boddy said. “It was years before the data stabilized, and MLBAM has public analysts to thank for doing tons of free work.” But in contrast to a decade ago, MLB is now providing very little detail about Statcast’s internal workings. Without greater disclosure from MLB, it’s impossible to know what issues Statcast is having, or when they may be resolved. (At times, their own analysts appear to find out about changes to the public data after the fact.) Until Statcast improves, television viewers and sabermetricians alike will have to take pitch-tracking measurements with a grain of salt.