Open data has contributed to dramatic improvements in a wide array of fields over the past few decades, affecting how we look at astronomy, genetics, climate change, sports and more. But until recently, crime has gone without the open analysis prevalent in other fields because crime data has been closely held by law enforcement agencies and has usually only been released in bulk at monthly, quarterly or annual intervals.
Now, thanks to efforts from the federal government and individual municipalities, crime analysis is positioned for a leap forward as cities place unprecedented quantities of data online. This data can be analyzed to help residents understand more about crime in their cities — for example, I recently used open data from New Orleans and Baltimore to separate shooting incidents from murders to get a fuller picture of gun violence in those cities.
Since the 1930s, publicly available crime data has traditionally revolved around the FBI’s Uniform Crime Reports. Police departments aggregate major crimes into eight categories known as UCR Part I1 and send that information to the FBI. The FBI collects the information and then publishes it many months later.2 That means official UCR statistics not only take a long time to come out but also reflect tallies made by the agencies themselves — and provide only a snapshot of selected major crimes over a given period.
Enter the Police Data Initiative. Born out of recommendations from President Obama’s Task Force on 21st Century Policing, the initiative was launched in May to encourage police departments to “better use data and technology to build community trust.” As of late November, 27 agencies had committed to providing public access to law enforcement data as part of the initiative. In addition, several large U.S. cities that are not actively participating in the Police Data Initiative have open data portals providing crime information.
Together, these efforts present an opportunity to change the way cities and residents view crime and police work by adding nuance and speed to the data analysis process. The data being made available includes information on crime location, suspects and perpetrators, police response and officer-involved shootings, although what cities report varies. All this information can be used to analyze local crime patterns in near-real time without waiting for law enforcement agencies to aggregate statistics and send them to the FBI for publication. For example, it is now possible to evaluate whether the Chicago Police Department is arresting a higher percentage of armed robbers this year.
But the quantity and quality of open crime data vary widely among cities, highlighting the need for better standards in data collection and reporting. Four of the cities that have joined the Police Data Initiative — Charlotte, North Carolina; Indianapolis; Newark, New Jersey; and Orlando, Florida — have not yet begun placing current incident data online, are providing only historical data, or are posting information on only certain types of incidents, such as when a department’s officers use force against a suspect.
And not all cities have embraced transparent crime data yet. Only 223 of the 79 American cities with a population of around 250,000 or more4 provide regularly updated incident data for public consumption. Of those 22 cities, 11 are formal members of the Police Data Initiative (they are Atlanta; Austin, Texas; Cincinnati; Dallas; Denver; Los Angeles; Louisville, Kentucky; New Orleans; Oakland, California; Philadelphia; and Seattle).
I wanted to compare the quantity and quality of publicly available crime incident data that cities are placing online, so I evaluated those 22 cities that provide regularly updated incident data and plotted the results in the chart below. (Among the cities that are providing regularly updated open crime data but are not participating in the Police Data Initiative are Boston; Kansas City, Missouri; and Pittsburgh.) The plot shows the average number of rows of data on crime incidents that each city places online per year versus the data’s quality in terms of the number of columns of information the city provides to describe each incident.
Most of these 22 cities providing transparent data at a minimum release incident information on individual UCR Part I and Part II5 offenses. The information typically includes an incident’s date, the type of offense, and some locational information at a minimum. Some cities release additional information. Boston, for example, identifies whether an incident was domestic and during which police shift it took place. Chicago has provided nearly 6 million records dating back to 2001, enabling analysis of longer-term crime trends.
Some of these cities provide data that is either poorly organized or does not go back very far. Oakland, California, has placed “calls for service” (essentially the metadata record of every action a police officer takes on the street) online in six-month increments from January 2013 through mid-October 2015. But the value of this information is limited because for each call, Oakland provides only an incident number, a description of the type of call (e.g., “ambulance requested”), and the date, leaving potential analysts with simply a list of calls to which the Oakland Police Department responded.
New Orleans is arguably the queen of open police data, at least in terms of the quantity of information it provides. New Orleans has placed its entire calls for service system online since 2011, offering the public a significant opportunity to evaluate its police force. A recent analysis of the data by several local media outlets found that the New Orleans Police Department’s response times had increased dramatically. Longer response times had led to fewer investigations, which in turn had deflated crime statistics by around 6 percent. After the news reports, the police department changed its policy regarding how crimes were reported.
Many others of the 22 cities place more modest quantities of high-quality data online. Sacramento, California, has provided several hundred thousand calls for service dispatches annually for the past three years. Sacramento’s open data portal is also impressive in the amount of detail it provides about each call for service — including when it occurred, when an officer arrived on the scene, the reporting officer’s badge number, and 22 additional data columns — but only goes back to 2013.
Dallas provides a staggering 97 columns of data. These — among other things — may describe injuries to a victim and his condition, or whether police say a gang was involved, or if a crime was drug-related. This detailed crime data goes back to June 2014, covering 145,000 UCR Part I and Part II incidents; the city has separately posted crime data with half as many columns covering January 2013 through May 2014.
Crime data for the 57 large U.S. cities that don’t post regularly updated information is of varying quality and usefulness for analyzing crime problems.
Some of the cities that produce only aggregated data do provide residents with other useful information. St. Louis has published a monthly crime summary from mid-2009 through October 2015 that contains detailed information on crime by neighborhood. Minneapolis produces a weekly map of calls for service for gunshots, in addition to providing aggregated UCR statistics for each month dating to 2001.
At the other end of the spectrum, several crime-heavy cities provide little crime data. Orlando’s police department currently provides only a running list of about 30 calls for service to which the department is actively responding and no other crime data or statistics.
Memphis had the second-most UCR Part I violent crimes per 100,000 residents in 20146 but similarly provides little publicly available crime data. While the Memphis Police Department’s website indicates that three annual reports — from 2009, 2010 and 2012 — can be accessed there, only the link to the 2012 report works.
Cities would be wise to place as much of their crime data online as possible — as New Orleans and Sacramento have — so that residents can analyze, understand crime trends and evaluate their police. Standardizing what data police departments provide the public and publicly highlighting the good this level of transparency can accomplish would maximize the impact of the open data movement.