Friday, March 23, 2012

Real Time Data Mining And The Future Of Policing

There was a recent discussion over at the IACA mailing list that kicked around the topic of predictive policing. One of the posts that started it included an article from The Atlantic Cities that discussed Memphis Police and their effort at real time data mining of crime data.

About seven years ago, researchers from the University of Memphis approached the city’s police department with the idea that they might be able to detect patterns in local crime – geographic hot spots on the city’s map and moments in time when they’re most likely to flare up – if they could just have access to the department’s crime data. Police departments produce reams of this stuff: arrest warrants, crime-scene reports, traffic citations, mug shots, dispatch transcripts and incident times. But that data has traditionally been painstaking to cross-reference, to mine for connections and even future trends.

The researchers ultimately turned the department onto an analytic software called SPSS, which had for years been used to crunch data in a host of disciplines not necessarily connected to crime. The department launched a pilot program with it to analyze trends, as part of a strategy of fighting crime by real-time data-mining.

I have been quite intrigued with being able to mine the crime data at my agency for meaningful information. We really do warehouse vast amounts of data. The tools to mine the data are getting more accessible all the time. Even your trusty Microsoft Excel is capable of some very sophisticated analysis.

Today I was working on a project to analyze over a half million Calls for Service records with Excel. I have also been experimenting with using the open source statistical software R, and a data mining GUI interface for it called Rattle.

I think that the future of crime analysis lies with technologies like data mining and predictive policing. Thankfully, there are lots of folks out there working on this very thing. What are you doing to become a data miner?

A tip of the hat to crime analyst Tom Scholten for posting the Atlantic Cities story.


  1. Scott,
    I have just started following your blog, and it has been very informative. Regarding this post, is there any research out there to suggest that a certain crime threshold is necessary in order for data-mining and predictive policing to be effective? It's a given that crime analysis can be applied anywhere- a few people and a few places are always going to be responsible for larger percentages of crime. But for smaller jurisdictions with lower levels of crime trying to be "big city," can there come a point when they try to do too much with too little? Wanting the Intel Core i7 when all they really need is the i3, so to speak.

  2. Tim,
    I haven't seen any yet. However, like in anything regarding statistics, the more data you have to base your analysis on, the more accurate it's likely to be.


I reserve the right to remove defamatory, libelous, inappropriate or otherwise stupid comments. If you are a spammer or are link baiting in the comments, a pox be upon you. The same goes for people trying to sell stuff. Your comment will be deleted without mercy.