HCS Blog

I’ve finally finished getting all the data imported and geocoded. There were a few months with obvious spikes in data, and a closer look revealed a bunch of weird dates. Such as 1976, 2000, etc. I can only guess these are erroneously entered dates which are normally filtered out… After cleaning those out it looks much better, and I will go back at some point to verify the other months don’t have such junk added in - but in the meantime it’s really not hurting anything.

I’ve also changed the monthly history graph and data to display a daily average instead of monthly totals. This makes the output more fair considering months can have anything from 26-31 days so it previously could appear a month was better or worse than another when it might actually not have been.

This coming weekend I should get some headway on bringing back a maps interface and will add premise stats and some other interesting data I think will be appealing. I doubt it will be done this weekend considering I have several other plans already - not to mention puppy-sitting for some newly-wed friends who will be on their honeymoon.

Stay tuned for more updates soon!

I’m moving the site to a different server - for more room and speed.

I haven’t had time to add much, except that all the data from Jan 2005 through Dec 2006 is now online and I’m re-geocoding each month using the new process to increase the number of entries with coordinates.

As mentioned in the very first post, the map interface has been removed - currently you are only able to receive a historical graph along with numbers of the total incidents for each month. Once I finish getting all the data re-geocoded and then get all of 2007’s data imported and geocoded, I will begin working on a new maps interface as well as more details and breakdown of the results for your inquiring minds.

Stay tuned!

I’ve updated some of the checks the geocoding process uses, as well as added a google maps API geocode lookup, with several different tests (mostly for wording/spelling, same as what I use for geocoder.us - such as changing FWY to Freeway)

I’m getting much better results now, and will go back and re-check the entries with no coordinates to see if I can bring down the failure percentage rates. I’m adding a DB Stats link on the home page so you can see the total number of records for a given month, along with the number of records that have no geocode info - and the resulting failure rate.

I’ve gotten the databases updated through Dec. 2006 and will work on 2007 in the coming week… Right now, I’ve started geocoding the recently added months, and you should see them slowly coming online soon!

Further checking the raw numbers indicates my failure rate for geocodes is currently about 12-13% (Number of total incidents per month compared to how many do NOT have map coordinates) this affects the overall accurateness (yes, that’s a word) as well as what shows up on a map or in a chart based on a given address and/or zip code. I’ll be working on this for sure, it’s gotta get better than that!

Yes, HoustonCrimeStats.org has been left behind - hardly any work has been done to it in over a year… So, my new years resolution is to restore this service to working order and slowly, as time permits, make changes and improvements.

The first step was to correct some permission issues, and to clear out the old temp files created when users generated a chart - I never got around to writing a cleanup script to wipe them out, this eventually caused problems ovbiously.

Now, I’ve decided to start this blog to keep everyone up to date on what’s happening with the site and any changes that are made.

Next, I plan on getting the crime stats data updated (current data stops in Mar 06) and will work on keeping it up to date after that. In the meantime, the google maps API will be removed because it needs to be completely re-done. In the meantime, only historical charts and monthly totals will be available.

I hope to eventually take a closer look at the geocode process I originally created and see if I can’t come up with a better (and hopefully faster) method. Right now it takes at least an hour to geocode a months worth of incidents and has a high failure rate even with multiple lookups and variations performed on the address.

Stay tuned for more… I hope to get the majority of the little things done in the next few weeks, but finishing the data imports might take longer as I don’t plan to sit in front of the computer waiting on it only to fire up the next month for geocoding.

Thanks for your support, ttyl!