craigslist

Scripts I use to get decent results from craigslist.

Download last N listings from craigslist
Find dups by exact title match
Attempt to grab search data from content (price, bedrooms, etc) and stuff into elasticsearch
Search based on my criteria
WIN

Requirements

Things I probably wont do since before actually finding an apartment

When analyzing listings, search for images within a listing, stream their contents, hash the contents and index those hashes. Use this to create a "unique images ratio" and consider anything less than a threshold ratio to be "duplicate". For instance if a listing has 10 images, but 9 of them appear in other listings consider this listing a duplicate and do not index it for search.
Learn how to create a better search using elasticsearch. Scoring, boosting, etc.
Make the load/analyze process 1 script/class
Use Twitter Storm to stream/analze in new listings and send new listings to myself via email when they match my criteria

Name		Name	Last commit message	Last commit date
Latest commit History 2 Commits
00_table.rb		00_table.rb
01_load.rb		01_load.rb
02_date.rb		02_date.rb
03_title_dups.rb		03_title_dups.rb
04_es.rb		04_es.rb
05_es_gimme.rb		05_es_gimme.rb
Gemfile		Gemfile
Gemfile.lock		Gemfile.lock
README.md		README.md