In the previous lecture I described scraping use cases, but in this lecture I want to show you.
Why I'm Leaving San Francisco (meta-study) One afternoon I got bored and decided to scrape 100s of blog posts by former SF residents who moved away. With the same tools taught in this course, I managed to find some interesting insights in the data.
My blog post and story went viral, appearing on the homepage of popular websites like Digg, and even scoring me a feature interview with the San Francisco Chronicle.
19,475 emails in 20 lines of code In this case I leveraged a tiny bit of web dev knowledge, specifically HTTP and Cookies, to spoof the login credentials for all ~20,000 members of a wholesale industry forum.
This was an awesome marketing win for my team at the time (a logistics/shipping startup) and an excellent real-world application combining the power of HTML, CSS, HTTP, and Cookie knowledge. No servers or databases were used, I simply let my web browser download the data straight to a CSV, something we learn together in this course.
Product Hunt Upvote Scraper One of the first scripts I ever wrote was to figure out which people have upvoted a given product on Product Hunt. I wanted this knowledge so I could target them in twitter advertising campaigns, etc (by their user handle).
Making this scraper was also my foray into web dev. To get it live for non technical people to use I learned how to quickly add scripts (such as the ones we'll build in this course) to a thin Sinatra server, then deploy it for free to Heroku.com. The web version of this scraper is included in the GitHub docs above.
Student Contributions Here I'd like to highlight students who built their own scraping solutions. Please email me or comment below to share yours!
Ryan Doyle -- scraping and hacking (dating app auto-matches, sales messaging, researching funded companies)
Alex Golovatenco - administrative bot that automatically finds free appointment slots on a foreign embassy's website (useful for tourists)
