Web scraper tutorials with Python

Web scrapers are invaluable tools for journalists, because they allow us to automate the retrieval of data from websites, PDFs, and the like.  Once we’ve got that data tucked away in a spreadsheet or database, we can start working with it.  

Here’s Scraperwiki’s tutorials (in this case, scrapers written in the programming language Python).  

Kelli Shaver: The Family that Codes Together


This should be a lot of fun, assuming we don’t kill each other in the process.

I’ve wanted to get into Python for a while now, but haven’t had the time/a good excuse to do so. Richard is getting back into programming and has decided that Python is where he’s going to start getting re-acquainted….


#python tag gets me too many pictures of actual snakes.

140+: Visualizing changes in popularity rankings of programming languages


Drew Conway ranked programming language by popularity on GitHub and Stackoverflow in 2010, and many people like it. I wrote this post after Stephen O’Grady’s Ranking made a new ranking September 2012. He compares new data to the original data from 2010, but I am not impressed and doubt his…

In Praise of Impractical Programming

By Jacob Harris, late of the New York Times.  ”…if you want to advance as a programmer, you need to take some impractical detours.”

Demo of Google “Blockly,” a drag-and-drop programming environment.  Two minutes.  

Let's Get Louder

As a member of the Python community, we pledge only to attend, speak at, assist, sponsor, or otherwise participate in conferences that publicly promote an anti-harassment and anti-discrimination code of conduct policy.

Interesting discussion and progress in the Python community aimed at cutting down on harassment and bigotry at conferences. 

Scraping for Journalism: A Guide for Collecting Data


These recipes may be most helpful to journalists who are trying to learn programming and already know the basics. If you’re already an experienced programmer, you might learn about a new library or tool you haven’t tried yet.

Leah Culver talks Python and Django at Girl Geek Dinner SF.  Ten minutes. 

Teach Kids to Program!

I keep a running list of learn-to-code resources aimed at kids and teens in the Life And Code Resource Guide.  Here’s a selection, but I update it frequently: 

Hackety Hack — teaches kids to program in Ruby. Good for former children, too. 
Scratch — a programming language and environment developed for kids; designed to encourage kids to create games and animations. has great video tutorials for Scratch, delivered in a very fetching French accent.  
Snake Wrangling for Kids — Free downloadable e-book teaches programming in Python.  
Python4Kids — tutorials for kids age 8 and up. 
How to initiate kids — or anyone — in coding - great list of resources.
Intro to Scratch for Kids — Scratch is a visual programming environment created for kids.  Lets them make games and animations to share online.  It’s really great. 
KidsRuby Have fun making games or hack your homework with Ruby 
AgentSheets lets you create agent-based games via a drag and drop interface 
Alice is a 3D programming environment aimed at kids who want to create games and animations 
Hack the Future is a hackathon/self-guided hacking experience for people from the ages of 10 to 19

"I’m going to give you a piece of advice when you’re trying to learn something new: Never listen to people who try to make beginners feel like losers."

Please Don’t Become Anything, Especially A Programmer,” Zed Shaw, author of Learn Python The Hard Way, responding to Jeff Atwood’s “Please Don’t Learn To Code. 

How To Make Text Adventures

Making one of these sounds like a really fun beginner project. 


Schnell Style (but in Python)

If you have made a text adventure in your mind that you really like, and you’re tired of dictating it and would rather make it in a computer language, this page is for you! I’m teaching you a shortcut to make a text adventure, for those who actually know Python.

"Recently I saw somebody asked a question in a forum, the question is “Which programming language should I learn first?”. Then someone answered this question. His answer:
To program in an expressive and powerful language: Python
To get a website up quickly: PHP
To mingle with programmers who call themselves “rockstars”: Ruby.
To really learn to program: C.
To achieve enlightenment: Scheme.
To feel depressed: SQL
To drop a chromosome: Microsoft Visual Basic
To get a guaranteed, mediocre, but well paying job writing financial applications in a cubicle under fluorescent lights: Java.
To do the same thing with certifications and letters after your name: C#
To achieve a magical sense of childlike wonder that you have a hard time differentiating from megalomania: Objective C
I could go on… but I’m not feeling hateful enough today.
I don’t know whether you agree with him or not. According to his logic, I can add one to this
If you want to say fxxk everyday—JavaScript
To be a god which can do everything—Assembly

arums: Which programming language should I learn first? 
About Me

Lisa Williams

Founder of | Winner of Knight News Challenge | Center for Civic Media, MIT Media Lab | Cambridge, MA | @lisawilliams on Twitter | lisawilliams on Github