How to realize web crawler with python

It's simple. I tried, just three steps, using the reptile framework scrapy.

Define project classification

Developing spiders (is the core)

Development pipeline

Look at the book Crazy python Lecture Notes, which is very helpful for learning python.