如果你使用 ubuntu，依赖以下二进制包：apt-get install python python-dev python-distribute python-pip libcurl4-openssl-dev libxml2-dev libxslt1-dev python-lxml

启动 ： pyspider all
脚本编辑：on_start 是脚本执行的入口， 点击run 会被调用
@every 表示这个任务是定时执行
self.crawl: 用于控制抓取连接， 使用callback = self.index_page指定用哪个函数解析抓取到的页面
def index_page(self

Read more

非堵塞io实现http请求

import socket, time 

from urllib.parse import urlparse 

from selectors import DefaultSelector, EVENT_READ, EVENT_WRITE 


# select +  回调 + 事件循环
# 并发性高
# 单线程模式/占用内存小
selector = DefaultSelector()

stop = False
# windows下会选择pool linux下会自动选择epool
urls = []

使用select完成http请求

class Fetcher:

Read more

Mozilla/5.0 (Linux; Android 6.0.1; HUAWEI RIO-AL00 Build/HuaweiRIO-AL00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/49.0.2623.105 Mobile Safari/537.36 JsSdk/2 VideoArticle/6.0.0 NetType/wifi
Mozilla/5.0 (Linux; Android 4.4.4; MI 2 Build/KTU84P) AppleWebKit/537.36 (KHTML, l

Read more

Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (iPhone; CPU OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.

Read more

Xpath表达式

/
    代表从顶端开始寻找标签一层层寻找
    例子:/html/head/title
    text()
    提取文本信息
    例子:/html/head/titile/text()
    @[]
    提取属性
    例子://li[@class=””]/a/@href       标签[@属性=值]
    //
    寻找所有标签
    例子://li
    / /*
    例子/div/* 选取属于div元素的所有子节点
    //*
    例子//* 选取所有元素
    //div[@*]
    选取所有带属性的title元素

Read more

Mozilla/5.0
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 1.0.3705; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, lik

Read more