scrapy抓取动态页面

Posted on | In

一、准备工作

1.安装splash

在windows环境下，splash可通过docker进行安装，安装方法在 这篇文章 中有详细讲解，在此不再赘述。

安装完成后在powershell中运行以下命令安装splash：

1
2
#从docker hub下载相关镜像文件
docker pull scrapinghub/splash、

使用docker启动服务命令启动Splash服务，在8050、8051、5023端口开启splash服务：

1
2
3
4
#启动splash服务，并通过http，https，telnet提供服务
#通常一般使用http模式 ，可以只启动一个8050就好  
#Splash 将运行在 0.0.0.0 at ports 8050 (http), 8051 (https) and 5023 (telnet).
docker run -p 5023:5023 -p 8050:8050 -p 8051:8051 scrapinghub/splash

在浏览器中输入localhost:8050查看服务启动情况：

img

2.安装scrapy-splash

使用pip安装scrapy-splash：

1
pip install scrapy-splash

完成以上步骤之后就可以在scrapy中使用splash了

二、编写爬虫

动态页面中的部分内容是浏览器运行页面中的javascript脚本生成的，相较于普通静态页面，爬取难度更大。以网站 http://quotes.toscrape.com/js/为例，爬取动态生成的名人名言信息。

在项目文件settings.py中配置scrapy-splash：

1
2
3
4
5
6
7
8
9
10
#定义splash服务器地址
SPLASH_URL = 'http://10.0.75.1:8050/'
#开启splash的下载中间件
DOWNLOADER_MIDDLEWARES = {
    'scrapy_splash.SplashCookiesMiddleware': 723,
    'scrapy_splash.SplashMiddleware': 725,
    'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}
#设置去重
DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'

编码实现：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# -*- coding: utf-8 -*-
import scrapy
from scrapy_splash import SplashRequest

class QuoteSpider(scrapy.Spider):
    name = 'quote'
    start_urls = ['http://quotes.toscrape.com/js/']

    def start_requests(self):
        for url in self.start_urls:
            yield SplashRequest(url, args={'images': 0})

    def parse(self, response):
        quotes = response.xpath('//div[@class="quote"]')
        for quote in quotes:
            text = quote.xpath('.//*[@class="text"]/text()').extract_first()
            author = quote.xpath('.//*[@class="author"]/text()').extract_first()
            yield {
                'text': text,
                'author': author,
            }
        next_page = response.xpath('//a[contains(text(),"Next")]/@href').extract_first()
        if next_page:
            absolute_url = response.urljoin(next_page)
            yield SplashRequest(absolute_url, args={'images': 0})

执行结果：

1
2
3
4
5
6
7
8
9
10
2018-02-14 15:17:31 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/js/>
{'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”', 'author': 'Albert Einstein'}
2018-02-14 15:17:31 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/js/>
{'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”', 'author': 'J.K. Rowling'}
....

2018-02-14 15:17:38 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/js/page/10/>
{'text': "“A person's a person, no matter how small.”", 'author': 'Dr. Seuss'}
2018-02-14 15:17:38 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/js/page/10/>
{'text': '“... a mind needs books as a sword needs a whetstone, if it is to keep its edge.”', 'author': 'George R.R. Martin'}

结果显示我们已成功爬取动态生成的名人名言信息。