Scrapy 抓取 Mediuem 网站 Android 标签下的文章

前面实现了对 简书上面的程序员专题抓取文章, 接下来开始 Medium 网站的 Android 标签的文章。

medium_spider.py 代码:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
#!/usr/bin/env python
# -*- coding:utf-8 -*-
import scrapy
class MediumSpider(scrapy.Spider):
"""docstring for MediumSpider"""
name = "mediumspider"
start_urls = ["https://medium.com/tag/android"]
def parse(self, response):
for postitem in response.css('.postItem'):
url = postitem.css('article a::attr(href)').extract()[0]
# get title
if len( postitem.css('article h2::text').extract() ) != 0 :
title = postitem.css('article h2::text').extract()[0]
elif len( postitem.css('article h3::text').extract() ) != 0 :
title = postitem.css('article h3::text').extract()[0]
elif len( postitem.css('article h4::text').extract()) != 0 :
title = postitem.css('article h4::text').extract()[0]
else:
title = "Error get title"
# get subtitle
if len( postitem.css('.section-inner h4::text').extract()) != 0 :
subtitle = postitem.css('.section-inner h4::text').extract()[0]
elif len( postitem.css('.section-inner p::text').extract()) != 0 :
subtitle = postitem.css('.section-inner p::text').extract()[0]
else:
subtitle = "No subtitle"
yield {
"title":title,
"subtitle":subtitle,
"url":url,
}

运行 scrapy runspider medium_spider.py -o res.json ,并且导出 json文件。
res.json

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
[
{
"url": "https://medium.com/sebs-top-tips/analyse-data-flows-without-the-debugger-android-studio-protips-3-ef2885aaffd9?source=tags---------1",
"subtitle": "Did you know that you can figure…",
"title": "Analyse data flows without the debugger (Android Studio protips #3)"
},
{
"url": "https://medium.com/imgur-engineering/design-at-imgur-from-188-colors-to-12-bb6a1a8a26d9?source=tags---------2",
"subtitle": "Let’s take a little look back in time, through the history of Imgur. In 2009, Imgur started…",
"title": "Design at Imgur: From 188 Colors to 12"
},
{
"url": "https://medium.com/@raveeshbhalla/android-n-notifications-a-design-analysis-cec09f1cc5bf?source=tags---------3",
"subtitle": "It’s been a few days since Google surprised us all by releasing a developer preview for Android N, more than two months before Google I/O. As ",
"title": "Android N Notifications: A Design Analysis"
},
{
"url": "https://medium.com/@shollmann/picasso-universal-image-loader-or-glide-that-s-the-question-af34fa7f5e63?source=tags---------4",
"subtitle": "No subtitle",
"title": "Error get title"
},
{
"url": "https://medium.com/@cesarmcferreira/building-android-apps-30-things-that-experience-made-me-learn-the-hard-way-313680430bf9?source=tags---------5",
"subtitle": "No subtitle",
"title": "Building Android Apps — 30 things that experience made me learn the hard way"
},
{
"url": "https://medium.com/@hitherejoe/android-n-introducing-picture-in-picture-for-android-tv-35f2392fb609?source=tags---------6",
"subtitle": "Last week we saw the surprise release of the…",
"title": "Android N: Introducing Picture-in-Picture for Android TV"
},
{
"url": "https://medium.com/@nielsz/android-quality-with-mvp-espresso-junit-jacoco-and-sonarqube-3430d9ee4a4a?source=tags---------7",
"subtitle": "In my ",
"title": "Android Quality with MVP, Espresso, JUnit, JaCoCo and SonarQube"
},
{
"url": "https://labs.ribot.co.uk/android-application-architecture-8b6e34acda65?source=tags---------8",
"subtitle": "Our journey from standard Activities and AsyncTasks to a modern MVP-based architecture powered by RxJava.",
"title": "Android Application Architecture"
},
{
"url": "https://medium.com/@hamen/android-library-aar-and-javadoc-6859898cad28?source=tags---------9",
"subtitle": "As an Android developer, I’m used to ask Android Studio/Intellij Idea for documentation constantly. I have even replaced the shortcut: now, it’s F1, the old-fashion help key. When I want to know about a method or a class, I hit F1 and the JavaDoc shows up: smooth. How does…",
"title": "Android Library AAR and Javadoc"
},
{
"url": "https://medium.com/@p.tournaris/rxjava-rxreplayingshare-emit-only-once-b19acd61b469?source=tags---------10",
"subtitle": "Following up on my ",
"title": "RxJava —RxReplayingShare, Emit only Once"
}
]

文章来自: https://hanks.pub