How my web crawler(python, Scrapy, Scrapy-splash) can crawl faster? - mongodb

Question

Develop Environment:
CentOS7
pip 18.1
Docker version 18.09.3, build 774a1f4
anaconda Command line client (version 1.7.2)
Python3.7
Scrapy 1.6.0
scrapy-splash
MongoDB(db version v4.0.6)
PyCharm
Server Specs:
CPU ->
processor: 22,
vendor_id: GenuineIntel,
cpu family: 6,
model: 45,
model name: Intel(R) Xeon(R) CPU E5-2430 0 # 2.20GHz
RAM -> Mem: 31960
64 bit
Hello.
I'm a php developer, and this is my first python project. I'm trying to use python because I heard that python has many benefits for web crawling.
I'm crawling one dynamic web site, and I need to crawl around 3,500 pages in every 5-15 seconds. For now, mine is too slow. It is crawl only 200 pages per minute.
My source is like this:
main.py
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings
from spiders.bot1 import Bot1Spider
from spiders.bot2 import Bot2Spider
from spiders.bot3 import Bot3Spider
from spiders.bot4 import Bot4Spider
from pprint import pprint
process = CrawlerProcess(get_project_settings())
process.crawl(Oddsbot1Spider)
process.crawl(Oddsbot2Spider)
process.crawl(Oddsbot3Spider)
process.crawl(Oddsbot4Spider)
process.start()
bot1.py
import scrapy
import datetime
import math
from scrapy_splash import SplashRequest
from pymongo import MongoClient
from pprint import pprint
class Bot1Spider(scrapy.Spider):
name = 'bot1'
client = MongoClient('localhost', 27017)
db = client.db
def start_requests(self):
count = int(self.db.games.find().count())
num = math.floor(count*0.25)
start_urls = self.db.games.find().limit(num-1)
for url in start_urls:
full_url = domain + list(url.values())[5]
yield SplashRequest(full_url, self.parse, args={'wait': 0.1}, meta={'oid': list(url.values())[0]})
def parse(self, response):
pass
settings.py
BOT_NAME = 'crawler'
SPIDER_MODULES = ['crawler.spiders']
NEWSPIDER_MODULE = 'crawler.spiders'
# Scrapy Configuration
SPLASH_URL = 'http://localhost:8050'
DOWNLOADER_MIDDLEWARES = {
'scrapy_splash.SplashCookiesMiddleware': 723,
'scrapy_splash.SplashMiddleware': 725,
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}
SPIDER_MIDDLEWARES = {
'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
}
DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = 'my-project-name (www.my.domain)'
# Obey robots.txt rules
ROBOTSTXT_OBEY = False
# Configure maximum concurrent requests performed by Scrapy (default: 16)
CONCURRENT_REQUESTS = 64
# Configure a delay for requests for the same website (default: 0)
# See https://doc.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
CONCURRENT_REQUESTS_PER_DOMAIN = 16
When execute these code, I'm using this command: python main.py
After seeing my code, please help me. I'll happily listen any saying.
1.How my spider can faster? I've tried to use threading, but it's seem not working right.
2.What is the best performance line up for web crawling?
3.Is that possible to crawl 3500 dynamic pages in every 5-15 seconds?
Thank you.