Install Baidu Cloud Crawler on Centos 7

The crawler runs under MySQL, Python 2.7, and Mysql-python, so install MySQL and MySQL-python first.

1. Install MySQL
Install Dependencies

 yum install libaio

Install MySQL

 wget http://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm yum localinstall mysql-community-release-el7-5.noarch.rpm yum install mysql-community-server

Start MySQL

 systemctl start mysqld

Set MySQL password

 mysql_secure_installation;

2. Firewall settings <br />Install iptables

 yum install iptables-services

Open port 3306

 vi /etc/sysconfig/iptables

Add to

 -A RH-Firewall-1-INPUT -m state –state NEW -m tcp -p tcp –dport 3306 -j ACCEPT -A RH-Firewall-1-INPUT -m state –state NEW -m udp -p udp –dport 3306 -j ACCEPT

Restart iptables

 service iptables restart

3. Install MySQL-python

 yum install MySQL-python

4. Set up the program

 wget https://github.com/x-spiders/baiduyun-spider/archive/master.zip unzip master.zip cd baiduyun-spider-master

Set the account and password for connecting to the database

打開bin/spider.py ，修改DB_HOST、DB_PORT、DB_USER、DB_PASS

Running the crawler for the first time

 python bin/spider.py --seed-user

Run the crawler

 python bin/spider.py

Source code: https://geekspider.org/senior/215.html