Install owllook online novel search engine in CentOS 7

1. Introduction to owllook

Owlook is a vertical novel search engine built on other search engines. Owlook aims to make reading simpler and more elegant, so that every reader has a comfortable reading experience, such as book search, reading, collection, follow-up, recommendation and other functions.

owllook uses MongoDB to store basic information generated during user use, such as registration information, novel search information, novel collection data, etc. For some necessary caches, redis is used for cache processing, such as novel cache and session cache. Note that for restricted data: all will be deleted within 24 hours

For novels on different websites, the page rules are not the same. I hope to be able to display them uniformly after code analysis, which is convenient and beautiful, instead of just jumping to the corresponding website. A fresh and concise reading experience is the most important.

Currently, we use search engines to retrieve results directly. I try to write a small number of rules to complete the analysis. For details, see the rule definition. If you encounter a novel website you like, you can also add your own analysis. owllook currently analyzes more than 200+ websites and 50+ chasing websites.

There are some places where crawlers are needed, such as rankings, some book information, etc. I don't want to use a heavyweight crawler framework to write it, so I wrote a very lightweight crawler framework in owllook to do this, see ruia

2. Installation of owllook on CentOS 7 system

Update the system first:

 yum -y update

Install the EPEL repository:

 yum -y install epel - release

Install Redis and some basic components:

 yum -y install redis git curl screen

Now run and set Redis to start automatically at boot:

 systemctl start redis
systemctl enable redis

Create a new repo source for installing mongodb:

 vi /etc/yum.repos.d/mongodb-org-4.0.repo​​​​​​​​​​​​​​​

Write:

 [ mongodb - org - 4.0 ]
name = MongoDB Repository
baseurl = https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/4.0/x86_64/​​
gpgcheck = 1
enabled = 1
gpgkey = https://www.mongodb.org/static/pgp/server-4.0.asc​​

Then install it directly with yum:

 yum -y install mongodb - org

Start mongodb and set it to start at boot:

 systemctl start mongod
systemctl enable mongod

Install the dependencies required to compile and install Python3:

 yum - y install gcc make zlib zlib - devel readline readline - devel readline - static openssl openssl - devel openssl - static sqlite - devel bzip2 - devel bzip2 - libs libffi - devel

Run the following script to install pyenv:

 curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash

To edit the account configuration file:

 vi ~/ .bash_profile

Add the following environment variables:

 export PATH = "/root/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

To make it effective:

 source ~/ .bash_profile

Install python3.6.7 using pyenv:

 pyenv install 3.6.7​

After completion, create a 3.6.7 virtual operating environment:

 pyenv virtualenv 3.6.7 owllook​

Remember your environment path as follows:

 /root/.pyenv/versions/3.6.7/envs/owllook/bin/python3​​​​​​​​​​​​​​​​

Now pull the project and go into the project directory:

 git clone https://github.com/howie6879/owllook
cd owllook

Set the Python environment of the project directory to the one we just created:

 pyenv local owllook

Install pipenv using pip:

 pip install pipenv

Then use pipenv to install the dependencies required for the project:

 pipenv install --python / root / .pyenv / versions / 3.6.7 / envs / owllook / bin / python3​​

The dependency installation is complete as shown in the figure. Please make sure there are no errors in the process:

After completion, enter the owllook directory:

 cd owllook

Edit the configuration file:

 vi config / config . py

Find the following code:

 VAL_HOST = os . getenv ( 'VAL_HOST' , 'true' )

to:

 VAL_HOST = os . getenv ( 'VAL_HOST' , 'false' )

Now create a new shell using screen:

 screen - S owllook

Run the program in a new shell:

 python server . py

If nothing unexpected happens, you can see a display similar to the following, which means the project is running:

Now we install Nginx for reverse proxy:

 yum -y install nginx

Create a new Nginx site configuration file:

 vi /etc/nginx/conf.d/book.conf​​​​​​​​​​​

Write:

 server {
    listen 80 ;
    server_name example . com www . example . com ;

location / {
    proxy_set_header X - Real - IP $remote_addr ;
    proxy_set_header X - Forwarded - For $proxy_add_x_forwarded_for ;
    proxy_set_header X - Forwarded - Proto $scheme ;
    proxy_set_header Host $http_host ;
    proxy_set_header X - NginX - Proxy true ;

    proxy_pass http://0.0.0.0:8001 ;
    proxy_redirect off ;

    proxy_http_version 1.1 ;
    proxy_set_header Upgrade $http_upgrade ;
    proxy_set_header Connection "upgrade" ;
    }
}

Restart Nginx:

 systemctl restart nginx

Disable firewall and SElinux:

 systemctl stop firewalld . service
systemctl disable firewalld . service
vi / etc / selinux / config
SELINUX = disabled
setenforce 0

Now open your site domain name and you should be able to access this program normally:

Show leaderboard:

Run spider_console.py:

 cd owllook / owllook / spiders

Then create a new shell using screen:

 screen - S spider_console

Execute in a new shell:

 python spider_console.py​

Owlook project address: https://github.com/howie6879/owllook

The full installation tutorial is transferred from: https://lala.im/4538.html