aiohttphandler

Time:2021-2-27

This module is self encapsulated. The client uses aiohttp to encapsulate the handler, and the server uses fastapi and aiofiles.

But if the log volume is too large, it still can’t hold

client

log.py

# -*-  coding:utf8 - *-

config.py

import logging

Server

log_server.py

# -*- coding: utf-8 -*-

Supervisor deployment server

/etc/supervisor/conf.d/log_server.conf

[program:server_5001]
directory = /home/ymserver/vhost/ym-crawler-ccs
command = /home/ymserver/.virtualenvs/ym-crawler-ccs/bin/gunicorn -w=10 -k uvicorn.workers.UvicornH11Worker utils.log_server:app -b 172.19.43.201:5001
process_name = %(program_name)s_%(process_num)02d
numprocs = 1
autostart = true
startsecs = 5
autorestart = true
killasgroup=true
stopasgroup=true
startretries = 3
user = ymserver
stdout_logfile_maxbytes = 0
stderr_logfile_maxbytes = 0
stderr_logfile = /data/log/ym-crawler-ccs/scripts/http_handler_server/server_5001.err.log
stdout_logfile = /data/log/ym-crawler-ccs/scripts/http_handler_server/server_5001.out.log
ding_at_mobiles=13763315112

[program:server_5002]
directory = /home/ymserver/vhost/ym-crawler-ccs
command = /home/ymserver/.virtualenvs/ym-crawler-ccs/bin/gunicorn -w=10 -k uvicorn.workers.UvicornH11Worker utils.log_server:app -b 172.19.43.201:5002
process_name = %(program_name)s_%(process_num)02d
numprocs = 1
autostart = true
startsecs = 5
autorestart = true
killasgroup=true
stopasgroup=true
startretries = 3
user = ymserver
stdout_logfile_maxbytes = 0
stderr_logfile_maxbytes = 0
stderr_logfile = /data/log/ym-crawler-ccs/scripts/http_handler_server/server_5002.err.log
stdout_logfile = /data/log/ym-crawler-ccs/scripts/http_handler_server/server_5002.out.log
ding_at_mobiles=13763315112

[program:server_5003]
directory = /home/ymserver/vhost/ym-crawler-ccs
command = /home/ymserver/.virtualenvs/ym-crawler-ccs/bin/gunicorn -w=10 -k uvicorn.workers.UvicornH11Worker utils.log_server:app -b 172.19.43.201:5003
process_name = %(program_name)s_%(process_num)02d
numprocs = 1
autostart = true
startsecs = 5
autorestart = true
killasgroup=true
stopasgroup=true
startretries = 3
user = ymserver
stdout_logfile_maxbytes = 0
stderr_logfile_maxbytes = 0
stderr_logfile = /data/log/ym-crawler-ccs/scripts/http_handler_server/server_5003.err.log
stdout_logfile = /data/log/ym-crawler-ccs/scripts/http_handler_server/server_5003.out.log
ding_at_mobiles=13763315112


[program:server_5004]
directory = /home/ymserver/vhost/ym-crawler-ccs
command = /home/ymserver/.virtualenvs/ym-crawler-ccs/bin/gunicorn -w=10 -k uvicorn.workers.UvicornH11Worker utils.log_server:app -b 172.19.43.201:5004
process_name = %(program_name)s_%(process_num)02d
numprocs = 1
autostart = true
startsecs = 5
autorestart = true
killasgroup=true
stopasgroup=true
startretries = 3
user = ymserver
stdout_logfile_maxbytes = 0
stderr_logfile_maxbytes = 0
stderr_logfile = /data/log/ym-crawler-ccs/scripts/http_handler_server/server_5004.err.log
stdout_logfile = /data/log/ym-crawler-ccs/scripts/http_handler_server/server_5004.out.log
ding_at_mobiles=13763315112

[group:log_server]
programs=server_5001,server_5002,server_5004,server_5003
priority=999

Nginx load balancing

/etc/nginx/sites-enabled/log-server.conf

upstream backserver {
        server 172.19.43.201:5001;
        server 172.19.43.201:5002;
        server 172.19.43.201:5003;
        server 172.19.43.201:5004;
        server 172.19.43.201:5005;
}

server {
    listen 5000;
    server_name _;
    location / {
        proxy_pass http://backserver;
        proxy_set_header Host $host:$server_port;
    }
}

This work adoptsCC agreementReprint must indicate the author and the link of this article