Request and log analysis in scrapy

Time:2020-1-13

Scrapy.http.Request

Automatic de duplication, according to the hash value of the URL, de duplication

 

attribute

Meta (dict) transfers data between different requests, Dict

Priority (int) the priority of this request (0 by default)

Don? Filter (Boolean) turn off automatic de duplication

Errback (callable) the function that will be called when any exception is thrown while processing the request

Flags (list) flag sent to the request, which can be used for logging or similar purposes

 

Method

Copy() returns a new request, which is a copy of the request

Replace() returns the request of an update pair

 

 

 

FormRequest

A formrequest class is built in the framework of the scratch, which extends the base class request and has the function of processing HTML forms

When post requests, use formrequest to pass data into the formdata parameter

 

 

 

Log usage

Log file log output file, if none, is printed on the console

Log? Enable log? True by default

Log & encoding date code, default UTF-8

Log level, default debug

Log & format log format

Log “dateformat” log date format

Log_stdout log standard output, default false, if true, all standard output will be written to the log

Log ﹣ short ﹣ names. The default is false. If true, the component name will not be output

 

Generally only added in settings profile