Request and log analysis in scrapy



Automatic de duplication, according to the hash value of the URL, de duplication



Meta (dict) transfers data between different requests, Dict

Priority (int) the priority of this request (0 by default)

Don? Filter (Boolean) turn off automatic de duplication

Errback (callable) the function that will be called when any exception is thrown while processing the request

Flags (list) flag sent to the request, which can be used for logging or similar purposes



Copy() returns a new request, which is a copy of the request

Replace() returns the request of an update pair





A formrequest class is built in the framework of the scratch, which extends the base class request and has the function of processing HTML forms

When post requests, use formrequest to pass data into the formdata parameter




Log usage

Log file log output file, if none, is printed on the console

Log? Enable log? True by default

Log & encoding date code, default UTF-8

Log level, default debug

Log & format log format

Log “dateformat” log date format

Log_stdout log standard output, default false, if true, all standard output will be written to the log

Log ﹣ short ﹣ names. The default is false. If true, the component name will not be output


Generally only added in settings profile