Nginx Log Custom Recording and Enabling Log Buffer Details

Time:2019-8-13

Preface

If you want to statistics the source information of the website visit, you can use PHP to get the information, record it in the form of database, or directly use the access log provided by nginx to record the details of the website visit. Administrators can analyze the source of users’visit and the details of users’ visit behavior by analyzing the access log of nginx. Site page access heat, etc. In addition, nginx itself also has error logs to facilitate operation and maintenance personnel to debug nginx. For logging behavior, if every disk operation, it will consume more resources. Based on this situation, the nginx log buffer can be opened. When the buffer is full or written regularly, it is time to write to the log again.

Access Log

Nginx writes information about client requests in the access log immediately after processing requests. By default, access the log bit logs/access.log, where information is written to the log in a predefined combination format.

To record access information accurately, you need to customize a more complete access log format, as follows:


http {
  log_format geoproxy
  '[$time_local] $remote_addr '
  '$realip_remote_addr $remote_user '
  '$request_method $server_protocol '
  '$scheme $server_name $uri $status '
  '$request_time $body_bytes_sent '
  '$geoip_city_country_code3 $geoip_region '
  '"$geoip_city" $http_x_forwarded_for '
  '$upstream_status $upstream_response_time '
  '"$http_referer" "$http_user_agent"';
  ...
}

This log configuration, named geoproxy, uses many nginx variables to demonstrate the logging function of nginx. Explain in detail the specific meaning of each variable in the configuration options:

When a user initiates a request, the server time is recorded, $time_local, $remote_user value is the user name authorized by the basic authority.

Open connection IP address and client IP address for nginx processing geoip_proxy and realip_header instructions;

Then record HTTP request method $request_method, protocol $server_protocol and HTTP method $scheme: HTTP or https.

Of course, there are server name $server_name, request URI and response status code.

In addition to the basic information, there are some statistical results data, including the millisecond time of request processing $request_time, the data block size of server response $body_bytes_send;

In addition, $geoip_city_country_code3, $geoip_region and $geoip_city are also recorded.

The variable $http_x_forwarded_for is used to record X-Forwarded-For header messages for requests initiated by other proxy servers.

Some data in the upstream module are also recorded in the log: the response status code of the proxy server, $upstream_status, $upstream_response_time for establishing links and receiving the last byte of the response body from the upstream server, $upstream_connection_time for establishing links with the upstream server, and establishing links with the upstream server. The time to connect the first byte from the upstream response header is $upstream_header_time.

Request source $http_referer and user agent $http_user_agent can also be logged.

The logging function of nginx is very powerful and flexible. It should be noted that the log_format instructions used to define the log format are only applicable to HTTP block-level instructions. All time values are measured in milliseconds with millisecond resolution.  。

Log configuration in this format will produce the following types of logs:

[25/Feb/2019:16:20:42 +0000] 10.0.1.16 192.168.0.122 Derek
GET HTTP/1.1 http www.example.com / 200 0.001 370 USA MI
“Ann Arbor” – 200 0.001 “-” “curl/7.47.0”

If you need to use this log configuration, you need to use the access_log instruction in conjunction with the access_log instruction, which receives a log directory and the configuration name used as parameters:


server {
  access_log /var/log/nginx/access.log geoproxy;
  ...
}

Access_log can be used in multiple contexts, each of which can define its own log directory and log record format.

Conclusion: The log module in nginx allows to configure the log format for different scenarios in order to view different log files.

In practice, it is very useful to configure different logs for different contexts. The log content can be simple or all necessary information can be recorded in detail. Not only that, but the log content also supports text.

It can also record data in JSON format and XML format. In fact, nginx logs help you understand information about server traffic, client usage, and client source. In addition, access logs can help you locate responses and problems related to upstream servers or specific uris; they are also useful for testing, which can be used to analyze traffic conditions and simulate real user interaction scenarios. Log plays an indispensable role in troubleshooting, debugging, application analysis and business adjustment.

Error log

In order to locate the error log of nginx accurately, the error_log instruction is used to define the error log directory and the level of recording the error log. The configuration is as follows:


error_log /var/log/nginx/error.log warn;

The error_log instruction configuration requires a required log directory and an optional error level option.

Except for the if instruction, the error_log instruction can be used in all contexts. Error log levels include:

Debug, info, notice, warn, error, crit, alert and emerg. Given logs

The ranking order is to record the smallest to the most rigorous ranking order of the log. Note that debug logs

When compiling the nginx server, you need to bring the — with-debug identifier to use it.

When a server configuration error occurs, you first need to view the error log to locate the problem. Error log

It is also a powerful tool for locating application servers, such as FastCGI services. Through the error log, we can debug the problems of worker process connection error, memory allocation, client IP and application server. Error log format does not support custom log format; however, it also records data such as current time, log level and specific information.

Note: The default settings for error logs apply globally. To override it, place the error_log instruction in the main (top-level) configuration context. Error_log adds the ability to specify multiple instructions at the same configuration level in the open source NGINX version 1.5.2.

Send logs to Unified Server via syslog

Since it is no longer necessary to write the log to a directory on disk, but to send it to a unified log server, the original directory part can be replaced by the server ip. The configuration is as follows:


error_log syslog:server=10.0.1.42 debug;
access_log syslog:server=10.0.1.42,tag=nginx,severity=info geoproxy;

#error_log server=unix:/var/log/nginx.sock debug;
#access_log syslog:server=[2001:db8::1]:1234,facility=local7,tag=nginx,severity=info;

The syslog parameters of the error_log and access_log instructions follow the colon: and some parameter options. Including: the required server tag indicates the IP, DNS name or UNIX socket that needs to be connected;

You can play with the high-end annotations above.

Optional parameters are facility, severity, tag:

The server parameter receives the IP address or DNS name with the port; the default is UDP 514 port.

The facility parameter sets the type facility of syslog, which is one of the 23 values defined by the syslog RFC standard, and the default value is local7. Other possible values are: auth, authpriv, daemon, cron, ftp, lpr, kern, mail, news, syslog, user, uucp, local0… local7

The tag parameter represents the title when displayed in the log file, and the default value is nginx.

Severity sets the severity of the message by default info level log.

Log Buffer

When the system is in load state, log buffer is enabled to reduce nginx worker process blocking. A large number of disk reads and writes and the use of CPU resources are also a huge consumption of server resources. Buffering log data into memory may be a small optimization method. Buffer parameter means the size of the buffer. Its function is that when the buffer is full, the log will be written to the file. Flush parameter means the longest time that the log in the buffer is stored in the buffer, that is, when the buffer is full, the function is the date in the buffer. If the log exceeds the maximum cache time, it will also be written to the file. The deficiency is that the log written to the log file is slightly delayed. The log buffer should be turned off in real-time debugging. The configuration is as follows:


http {
  access_log /var/log/nginx/access.log main buffer=32k flush=1m;
}

Reference link:

  1. nginx cook book
  2. Nginx configuration log

summary

Above is the whole content of this article. I hope the content of this article has some reference value for your study or work. Thank you for your support to developpaer.