Nginx

This topic explains how to collect Nginx logs with NXLog.

The Nginx web server supports error and access logging. Both types of logs can be written to file, or forwarded as Syslog via UDP, or written as Syslog to a Unix domain socket. The sections below provide a brief overview; see the Logging section of the Nginx documentation for more detailed information.

NXLog can be configured to collect Nginx logs.

Nginx error log

The error_log directive configures the destination and log level for the error log. This directive can be given in the main (top-level) configuration context to override the default. It can also be specified at the http, stream, server, and location levels, where it will override the inherited setting from the higher levels.

Example 1. Collecting Nginx error logs from file

With the following directive, Nginx will log all messages of "warn" severity or higher to the specified log file.

nginx.conf
error_log /var/log/nginx/error.log warn;

Following is a log message generated by Nginx, an NXLog configuration for parsing it, and the resulting JSON.

Log sample
2017/08/07 04:37:16 [emerg] 17479#17479: epoll_create() failed (24: Too many open files)
nxlog.conf
<Input nginx_error>
    Module  im_file
    File    '/var/log/nginx/error.log'
    <Exec>
        if $raw_event =~ /^(\S+ \S+) \[(\S+)\] (\d+)\#(\d+): (\*(\d+) )?(.+)$/
        {  
            $EventTime = strptime($1, '%Y/%m/%d %H:%M:%S');
            $NginxLogLevel = $2;
            $NginxPID = $3;
            $NginxTID = $4;
            if $6 != '' $NginxCID = $6;
            $Message = $7;
        } 
    </Exec>
</Input>
Output sample
{
  "EventReceivedTime": "2017-08-07T04:37:16.245375+02:00",
  "SourceModuleName": "nginx_error",
  "SourceModuleType": "im_file",
  "EventTime": "2017-08-07T04:37:16.000000+02:00",
  "NginxLogLevel": "emerg",
  "NginxPID": "17479",
  "NginxTID": "17479",
  "Message": "epoll_create() failed (24: Too many open files)"
}
Example 2. Collecting Nginx error logs via Syslog

With this directive, Nginx will forward all messages of "warn" severity or higher to the specified Syslog server. The messages will be generated with the "local7" facility.

nginx.conf
error_log syslog:server=192.168.1.1:514,facility=local7 warn;

This NXLog configuration can be used to parse the logs.

nxlog.conf
<Input nginx_error>
    Module  im_udp
    Host    0.0.0.0
    Port    514
    <Exec>
        parse_syslog();
        if $Message =~ /^\S+ \S+ \[\S+\] (\d+)\#(\d+): (\*(\d+) )?(.+)$/
        {
            $NginxPID = $1;
            $NginxTID = $2;
            if $4 != '' $NginxCID = $4;
            $Message = $5;
        }
    </Exec>
</Input>
Output sample
{
  "MessageSourceAddress": "192.168.1.12",
  "EventReceivedTime": "2017-08-07T04:37:16.441368+02:00",
  "SourceModuleName": "nginx_error",
  "SourceModuleType": "im_udp",
  "SyslogFacilityValue": 23,
  "SyslogFacility": "LOCAL7",
  "SyslogSeverityValue": 1,
  "SyslogSeverity": "ALERT",
  "SeverityValue": 5,
  "Severity": "CRITICAL",
  "Hostname": "nginx-host",
  "EventTime": "2017-08-07T04:37:16.000000+02:00",
  "SourceName": "nginx",
  "Message": "epoll_create() failed (24: Too many open files)",
  "NginxPID": "17479",
  "NginxTID": "17479"
}
Example 3. Collecting Nginx error logs via Unix Domain Socket

With this directive, Nginx will forward all messages of "warn" severity or higher to the specified Unix domain socket. The messages will be sent in Syslog format with the "local7" Syslog facility.

nginx.conf
error_log syslog:server=unix:/var/log/nginx/error.sock,facility=local7 warn;
nxlog.conf
<Input nginx_error>
    Module  im_uds
    UDS     /var/log/nginx/error.sock
    <Exec>
        parse_syslog();
        if $Message =~ /^\S+ \S+ \[\S+\] (\d+)\#(\d+): (\*(\d+) )?(.+)$/
        {
            $NginxPID = $1;
            $NginxTID = $2;
            if $4 != '' $NginxCID = $4;
            $Message = $5;
        }
    </Exec>
</Input>

Nginx access log

By default, Nginx access logs are written to logs/access.log in the Nginx Combined Log Format. An NXLog configuration example for parsing this can be found in the Common & Combined Log Formats section. Access logs can also be forwarded in Syslog format via UDP or a Unix domain socket, as shown below.

The Nginx log format can be customized by setting the log_format directive; see the Nginx documentation for more information.

Example 4. Collecting Nginx access logs via Syslog

With this directive, Nginx will forward access logs to the specified Syslog server. The messages will be generated with the "local7" facility and the "info" severity.

nginx.conf
access_log syslog:server=192.168.1.1:514,facility=local7,severity=info;

This NXLog configuration can be used to parse the logs.

nxlog.conf
<Input nginx_access>
    Module  im_udp
    Host    0.0.0.0
    Port    514
    <Exec>
        parse_syslog();
        if $Message =~ /(?x)^(\S+)\ \S+\ (\S+)\ \[([^\]]+)\]\ \"(\S+)\ (.+)
                          \ HTTP\/\d\.\d\"\ (\S+)\ (\S+)\ \"([^\"]+)\"
                          \ \"([^\"]+)\"/
        {  
            $Hostname = $1;
            if $2 != '-' $AccountName = $2;
            $EventTime = parsedate($3);
            $HTTPMethod = $4;
            $HTTPURL = $5;
            $HTTPResponseStatus = $6;
            if $7 != '-' $FileSize = $7;
            if $8 != '-' $HTTPReferer = $8;
            if $9 != '-' $HTTPUserAgent = $9;
            delete($Message);
        }
    </Exec>
</Input>
Output sample
{
  "MessageSourceAddress": "192.168.1.12",
  "EventReceivedTime": "2017-08-07T06:15:55.662319+02:00",
  "SourceModuleName": "nginx_access",
  "SourceModuleType": "im_udp",
  "SyslogFacilityValue": 23,
  "SyslogFacility": "LOCAL7",
  "SyslogSeverityValue": 6,
  "SyslogSeverity": "INFO",
  "SeverityValue": 2,
  "Severity": "INFO",
  "Hostname": "192.168.1.12",
  "EventTime": "2017-08-07T06:15:55.000000+02:00",
  "SourceName": "nginx",
  "HTTPMethod": "GET",
  "HTTPURL": "/",
  "HTTPResponseStatus": "304",
  "FileSize": "0",
  "HTTPUserAgent": "Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0"
}
Example 5. Collecting Nginx access logs via Unix Domain Socket

With this directive, Nginx will forward all logs of "warn" severity or higher to the specified Unix domain socket. The messages will be sent in Syslog format with the "local7" Syslog facility.

nginx.conf
access_log syslog:server=unix:/var/log/nginx/access.sock,facility=local7,severity=info;
nxlog.conf
<Input nginx_access>
    Module  im_uds
    UDS     /var/log/nginx/access.sock
    <Exec>
        parse_syslog();
        if $Message =~ /(?x)^(\S+)\ \S+\ (\S+)\ \[([^\]]+)\]\ \"(\S+)\ (.+)
                          \ HTTP\/\d\.\d\"\ (\S+)\ (\S+)\ \"([^\"]+)\"
                          \ \"([^\"]+)\"/
        {  
            $Hostname = $1;
            if $2 != '-' $AccountName = $2;
            $EventTime = parsedate($3);
            $HTTPMethod = $4;
            $HTTPURL = $5;
            $HTTPResponseStatus = $6;
            if $7 != '-' $FileSize = $7;
            if $8 != '-' $HTTPReferer = $8;
            if $9 != '-' $HTTPUserAgent = $9;
            delete($Message);
        }
    </Exec>
</Input>
Disclaimer

While we endeavor to keep the information in this topic up to date and correct, NXLog makes no representations or warranties of any kind, express or implied about the completeness, accuracy, reliability, suitability, or availability of the content represented here.

Last revision: 17 September 2018