Nginx
This topic explains how to collect Nginx logs with NXLog.
The Nginx web server supports error and access logging. Both types of logs can be written to file, or forwarded as Syslog via UDP, or written as Syslog to a Unix domain socket. The sections below provide a brief overview; see the Logging section of the Nginx documentation for more detailed information.
NXLog can be configured to collect Nginx logs.
Nginx error log
The error_log directive configures the destination and log level for the
error log. This directive can be given in the main (top-level) configuration
context to override the default. It can also be specified at the http,
stream, server, and location levels, where it will override the inherited
setting from the higher levels.
With the following directive, Nginx will log all messages of "warn" severity or higher to the specified log file.
error_log /var/log/nginx/error.log warn;
Following is a log message generated by Nginx, an NXLog configuration for parsing it, and the resulting JSON.
2017/08/07 04:37:16 [emerg] 17479#17479: epoll_create() failed (24: Too many open files)<Input nginx_error>
Module im_file
File '/var/log/nginx/error.log'
<Exec>
if $raw_event =~ /^(\S+ \S+) \[(\S+)\] (\d+)\#(\d+): (\*(\d+) )?(.+)$/
{
$EventTime = strptime($1, '%Y/%m/%d %H:%M:%S');
$NginxLogLevel = $2;
$NginxPID = $3;
$NginxTID = $4;
if $6 != '' $NginxCID = $6;
$Message = $7;
}
</Exec>
</Input>{
"EventReceivedTime": "2017-08-07T04:37:16.245375+02:00",
"SourceModuleName": "nginx_error",
"SourceModuleType": "im_file",
"EventTime": "2017-08-07T04:37:16.000000+02:00",
"NginxLogLevel": "emerg",
"NginxPID": "17479",
"NginxTID": "17479",
"Message": "epoll_create() failed (24: Too many open files)"
}With this directive, Nginx will forward all messages of "warn" severity or higher to the specified Syslog server. The messages will be generated with the "local7" facility.
error_log syslog:server=192.168.1.1:514,facility=local7 warn;
This NXLog configuration can be used to parse the logs.
<Input nginx_error>
Module im_udp
Host 0.0.0.0
Port 514
<Exec>
parse_syslog();
if $Message =~ /^\S+ \S+ \[\S+\] (\d+)\#(\d+): (\*(\d+) )?(.+)$/
{
$NginxPID = $1;
$NginxTID = $2;
if $4 != '' $NginxCID = $4;
$Message = $5;
}
</Exec>
</Input>{
"MessageSourceAddress": "192.168.1.12",
"EventReceivedTime": "2017-08-07T04:37:16.441368+02:00",
"SourceModuleName": "nginx_error",
"SourceModuleType": "im_udp",
"SyslogFacilityValue": 23,
"SyslogFacility": "LOCAL7",
"SyslogSeverityValue": 1,
"SyslogSeverity": "ALERT",
"SeverityValue": 5,
"Severity": "CRITICAL",
"Hostname": "nginx-host",
"EventTime": "2017-08-07T04:37:16.000000+02:00",
"SourceName": "nginx",
"Message": "epoll_create() failed (24: Too many open files)",
"NginxPID": "17479",
"NginxTID": "17479"
}With this directive, Nginx will forward all messages of "warn" severity or higher to the specified Unix domain socket. The messages will be sent in Syslog format with the "local7" Syslog facility.
error_log syslog:server=unix:/var/log/nginx/error.sock,facility=local7 warn;
<Input nginx_error>
Module im_uds
UDS /var/log/nginx/error.sock
<Exec>
parse_syslog();
if $Message =~ /^\S+ \S+ \[\S+\] (\d+)\#(\d+): (\*(\d+) )?(.+)$/
{
$NginxPID = $1;
$NginxTID = $2;
if $4 != '' $NginxCID = $4;
$Message = $5;
}
</Exec>
</Input>Nginx access log
By default, Nginx access logs are written to logs/access.log in the Nginx
Combined Log Format. An NXLog configuration example for parsing this
can be found in the Common & Combined Log Formats section. Access logs
can also be forwarded in Syslog format via UDP or a Unix domain socket, as shown
below.
The Nginx log format can be customized by setting the log_format directive;
see the
Nginx documentation for more information.
With this directive, Nginx will forward access logs to the specified Syslog server. The messages will be generated with the "local7" facility and the "info" severity.
access_log syslog:server=192.168.1.1:514,facility=local7,severity=info;
This NXLog configuration can be used to parse the logs.
<Input nginx_access>
Module im_udp
Host 0.0.0.0
Port 514
<Exec>
parse_syslog();
if $Message =~ /(?x)^(\S+)\ \S+\ (\S+)\ \[([^\]]+)\]\ \"(\S+)\ (.+)
\ HTTP\/\d\.\d\"\ (\S+)\ (\S+)\ \"([^\"]+)\"
\ \"([^\"]+)\"/
{
$Hostname = $1;
if $2 != '-' $AccountName = $2;
$EventTime = parsedate($3);
$HTTPMethod = $4;
$HTTPURL = $5;
$HTTPResponseStatus = $6;
if $7 != '-' $FileSize = $7;
if $8 != '-' $HTTPReferer = $8;
if $9 != '-' $HTTPUserAgent = $9;
delete($Message);
}
</Exec>
</Input>{
"MessageSourceAddress": "192.168.1.12",
"EventReceivedTime": "2017-08-07T06:15:55.662319+02:00",
"SourceModuleName": "nginx_access",
"SourceModuleType": "im_udp",
"SyslogFacilityValue": 23,
"SyslogFacility": "LOCAL7",
"SyslogSeverityValue": 6,
"SyslogSeverity": "INFO",
"SeverityValue": 2,
"Severity": "INFO",
"Hostname": "192.168.1.12",
"EventTime": "2017-08-07T06:15:55.000000+02:00",
"SourceName": "nginx",
"HTTPMethod": "GET",
"HTTPURL": "/",
"HTTPResponseStatus": "304",
"FileSize": "0",
"HTTPUserAgent": "Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0"
}With this directive, Nginx will forward all logs of "warn" severity or higher to the specified Unix domain socket. The messages will be sent in Syslog format with the "local7" Syslog facility.
access_log syslog:server=unix:/var/log/nginx/access.sock,facility=local7,severity=info;
<Input nginx_access>
Module im_uds
UDS /var/log/nginx/access.sock
<Exec>
parse_syslog();
if $Message =~ /(?x)^(\S+)\ \S+\ (\S+)\ \[([^\]]+)\]\ \"(\S+)\ (.+)
\ HTTP\/\d\.\d\"\ (\S+)\ (\S+)\ \"([^\"]+)\"
\ \"([^\"]+)\"/
{
$Hostname = $1;
if $2 != '-' $AccountName = $2;
$EventTime = parsedate($3);
$HTTPMethod = $4;
$HTTPURL = $5;
$HTTPResponseStatus = $6;
if $7 != '-' $FileSize = $7;
if $8 != '-' $HTTPReferer = $8;
if $9 != '-' $HTTPUserAgent = $9;
delete($Message);
}
</Exec>
</Input>