Syslog (xm_syslog)
This module provides support for the legacy BSD Syslog protocol as defined in RFC 3164 and the current IETF standard defined by RFCs 5424-5426. This is achieved by exporting functions and procedures usable from the NXLog language. The transport is handled by the respective input and output modules (such as im_udp), this module only provides a parser and helper functions to create syslog messages and handle facility and severity values.
The older but still widespread BSD Syslog standard defines both the format and the transport protocol in RFC 3164. The transport protocol is UDP, but to provide reliability and security, this line-based format is also commonly transferred over TCP and SSL. There is a newer standard defined in RFC 5424, also known as the IETF Syslog format, which obsoletes the BSD Syslog format. This format overcomes most of the limitations of BSD Syslog and allows multiline messages and proper timestamps. The transport method is defined in RFC 5426 for UDP and RFC 5425 for TLS/SSL.
Because the IETF Syslog format supports multiline messages, RFC 5425 defines a special format to encapsulate these by prefixing the payload size in ASCII to the IETF Syslog message. Messages transferred in UDP packets are self-contained and do not need this additional framing. The following input reader and output writer functions are provided by the xm_syslog module to support this TLS transport defined in RFC 5425. While RFC 5425 explicitly defines that the TLS network transport protocol is to be used, pure TCP may be used if security is not a requirement. Syslog messages can also be written to file with this framing format using these functions.
- InputType Syslog_TLS
-
This input reader function parses the payload size and then reads the message according to this value. It is required to support Syslog TLS transport defined in RFC 5425.
- OutputType Syslog_TLS
-
This output writer function prepends the payload size to the message. It is required to support Syslog TLS transport defined in RFC 5425.
The Syslog_TLS InputType/OutputType can work with any input/output such as im_tcp or im_file and does not depend on SSL transport at all. The name Syslog_TLS was chosen to refer to the octet-framing method described in RFC 5425 used for TLS transport. |
Structured data in IETF Syslog messages is parsed and put into
NXLog Agent fields. The SD-ID will be prepended to the field name
with a dot unless it is NXLOG@XXXX
. Consider the following Syslog
message:
<30>1 2011-12-04T21:16:10.000000+02:00 host app procid msgid [exampleSDID@32473 eventSource="Application" eventID="1011"] Message part
After this IETF-formatted Syslog message is parsed with
parse_syslog_ietf(), there will
be two additional fields: $exampleSDID.eventID
and
$exampleSDID.eventSource
. When SD-ID is NXLOG
, the field name will
be the same as the SD-PARAM name. The two additional fields extracted
from the structured data part of the following IETF syslog messages are
$eventID
and $eventSource
:
<30>1 2011-12-04T21:16:10.000000+02:00 host app procid msgid [NXLOG@32473 eventSource="Application" eventID="1011"] Message part
All fields in the structured data part are parsed as strings.
Configuration
The xm_syslog module accepts the following directives in addition to the common module directives.
Optional directives
This is an alias for the UTCTimestamp directive below. |
|
This optional directive specifies a character with which to replace line breaks in the Syslog message when generating Syslog events with to_syslog_bsd(), to_syslog_ietf(), and to_syslog_snare().
The default is a space.
To retain line breaks in Syslog messages, set this to |
|
This optional directive takes a single character (see below) as argument.
This character is used by the to_syslog_snare() procedure to separate fields.
If this directive is not specified, the default delimiter character is the tab ( |
|
This optional directive takes a single character (see below) as argument.
This character is used by the to_syslog_snare() procedure to replace occurrences of the delimiter character inside the |
|
This optional boolean directive can be used to format the timestamps produced by to_syslog_ietf() in UTC/GMT instead of local time.
The default is |
Specifying Quote, Escape, and Delimiter Characters
The SnareDelimiter and SnareReplacement directives can be specified in several ways.
- Unquoted single character
-
Any printable character can be specified as an unquoted character, except for the backslash (
\
):Delimiter ;
- Control characters
-
The following non-printable characters can be specified with escape sequences:
- \a
-
audible alert (bell)
- \b
-
backspace
- \t
-
horizontal tab
- \n
-
newline
- \v
-
vertical tab
- \f
-
formfeed
- \r
-
carriage return
For example, to use TAB delimiting:
Delimiter \t
- A character in single quotes
-
The configuration parser strips whitespace, so it is not possible to define a space as the delimiter unless it is enclosed within quotes:
Delimiter ' '
Printable characters can also be enclosed:
Delimiter ';'
The backslash can be specified when enclosed within quotes:
Delimiter '\'
- A character in double quotes
-
Double quotes can be used like single quotes:
Delimiter " "
The backslash can be specified when enclosed within double quotes:
Delimiter "\"
- A hexadecimal ASCII code
-
Hexadecimal ASCII character codes can be used prefixed with
0x
. For example, the space can be specified as:Delimiter 0x20
This is equivalent to:
Delimiter " "
Procedures
The following procedures are exported by xm_syslog.
parse_syslog();
-
Parse the $raw_event field as either BSD Syslog (RFC 3164) or IETF Syslog (RFC 5424) format.
parse_syslog(string source);
-
Parse the given string as either BSD Syslog (RFC 3164) or IETF Syslog (RFC 5424) format.
parse_syslog_bsd();
-
Parse the $raw_event field as BSD Syslog (RFC 3164) format.
parse_syslog_bsd(string source);
-
Parse the given string as BSD Syslog (RFC 3164) format.
parse_syslog_ietf();
-
Parse the $raw_event field as IETF Syslog (RFC 5424) format.
parse_syslog_ietf(string source);
-
Parse the given string as IETF Syslog (RFC 5424) format.
to_snare();
-
Create a SNARE formatted log message in $raw_event. The following fields are used to construct the $raw_event field: $EventTime, $Hostname, $SeverityValue, $FileName, $Channel, $SourceName, $AccountName, $AccountType, $EventType, $Category, $RecordNumber, and $Message.
to_syslog_bsd();
-
Create a BSD Syslog formatted log message in $raw_event from the fields of the event. The following fields are used to construct the $raw_event field: $EventTime; $Hostname; $SourceName; $ProcessID or $ExecutionProcessID; $Message or $raw_event; $SyslogSeverity, $SyslogSeverityValue, $Severity, or $SeverityValue; and $SyslogFacility or $SyslogFacilityValue. If the fields are not present, a sensible default is used.
to_syslog_ietf();
-
Create an IETF Syslog (RFC 5424) formatted log message in $raw_event from the fields of the event. The following fields are used to construct the $raw_event field: $EventTime; $Hostname; $SourceName; $ProcessID or $ExecutionProcessID; $Message or $raw_event; $SyslogSeverity, $SyslogSeverityValue, $Severity, or $SeverityValue; and $SyslogFacility or $SyslogFacilityValue. If the fields are not present, a sensible default is used.
to_syslog_snare();
-
Create a SNARE Syslog formatted log message in $raw_event. This procedure is compatible with the Snare format for Windows Event Log. The following fields are used to construct the $raw_event field: $EventTime, $Hostname, $SeverityValue, $FileName, $Channel, $SourceName, $AccountName, $AccountType, $EventType, $Category, $RecordNumber, and $Message.
Fields
The following fields are used by xm_syslog.
In addition to the fields listed below, the
parse_syslog() and
parse_syslog_ietf()
procedures will create fields from the Structured Data part of an IETF
Syslog message. If the SD-ID in this case is not "NXLOG", these fields
will be prefixed by the SD-ID (for example, $mySDID.CustomField
).
$raw_event
(type: string)-
A Syslog formatted string, set after to_syslog_bsd(), to_syslog_ietf(), or to_snare(), or to_syslog_snare() is invoked.
$EventTime
(type: datetime)-
The timestamp found in the Syslog message, set after parse_syslog(), parse_syslog_bsd(), or parse_syslog_ietf() is called. If the year value is missing, it is set as described in the core fix_year() function.
$Hostname
(type: string)-
The hostname part of the Syslog line, set after parse_syslog(), parse_syslog_bsd(), or parse_syslog_ietf() is called.
$Message
(type: string)-
The message part of the Syslog line, set after parse_syslog(), parse_syslog_bsd(), or parse_syslog_ietf() is called.
$MessageID
(type: string)-
The MSGID part of the syslog message, set after parse_syslog_ietf() is called.
$ProcessID
(type: string)-
The process ID in the Syslog line, set after parse_syslog(), parse_syslog_bsd(), or parse_syslog_ietf() is called.
$Severity
(type: string)-
The normalized severity name of the event. See $SeverityValue.
$SeverityValue
(type: integer)-
The normalized severity number of the event, mapped as follows.
Syslog Severity Normalized Severity 0/emerg
5/critical
1/alert
5/critical
2/crit
5/critical
3/err
4/error
4/warning
3/warning
5/notice
2/info
6/info
2/info
7/debug
1/debug
$SourceName
(type: string)-
The application/program part of the Syslog line, set after parse_syslog(), parse_syslog_bsd(), or parse_syslog_ietf() is called.
$SyslogFacility
(type: string)-
The facility name of the Syslog line, set after parse_syslog(), parse_syslog_bsd(), or parse_syslog_ietf() is called. The default facility is
user
.
$SyslogFacilityValue
(type: integer)-
The facility code of the Syslog line, set after parse_syslog(), parse_syslog_bsd(), or parse_syslog_ietf() is called. The default facility is
1
(user).
$SyslogSeverity
(type: string)-
The severity name of the Syslog line, set after parse_syslog(), parse_syslog_bsd(), or parse_syslog_ietf() is called. The default severity is
notice
. See $SeverityValue.
$SyslogSeverityValue
(type: integer)-
The severity code of the Syslog line, set after parse_syslog(), parse_syslog_bsd(), or parse_syslog_ietf() is called. The default severity is
5
(notice). See $SeverityValue.
Examples
In this example, logs are collected from files, converted to BSD Syslog format with the to_syslog_bsd() procedure, and sent over UDP with the om_udp module.
<Extension syslog>
Module xm_syslog
</Extension>
<Input file>
Module im_file
# We monitor all files matching the wildcard.
# Every line is read into the $raw_event field.
File "/var/log/app*.log"
<Exec>
# Set the $EventTime field usually found in the logs by
# extracting it with a regexp. If this is not set, the current
# system time will be used which might be a little off.
if $raw_event =~ /(\d\d\d\d\-\d\d-\d\d \d\d:\d\d:\d\d)/
{
$EventTime = parsedate($1);
}
# Now set the severity to something custom. This defaults to
# 'INFO' if unset.
if $raw_event =~ /ERROR/ $Severity = 'ERROR';
else $Severity = 'INFO';
# The facility can be also set, otherwise the default value is
# 'USER'.
$SyslogFacility = 'AUDIT';
# The SourceName field is called the TAG in RFC 3164
# terminology and is usually the process name.
$SourceName = 'my_application';
# It is also possible to rewrite the Hostname if you do not
# want to use the system hostname.
$Hostname = 'myhost';
# The Message field is used if present, otherwise the current
# $raw_event is prepended with the Syslog headers. You can do
# some modifications on the Message if required. Here we add
# the full path of the source file to the end of message line.
$Message = $raw_event + ' [' + file_name() + ']';
# Now create our RFC 3164 compliant Syslog line using the
# fields set above and/or use sensible defaults where
# possible. The result will be in $raw_event.
to_syslog_bsd();
</Exec>
</Input>
<Output udp>
# This module just sends the contents of the $raw_event field to
# the destination defined here, one UDP packet per message.
Module om_udp
Host 192.168.1.42:1514
</Output>
<Route file_to_udp>
Path file => udp
</Route>
To collect BSD Syslog messages over UDP, use the parse_syslog_bsd() procedure coupled with the im_udp module as in the following example.
<Extension syslog>
Module xm_syslog
</Extension>
<Input udp>
Module im_udp
ListenAddr 0.0.0.0:514
Exec parse_syslog_bsd();
</Input>
<Output file>
Module om_file
File "/var/log/logmsg.txt"
</Output>
<Route syslog_to_file>
Path udp => file
</Route>
To collect IETF Syslog messages over UDP as defined by RFC 5424 and RFC 5426, use the parse_syslog_ietf() procedure coupled with the im_udp module as in the following example. Note that, as for BSD Syslog, the default port is 514 (as defined by RFC 5426).
<Extension syslog>
Module xm_syslog
</Extension>
<Input ietf>
Module im_udp
ListenAddr 0.0.0.0:514
Exec parse_syslog_ietf();
</Input>
<Output file>
Module om_file
File "/var/log/logmsg.txt"
</Output>
<Route ietf_to_file>
Path ietf => file
</Route>
To collect both IETF and BSD Syslog messages over UDP, use the parse_syslog() procedure coupled with the im_udp module as in the following example. This procedure is capable of detecting and parsing both Syslog formats. Since 514 is the default UDP port number for both BSD and IETF Syslog, this port can be useful to collect both formats simultaneously. To accept both formats on different ports, the appropriate parsers can be used as in the previous two examples.
<Extension syslog>
Module xm_syslog
</Extension>
<Input udp>
Module im_udp
ListenAddr 0.0.0.0:514
Exec parse_syslog();
</Input>
<Output file>
Module om_file
File "/var/log/logmsg.txt"
</Output>
<Route syslog_to_file>
Path udp => file
</Route>
To collect IETF Syslog messages over TLS/SSL as defined by RFC 5424 and RFC 5425, use the parse_syslog_ietf() procedure coupled with the im_ssl module as in this example. Note that the default port is 6514 in this case (as defined by RFC 5425). The payload format parser is handled by the Syslog_TLS input reader.
<Extension syslog>
Module xm_syslog
</Extension>
<Input ssl>
Module im_ssl
ListenAddr localhost:6514
CAFile %CERTDIR%/ca.pem
CertFile %CERTDIR%/client-cert.pem
CertKeyFile %CERTDIR%/client-key.pem
KeyPass secret
InputType Syslog_TLS
Exec parse_syslog_ietf();
</Input>
<Output file>
Module om_file
File "/var/log/logmsg.txt"
</Output>
<Route ssl_to_file>
Path ssl => file
</Route>
The following configuration uses the to_syslog_ietf() procedure to convert input to IETF Syslog and forward it over TCP.
<Extension syslog>
Module xm_syslog
</Extension>
<Input file>
Module im_file
File "/var/log/input.txt"
Exec $TestField = "test value"; $Message = $raw_event;
</Input>
<Output tcp>
Module om_tcp
Host 127.0.0.1:1514
Exec to_syslog_ietf();
OutputType Syslog_TLS
</Output>
<Route file_to_syslog>
Path file => tcp
</Route>
Because of the Syslog_TLS framing, the raw data sent over TCP will look like the following.
130 <13>1 2012-01-01T16:15:52.873750Z - - - [NXLOG@14506 EventReceivedTime="2012-01-01 17:15:52" TestField="test value"] test message
This example shows that all fields—except those which are filled by the Syslog parser—are added to the structured data part.
If the message part of the syslog event matches the regular expression, the $SeverityValue
field will be set to the "error" syslog severity integer value (which is provided by the syslog_severity_value() function).
<Extension syslog>
Module xm_syslog
</Extension>
<Input udp>
Module im_udp
ListenAddr 0.0.0.0:514
Exec parse_syslog_bsd();
</Input>
<Output file>
Module om_file
File "/var/log/logmsg.txt"
Exec if $Message =~ /error/ $SeverityValue = syslog_severity_value("error");
Exec to_syslog_bsd();
</Output>
<Route syslog_to_file>
Path udp => file
</Route>
The following example does almost the same thing as the previous example, except that the syslog parsing and rewrite is moved to a processor module and the rewrite only occurs if the facility was modified. This can make processing faster on multi-core systems because the processor module runs in a separate thread. This method can also minimize UDP packet loss because the input module does not need to parse Syslog messages and therefore can process UDP packets faster.
<Extension syslog>
Module xm_syslog
</Extension>
<Input udp>
Module im_udp
ListenAddr 0.0.0.0:514
</Input>
<Processor rewrite>
Module pm_null
<Exec>
parse_syslog_bsd();
if $Message =~ /error/
{
$SeverityValue = syslog_severity_value("error");
to_syslog_bsd();
}
</Exec>
</Processor>
<Output file>
Module om_file
File "/var/log/logmsg.txt"
</Output>
<Route syslog_to_file>
Path udp => rewrite => file
</Route>