Syslog (xm_syslog)

This module provides support for the legacy BSD Syslog protocol as defined in RFC 3164 and the current IETF standard defined by RFCs 5424-5426. This is achieved by exporting functions and procedures usable from the NXLog language. The transport is handled by the respective input and output modules (such as im_udp), this module only provides a parser and helper functions to create syslog messages and handle facility and severity values.

The older but still widespread BSD Syslog standard defines both the format and the transport protocol in RFC 3164. The transport protocol is UDP, but to provide reliability and security, this line-based format is also commonly transferred over TCP and SSL. There is a newer standard defined in RFC 5424, also known as the IETF Syslog format, which obsoletes the BSD Syslog format. This format overcomes most of the limitations of BSD Syslog and allows multiline messages and proper timestamps. The transport method is defined in RFC 5426 for UDP and RFC 5425 for TLS/SSL.

Because the IETF Syslog format supports multiline messages, RFC 5425 defines a special format to encapsulate these by prefixing the payload size in ASCII to the IETF Syslog message. Messages transferred in UDP packets are self-contained and do not need this additional framing. The following input reader and output writer functions are provided by the xm_syslog module to support this TLS transport defined in RFC 5425. While RFC 5425 explicitly defines that the TLS network transport protocol is to be used, pure TCP may be used if security is not a requirement. Syslog messages can also be written to file with this framing format using these functions.

InputType Syslog_TLS

This input reader function parses the payload size and then reads the message according to this value. It is required to support Syslog TLS transport defined in RFC 5425.

OutputType Syslog_TLS

This output writer function prepends the payload size to the message. It is required to support Syslog TLS transport defined in RFC 5425.

The Syslog_TLS InputType/OutputType can work with any input/output such as im_tcp or im_file and does not depend on SSL transport at all. The name Syslog_TLS was chosen to refer to the octet-framing method described in RFC 5425 used for TLS transport.

Structured data in IETF Syslog messages is parsed and put into NXLog Agent fields. The SD-ID will be prepended to the field name with a dot unless it is NXLOG@XXXX. Consider the following Syslog message:

<30>1 2011-12-04T21:16:10.000000+02:00 host app procid msgid [exampleSDID@32473 eventSource="Application" eventID="1011"] Message part

After this IETF-formatted Syslog message is parsed with parse_syslog_ietf(), there will be two additional fields: $exampleSDID.eventID and $exampleSDID.eventSource. When SD-ID is NXLOG, the field name will be the same as the SD-PARAM name. The two additional fields extracted from the structured data part of the following IETF Syslog message are $eventID and $eventSource:

<30>1 2011-12-04T21:16:10.000000+02:00 host app procid msgid [NXLOG@32473 eventSource="Application" eventID="1011"] Message part

All fields in the structured data part are parsed as strings.

Configuration

The xm_syslog module accepts the following directives in addition to the common module directives.

Optional directives

IETFTimestampInGMT

This is an alias for the UTCTimestamp directive below.

ReplaceLineBreaks

This optional directive specifies a character with which to replace line breaks in the Syslog message when generating Syslog events with to_syslog_bsd(), to_syslog_ietf(), and to_syslog_snare(). The default is a space. To retain line breaks in Syslog messages, set this to \n.

SnareDelimiter

This optional directive takes a single character (see below) as argument. This character is used by the to_syslog_snare() procedure to separate fields. If this directive is not specified, the default delimiter character is the tab (\t). In latter versions of Snare 4 this has changed to the hash mark (#); this directive can be used to specify the alternative delimiter. Note that there is no delimiter after the last field.

SnareReplacement

This optional directive takes a single character (see below) as argument. This character is used by the to_syslog_snare() procedure to replace occurrences of the delimiter character inside the $Message field. If this directive is not specified, the default replacement character is the space.

UTCTimestamp

This optional boolean directive can be used to format the timestamps produced by to_syslog_ietf() in UTC/GMT instead of local time. The default is FALSE: local time is used with a timezone indicator.

Specifying Quote, Escape, and Delimiter Characters

The SnareDelimiter and SnareReplacement directives can be specified in several ways.

Unquoted single character

Any printable character can be specified as an unquoted character, except for the backslash (\):

Delimiter ;
Control characters

The following non-printable characters can be specified with escape sequences:

\a

audible alert (bell)

\b

backspace

\t

horizontal tab

\n

newline

\v

vertical tab

\f

formfeed

\r

carriage return

For example, to use TAB delimiting:

Delimiter \t
A character in single quotes

The configuration parser strips whitespace, so it is not possible to define a space as the delimiter unless it is enclosed within quotes:

Delimiter ' '

Printable characters can also be enclosed:

Delimiter ';'

The backslash can be specified when enclosed within quotes:

Delimiter '\'
A character in double quotes

Double quotes can be used like single quotes:

Delimiter " "

The backslash can be specified when enclosed within double quotes:

Delimiter "\"
A hexadecimal ASCII code

Hexadecimal ASCII character codes can be used prefixed with 0x. For example, the space can be specified as:

Delimiter 0x20

This is equivalent to:

Delimiter " "

Functions

The following functions are exported by xm_syslog.

string syslog_facility_string(integer arg)

Convert a Syslog facility value to a string.

integer syslog_facility_value(string arg)

Convert a Syslog facility string to an integer.

string syslog_severity_string(integer arg)

Convert a Syslog severity value to a string.

integer syslog_severity_value(string arg)

Convert a Syslog severity string to an integer.

Procedures

The following procedures are exported by xm_syslog.

parse_syslog();

Parse the $raw_event field as either BSD Syslog (RFC 3164) or IETF Syslog (RFC 5424) format.

parse_syslog(string source);

Parse the given string as either BSD Syslog (RFC 3164) or IETF Syslog (RFC 5424) format.

parse_syslog_bsd();

Parse the $raw_event field as BSD Syslog (RFC 3164) format.

parse_syslog_bsd(string source);

Parse the given string as BSD Syslog (RFC 3164) format.

parse_syslog_ietf();

Parse the $raw_event field as IETF Syslog (RFC 5424) format.

parse_syslog_ietf(string source);

Parse the given string as IETF Syslog (RFC 5424) format.

to_snare();

Create a SNARE formatted log message in $raw_event. The following fields are used to construct the $raw_event field: $EventTime, $Hostname, $SeverityValue, $FileName, $Channel, $SourceName, $AccountName, $AccountType, $EventType, $Category, $RecordNumber, and $Message.

to_syslog_bsd();

Create a BSD Syslog formatted log message in $raw_event from the fields of the event. The following fields are used to construct the $raw_event field: $EventTime; $Hostname; $SourceName; $ProcessID or $ExecutionProcessID; $Message or $raw_event; $SyslogSeverity, $SyslogSeverityValue, $Severity, or $SeverityValue; and $SyslogFacility or $SyslogFacilityValue. If the fields are not present, a sensible default is used.

to_syslog_ietf();

Create an IETF Syslog (RFC 5424) formatted log message in $raw_event from the fields of the event. The following fields are used to construct the $raw_event field: $EventTime; $Hostname; $SourceName; $ProcessID or $ExecutionProcessID; $Message or $raw_event; $SyslogSeverity, $SyslogSeverityValue, $Severity, or $SeverityValue; and $SyslogFacility or $SyslogFacilityValue. If the fields are not present, a sensible default is used.

to_syslog_snare();

Create a SNARE Syslog formatted log message in $raw_event. This procedure is compatible with the Snare format for Windows Event Log. The following fields are used to construct the $raw_event field: $EventTime, $Hostname, $SeverityValue, $FileName, $Channel, $SourceName, $AccountName, $AccountType, $EventType, $Category, $RecordNumber, and $Message.

Fields

The following fields are used by xm_syslog.

In addition to the fields listed below, the parse_syslog() and parse_syslog_ietf() procedures will create fields from the Structured Data part of an IETF Syslog message. If the SD-ID in this case is not "NXLOG", these fields will be prefixed by the SD-ID (for example, $mySDID.CustomField).

$raw_event (type: string)

A Syslog formatted string, set after to_syslog_bsd(), to_syslog_ietf(), or to_snare(), or to_syslog_snare() is invoked.

$EventTime (type: datetime)

The timestamp found in the Syslog message, set after parse_syslog(), parse_syslog_bsd(), or parse_syslog_ietf() is called. If the year value is missing, it is set as described in the core fix_year() function.

$Hostname (type: string)

The hostname part of the Syslog line, set after parse_syslog(), parse_syslog_bsd(), or parse_syslog_ietf() is called.

$Message (type: string)

The message part of the Syslog line, set after parse_syslog(), parse_syslog_bsd(), or parse_syslog_ietf() is called.

$MessageID (type: string)

The MSGID part of the syslog message, set after parse_syslog_ietf() is called.

$ProcessID (type: string)

The process ID in the Syslog line, set after parse_syslog(), parse_syslog_bsd(), or parse_syslog_ietf() is called.

$Severity (type: string)

The normalized severity name of the event. See $SeverityValue.

$SeverityValue (type: integer)

The normalized severity number of the event, mapped as follows.

Syslog Severity Normalized Severity

0/emerg

5/critical

1/alert

5/critical

2/crit

5/critical

3/err

4/error

4/warning

3/warning

5/notice

2/info

6/info

2/info

7/debug

1/debug

$SourceName (type: string)

The application/program part of the Syslog line, set after parse_syslog(), parse_syslog_bsd(), or parse_syslog_ietf() is called.

$SyslogFacility (type: string)

The facility name of the Syslog line, set after parse_syslog(), parse_syslog_bsd(), or parse_syslog_ietf() is called. The default facility is user.

$SyslogFacilityValue (type: integer)

The facility code of the Syslog line, set after parse_syslog(), parse_syslog_bsd(), or parse_syslog_ietf() is called. The default facility is 1 (user).

$SyslogSeverity (type: string)

The severity name of the Syslog line, set after parse_syslog(), parse_syslog_bsd(), or parse_syslog_ietf() is called. The default severity is notice. See $SeverityValue.

$SyslogSeverityValue (type: integer)

The severity code of the Syslog line, set after parse_syslog(), parse_syslog_bsd(), or parse_syslog_ietf() is called. The default severity is 5 (notice). See $SeverityValue.

Examples

Example 1. Sending a file as BSD Syslog over UDP

In this example, logs are collected from files, converted to BSD Syslog format with the to_syslog_bsd() procedure, and sent over UDP with the om_udp module.

nxlog.conf
<Extension syslog>
    Module  xm_syslog
</Extension>

<Input file>
    Module  im_file

    # We monitor all files matching the wildcard.
    # Every line is read into the $raw_event field.
    File    "/var/log/app*.log"

    <Exec>
        # Set the $EventTime field usually found in the logs by
        # extracting it with a regexp. If this is not set, the current
        # system time will be used which might be a little off.
        if $raw_event =~ /(\d\d\d\d\-\d\d-\d\d \d\d:\d\d:\d\d)/
        {
            $EventTime = parsedate($1);
        }

        # Now set the severity to something custom. This defaults to
        # 'INFO' if unset.
        if $raw_event =~ /ERROR/ $Severity = 'ERROR';
        else $Severity = 'INFO';

        # The facility can be also set, otherwise the default value is
        # 'USER'.
        $SyslogFacility = 'AUDIT';

        # The SourceName field is called the TAG in RFC 3164
        # terminology and is usually the process name.
        $SourceName = 'my_application';

        # It is also possible to rewrite the Hostname if you do not
        # want to use the system hostname.
        $Hostname = 'myhost';

        # The Message field is used if present, otherwise the current
        # $raw_event is prepended with the Syslog headers. You can do
        # some modifications on the Message if required. Here we add
        # the full path of the source file to the end of message line.
        $Message = $raw_event + ' [' + file_name() + ']';

        # Now create our RFC 3164 compliant Syslog line using the
        # fields set above and/or use sensible defaults where
        # possible. The result will be in $raw_event.
        to_syslog_bsd();
    </Exec>
</Input>

<Output udp>
    # This module just sends the contents of the $raw_event field to
    # the destination defined here, one UDP packet per message.
    Module  om_udp
    Host    192.168.1.42:1514
</Output>

<Route file_to_udp>
    Path    file => udp
</Route>
Example 2. Collecting BSD style syslog messages over UDP

To collect BSD Syslog messages over UDP, use the parse_syslog_bsd() procedure coupled with the im_udp module as in the following example.

nxlog.conf
<Extension syslog>
    Module      xm_syslog
</Extension>

<Input udp>
    Module      im_udp
    ListenAddr  0.0.0.0:514
    Exec        parse_syslog_bsd();
</Input>

<Output file>
    Module      om_file
    File        "/var/log/logmsg.txt"
</Output>

<Route syslog_to_file>
    Path        udp => file
</Route>
Example 3. Collecting IETF style syslog messages over UDP

To collect IETF Syslog messages over UDP as defined by RFC 5424 and RFC 5426, use the parse_syslog_ietf() procedure coupled with the im_udp module as in the following example. Note that, as for BSD Syslog, the default port is 514 (as defined by RFC 5426).

nxlog.conf
<Extension syslog>
    Module      xm_syslog
</Extension>

<Input ietf>
    Module      im_udp
    ListenAddr  0.0.0.0:514
    Exec        parse_syslog_ietf();
</Input>

<Output file>
    Module      om_file
    File        "/var/log/logmsg.txt"
</Output>

<Route ietf_to_file>
    Path        ietf => file
</Route>
Example 4. Collecting both IETF and BSD Syslog messages over the same UDP port

To collect both IETF and BSD Syslog messages over UDP, use the parse_syslog() procedure coupled with the im_udp module as in the following example. This procedure is capable of detecting and parsing both Syslog formats. Since 514 is the default UDP port number for both BSD and IETF Syslog, this port can be useful to collect both formats simultaneously. To accept both formats on different ports, the appropriate parsers can be used as in the previous two examples.

nxlog.conf
<Extension syslog>
    Module      xm_syslog
</Extension>

<Input udp>
    Module      im_udp
    ListenAddr  0.0.0.0:514
    Exec        parse_syslog();
</Input>

<Output file>
    Module      om_file
    File        "/var/log/logmsg.txt"
</Output>

<Route syslog_to_file>
    Path        udp => file
</Route>
Example 5. Collecting IETF Syslog messages over TLS/SSL

To collect IETF Syslog messages over TLS/SSL as defined by RFC 5424 and RFC 5425, use the parse_syslog_ietf() procedure coupled with the im_ssl module as in this example. Note that the default port is 6514 in this case (as defined by RFC 5425). The payload format parser is handled by the Syslog_TLS input reader.

nxlog.conf
<Extension syslog>
    Module          xm_syslog
</Extension>

<Input ssl>
    Module          im_ssl
    ListenAddr      localhost:6514
    CAFile          %CERTDIR%/ca.pem
    CertFile        %CERTDIR%/client-cert.pem
    CertKeyFile     %CERTDIR%/client-key.pem
    KeyPass         secret
    InputType       Syslog_TLS
    Exec            parse_syslog_ietf();
</Input>

<Output file>
    Module          om_file
    File            "/var/log/logmsg.txt"
</Output>

<Route ssl_to_file>
    Path            ssl => file
</Route>
Example 6. Forwarding IETF Syslog over TCP

The following configuration uses the to_syslog_ietf() procedure to convert input to IETF Syslog and forward it over TCP.

nxlog.conf
<Extension syslog>
    Module      xm_syslog
</Extension>

<Input file>
    Module      im_file
    File        "/var/log/input.txt"
    Exec        $TestField = "test value"; $Message = $raw_event;
</Input>

<Output tcp>
    Module      om_tcp
    Host        127.0.0.1:1514
    Exec        to_syslog_ietf();
    OutputType  Syslog_TLS
</Output>

<Route file_to_syslog>
    Path        file => tcp
</Route>

Because of the Syslog_TLS framing, the raw data sent over TCP will look like the following.

Output sample
130 <13>1 2012-01-01T16:15:52.873750Z  - - - [NXLOG@14506 EventReceivedTime="2012-01-01 17:15:52" TestField="test value"] test message

This example shows that all fields—​except those which are filled by the Syslog parser—​are added to the structured data part.

Example 7. Conditional rewrite of the Syslog Facility—​Version 1

If the message part of the syslog event matches the regular expression, the $SeverityValue field will be set to the "error" syslog severity integer value (which is provided by the syslog_severity_value() function).

nxlog.conf
<Extension syslog>
    Module      xm_syslog
</Extension>

<Input udp>
    Module      im_udp
    ListenAddr  0.0.0.0:514
    Exec        parse_syslog_bsd();
</Input>

<Output file>
    Module      om_file
    File        "/var/log/logmsg.txt"
    Exec        if $Message =~ /error/ $SeverityValue = syslog_severity_value("error");
    Exec        to_syslog_bsd();
</Output>

<Route syslog_to_file>
    Path        udp => file
</Route>
Example 8. Conditional Rewrite of the Syslog Facility—​Version 2

The following example does almost the same thing as the previous example, except that the syslog parsing and rewrite is moved to a processor module and the rewrite only occurs if the facility was modified. This can make processing faster on multi-core systems because the processor module runs in a separate thread. This method can also minimize UDP packet loss because the input module does not need to parse Syslog messages and therefore can process UDP packets faster.

nxlog.conf
<Extension syslog>
    Module      xm_syslog
</Extension>

<Input udp>
    Module      im_udp
    ListenAddr  0.0.0.0:514
</Input>

<Processor rewrite>
    Module      pm_null
    <Exec>
        parse_syslog_bsd();
        if $Message =~ /error/
        {
            $SeverityValue = syslog_severity_value("error");
            to_syslog_bsd();
        }
    </Exec>
</Processor>

<Output file>
    Module      om_file
    File        "/var/log/logmsg.txt"
</Output>

<Route syslog_to_file>
    Path        udp => rewrite => file
</Route>