GELF (xm_gelf)

This module provides reader and writer functions that can be used for processing logs in the Graylog Extended Log Format (GELF) for Graylog or GELF-compliant tools.

Unlike syslog format (with Snare Agent, for example), the GELF format contains structured data in JSON so that the fields are available for analysis. This is especially convenient with sources such as the Windows Event Log which generates logs in a structured format.

The xm_gelf module provides the following reader and writer functions.

InputType GELF_TCP: This input reader generates GELF for use with TCP (use with the TCP (im_tcp) input module).

InputType GELF_UDP: This input reader generates GELF for use with UDP (use with the UDP (im_udp) input module).

InputType GELF: This type is equivalent to the GELF_UDP reader.

OutputType GELF_TCP: This output writer generates GELF for use with TCP (use with the om_tcp output module).

OutputType GELF_UDP: This output writer generates GELF for use with UDP (use with the om_udp output module).

OutputType GELF: This type is equivalent to the GELF_UDP writer.

Configuring NXLog Agent to process GELF input or output requires loading the xm_gelf extension module and then setting the corresponding InputType or OutputType in the Input or Output module instance. See the examples below.

The GELF output generated by this module includes all fields, except for the $raw_event field and any field having a leading dot (.) or underscore (_).

Configuration

The xm_gelf module accepts the following directives in addition to the common module directives.

Optional directives

IncludeHiddenFields

This boolean directive specifies that the GELF output should include fields having a leading dot (.) or underscore (_) in their names. The default is TRUE. If IncludeHiddenFields is set to TRUE, then the generated GELF JSON will contain these otherwise excluded fields. In this case field name _fld1 will become __fld1 and .fld2 will become _.fld2 in GELF JSON.

ShortMessageLength

This optional directive can be used to specify the length of the short_message field for the GELF output writers. This defaults to 64 if the directive is not explicitly specified. If the field short_message or ShortMessage is present, it will not be truncated.

UseNullDelimiter

If this optional boolean directive is TRUE, the GELF_TCP output writer will use the NUL delimiter. If this directive is FALSE, it will use the newline delimiter. The default is TRUE.

Fields

The following fields are used by xm_gelf.

In addition to the fields listed below, if the GELF input contains custom user fields (those prefixed with the _ character), those fields will be available without the _ prefix. For example, the GELF record {"_foo": "bar"} will generate the field $foo containing the value "bar".

$EventTime (type: datetime): The time when the GELF message was created. This is called timestamp in the GELF specification.

$FullMessage (type: string): A long message that might contain a backtrace, a listing of environment variables, and so on.

$Hostname (type: string): The name of the host that sent the GELF message. This is called host in the GELF specification.

$SeverityValue (type: integer): The standard syslog severity level. This is called level in the GELF specification.

$ShortMessage (type: string): A short message with a brief description of the event. If the "short_message" JSON field is not present in the incoming GELF message, the module uses the truncated value of $Message or $raw_event.

$SourceLine (type: integer): The line in a file that caused the event. This is called line in the GELF specification.

$SyslogFacility (type: string): The syslog facility that created the event. This is called facility in the GELF specification.

$version (type: string): The GELF specification version as present in the input, e.g. 1.1.

Examples

Example 1. Parsing GELF logs collected via TCP

This configuration uses the im_tcp module to collect logs over TCP port 12201 and the xm_gelf module to parse them.

nxlog.conf

<Extension gelf>
    Module        xm_gelf
</Extension>

<Input tcpin>
    Module        im_tcp
    ListenAddr    0.0.0.0:12001
    InputType     GELF_TCP
</Input>

Example 2. Sending Windows Event Log to Graylog in GELF

The following configuration reads the Windows Event Log and sends it to a Graylog server in GELF format.

nxlog.conf

<Extension gelf>
    Module        xm_gelf
</Extension>

<Input eventlog>
    # Use 'im_mseventlog' for Windows XP, 2000 and 2003
    Module        im_msvistalog
    # Uncomment the following to collect specific event logs only
    # but make sure not to leave any `#` as only <!-- --> style comments
    # are supported inside the XML.
    #Query        <QueryList>\
    #                 <Query Id="0">\
    #                     <Select Path="Application">*</Select>\
    #                     <Select Path="System">*</Select>\
    #                     <Select Path="Security">*</Select>\
    #                 </Query>\
    #             </QueryList>
</Input>

<Output udp>
    Module        om_udp
    Host          192.168.1.1:12201
    OutputType    GELF_UDP
</Output>

<Route eventlog_to_udp>
    Path          eventlog => udp
</Route>

Example 3. Forwarding custom log files to Graylog in GELF

In this example, custom application logs are collected and sent out in GELF, with custom fields set to make the data more useful for the receiver.

nxlog.conf

<Extension gelf>
    Module        xm_gelf
</Extension>

<Input file>
    Module        im_file
    File          "/var/log/app*.log"

    <Exec>
        # Set the $EventTime field usually found in the logs by
        # extracting it with a regexp. If this is not set, the current
        # system time will be used which might be a little off.
        if $raw_event =~ /(\d\d\d\d\-\d\d-\d\d \d\d:\d\d:\d\d)/
            $EventTime = parsedate($1);

        # Explicitly set the Hostname. This defaults to the system's
        # hostname if unset.
        $Hostname = 'myhost';

        # Now set the severity level to something custom. This defaults
        # to 'INFO' if unset. We can use the following numeric values
        # here which are the standard syslog values: ALERT: 1, CRITICAL:
        # 2, ERROR: 3, WARNING: 4, NOTICE: 5, INFO: 6, DEBUG: 7
        if $raw_event =~ /ERROR/ $SyslogSeverityValue = 3;
        else $SyslogSeverityValue = 6;

        # Set a field to contain the name of the source file
        $FileName = file_name();

        # To set a custom message, use the $Message field. The
        # $raw_event field is used if $Message is unset.
        if $raw_event =~ /something important/
            $Message = 'IMPORTANT!! ' + $raw_event;
    </Exec>
</Input>

<Output udp>
    Module        om_udp
    Host          192.168.1.1:12201
    OutputType    GELF_UDP
</Output>

<Route file_to_gelf>
    Path          file => udp
</Route>

Example 4. Parsing a CSV file and sending it to Graylog in GELF

With this configuration, NXLog Agent will read a CSV file containing three fields and forward the data in GELF so that the fields will be available on the server.

nxlog.conf

<Extension gelf>
    Module        xm_gelf
</Extension>

<Extension csv>
    Module        xm_csv
    Fields        $name, $number, $location
    FieldTypes    string, integer, string
    Delimiter     ,
</Extension>

<Input file>
    Module        im_file
    File          "/var/log/app/csv.log"
    Exec          csv->parse_csv();
</Input>

<Output udp>
    Module        om_udp
    Host          192.168.1.1:12201
    OutputType    GELF_UDP
</Output>

<Route csv_to_gelf>
    Path          file => udp
</Route>