Rewriting and modifying logs

NXLog provides various ways to modify log records.

Simple rewrite of log message

A simple rewrite can be done by modifying the $raw_event field without parsing the message (with Syslog, for example). Regular expression capturing can be used for this.

Example 1. Simple rewrite statement

This statement, when used in an Exec directive, will apply the replacement directly to the $raw_event field. In this case, a parsing procedure like parse_syslog() would not be used.

if $raw_event =~ /^(aaaa)(replaceME)(.+)/
    $raw_event = $1 + 'replaceMENT' + $3;

Example 2. Converting the timestamp format of a log message

This example will convert a timestamp field to a different format. Like the previous example, the goal is to modify the $raw_event field directly, rather than use other fields and then a procedure like to_json() to update $raw_event.

The input log format is line-based, with whitespace-separated fields. The first field is a timestamp expressed as seconds since the epoch.

Input sample

1301471167.225121 AChBVvgs1dfHjwhG8 141.143.210.102 5353 224.0.0.251 5353 udp dns - - - S0 - - 0 D 1 73 0 0 (empty)

In the output module instance Exec directive, the regular expression will match and capture the first field from the line, and remove it. This captured portion is parsed with the parsedate() function and used to set the $EventTime field. This field is then prepended to the $raw_event field to replace the previously removed field.

nxlog.conf

<Input in>
    Module  im_file
    File    "conn.log"
</Input>

<Output out>
    Module  om_tcp
    Host    192.168.0.1
    Port    1514
    <Exec>
        if $raw_event =~ s/^(\S+)//
        {
            $EventTime = parsedate($1);
            $raw_event = strftime($EventTime, 'YYYY-MM-DDThh:mm:ss.sTZ') +
                         $raw_event;
        }
    </Exec>
</Output>

Output sample

2011-03-30T00:46:07.225121-07:00 AChBVvgs1dfHjwhG8 141.143.210.102 5353 224.0.0.251 5353 udp dns - - - S0 - - 0 D 1 73 0 0 (empty)

Modifying fields in a log message

A more complex method is to parse the log message into fields, modify some fields, and finally reconstruct the event record from the fields. This method is much more versatile: it allows rewriting to be done regardless of input and output formats of the log.

Example 3. Rewrite using fields

In this example, each Syslog message is received via UDP and parsed with parse_syslog_bsd(). Then, if the $Message field matches the regular expression, the $SeverityValue field is modified. Finally, the to_syslog_bsd() procedure generates $raw_event from the fields.

nxlog.conf

<Extension _syslog>
    Module  xm_syslog
</Extension>

<Input udp>
    Module  im_udp
    Port    514
    Host    0.0.0.0
    Exec    parse_syslog_bsd();
</Input>

<Output file>
    Module  om_file
    File    "/var/log/logmsg.txt"
    <Exec>
        if $Message =~ /error/ $SeverityValue = syslog_severity_value("error");
        to_syslog_bsd();
    </Exec>
</Output>

<Route syslog_to_file>
    Path    udp => file
</Route>

Renaming and deleting fields in a log message

NXLog Enterprise Edition exclusive feature

In some cases it may be necessary to rename or delete fields in an event record.

The simplest way is to use the NXLog language and the Exec directive.

Example 4. Simple field rename

This statement uses the rename_field() procedure to rename the $user field to $AccountName.

rename_field($user, $AccountName);

Example 5. Simple field deletion

This statement uses the delete() procedure to delete the $Serial field.

delete($Serial);

Alternatively, the xm_rewrite extension module (available in NXLog Enterprise Edition) can be used to rename or delete fields in a log record.

Example 6. Using xm_rewrite to whitelist and rename fields

This example uses the parse_syslog() procedure to create a set of Syslog fields in the event record. It then uses the Keep directive to whitelist a set of fields, deleting any field that is not in the list. Finally the Rename directive is used to rename the $EventTime field to $Timestamp. The resulting event record is converted to JSON and sent out via TCP.

nxlog.conf

<Extension json>
    Module  xm_json
</Extension>

<Extension rewrite>
    Module  xm_rewrite
    Keep    EventTime, Severity, Hostname, SourceName, Message
    Rename  EventTime, Timestamp
</Extension>

<Input in>
    Module  im_file
    File    '/var/log/messages'
    Exec    parse_syslog(); rewrite->process();
</Input>

<Output out>
    Module  om_tcp
    Host    10.0.0.1
    Port    1514
    Exec    to_json();
</Output>

<Route r>
    Path    in => out
</Route>

Example 7. Using xm_rewrite to remove fields

Here is an example Extension block that uses the Delete directive to delete all the severity fields. This could be used to prevent severity-based matching (during later processing) on a log source that does not set severity values correctly.

nxlog.conf

<Extension rewrite>
    Module  xm_rewrite
    Delete  SyslogSeverityValue, SyslogSeverity, SeverityValue, Severity
</Extension>