Rewrite (xm_rewrite)

This module can be used to transform event records by:

renaming fields,
deleting specified fields (blacklist),
keeping only a list of specified fields (whitelist), and
evaluating additional statements.

The xm_rewrite module provides Delete, Keep, and Rename directives for modifying event records. With the Exec directive of this module, it is possible to invoke functions and procedures from other modules. This allows all data transformation to be configured in a single module instance to simplify the configuration. Then the transformation can be referenced from another module by adding:

Exec rewrite->process();

This same statement can be used by more than one module instance if necessary, rather than duplicating configuration.

Configuration

The xm_rewrite module accepts the following directives in addition to the common module directives.

The order of the action directives is significant as the module executes them in the order of appearance. It is possible to configure an xm_rewrite instance with no directives (other than the Module directive).

Optional directives

Delete

This directive takes a field name or a list of fields. The fields specified will be removed from the event record. This can be used to blacklist specific fields that are not wanted in the event record. This is equivalent to using delete() in Exec.

Exec

This directive works the same way as the Exec directive in other modules: the statement(s) provided in the argument/block will be evaluated in the context of the module that called process() (i.e., as though the statement(s) from this Exec directive/block were inserted into the caller’s Exec directive/block, at the location of the process() call).

Keep

This directive takes a field name or a list of fields. The fields specified will be kept and all other fields not appearing in the list will be removed from the event record. This can be used to whitelist specific fields.

To retain only the $raw_event field, use Keep raw_event (it is not possible to delete the $raw_event field). This can help to discard extra event fields after $raw_event has been set (with to_json(), for example) and before an output module that operates on all fields in the event record (such as om_batchcompress).

Rename

This directive takes two fields. The field in the first argument will be renamed to the name in the second. This is equivalent to using rename_field() in Exec.

Procedures

The following procedures are exported by xm_rewrite.

process();: This procedure invokes the data processing as specified in the configuration of the xm_rewrite module instance.

Examples

Example 1. Using xm_rewrite to transform syslog data read from file

The following configuration parses syslog data from a file, invokes the process() procedure of the xm_rewrite instance to keep and rename whitelisted fields, then writes JSON-formatted output to a file.

nxlog.conf

<Extension syslog>
    Module          xm_syslog
</Extension>

<Extension rewrite>
    Module          xm_rewrite
    Keep            EventTime, Severity, Hostname, SourceName, Message
    Rename          EventTime, timestamp
    Rename          Hostname, host
    Rename          SourceName, src
    Rename          Message, msg
    Rename          Severity, sev
    Exec            if $msg =~ /error/ $sev = 'ERROR';
</Extension>

<Extension json>
    Module          xm_json
</Extension>

<Input syslogfile>
    Module          im_file
    File            "modules/extension/rewrite/xm_rewrite.in"
    Exec            parse_syslog();
    Exec            rewrite->process();
</Input>

<Output fileout>
    Module          om_file
    File            'tmp/output'
    Exec            to_json();
</Output>

<Route rewrite_fields>
    Path            syslogfile => fileout
</Route>

Input Sample

<0>2010-10-12 12:49:06 mybox app[12345]: kernel message
<30>2010-10-12 12:49:06 mybox app[12345]: daemon - info
<27>2010-10-12 12:49:06 mybox app[12345]: daemon - error
<30>2010-10-12 13:19:11 mybox app[12345]: There was an error

Output Sample

{"sev":"CRITICAL","host":"mybox","timestamp":"2010-10-12 12:49:06","src":"app","msg":"kernel message"}
{"sev":"INFO","host":"mybox","timestamp":"2010-10-12 12:49:06","src":"app","msg":"daemon - info"}
{"sev":"ERROR","host":"mybox","timestamp":"2010-10-12 12:49:06","src":"app","msg":"daemon - error"}
{"sev":"ERROR","host":"mybox","timestamp":"2010-10-12 13:19:11","src":"app","msg":"There was an error"}

Example 2. Performing additional parsing in an xm_rewrite module instance

The following configuration does the exact same processing. In this case, however, the Syslog parsing is moved into the xm_rewrite module instance so the input module only needs to invoke the process() procedure.

nxlog.conf

<Extension syslog>
    Module          xm_syslog
</Extension>

<Extension rewrite>
    Module          xm_rewrite
    Exec            parse_syslog();
    Keep            EventTime, Severity, Hostname, SourceName, Message
    Rename          EventTime, timestamp
    Rename          Hostname, host
    Rename          SourceName, src
    Rename          Message, msg
    Rename          Severity, sev
    Exec            if $msg =~ /error/ $sev = 'ERROR';
</Extension>

<Extension json>
    Module          xm_json
</Extension>

<Input syslogfile>
    Module          im_file
    File            "modules/extension/rewrite/xm_rewrite.in"
    Exec            rewrite->process();
</Input>

<Output fileout>
    Module          om_file
    File            'tmp/output'
    Exec            to_json();
</Output>

<Route rewrite_fields>
    Path            syslogfile => fileout
</Route>