Rewrite (xm_rewrite)

This module can be used to transform event records by:

renaming fields,
removing specified fields,
keeping only a list of specified fields, and
evaluating additional statements.

The xm_rewrite module provides Delete, Keep, and Rename directives for modifying event records. With the Exec directive of this module, it is possible to invoke functions and procedures from other modules. This allows all data transformation to be configured in a single module instance to simplify the configuration. Then the transformation can be referenced from another module by adding:

Exec rewrite->process();

This same statement can be used by more than one module instance if necessary, rather than duplicating configuration.

To examine the supported platforms, see the list of installation packages.

Processing sequence

The module processes directives one after another in the order they appear. Each directive operates on the record that results from the previous directives. You can use the same directive multiple times in a single instance.

When multiple Keep directives appear in a sequence, they are not cumulative. For example, if a record consists of two fields, field_a and field_b:

Keep field_a, field_b retains both field_a and field_b.
Keep field_a on one line and Keep field_b on another line discards both fields.

In contrast, sequential Delete directives are cumulative:

Delete field_a followed by Delete field_b is equivalent to Delete field_a, field_b.

This sequential approach allows you to use patterns that would otherwise be impossible. For example, you can use Keep, then an Exec that adds new fields, and then another Keep to process those fields.

If a directive specifies a field that does not exist in the record, the module silently ignores it.

Configuration

The xm_rewrite module accepts the following directives in addition to the common module directives.

Optional directives

Delete

This directive takes a field name or a list of fields. The fields specified will be removed from the event record. This can be used to remove specific fields that are not wanted in the event record. This is equivalent to using delete() in Exec.

Exec

This directive works the same way as the Exec directive in other modules: the statement(s) provided in the argument/block will be evaluated in the context of the module that called process() (i.e., as though the statement(s) from this Exec directive/block were inserted into the caller’s Exec directive/block, at the location of the process() call).

Keep

This directive takes a field name or a list of fields. The fields specified will be kept and all other fields not appearing in the list will be removed from the event record.

To retain only the $raw_event field, use Keep raw_event (it is not possible to delete the $raw_event field). This can help to discard extra event fields after $raw_event has been set (with to_json(), for example) and before an output module that operates on all fields in the event record (such as om_batchcompress).

Rename

This directive takes two fields. The field in the first argument will be renamed to the name in the second. This is equivalent to using rename_field() in Exec.

Procedures

The following procedures are exported by xm_rewrite.

process();: This procedure invokes the data processing as specified in the configuration of the xm_rewrite module instance. You must call this procedure using the -> operator. See Calling a function of a specific module instance for more information.

Examples

Example 1. Using xm_rewrite to transform syslog data read from file

The following configuration parses syslog data from a file, invokes the process() procedure of the xm_rewrite instance to keep and rename whitelisted fields, then writes JSON-formatted output to a file.

nxlog.conf

<Extension syslog>
    Module          xm_syslog
</Extension>

<Extension rewrite>
    Module          xm_rewrite
    Keep            EventTime, Severity, Hostname, SourceName, Message
    Rename          EventTime, timestamp
    Rename          Hostname, host
    Rename          SourceName, src
    Rename          Message, msg
    Rename          Severity, sev
    Exec            if $msg =~ /error/ $sev = 'ERROR';
</Extension>

<Extension json>
    Module          xm_json
</Extension>

<Input syslogfile>
    Module          im_file
    File            "modules/extension/rewrite/xm_rewrite.in"
    Exec            parse_syslog();
    Exec            rewrite->process();
</Input>

<Output fileout>
    Module          om_file
    File            'tmp/output'
    Exec            to_json();
</Output>

<Route rewrite_fields>
    Path            syslogfile => fileout
</Route>

Input Sample

<0>2010-10-12 12:49:06 mybox app[12345]: kernel message
<30>2010-10-12 12:49:06 mybox app[12345]: daemon - info
<27>2010-10-12 12:49:06 mybox app[12345]: daemon - error
<30>2010-10-12 13:19:11 mybox app[12345]: There was an error

Output Sample

{"sev":"CRITICAL","host":"mybox","timestamp":"2010-10-12 12:49:06","src":"app","msg":"kernel message"}
{"sev":"INFO","host":"mybox","timestamp":"2010-10-12 12:49:06","src":"app","msg":"daemon - info"}
{"sev":"ERROR","host":"mybox","timestamp":"2010-10-12 12:49:06","src":"app","msg":"daemon - error"}
{"sev":"ERROR","host":"mybox","timestamp":"2010-10-12 13:19:11","src":"app","msg":"There was an error"}

Example 2. Performing additional parsing in an xm_rewrite module instance

The following configuration does the exact same processing. In this case, however, the Syslog parsing is moved into the xm_rewrite module instance so the input module only needs to invoke the process() procedure.

nxlog.conf

<Extension syslog>
    Module          xm_syslog
</Extension>

<Extension rewrite>
    Module          xm_rewrite
    Exec            parse_syslog();
    Keep            EventTime, Severity, Hostname, SourceName, Message
    Rename          EventTime, timestamp
    Rename          Hostname, host
    Rename          SourceName, src
    Rename          Message, msg
    Rename          Severity, sev
    Exec            if $msg =~ /error/ $sev = 'ERROR';
</Extension>

<Extension json>
    Module          xm_json
</Extension>

<Input syslogfile>
    Module          im_file
    File            "modules/extension/rewrite/xm_rewrite.in"
    Exec            rewrite->process();
</Input>

<Output fileout>
    Module          om_file
    File            'tmp/output'
    Exec            to_json();
</Output>

<Route rewrite_fields>
    Path            syslogfile => fileout
</Route>