Rewrite (xm_rewrite)
This module can be used to transform event records by:
-
renaming fields,
-
deleting specified fields (blacklist),
-
keeping only a list of specified fields (whitelist), and
-
evaluating additional statements.
The xm_rewrite module provides Delete, Keep, and Rename directives for modifying event records. With the Exec directive of this module, it is possible to invoke functions and procedures from other modules. This allows all data transformation to be configured in a single module instance to simplify the configuration. Then the transformation can be referenced from another module by adding:
Exec rewrite->process();
This same statement can be used by more than one module instance if necessary, rather than duplicating configuration.
Configuration
The xm_rewrite module accepts the following directives in addition to the common module directives.
The order of the action directives is significant as the module executes them in the order of appearance. It is possible to configure an xm_rewrite instance with no directives (other than the Module directive). In this case, the corresponding process() procedure will do nothing.
Optional directives
This directive takes a field name or a list of fields. The fields specified will be removed from the event record. This can be used to blacklist specific fields that are not wanted in the event record. This is equivalent to using delete() in Exec. |
|
This directive works the same way as the Exec directive in other modules: the statement(s) provided in the argument/block will be evaluated in the context of the module that called process() (i.e., as though the statement(s) from this Exec directive/block were inserted into the caller’s Exec directive/block, at the location of the process() call). |
|
This directive takes a field name or a list of fields. The fields specified will be kept and all other fields not appearing in the list will be removed from the event record. This can be used to whitelist specific fields. To retain only the |
|
This directive takes two fields. The field in the first argument will be renamed to the name in the second. This is equivalent to using rename_field() in Exec. |
Procedures
The following procedures are exported by xm_rewrite.
process();
-
This procedure invokes the data processing as specified in the configuration of the xm_rewrite module instance.
Examples
The following configuration parses syslog data from a file, invokes the process() procedure of the xm_rewrite instance to keep and rename whitelisted fields, then writes JSON-formatted output to a file.
<Extension syslog>
Module xm_syslog
</Extension>
<Extension rewrite>
Module xm_rewrite
Keep EventTime, Severity, Hostname, SourceName, Message
Rename EventTime, timestamp
Rename Hostname, host
Rename SourceName, src
Rename Message, msg
Rename Severity, sev
Exec if $msg =~ /error/ $sev = 'ERROR';
</Extension>
<Extension json>
Module xm_json
</Extension>
<Input syslogfile>
Module im_file
File "modules/extension/rewrite/xm_rewrite.in"
Exec parse_syslog();
Exec rewrite->process();
</Input>
<Output fileout>
Module om_file
File 'tmp/output'
Exec to_json();
</Output>
<Route rewrite_fields>
Path syslogfile => fileout
</Route>
<0>2010-10-12 12:49:06 mybox app[12345]: kernel message
<30>2010-10-12 12:49:06 mybox app[12345]: daemon - info
<27>2010-10-12 12:49:06 mybox app[12345]: daemon - error
<30>2010-10-12 13:19:11 mybox app[12345]: There was an error
{"sev":"CRITICAL","host":"mybox","timestamp":"2010-10-12 12:49:06","src":"app","msg":"kernel message"}
{"sev":"INFO","host":"mybox","timestamp":"2010-10-12 12:49:06","src":"app","msg":"daemon - info"}
{"sev":"ERROR","host":"mybox","timestamp":"2010-10-12 12:49:06","src":"app","msg":"daemon - error"}
{"sev":"ERROR","host":"mybox","timestamp":"2010-10-12 13:19:11","src":"app","msg":"There was an error"}
The following configuration does the exact same processing. In this case, however, the Syslog parsing is moved into the xm_rewrite module instance so the input module only needs to invoke the process() procedure.
<Extension syslog>
Module xm_syslog
</Extension>
<Extension rewrite>
Module xm_rewrite
Exec parse_syslog();
Keep EventTime, Severity, Hostname, SourceName, Message
Rename EventTime, timestamp
Rename Hostname, host
Rename SourceName, src
Rename Message, msg
Rename Severity, sev
Exec if $msg =~ /error/ $sev = 'ERROR';
</Extension>
<Extension json>
Module xm_json
</Extension>
<Input syslogfile>
Module im_file
File "modules/extension/rewrite/xm_rewrite.in"
Exec rewrite->process();
</Input>
<Output fileout>
Module om_file
File 'tmp/output'
Exec to_json();
</Output>
<Route rewrite_fields>
Path syslogfile => fileout
</Route>