Format Converter (pm_transformer)

The pm_transformer module can parse and convert logs to BSD syslog, IETF syslog, CSV, JSON, and XML data formats.

This module is being phased out and will be removed in a future release. Use the xm_syslog, xm_csv, xm_json, and xm_xml modules instead.

To examine the supported platforms, see the list of installer packages in the Available Modules chapter.

Configuration

The pm_transformer module accepts the following directives in addition to the common module directives. For conversion to occur, the InputFormat and OutputFormat directives must be specified.

InputFormat

This directive specifies the input format of the $raw_event field so that it is further parsed into fields. If this directive is not specified, no parsing will be performed.

CSV: Input is parsed as a comma-separated list of values. See xm_csv for similar functionality. The input fields must be defined by CSVInputFields.

JSON: Input is parsed as JSON. This does the same as the parse_json() procedure.

syslog_bsd: Same as syslog_rfc3164.

syslog_ietf: Same as syslog_rfc5424.

syslog_rfc3164: Input is parsed in the BSD Syslog format as defined by RFC 3164. This does the same as the parse_syslog_bsd() procedure.

syslog_rfc5424: Input is parsed in the IETF Syslog format as defined by RFC 5424. This does the same as the parse_syslog_ietf() procedure.

XML: Input is parsed as XML. This does the same as the parse_xml() procedure.

OutputFormat

This directive specifies the output transformation. If this directive is not specified, fields are not converted and $raw_event is left unmodified.

CSV: Output in $raw_event is formatted as a comma-separated list of values. See xm_csv for similar functionality.

JSON: Output in $raw_event is formatted as JSON. This does the same as the to_json() procedure.

syslog_bsd: Same as syslog_rfc3164.

syslog_ietf: Same as syslog_rfc5424.

syslog_rfc3164: Output in $raw_event is formatted in the BSD Syslog format as defined by RFC 3164. This does the same as the to_syslog_bsd() procedure.

syslog_rfc5424: Output in $raw_event is formatted in the IETF Syslog format as defined by RFC 5424. This does the same as the to_syslog_ietf() procedure.

syslog_snare: Output in $raw_event is formatted in the SNARE Syslog format. This does the same as the to_syslog_snare() procedure. This should be used in conjunction with the im_mseventlog or im_msvistalog module to produce an output compatible with Snare Agent for Windows.

XML: Output in $raw_event is formatted in XML. This does the same as the to_xml() procedure.

CSVInputFields: This is a comma-separated list of fields which will be set from the input parsed. The field names must have the dollar sign ($) prepended.

CSVInputFieldTypes: This optional directive specifies the list of types corresponding to the field names defined in CSVInputFields. If specified, the number of types must match the number of field names specified with CSVInputFields. If this directive is omitted, all fields will be stored as strings. This directive has no effect on the fields-to-CSV conversion.

CSVOutputFields: This is a comma-separated list of message fields which are placed in the CSV lines. The field names must have the dollar sign ($) prepended.

Examples

Example 1. Using the pm_transformer Module

This configuration reads BSD Syslog messages from file and writes them to another file in CSV format.

nxlog.conf

<Extension syslog>
    Module          xm_syslog
</Extension>

<Input filein>
    Module          im_file
    File            "tmp/input"
</Input>

<Processor transformer>
    Module          pm_transformer
    InputFormat     syslog_rfc3164
    OutputFormat    csv
    CSVOutputFields $facility, $severity, $timestamp, $hostname, \
                    $application, $pid, $message
</Processor>

<Output fileout>
    Module          om_file
    File            "tmp/output"
</Output>

<Route filein_to_fileout>
    Path            filein => transformer => fileout
</Route>