NXLog search
Let's Talk Start free
Let's Talk
6.16

Automatic Data Parser (xm_autoparse)

This module is experimental. Its detection rules, the fields it creates, and its behavior can change in future releases. We recommend testing it against your data before you use it in production.

This module automatically detects the format of incoming telemetry data and parses it into structured fields. It supports the following formats:

  • CEF

  • CSV

  • JSON

  • Key-value pairs

  • LEEF

  • BSD syslog (RFC 3164, including Snare-formatted messages), IETF syslog (RFC 5424), and IETF syslog over TLS (RFC 5425).

The module processes each line of input separately. It removes the line terminator and any leading or trailing whitespace and control characters, and skips empty lines. It also skips a UTF-8 byte order mark (BOM) at the start of a file. If the parser for the detected format fails, the module tries the other supported formats. If none of them succeeds, NXLog Agent treats the line as plain text. The module sets the $NXFormat field to the name of the format used, or plaintext if no format matched.

Be aware of these limitations:

  • The module doesn’t reassemble records that span multiple lines, such as pretty-printed JSON, and parses each line separately.

  • The module parses IETF syslog over TLS (RFC 5425) messages only if each message is on a single line. It doesn’t use the byte count at the start of each message to find where the message ends, so it can’t parse messages that contain line breaks.

  • The module doesn’t process lines without a line terminator, such as the last line of a file that doesn’t end with a newline.

  • The module parses each CSV line separately, so a header row doesn’t name the fields of the rows that follow it. Instead, the module creates a separate record for the header row and names the fields of the other rows $field0, $field1, and so on.

  • The module doesn’t limit line length. If the input never contains a line terminator, NXLog Agent buffers the data in memory until the file or connection closes, or the process runs out of memory.

  • The module detects XML but doesn’t parse it. It sets the $NXFormat field to xml and keeps the content in the $raw_event field only.

Set the InputType directive of your input module to the name of an xm_autoparse module instance to automatically detect and parse data.

The xm_autoparse module accepts only the common module directives.

To examine the supported platforms, see the list of installation packages.

Fields

The following fields are created by xm_autoparse.

Each record includes the $raw_event and $NXFormat fields described below. Other fields vary depending on the format and content of the data record.

$raw_event (type: string)

The raw input line. If the line contains a JSON array, the text of an array element.

$NXFormat (type: string)

The format the module detected and used to parse the line. The possible values are cef, csv, json, kvp, leef, plaintext, syslog3164, syslog5424, syslog5425, and xml. If a record already has a field called NXFormat, the module changes it to an array with the original value and the detected format.

Examples

Example 1. Automatically parsing telemetry data received over TCP

This configuration listens for data over TCP and uses an xm_autoparse module instance as the InputType of the input module. NXLog Agent automatically detects the format of each line and parses it into structured fields.

nxlog.conf
<Extension autoparse>
    Module        xm_autoparse
</Extension>

<Input tcp_listen>
    Module        im_tcp
    ListenAddr    0.0.0.0:1514
    InputType     autoparse
</Input>

The following is a BSD syslog message.

Input sample
<30>Sep 25 15:40:27 myserver sshd[26459]: Accepted publickey for john from 192.168.1.1 port 41193 ssh2

When the NXLog Agent configuration above processes this message, it adds the following fields to the log record.

Field Value

$EventReceivedTime

2026-09-25 15:40:32

$SourceModuleName

tcp_listen

$SourceModuleType

im_tcp

$Hostname

myserver

$SyslogFacilityValue

3

$SyslogFacility

DAEMON

$SyslogSeverityValue

6

$SyslogSeverity

INFO

$SeverityValue

2

$Severity

INFO

$EventTime

2026-09-25 15:40:27

$SourceName

sshd

$ProcessID

26459

$Message

Accepted publickey for john from 192.168.1.1 port 41193 ssh2

$NXFormat

syslog3164

$raw_event

<30>Sep 25 15:40:27 myserver sshd[26459]: Accepted publickey for john from 192.168.1.1 port 41193 ssh2
Example 2. Automatically parsing UTF-16 encoded log files

The xm_autoparse extension expects UTF-8 input. This configuration reads UTF-16LE encoded log files and uses the Character Set Conversion extension to convert the data to UTF-8. The converter must come before the xm_autoparse instance in the InputType directive. NXLog Agent then detects the format of each line and parses it into structured fields.

nxlog.conf
<Extension utf16>
    Module           xm_charconv
    InputEncoding    UTF-16LE
</Extension>

<Extension autoparse>
    Module           xm_autoparse
</Extension>

<Input app_logs>
    Module           im_file
    File             'C:\Program Files\app\*.log'
    InputType        utf16.convert, autoparse
</Input>
Example 3. Automatically parsing and normalizing telemetry data

This configuration reads log files that contain records in different formats and uses the $NXFormat field to normalize them into a common schema.

nxlog.conf
<Extension autoparse>
    Module       xm_autoparse
</Extension>

<Extension json>
    Module       xm_json
</Extension>

<Input app_logs>
    Module       im_file
    File         '/var/log/app/*.log'
    InputType    autoparse
    <Exec>
        if ($NXFormat == 'plaintext') {
            drop(); (1)
        }
        else if ($NXFormat == 'cef') {
            $Message = $Name;
            rename_field($src, $SourceIP);
            rename_field($dst, $DestinationIP);
        }
        else if ($NXFormat == 'csv') {
            $EventTime = parsedate($field0);
            $Hostname = $field1;
            $Message = $field2 + ' ' + $field3;
            $StatusCode = integer($field4);

            delete($field0);
            delete($field1);
            delete($field2);
            delete($field3);
            delete($field4);
        }

        delete($EventReceivedTime); (2)
        delete($SourceModuleName);
        delete($SourceModuleType);

        to_json(); (3)
    </Exec>
</Input>
1 Drops records that NXLog Agent can’t parse and have the $NXFormat field set to plaintext.
2 Removes core fields that NXLog Agent adds to every record.
3 The to_json() procedure converts the record to JSON and writes it to the $raw_event field.

The following input sample contains a CEF record, a CSV record, a plain-text line, and a JSON array.

Input sample
CEF:0|Security|threatmanager|1.0|100|worm successfully stopped|10|src=192.168.0.100 dst=192.168.0.120 spt=1232
2026-09-25 10:15:00,SRV01,GET,/index.html,200
--- MARK ---
[{"user":"alice","action":"login"},{"user":"bob","action":"logout"}]

When the NXLog Agent configuration above processes these records, it drops the plain-text line and outputs the following JSON records.

Output sample
{"CEFVersion":"0","DeviceVendor":"Security","DeviceProduct":"threatmanager","DeviceVersion":"1.0","SignatureID":"100","Name":"worm successfully stopped","Severity":"10","SourceIP":"192.168.0.100","DestinationIP":"192.168.0.120","spt":"1232","NXFormat":"cef","Hostname":"SRV01","Message":"worm successfully stopped"}
{"NXFormat":"csv","Hostname":"SRV01","EventTime":"2026-09-25T10:15:00.000000+00:00","Message":"GET /index.html","StatusCode":200}
{"action":"login","user":"alice","NXFormat":"json","Hostname":"SRV01"}
{"action":"logout","user":"bob","NXFormat":"json","Hostname":"SRV01"}

© Copyright NXLog Ltd.

PRIVACY POLICY GENERAL TERMS OF BUSINESS