JSON (xm_json)

This module provides features to parse or generate JSON data.

Logs in JSON format can be parsed into structured data using one of two methods:

By calling the parse_json() procedure, which expects the $raw_event field or the string passed as a parameter to contain a valid JSON object. The most common use case for this is when an input module reads line-based data and each line is JSON object.
By specifying the name of an xm_json module instance in the InputType directive of the input module, which instructs the module to parse the raw data directly. This method supports JSON spanning multiple lines (e.g. prettified JSON), JSON arrays, and newline-delimited JSON. See the example on Processing multiple JSON objects.

Log records can be converted to JSON format by calling the to_json() function or procedure. The records can be produced as JSON array elements by specifying the name of an xm_json module instance in the OutputType directive of the output module. See Produce JSON array below for an example.

When using this module keep in mind the following limitations:

The JSON specification does not define a type for datetime values, therefore, these are represented as JSON strings. The JSON parser in this module can automatically detect datetime values, therefore, it is not necessary to explicitly use parsedate().
The parse_json() and to_json() functions do not support floating point or integer values that exceed the limits of the 32-bit or 64-bit instruction set architecture of the computer where NXLog Agent runs. Those values are converted into strings to preserve the original data.
The length of JSON keys is limited to a maximum of 499 characters. Any attempt to use a longer key will result in the current operation failing and probable data loss.

To examine the supported platforms, see the list of installation packages.

Configuration

The xm_json module accepts the following directives in addition to the common module directives.

Optional directives

DateFormat

This optional directive can be used to set the format of the datetime strings in the generated JSON. It is similar to the global DateFormat but is independent of it: is defined separately and has its own default. If this directive is not specified, the default is YYYY-MM-DDThh:mm:ss.sTZ.

DetectNestedJSON

This optional directive can be used to disable the autodetection of nested JSON strings when calling the to_json() function or the to_json() procedure. For example, consider a field $key which contains the string value of {"subkey":42}. If DetectNestedJSON is set to FALSE, to_json() will produce {"key":"{\"subkey\":42}"}. If DetectNestedJSON is set to TRUE (the default), the result is {"key":{"subkey":42}}—a valid nested JSON record.

Flatten

This optional boolean directive specifies that the parse_json() procedure should flatten nested JSON, creating field names with dot notation. The default is FALSE. If Flatten is set to TRUE, the following JSON will populate the fields $event.time and $event.severity:

{"event":{"time":"2015-01-01T00:00:00.000Z","severity":"ERROR"}}

ForceUTF8

This optional boolean directive specifies whether the generated JSON should be valid UTF-8. The JSON specification requires JSON records to be UTF-8 encoded, and some tools fail to parse JSON if it is not valid UTF-8. If ForceUTF8 is set to TRUE, the generated JSON will be validated and any invalid character will be replaced with a question mark (?). The default is FALSE.

IncludeHiddenFields

This boolean directive specifies that the to_json() function or the to_json() procedure should include fields having a leading dot (.) or underscore (_) in their names. The default is TRUE. If IncludeHiddenFields is set to TRUE, then generated JSON will contain these otherwise excluded fields.

ParseDate

If this boolean directive is set to TRUE, xm_json will attempt to parse as a timestamp any string that appears to begin with a 4-digit year (as a regular expression, ^[12][0-9]{3}-). If this directive is set to FALSE, xm_json will not attempt to parse these strings. The default is TRUE.

PrettyPrint

If set to TRUE, this optional boolean directive specifies that the generated JSON should be pretty-printed, where each key-value is printed on a new indented line. Note that this adds line breaks to the JSON records, which can cause parser errors in some tools that expect single-line JSON. If this directive is not specified, the default is FALSE.

UnFlatten

This optional boolean directive specifies that the to_json() procedure should generate nested JSON when field names exist containing the dot (.). For example, if UnFlatten is set to TRUE, the two fields $event.time and $event.severity will be converted to JSON as follows:

{"event":{"time":"2015-01-01T00:00:00.000Z","severity":"ERROR"}}

When UnFlatten is set to FALSE (the default if not specified), the following JSON would result:

{"event.time":"2015-01-01T00:00:00.000Z","event.severity":"ERROR"}

Functions

The following functions are exported by xm_json.

type: string extract_json(type: string jsonpath): Search the $raw_event field using the specified JSONPath expression. If successfully matched, it returns the value of the matched node. If no match is found, it returns an empty string.
type: string extract_json(type: string jsonpath, type: string json_data): Search json_data using the specified JSONPath expression. If successfully matched, it returns the value of the matched node. If no match is found, it returns an empty string.
type: unknown parse_json(type: string source): Parse the parameter string as JSON format into a simple or compound returning value.
type: string to_json(): Convert the fields to JSON and return this as a string value. Any field having a leading dot (.) or underscore (_) will be automatically excluded unless IncludeHiddenFields directive is set to TRUE. The existing $raw_event field is never included in the generated JSON.
type: string to_json(type: unknown value): Converts the value parameter of any simple or compound type to JSON and returns this as a string value.

Procedures

The following procedures are exported by xm_json.

extract_json(type: string jsonpath);: Search the $raw_event field using the specified JSONPath expression. If successfully matched, it rewrites the $raw_event field with the value of the matched node. If no match is found, $raw_event will be set to an empty string.
extract_json(type: string jsonpath, type: string json_data);: Search json_data using the specified JSONPath expression. If successfully matched, it rewrites the $raw_event field with the value of the matched node. If no match is found, $raw_event will be set to an empty string.
parse_json();: Parse the $raw_event field as JSON input.
parse_json(type: string source);: Parse the given string as JSON format.
to_json();: Convert the fields to JSON and write it to the $raw_event field. Any field starting with a dot (.) or underscore (_) will be automatically excluded unless the IncludeHiddenFields directive is set to TRUE. The $raw_event core field is not included when converting fields to JSON.

Creating and populating fields

The parse_json() procedure parses a string containing a JSON object into structured data. It expects the $raw_event field or the string passed as a parameter to be in the following format:

{"Key1":"Value1","Key2":"Value2"}

Once a log record is parsed with this procedure, fields are created according to the keys and values in the JSON object. The fields can be used for further log processing or to convert the log record into a different output format. For an example of how to parse JSON log records and manipulate fields, see Parsing JSON below.

Input modules may create additional fields containing various information. When converting to a different format, such fields will be included in the output log record, which may consume additional memory and bandwidth. For efficient handling of log records, consult the Fields section in the documentation of input modules and test the configuration before deployment. To delete any unwanted fields, use the delete() procedure or the xm_rewrite extension.

Extracting values from JSON

The extract_json() function or procedure extracts a node’s value from a JSON tree.

Both function and procedure accept a JSONPath expression to select the required node and an optional string containing the JSON data to parse. If no JSON data is specified, the content of the $raw_event core field is parsed.

Accepted JSONPath expressions

extract_json() implements a subset of JSONPath that is most commonly used in $.a.['b']['c'].d[1].['e'][1]..f notation:

Node selection by name, e.g., $.a or $['a'].
Array elements selection by index, including multi-dimensional arrays, e.g., $.a[1] or $.a[1][1][1] or $['a'][1][1][1].
Search for a named node in one nesting layer, e.g., $..a.
Any sequence of the above expressions.

When specifying a $..a-type expression, extract_json() does not perform a recursive search for nodes but limits the scope to only child nodes of the current level. The first match is returned.

Returned data

The extract_json() function returns a string containing the node’s value or JSON if the matched node contains a JSON object. The extract_json() procedure sets the value of the $raw_event field instead of returning a value.

Since data is parsed and rewritten as a string, floating-point values always include 6 digits after the point.

See Extracting values from JSON below for an example.

Examples

Converting logs to JSON

Example 1. Converting syslog to JSON format

This configuration receives logs over TCP and parses records into structured data with the parse_syslog() procedure. It then converts records to JSON using the to_json() procedure.

nxlog.conf

<Extension syslog>
    Module        xm_syslog
</Extension>

<Extension json>
    Module        xm_json
</Extension>

<Input tcp_listen>
    Module        im_tcp
    ListenAddr    0.0.0.0:1514
    <Exec>
        parse_syslog();
        to_json();
    </Exec>
</Input>

The following is a syslog message in BSD format.

Input sample

<30>Sep 30 15:45:43 SRV01.example.com acpid: 1 client rule loaded

The following JSON object shows the same event after NXLog Agent processed it.

Output sample

{
  "MessageSourceAddress": "127.0.0.1",
  "EventReceivedTime": "2025-09-30T15:46:24.386367+02:00",
  "SourceModuleName": "tcp_listen",
  "SourceModuleType": "im_tcp",
  "Hostname": "SRV01.example.com",
  "SyslogFacilityValue": 3,
  "SyslogFacility": "DAEMON",
  "SyslogSeverityValue": 6,
  "SyslogSeverity": "INFO",
  "SeverityValue": 2,
  "Severity": "INFO",
  "EventTime": "2025-09-30T15:45:43.000000+02:00",
  "SourceName": "acpid",
  "Message": "1 client rule loaded"
}

Example 2. Converting Windows events to syslog-encapsulated JSON

This configuration collects Windows events and converts them to JSON format using the to_json() procedure. It then converts records to BSD syslog format using the to_syslog_bsd() procedure.

nxlog.conf

<Extension syslog>
    Module    xm_syslog
</Extension>

<Extension json>
    Module    xm_json
</Extension>

<Input windows_events>
    Module    im_msvistalog
    <Exec>
        $Message = to_json(); (1)
        to_syslog_bsd(); (2)
    </Exec>
</Input>

1	Converts the event to JSON format and writes the value to the `$Message` field.
2	Converts the record to BSD syslog, which uses the $Message field to populate the message part.

Output sample

<14>Sep  30 14:40:11 SRV01 Service_Control_Manager[1332]: {"EventTime":"2025-09-30T14:40:11.770503+02:00","Hostname":"SRV01","EventType":"INFO","SeverityValue":2,"Severity":"INFO","SourceName":"Service Control Manager","FileName":"System","EventID":7036,"CategoryNumber":0,"RecordNumber":6788,"Message":"The nxlog service entered the running state. ","EventReceivedTime":"2025-09-30T14:40:12.770211+02:00"}

Parsing JSON logs

Example 3. Parsing simple JSON

This configuration collects events in JSON format from a file and parses records into structured data using the parse_json() procedure. It processes the events and converts them back to JSON using the to_json() procedure.

nxlog.conf

<Extension json>
    Module      xm_json
</Extension>

<Input server_logs>
    Module      im_file
    File        '/tmp/server_logs'
    <Exec>
        parse_json();

        delete($SourceModuleType); (1)
        delete($SourceModuleName);

        to_json();
    </Exec>
</Input>

1	Deletes unnecessary core fields.

The following is an event in JSON format. Although it is pretty printed for readability, the configuration expects one record per line.

Input sample

{
  "Timestamp": "2025-09-30T22:27:14+02:00",
  "Hostname": "SRV01",
  "Message": "The server has started.",
  "Severity": "INFO"
}

The following JSON object shows the same event after NXLog Agent processed it.

Output sample

{
  "EventReceivedTime":"2025-09-30T22:27:25.902492+02:00",
  "Hostname":"SRV01",
  "Timestamp":"2025-09-30T22:27:14.000000+02:00",
  "Message":"The server has started.",
  "Severity":"INFO"
}

Example 4. Parsing multiple JSON objects and arrays

The following input samples consist of the same events in different JSON objects.

JSON objects spanning multiple lines

{
  "Date": "2025-09-30", "Time": "22:50:12",
  "Message": "The process has started"
},
{
  "Message": "The process has stopped!"
}

Array of JSON objects

[{"Date": "2025-09-30", "Time": "22:50:12", "Message": "The process has started"}, {"Message": "The process has stopped!"}]

Newline-delimited JSON objects

{"Date": "2025-09-30", "Time": "22:50:12", "Message": "The process has started"}
{"Message": "The process has stopped!"}

NXLog Agent can parse all of these inputs by setting the InputType of the input module instance to an xm_json instance.

nxlog.conf

<Extension json>
    Module      xm_json
</Extension>

<Input myapp>
    Module      im_file
    File        'C:\MyApp\app.log'
    InputType   json (1)
    <Exec>
        if defined $Date { (2)
            $EventTime = strptime($Date + $Time, "%Y-%m-%d %T");
        }
        else {
            $EventTime = $EventReceivedTime;
        }

        delete($Date); (3)
        delete($Time);
        delete($EventReceivedTime);
        delete($SourceModuleType);

        to_json(); (4)
    </Exec>
</Input>

1	Sets the InputType to the xm_json module instance.
2	If the `$Date` field exists, it combines it with the `$Time` field to create the `$EventTime` field. Otherwise, it sets the `$EventTime` field to the value of the `$EventReceivedTime` core field.
3	Deletes unnecessary fields.
4	Converts the event back to JSON using the to_json() procedure.

The following JSON objects show the same events after NXLog Agent processed them. The output is the same for all three input samples above.

Output sample

{"Hostname":"SRV01","Message":"The process has started","EventTime":"2025-09-30T22:50:12.000000+02:00","SourceModuleName":"myapp"}
{"Hostname":"SRV01","Message":"The process has stopped!","EventTime":"2025-09-30T22:51:26.979726+02:00","SourceModuleName":"myapp"}

Example 5. Extracting fields from JSON objects

This configuration collects JSON-formatted events using the File input module, which populates the $raw_event core field with the log line read from the file. It then uses the extract_json() procedure to extract the EventData node from the original event and rewrite the $raw_event field.

nxlog.conf

<Extension json>
    Module    xm_json
</Extension>

<Input myapp>
    Module    im_file
    File      '/var/log/myapp'
    Exec      extract_json("$.EventData");
</Input>

The following is an event in JSON format. Although it is pretty printed for readability, the configuration expects one record per line.

Input sample

{
  "Stream": "stdout",
  "EventData": {
    "EventTime": "2025-09-30 09:35:12",
    "Message": "The process has started",
    "Severity": "INFO"
  }
}

The following JSON object shows the same event after NXLog Agent processed it.

Output sample

{
  "EventTime": "2025-09-30 09:35:12",
  "Message": "The process has started",
  "Severity": "INFO"
}

Example 6. Parsing compound data

This example demonstrates how to use the parse_json() function to process JSON logs containing composite data. The function converts the raw JSON event into an object, allowing you to access individual fields.

Input sample

{"message": "Session opened for user jdoe","metadata":{"event_type":"SSH Login","log_level":3}}
{"message": "Session closed"}

This configuration collects logs from a file and parses log records using the parse_json() function. Variables prefixed with `$$`are module variables, which means they are only accessible by that module instance and not propagated through the data processing pipeline.

nxlog.conf

<Extension json>
    Module    xm_json
</Extension>

<Input input_file>
    Module    im_file
    File      '/path/to/log/file'
    <Exec>
        $$parsed = json->parse_json($raw_event);             (1)
        log_info("Parsed type: " + type($$parsed));          (2)
        log_info("Message value: " + $$parsed("message"));   (3)
        log_info("Metadata value: " + $$parsed("metadata")); (4)
    </Exec>
</Input>

1	Parses the raw JSON event and stores the structured result in the `$$parsed` variable. The `parse_json()` function automatically detects whether the input is a JSON object or an array.
2	Logs the data type of the `$$parsed` variable, which should be `hash` for objects and `array` for lists. This verifies that the data was parsed correctly.
3	Extracts the value of the `Message` key from the parsed JSON. If the key is missing, it logs an empty value.
4	Retrieves the `Attributes` field from the JSON object. If this field contains a nested object, the structure is preserved. If the key is missing, it logs an empty value.

Notice that the second log entry in the output sample below has an empty Attributes value. If a field is missing from the JSON input, the parse_json()`function sets its value to `undef.

Output sample

2025-10-07 17:58:42 INFO [im_file|system_logs] Parsed type: hash
2025-10-07 17:58:42 INFO [im_file|system_logs] Message value: Session opened for user jdoe
2025-10-07 17:58:42 INFO [im_file|system_logs] Metadata value: ( 'event_type' => 'SSH Login', 'log_level' => 3 )
2025-10-07 17:58:42 INFO [im_file|system_logs] Parsed type: hash
2025-10-07 17:58:42 INFO [im_file|system_logs] Message value: Session closed
2025-10-07 17:58:42 INFO [im_file|system_logs] Metadata value:

Example 7. Parse JSON arrays

The parse_json() procedure can automatically detect JSON arrays.

Input sample

[[101, "jdoe", "ssh_login"], [102, "asmith", "failed_ssh"], [103, "mjane", "session_closed"]]

nxlog.conf

<Extension json>
    Module    xm_json
</Extension>

<Input input_file>
    Module    im_file
    File      '/path/to/log/file'
    <Exec>
        $$parsed = json->parse_json($raw_event);                        (1)
        log_info("Parsed type: " + type($$parsed));                     (2)
        log_info("Second array item: " + $$parsed[1]);                  (3)
        log_info("First value of third array item: " + $$parsed[2][0]); (4)
    </Exec>
</Input>

1	Parses the raw JSON event and stores the structured result in the `$$parsed` variable. The `parse_json()` function automatically detects whether the input is a JSON object or an array.
2	Logs the data type of the `$$parsed` variable, which should be `array` in this case. This verifies that the data was parsed correctly.
3	Extracts the second element from the top-level array. Since JSON arrays use zero-based indexing, `$$parsed[1]` refers to the second item in the list.
4	Outputs the first element of the third nested array. If the data structure is different or the index is out of range, it will result in an error or an empty value.

Output sample

2025-10-07 18:14:40 INFO [im_file|in1] Parsed type: array
2025-10-07 18:14:40 INFO [im_file|in1] Second array item: [ 102, "asmith", "failed_ssh" ]
2025-10-07 18:14:40 INFO [im_file|in1] First value of third array item: 103

Output logs in JSON format

Example 8. Output a JSON array

This configuration collects events in JSON format from a file and parses records into structured data by specifying an xm_json module instance as the InputType. It processes the events and outputs them as a JSON array by specifying an xm_json module instance as the OutputType.

nxlog.conf

<Extension json>
    Module       xm_json
</Extension>

<Input myapp>
    Module       im_file
    File         'C:\MyApp\app.log'
    InputType    json (1)
</Input>

<Output file>
    Module       om_file

    <Exec>
        delete($EventReceivedTime); (2)
        delete($SourceModuleType);
    </Exec>

    OutputType   json (3)
    File         'C:\temp\nxlog-output.log'
</Output>

<Route r1>
    Path         myapp => file
</Route>

1	Sets the InputType to the xm_json module instance.
2	Deletes unnecessary core fields.
3	Sets the OutputType to the xm_json module instance.

The following are sample events in JSON format.

Input sample

{ "uid": 175, "user": "terry", "os": "linux", "active": false, "ssh": null }
[ { "uid": 235, "user": "alex", "os": "linux", "active": true, "ssh": null }, { "uid": 333, "user": "roger", "os": "windows", "active": false, "ssh": null } ]
{ "uid": 853, "user": "ben", "os": "windows", "active": false, "ssh": null }

The following JSON array shows the same events after NXLog Agent processed them. Note that the closing bracket of the array is added to the stream only when NXLog Agent closes the file.

Output sample

[ {"uid":175,"user":"terry","os":"linux","active":false,"ssh":null,"SourceModuleName":"myapp"}, {"uid":235,"user":"alex","os":"linux","active":true,"ssh":null,"SourceModuleName":"myapp"}, {"uid":333,"user":"roger","os":"windows","active":false,"ssh":null,"SourceModuleName":"myapp"}, {"uid":853,"user":"ben","os":"windows","active":false,"ssh":null,"SourceModuleName":"myapp"} ]

Example 9. Output compound data

Some SIEMs require sending data using a specific schema, which may include compound values. There are multiple ways to output compound values with the xm_json module. The configuration in these examples uses the Test Generator module to generate events in the following format.

Input sample

0@Thu Oct 02 13:17:46 2025

The first method is to use the to_json() function. However, this method may not process complex structures correctly, such as JSON objects inside arrays or nested arrays in JSON objects.

nxlog.conf

<Extension json>
    Module      xm_json
</Extension>

<Input test>
    Module      im_testgen
    MaxCount    1
    <Exec>
        $generated = ( 'Message' => 'This is a test', (1)
                       'Attributes' => ( 'Type' => 'debug',
                                         'Value' => 10
                                       )
                     );

        log_info("Generated type: " + type($generated)); (2)
    </Exec>
</Input>

<Output file>
    Module      om_file
    File        '/var/log/nxlog-output.json'
    Exec        to_json(); (3)
</Output>

<Route r1>
    Path        test => file
</Route>

1	Creates a Hash field called `$generated` comprising two keys: `Message` and `Attributes`.
2	Logs the data type of the `$generated` field. This verifies the data is stored as a hash.
3	Converts the record, including the `$generated` field, into a JSON-formatted string and writes it to the `$raw_event` core field.

The following JSON object shows the event after NXLog Agent processed it.

Output sample

{
  "SeverityValue": 2,
  "EventTime": "2025-10-02T13:17:46.014673+02:00",
  "SourceName": "nxlog",
  "ProcessID": 26368,
  "EventReceivedTime": "2025-10-02T13:17:46.014673+02:00",
  "SourceModuleName": "test",
  "SourceModuleType": "im_testgen",
  "Hostname": "PC01",
  "generated": {
    "Message": "This is a test",
    "Attributes": {
      "Type": "debug",
      "Value": 10
    }
  }
}

An alternative approach is to use the OutputType directive. This method structures the output differently but does not allow for formatting adjustments.

nxlog.conf

<Extension json>
    Module       xm_json
</Extension>

<Input test>
    Module       im_testgen
    MaxCount     1
    <Exec>
        $generated = ( 'Message' => 'This is a test', (1)
                       'Attributes' => ( 'Type' => 'debug',
                                         'Value' => 10
                                       )
                     );

        log_info("Generated type:" + type($generated)); (2)
    </Exec>
</Input>

<Output file>
    Module        om_file
    File          '/var/log/nxlog-output.json'
    OutputType    json (3)
</Output>

<Route r1>
    Path        test => file
</Route>

1	Creates a Hash field called `$generated` comprising two keys: `Message` and `Attributes`.
2	Logs the data type of the `$generated` field. This verifies the data is stored as a hash.
3	Specifies that the output format should match the structure of events from the `json` (xm_json) module instance. This method maintains the event’s original structure, including any automatically added event fields, but does not allow for custom formatting or pretty-printing.

The following JSON array shows the event after NXLog Agent processed it. Note that the closing bracket of the array is added to the stream only when NXLog Agent closes the file.

Output sample

[
  {
    "SeverityValue": 2,
    "EventTime": "2025-10-02 13:17:46",
    "SourceName": "nxlog",
    "ProcessID": 26368,
    "EventReceivedTime": "2025-10-02 13:17:46",
    "SourceModuleName": "test",
    "SourceModuleType": "im_testgen",
    "Hostname": "PC01",
    "generated": {
      "Message": "This is a test",
      "Attributes": {
        "Type": "debug",
        "Value": 10
      }
    }
  }
]