Parse JSON logs
JSON (JavaScript Object Notation) is a standard data-interchange text format consisting of key-value pairs and arrays. NXLog Agent includes a JSON parser that significantly simplifies JSON log processing.
Below, we provide examples of collecting and parsing different JSON formats with NXLog Agent.
Parse simple JSON
NXLog Agent’s xm_json module parses dates in the YYYY-MM-DDThh:mm:ss.sTZ format and includes hidden fields by default.
This configuration collects JSON logs from a file with the im_file input module and parses records with the xm_json module using the default settings.
<Extension json_parser>
Module xm_json
</Extension>
<Input json_logs>
Module im_file
File '/path/to/file'
Exec parse_json(); (1)
</Input>| 1 | Calls the parse_json() procedure to parse the record into structured data. |
The following is a JSON log sample containing hidden fields. Although it is pretty printed for legibility, the configuration expects one record per line.
{
"EventTime": "2024-02-10T11:50:26.979726+01:00",
"Hostname": "NXLog-Server-1",
"Message": "The service has started.",
"Severity": "NOTICE",
"_meta": {
"_os": "Linux Ubuntu 22.04",
"_application": "nxlog"
}
}When the NXLog Agent configuration above processes this log event, it adds the following fields to the log record in addition to the core fields.
| Field | Value |
|---|---|
$EventTime |
2024-02-10T11:50:26.979726+01:00 |
$Hostname |
NXLog-Server-1 |
$Message |
The service has started. |
$Severity |
NOTICE |
$_meta |
{"_os":"Linux Ubuntu 22.04","_application":"nxlog"} |
Parse nested JSON
JSON logs often contain data in nested objects. NXLog Agent’s xm_json module supports parsing nested objects and flattening the structure to a single level.
This configuration collects JSON logs from a file with the im_file input module and parses records with the xm_json module.
<Extension json_parser>
Module xm_json
Flatten TRUE (1)
</Extension>
<Input json_logs>
Module im_file
File '/path/to/file'
Exec parse_json(); (2)
</Input>| 1 | Enables the Flatten directive to create nested fields using the dot notation. |
| 2 | Calls the parse_json() procedure to parse the record into structured data. |
The following is a JSON log sample containing a nested object. Although it is pretty printed for legibility, the configuration expects one record per line.
{
"EventTime": "2024-02-10T11:50:26.979726+01:00",
"Hostname": "NXLog-Server-1",
"Message": "The service has started.",
"Severity": "NOTICE",
"_meta": {
"_os": "Linux Ubuntu 22.04",
"_application": "nxlog"
}
}When the NXLog Agent configuration above processes this log event, it adds the following fields to the log record in addition to the core fields.
| Field | Value |
|---|---|
$EventTime |
2024-02-10T11:50:26.979726+01:00 |
$Hostname |
NXLog-Server-1 |
$Message |
The service has started. |
$Severity |
NOTICE |
${_meta._os} |
Linux Ubuntu 22.04 |
${_meta._application} |
nxlog |
This configuration collects JSON logs from a file with the im_file input module and parses records with the xm_json module.
It expects log records containing a _meta object.
<Extension json_parser>
Module xm_json
</Extension>
<Input json_logs>
Module im_file
File '/path/to/file'
<Exec>
parse_json(); (1)
$os = extract_json("$.['_meta']['_os']"); (2)
$application = extract_json("$.['_meta']['_application']");
delete($_meta); (3)
</Exec>
</Input>| 1 | Calls the parse_json() procedure to parse the record into structured data. |
| 2 | Uses the extract_json() function to extract nested fields. |
| 3 | Deletes the original _meta object. |
The following is a JSON log sample containing a _meta nested object.
Although it is pretty printed for legibility, the configuration expects one record per line.
{
"EventTime": "2024-02-10T11:50:26.979726+01:00",
"Hostname": "NXLog-Server-1",
"Message": "The service has started.",
"Severity": "NOTICE",
"_meta": {
"_os": "Linux Ubuntu 22.04",
"_application": "nxlog"
}
}When the NXLog Agent configuration above processes this log event, it adds the following fields to the log record in addition to the core fields.
| Field | Value |
|---|---|
$EventTime |
2024-02-10T11:50:26.979726+01:00 |
$Hostname |
NXLog-Server-1 |
$Message |
The service has started. |
$Severity |
NOTICE |
$os |
Linux Ubuntu 22.04 |
$application |
nxlog |
Parse a JSON array
NXLog Agent’s xm_json module registers an input reader function that supports JSON arrays and JSON records spanning multiple lines (pretty-printed).
This configuration collects JSON logs from a file with the im_file input module and parses records with xm_json's input reader function.
<Extension json_parser>
Module xm_json
</Extension>
<Input json_logs>
Module im_file
File '/path/to/file'
InputType json_parser (1)
</Input>| 1 | Sets the InputType to the xm_json module instance. |
The input samples presented below contain a pair of JSON objects.
[{"EventTime":"2024-02-10T11:50:26.979726+01:00","Hostname":"NXLog-Server-1","Message":"The service has started.","Severity":"NOTICE"}, {"EventTime":"2024-02-10T12:37:15.501640+01:00","Hostname":"NXLog-Server-1","Message":"The service has stopped.","Severity":"WARNING"}]{
"EventTime": "2024-02-10T11:50:26.979726+01:00",
"Hostname": "NXLog-Server-1",
"Message": "The service has started.",
"Severity": "NOTICE"
}
{
"EventTime": "2024-02-10T12:37:15.501640+01:00",
"Hostname": "NXLog-Server-1",
"Message": "The service has stopped.",
"Severity": "WARNING"
}The output is the same for both input samples. When the NXLog Agent configuration above processes these log events, it adds the following fields to the log record in addition to the core fields.
| Field | Value |
|---|---|
$EventTime |
2024-02-10T11:50:26.979726+01:00 |
$Hostname |
NXLog-Server-1 |
$Message |
The service has started. |
$Severity |
NOTICE |