Parse CSV logs

CSV is a text-based format commonly used for logging. It stores tabular data delimited by a character such as the comma, space, or semicolon. There is no official CSV logging standard; every vendor defines their own format, such as the fields, delimiter, quote character, etc.

Below, we provide examples of collecting and parsing different CSV formats with NXLog Agent.

Example 1. Parsing simple CSV logs

This configuration uses the im_file input module to collect CSV logs from a file and parses records with the xm_csv module.

nxlog.conf

<Extension csv_parser> (1)
    Module      xm_csv
    Fields      $EventTime, $Severity, $Message
</Extension>

<Input csv_log>
    Module      im_file
    File        'path/to/logs/*'
    Exec        parse_csv(); (2)
</Input>

1	Defines the basic settings for parsing the CSV records.
2	Calls the parse_csv() procedure to parse the record into structured data.

Input sample

2024-02-08T10:27:45.919858+01:00,ERROR,File not found

When the NXLog Agent configuration above processes this log event, it adds the following fields to the log record in addition to the core fields.

Field	Value
$EventTime	2024-02-08T10:27:45.919858+01:00
$Severity	ERROR
$Message	File not found

Field

Value

$EventTime

2024-02-08T10:27:45.919858+01:00

$Severity

ERROR

$Message

File not found

Example 2. Parsing complex CSV logs

This configuration uses the im_file input module to collect CSV logs from a file and parses records with the xm_csv module.

nxlog.conf

<Extension csv_parser> (1)
    Module        xm_csv
    Fields        $id, $username, $name, $surname, $timestamp, $message
    FieldTypes    integer, string, string, string, datetime, string
    Delimiter     ' '
    QuoteChar     '"'
    UndefValue    -
</Extension>

<Input csv_log>
    Module        im_file
    File          '/path/to/logs/*'
    Exec          parse_csv(); (2)
</Input>

1	Defines granular settings for parsing the CSV records.
2	Calls the parse_csv() procedure to parse the record into structured data.

Input sample

123 "jdoe" "John K." "Doe" 2024-02-08T10:05:32+01:00 "User login"

When the NXLog Agent configuration above processes this log event, it adds the following fields to the log record in addition to the core fields.

Field	Value
$id	123
$username	jdoe
$name	John K.
$surname	Doe
$timestamp	2024-02-08T10:05:32.000000+01:00
$message	User login

Field

Value

$id

123

$username

jdoe

$name

John K.

$surname

Doe

$timestamp

2024-02-08T10:05:32.000000+01:00

$message

User login