Parse timestamps

The NXLog language provides two functions to convert a string to datetime:

parsedate() automatically converts well-known date and time formats.
strptime() converts a timestamp according to the specified format.

Below, we provide examples of parsing different date and time formats with NXLog Agent.

Parse standard date formats

The parsedate() function recognizes most date and time formats, including ISO 8601, Windows, and Unix epoch timestamps. It treats timestamps without timezone information as local time and will add the current year to timestamps that do not contain one.

Example 1. Parsing standard date formats

The following log sample starts with an ISO 8601 timestamp followed by a tab.

Input sample

2024-02-14T10:11:15.003Z	SERVER-1	An account failed to log on.

This configuration uses a regular expression to extract the timestamp from the above input sample. It then uses the parsedate() function to parse the captured string and set the $EventTime field. Finally, it converts the log record to JSON for demonstration purposes.

nxlog.conf

<Extension json>
    Module    xm_json
</Extension>

<Input auth_events>
    Module    im_file
    File      '/path/to/log/file'
    <Exec>     
        if $raw_event =~ /^([^\t]+)\t(.*)/ (1)
        {
            $EventTime = parsedate($1); (2)
            $Message = $2;
            to_json();
        }
    </Exec>         
</Input>

1	This regular expression parses the event into a timestamp and message.
2	Creates fields from the regex capturing groups.

The following JSON object shows the same log record after NXLog Agent processed it. The to_json() procedure transforms timestamps into local time by default.

Output sample

{
  "EventReceivedTime": "2024-02-14T11:14:59.522067+01:00",
  "SourceModuleName": "auth_events",
  "SourceModuleType": "im_file",
  "Hostname": "SERVER-1",
  "EventTime": "2024-02-14T11:11:15.003000+01:00",
  "Message": "SERVER-1\tAn account failed to log on."
}

Set a fallback timestamp

Sometimes, log events may contain an invalid or unexpected timestamp format. The parsedate() function returns undef if it does not recognize the format, which allows you to configure a fallback timestamp for when this happens.

Example 2. Falling back to the current date and time

The following log sample starts with an invalid date followed by the time and message.

Input sample

02-24 10:11:15	SERVER-1	An account failed to log on.

This configuration uses a regular expression to extract the timestamp. It expects events with a <date> <time> <message> format. It then uses the parsedate() function to parse the captured strings and set the $EventTime field. If the function fails to parse the timestamp, it instead sets the $EventTime to the current date and time. Finally, it converts the log record to JSON for demonstration purposes.

nxlog.conf

<Extension json>
    Module    xm_json
</Extension>

<Input auth_events>
    Module    im_file
    File      '/path/to/log/file'
    <Exec>
        if $raw_event =~ /^(\S+)\s+(\S+)\t(.*)/  (1)
        {
            $EventTime = parsedate($1 + " " + $2); (2)
            $Message = $3;

            if not defined($EventTime)  (3)
            { 
                $EventTime = now();
                $Message = $raw_event;
            }

            to_json();
        }
    </Exec>      
</Input>

1	This regular expression parses the event into a date, time, and message.
2	Creates fields from the regex capturing groups.
3	This condition checks whether the timestamp is available. If not, it sets the `$EventTime` to the current date and the `$Message` to the original event.

The following JSON object shows the same log record after NXLog Agent processed it. Notice that the $Message field contains the original event text and $EventTime is the date and time when NXLog Agent processed the log record. The to_json() procedure transforms timestamps into local time by default.

Output sample

{
  "EventReceivedTime": "2024-02-14T17:32:05.251791+01:00",
  "SourceModuleName": "auth_events",
  "SourceModuleType": "im_file",
  "Hostname": "SERVER-1",
  "EventTime": "2024-02-14T17:32:05.251844+01:00",
  "Message": "02-24 10:11:15\tSERVER-1\tAn account failed to log on."
}

Parse custom date formats

The strptime() function allows you to parse timestamps in unconventional formats. It accepts a date and time format in the strptime(3) specification.

Example 3. Parsing a custom date format

The following log sample starts with the date followed by a tab and the time in single-digit format.

Input sample

2024-02-14	4:3:2	SERVER-1	An account failed to log on.

This configuration uses a regular expression to extract the timestamp from the above input sample. It then uses the strptime() function to parse the captured string and set the $EventTime field. Finally, it converts the log record to JSON for demonstration purposes.

nxlog.conf

<Extension json>
    Module    xm_json
</Extension>

<Input auth_events>
    Module    im_file
    File      '/path/to/log/file'
    <Exec>
        if $raw_event =~ /^(\d+-\d+-\d+\t\d+:\d+:\d+)\s(.*)/ (1)
        {
            $EventTime = strptime($1, '%Y-%m-%d%t%H:%M:%S'); (2)
            $Message = $2;
            to_json();
        }
    </Exec>
</Input>

1	This regular expression parses the event into a timestamp and message.
2	Creates fields from the regex capturing groups.

The following JSON object shows the same log record after NXLog Agent processed it. The to_json() procedure transforms timestamps into local time by default.

Output sample

{
  "EventReceivedTime": "2024-02-14T10:45:17.533409+01:00",
  "SourceModuleName": "auth_events",
  "SourceModuleType": "im_file",
  "Hostname": "SERVER-1",
  "EventTime": "2024-02-14T04:03:02.000000+01:00",
  "Message": "SERVER-1\tAn account failed to log on."
}