Preserve timestamps across time zones

NXLog Agent outputs timestamps in local time by default. However, you can change this behavior to normalize timestamps to UTC. To do so, you must consider both how you parse timestamps on collection as well as how you output timestamps.

Below, we provide several examples of generating UTC timestamps with NXLog Agent. In these examples, we execute the configurations on a machine with the timezone set to UTC+1.

Parsing timestamps without timezone information

The parsedate() function treats timestamps that do not contain timezone information as local time by default. However, the function also accepts a second boolean argument, utc. When this argument is TRUE, the function treats timestamps without timezone information as UTC.

Example 1. Parsing timestamps as UTC

The following log sample contains a timestamp without a UTC offset.

Input sample

2024-02-22T20:14:15.003	SERVER-1	An account failed to log on.

This configuration reads logs from a file and parses records to structured data with a regular expression. It then uses the parsedate() function to parse the captured string and set the $EventTime field. Finally, it converts the log record to JSON for demonstration purposes.

nxlog.conf

<Extension json>
    Module        xm_json
    DateFormat    YYYY-MM-DDThh:mm:ss.sUTC  (1)
</Extension>

<Input auth_events>
    Module        im_file
    File          '/path/to/log/file'
    <Exec>
        if $raw_event =~ /^(.+)\t(.+)\t(.+)/
        {
            $EventTime = parsedate($1, TRUE);  (2)
            $Hostname = $2;
            $Message = $3;
            to_json();  (3)
        }
    </Exec>
</Input>

1	The DateFormat directive specifies the output date format when using xm_json's functions and procedures.
2	Calls the `parsedate()` function with the `utc` argument set to `TRUE`. This argument specifies that if the timestamp does not contain timezone information, the function should treat it as UTC.
3	The to_json() procedure of the xm_json module converts the log record to JSON format and sets the `$raw_event` field.

The following JSON object shows the same log record after NXLog Agent processed it.

Output sample

{
  "EventReceivedTime": "2024-02-22T20:16:04.573668Z",
  "SourceModuleName": "auth_events",
  "SourceModuleType": "im_file",
  "Hostname": "SERVER-1",
  "EventTime": "2024-02-22T20:14:15.003000Z",
  "Message": "An account failed to log on."
}

Since JSON does not contain a datetime type, the string type is commonly used for timestamps in JSON logs. By default, the xm_json module attempts to parse strings that begin with four digits as datetime and will treat timestamps without a UTC offset as local time. You can change this behavior with the ParseDate directive.

Example 2. Parsing timestamp fields in JSON logs as UTC

The following JSON log sample contains a timestamp without a UTC offset.

Input sample

{
  "EventTime": "2024-02-22T20:14:15.003",
  "Hostname": "SERVER-1",
  "Message": "An account failed to log on."
}

This configuration reads logs from a file and parses records to structured data with the parse_json() procedure of the xm_json module. It then uses the parsedate() function to convert the captured timestamp string to datetime. Finally, it converts the log record back to JSON for demonstration purposes.

nxlog.conf

<Extension json>
    Module        xm_json
    ParseDate     FALSE  (1)
    DateFormat    YYYY-MM-DDThh:mm:ss.sUTC  (2)
</Extension>

<Input auth_events>
    Module        im_file
    File          '/path/to/log/file'
    <Exec>
        parse_json();
        $EventTime = parsedate($EventTime, TRUE);  (3)
        to_json();  (4)
    </Exec>
</Input>

1	Sets the ParseDate directive to `FALSE` to switch off automatic date parsing.
2	The DateFormat directive specifies the output date format when using xm_json's functions and procedures.
3	Calls the `parsedate()` function with the `utc` argument set to `TRUE`. This argument specifies that if the timestamp does not contain timezone information, the function should treat it as UTC.
4	The to_json() procedure of the xm_json module converts the log record to JSON format and sets the `$raw_event` field.

The following JSON object shows the same log record after NXLog Agent processed it.

Output sample

{
    "EventReceivedTime": "2024-02-22T20:16:04.573668Z",
    "SourceModuleName": "auth_events",
    "SourceModuleType": "im_file",
    "Hostname": "SERVER-1",
    "EventTime": "2024-02-22T19:14:15.003000Z",
    "Message": "An account failed to log on."
  }

Generating timestamps in UTC

Converting a datetime value to a string depends on the function, procedure, and module you use. The global DateFormat directive applies to most functions and procedures. However, modules like xm_json and xm_syslog handle datetime values differently.

The examples below demonstrate how to output dates in UTC when generating CSV, JSON, and syslog formats.

The global DateFormat directive also applies to the timestamp format in the NXLog Agent LogFile.

Example 3. Generating UTC timestamps with the global DateFormat directive

The following log sample contains a timestamp without a UTC offset.

Input sample

2024-02-22T20:14:15.003	SERVER-1	An account failed to log on.

This configuration reads logs from a file and parses records to structured data with a regular expression. It then uses the parsedate() function to parse the captured string and set the $EventTime field. Unless the event time contains timezone information, this function treats the timestamp as local time. Finally, it converts the log record to CSV for demonstration purposes.

nxlog.conf

DateFormat    YYYY-MM-DDThh:mm:ss.sUTC  (1)

<Extension csv>
    Module    xm_csv
    Fields    $EventTime, $Hostname, $Message
</Extension>

<Input auth_events>
    Module    im_file
    File      '/path/to/log/file'
    <Exec>
        if $raw_event =~ /^(.+)\t(.+)\t(.+)/
        {
            $EventTime = parsedate($1);
            $Hostname = $2;
            $Message = $3;
            to_csv();  (2)
        }
    </Exec>
</Input>

1	The DateFormat directive specifies the output date format.
2	The to_csv() procedure of the xm_csv module converts the log record to CSV and sets the `$raw_event` field.

The following output shows the same log record after NXLog Agent processed it. These timestamp variations would produce the same output:

2024-02-22T21:14:15.003+02:00
2024-02-22T19:14:15.003Z

Output sample

2024-02-22T19:14:15.003000Z,"SERVER-1","An account failed to log on."

Example 4. Generating JSON logs with UTC timestamps

The following log sample contains a timestamp without a UTC offset.

Input sample

2024-02-22T20:14:15.003	SERVER-1	An account failed to log on.

This configuration reads logs from a file and parses records to structured data with a regular expression. It then uses the parsedate() function to parse the captured string and set the $EventTime field. Unless the event time contains timezone information, this function treats the timestamp as local time. Finally, it converts the log record to JSON format.

nxlog.conf

<Extension json>
    Module        xm_json
    DateFormat    YYYY-MM-DDThh:mm:ss.sUTC  (1)
</Extension>

<Input auth_events>
    Module        im_file
    File          '/path/to/log/file'
    <Exec>
        if $raw_event =~ /^(.+)\t(.+)\t(.+)/
        {
            $EventTime = parsedate($1);
            $Hostname = $2;
            $Message = $3;
            to_json();  (2)
        }
    </Exec>
</Input>

1	Sets the DateFormat directive of the xm_json module instance to an ISO 8601 date format with the `UTC` suffix.
2	The to_json() procedure of the xm_json module converts the log record to JSON and sets the `$raw_event` field.

The following JSON object shows the same log record after NXLog Agent processed it.

Output sample

{
    "EventReceivedTime": "2024-02-22T20:16:04.573668Z",
    "SourceModuleName": "auth_events",
    "SourceModuleType": "im_file",
    "Hostname": "SERVER-1",
    "EventTime": "2024-02-22T19:14:15.003000Z",
    "Message": "An account failed to log on."
  }

Example 5. Generating syslog messages with UTC timestamps

The following log sample contains a timestamp without a UTC offset.

Input sample

2024-02-22T20:14:15.003	SERVER-1	An account failed to log on.

This configuration reads logs from a file and parses records to structured data with a regular expression. It then uses the parsedate() function to parse the captured string and set the $EventTime field. Unless the event time contains timezone information, this function treats the timestamp as local time. Finally, it converts the log record to BSD syslog.

nxlog.conf

<Extension syslog>
    Module          xm_syslog
    UTCTimestamp    TRUE  (1)
</Extension>

<Input auth_events>
    Module          im_file
    File            '/path/to/log/file'
    <Exec>
        if $raw_event =~ /^(.+)\t(.+)\t(.+)/
        {
            $EventTime = parsedate($1);
            $Hostname = $2;
            $Message = $3;
            to_syslog_bsd();  (2)
        }
    </Exec>
</Input>

1	Sets the UTCTimestamp directive to `TRUE` to output timestamps in UTC.
2	The to_syslog_bsd() procedure of the xm_syslog module converts the log record to syslog and sets the `$raw_event` field.

The following output shows the same log record after NXLog Agent processed it.

Output sample

<13>Feb 22 19:14:15 SERVER-1 An account failed to log on.

Example 6. Generating UTC integer timestamps

The following log sample contains a timestamp without a UTC offset.

Input sample

2024-02-22T20:14:15.003	SERVER-1	An account failed to log on.

This configuration reads logs from a file and parses records to structured data with a regular expression. It then uses the parsedate() function to parse the captured string and set the $EventTime field. Unless the event time contains timezone information, this function treats the timestamp as local time. Finally, it converts the log record to CSV for demonstration purposes.

nxlog.conf

<Extension csv>
    Module    xm_csv
    Fields    $EventTime, $UTCTimestamp, $Hostname, $Message
</Extension>

<Input auth_events>
    Module    im_file
    File      '/path/to/log/file'
    <Exec>
        if $raw_event =~ /^(.+)\t(.+)\t(.+)/
        {
            $EventTime = parsedate($1);
            $UTCTimestamp = integer($EventTime);  (1)
            $Hostname = $2;
            $Message = $3;
            to_csv();  (2)
        }
    </Exec>
</Input>

1	The integer() function converts a datetime to a UTC timestamp in the Unix time format.
2	The to_csv() procedure of the xm_csv module converts the log record to CSV and sets the `$raw_event` field.

The following output shows the same log record after NXLog Agent processed it.

Output sample

2024-02-22 20:14:15,1708629255003000,"SERVER-1","An account failed to log on."

Configure global timestamp settings

NXLog Agent uses then`YYYY-MM-DD hh:mm:ss` date and time format by default and provides the following global directives to parse and generate timestamps in UTC:

ParseDateInUTC is equivalent to using parsedate(arg, TRUE) to parse timestamps without timezone information as UTC.
GenerateDateInUTC is equivalent to specifying DateFormat YYYY-MM-DD hh:mm:ssUTC to output timestamps in UTC.

Example 7. Using the GenerateDateInUTC and ParseDateInUTC directives

The following log record contains a timestamp without a timezone offset.

Input sample

2024-02-22T20:14:15.003	SERVER-1	An account failed to log on.

This configuration sets the ParseDateInUTC and GenerateDateInUTC global directives to TRUE. It reads logs from a file and parses records to structured data with a regular expression. It then uses the parsedate() function to parse the captured string and set the $EventTime field. Unless the event time contains timezone information, this function will treat timestamps as UTC according to the ParseDateInUTC directive. Finally, it converts the log record to CSV for demonstration purposes.

nxlog.conf

GenerateDateInUTC    TRUE
ParseDateInUTC       TRUE

<Extension csv>
    Module    xm_csv
    Fields    $EventTime, $Hostname, $Message
</Extension>

<Input auth_events>
    Module    im_file
    File      '/path/to/log/file'
    <Exec>
        if $raw_event =~ /^(.+)\t(.+)\t(.+)/
        {
            $EventTime = parsedate($1);
            $Hostname = $2;
            $Message = $3;
            to_csv();  (1)
        }
    </Exec>
</Input>

1	The to_csv() procedure of the xm_csv module converts the log record to CSV and sets the `$raw_event` field. It will output timestamps without timezone information in UTC according to the `GenerateDateInUTC` directive.

The following output shows the same log record after NXLog Agent processed it. These timestamp variations would produce the same output:

2024-02-22 21:14:15+01:00
2024-02-22 20:14:15Z

Output sample

2024-02-22 20:14:15,"SERVER-1","An account failed to log on."

The configuration above does not apply when parsing logs with the parse_json() procedure. The next example demonstrates using the ParseDateInUTC global directive when parsing JSON logs.

Example 8. Using the ParseDateInUTC directive when parsing JSON logs

The following is a log record containing a timestamp without a timezone offset. In this example, we used this sample as input data and executed the configuration on a machine with the timezone set to UTC+2.

Input sample

{
  "EventTime": "2024-02-22T20:14:15",
  "Hostname": "SERVER-1",
  "Message": "An account failed to log on."
}

This configuration sets the ParseDateInUTC and GenerateDateInUTC global directives to TRUE. It reads logs from a file and parses records to structured data with the parse_json() procedure of the xm_json module. It then uses the parsedate() function to parse the captured string and set the $EventTime field. Unless the event time contains timezone information, this function will treat timestamps as UTC according to the ParseDateInUTC directive. Finally, it converts the log record to CSV for demonstration purposes.

nxlog.conf

GenerateDateInUTC    TRUE
ParseDateInUTC       TRUE

<Extension json>
    Module       xm_json
    ParseDate    FALSE  (1)
</Extension>

<Extension csv>
    Module       xm_csv
    Fields       $EventTime, $Hostname, $Message
</Extension>

<Input auth_events>
    Module       im_file
    File         '/path/to/log/file'
    <Exec>
        parse_json();
        $EventTime = parsedate($EventTime);
        to_csv(); (2)
    </Exec>
</Input>

1	Sets the ParseDate directive to `FALSE` to switch off automatic date parsing.
2	The to_csv() procedure of the xm_csv module converts the log record to CSV and sets the `$raw_event` field. It will output timestamps without timezone information in UTC according to the `GenerateDateInUTC` directive.

The following output shows the same log record after NXLog Agent processed it. These timestamp variations would produce the same output:

2021-09-22 17:14:15+02:00
2021-09-22 15:14:15Z

Output sample

2024-02-22 20:14:15,"SERVER-1","An account failed to log on."

GenerateDateInUTC does not apply when creating JSON output with the to_json() procedure or function. Use the xm_json DateFormat directive instead. See Generating JSON logs with UTC timestamps above.