Log event timestamps

Knowing the kind of timestamps your log sources produce is essential. Handling all variables associated with timestamps will ensure that events reach your log repository with the correct timestamp.

There are many variables to consider when dealing with timestamps, such as:

Different systems produce timestamps in different formats.
A timestamp can be a string or an integer.
It may or may not contain a timezone offset.
Timestamps may be precise up to the second, millisecond, or microsecond.

NXLog Agent provides functions for converting timestamps from datetime values to a formatted string and vice versa. These functions help you output timestamps in a unified format, fix broken formats such as timestamps without a year in syslog BSD messages, and normalize timestamps across different time zones.

See also Process log event timestamps for configuration examples.

Default timestamp handling

NXLog Agent internally stores datetime values as microseconds since the Unix epoch. However, when converting to a string, such as when writing output to a file, it presents the value in a human-readable format. It’s important to note that NXLog Agent writes timestamps in local time by default. The format is typically YYYY-MM-DD hh:mm:ss unless you use specific modules that handle timestamps differently.

The following modules have different default timestamp handling than the above:

JSON (xm_json)

JSON does not contain a datetime type, so timestamps in JSON logs are often of string type. The module attempts to parse strings that begin with four digits as datetime and treats timestamps without a timezone offset as local time. By default, the to_json() procedure and function convert datetime values to a string in the YYYY-MM-DDThh:mm:ss.sTZ format.

Syslog (xm_syslog)

The module parses the message timestamp to set the $EventTime field. Since BSD syslog timestamps do not include the year and timezone information, the module adds the year as explained in the fix_year() description and assumes it is in local time. When converting the $EventTime field to a string:

The to_syslog_bsd() procedure uses the Mmm dd hh:mm:ss format in local time according to the RFC 3164 specification.
The to_syslog_ietf() procedure uses the YYYY-MM-DDThh:mm:ss.sTZ format according to the RFC 5424 specification.

We recommend checking the documentation of the relevant modules in the NXLog Agent Reference Manual to ensure your configuration produces the expected result.

Parsing custom timestamp formats

The NXLog language contains two functions for converting a string to datetime:

parsedate: The parsedate() function automatically parses most standard date and time formats. It is especially useful when parsing logs with different timestamp formats.
strptime: The strptime() function converts a string to datetime according to the specified format. It supports the same format descriptors as the Linux strptime(3) function.

Both functions return undef if they fail to parse the specified string, allowing you to configure a fallback timestamp.

The JSON (xm_json) module implements additional timestamp parsing. You can turn off automatic date parsing by setting the ParseDate directive to FALSE and use one of the above functions instead.

Formatting timestamps

You can use the DateFormat global directive to define a custom date and time format when converting a datetime value to a string.

In addition, the NXLog language contains the strftime() function for converting a datetime value to a string in the specified format.

Some modules implement different timestamp formatting rules.

JSON (xm_json): Set the DateFormat module directive to specify a custom output format. The to_json() procedure and function will convert datetime values to a string in this format.
Syslog (xm_syslog): The to_syslog_bsd() and to_syslog_ietf() procedures generate the message timestamp from the $EventTime field as specified by the RCF 3164 or RFC 5424 specification accordingly, regardless of the DateFormat global directive.

UTC timestamps

NXLog Agent can produce timestamps with the timezone offset. However, there are instances when you may want to normalize timestamps to UTC, such as if the receiving end doesn’t handle timezone offset and your log sources are in different time zones.

When normalizing timestamps to UTC, it’s essential to consider both the parsing of timestamps during log collection and how NXLog Agent writes timestamps in the output. The following table lists configuration settings and functions for parsing and generating timestamps in UTC with NXLog Agent.

Table 1. NXLog Agent settings and functions for handling UTC timestamps
Option	Type	Description
integer()	Core function	Calling this function with a datetime argument returns an integer representing the number of microseconds that have elapsed since the epoch, also known as Unix time. This format is always in UTC.
parsedate()	Core function	When you call this function with the utc argument set to `TRUE`, it treats timestamps without a timezone offset as UTC.
DateFormat	Global directive	This directive defines the date format for converting datetime values to string. NXLog Agent writes the timestamp in UTC if you specify a date format with the UTC suffix.
GenerateDateInUTC	Global directive	When this directive is `TRUE`, NXLog Agent outputs timestamps in UTC in `YYYY-MM-DD hh:mm:ss` format. It does not apply when creating JSON output with the `to_json()` procedure or function.
ParseDateInUTC	Global directive	When this directive is `TRUE`, NXLog Agent treats timestamps in `YYYY-MM-DD hh:mm:ss` format without a timezone offset as UTC. It does not affect timestamps in other date formats.
DateFormat	Module directive	This directive is available for the xm_json module to define the date format when converting data to JSON. You can set it differently for each xm_json module instance, and it is independent of the global DateFormat directive.
UTCTimestamp	Module directive	This directive is available for the xm_syslog module. When set to `TRUE`, NXLog Agent writes the timestamp in the syslog header in UTC.