Adjust timestamps
Sometimes, log events may have an incorrect or incomplete timestamp. For example, BSD syslog messages do not include the year or timezone information, and some network devices may have the wrong time immediately after a reboot.
Below, we provide examples of adjusting timestamps with NXLog Agent.
Replace the original timestamp
A simple solution for fixing dubious timestamps is to replace them with the time when NXLog Agent received the event. This option is most suitable when the log source sends events to NXLog Agent in real time.
The following is a standard BSD syslog message.
<30>Feb 19 11:40:27 SERVER-1 sshd[26459]: Accepted publickey for john from 192.168.1.1 port 41193 ssh2This configuration listens for syslog messages over UDP and parses them into structured data using the parse_syslog() procedure of the xm_syslog module. This procedure populates the $EventTime field with the syslog message timestamp.
It then replaces the $EventTime with the value of the $EventReceivedTime core field.
Finally, it converts the log record to JSON for demonstration purposes.
<Extension syslog>
Module xm_syslog
</Extension>
<Extension json>
Module xm_json
</Extension>
<Input udp_listen>
Module im_udp
<Exec>
parse_syslog();
$EventTime = $EventReceivedTime;
to_json();
</Exec>
</Input>The following JSON object shows the same log record after NXLog Agent processed it. The to_json() procedure transforms timestamps into local time by default.
{
"EventReceivedTime":"2024-02-19T12:41:14.849544+02:00",
"SourceModuleName":"udp_listen",
"SourceModuleType":"im_udp",
"Hostname":"SERVER-1",
"SyslogFacilityValue":3,
"SyslogFacility":"DAEMON",
"SyslogSeverityValue":6,
"SyslogSeverity":"INFO",
"SeverityValue":2,
"Severity":"INFO",
"EventTime":"2024-02-19T12:41:14.849544+02:00",
"SourceName":"sshd",
"ProcessID":26459,
"Message":"Accepted publickey for john from 192.168.1.1 port 41193 ssh2"
}Reverse a timezone offset
In some edge cases, you may need to prevent a timestamp without timezone information from being parsed as local time. For example, when collecting BSD syslog messages with a UTC timestamp. In this case, you must either re-parse the timestamp or apply a corresponding reverse offset.
The following is a standard BSD syslog message. For this example, we assume the agent is installed on a machine with a UTC +01:00 timezone, but the event timestamp is in UTC.
<30>Feb 19 11:40:27 SERVER-1 sshd[26459]: Accepted publickey for john from 192.168.1.1 port 41193 ssh2This configuration collects syslog messages from a file and parses them into structured data using the parse_syslog() procedure of the xm_syslog module. This procedure populates the $EventTime field with the syslog message timestamp. However, it incorrectly assumes that the timestamp is in local time.
It then uses the parsedate() and strftime() functions to reverse the incorrect UTC offset. Doing this conversion in the input module instance is best to reduce the probability of an incorrect offset during the daylight saving time (DST) transition. Finally, it converts the log record to JSON for demonstration purposes.
<Extension syslog>
Module xm_syslog
</Extension>
<Extension json>
Module xm_json
</Extension>
<Input system_log>
Module im_file
File '/path/to/log/file'
<Exec>
parse_syslog(); (1)
$EventTime = parsedate(strftime($EventTime, '%Y-%m-%d %H:%M:%SZ')); (2)
to_json();
</Exec>
</Input>| 1 | The parse_syslog() procedure incorrectly assumes the timestamp is in local time and sets $EventTime to 2024-02-21T11:40:27+01:00. |
| 2 | This statement reverses the incorrect offset to convert the timestamp back to UTC. |
The following JSON object shows the same log record after NXLog Agent processed it. The to_json() procedure transforms timestamps into local time by default.
{
"EventReceivedTime": "2024-02-19T12:41:26.679575+01:00",
"SourceModuleName": "system_log",
"SourceModuleType": "im_file",
"Hostname": "SERVER-1",
"SyslogFacilityValue": 3,
"SyslogFacility": "DAEMON",
"SyslogSeverityValue": 6,
"SyslogSeverity": "INFO",
"SeverityValue": 2,
"Severity": "INFO",
"EventTime": "2024-02-19T12:40:27.000000+01:00",
"SourceName": "sshd",
"ProcessID": 26459,
"Message": "Accepted publickey for john from 192.168.1.1 port 41193 ssh2"
}Modify a timestamp
For general cases, you can use the plus (+) and minus (-) operators to add or subtract seconds from a timestamp.
This simple method may not be suitable for correcting a timezone that uses daylight saving time (DST) since the UTC offset depends on whether DST is in effect.
The following is a standard BSD syslog message.
<30>Feb 19 11:40:27 SERVER-1 sshd[26459]: Accepted publickey for john from 192.168.1.1 port 41193 ssh2This configuration collects syslog messages from a file and parses them into structured data using the parse_syslog() procedure of the xm_syslog module. This procedure populates the $EventTime field with the syslog message timestamp.
It then increases the timestamp by two hours. Finally, it converts the log record to JSON for demonstration purposes.
<Extension syslog>
Module xm_syslog
</Extension>
<Extension json>
Module xm_json
</Extension>
<Input system_log>
Module im_file
File '/path/to/log/file'
<Exec>
parse_syslog();
$EventTime = $EventTime + (2 * 3600); (1)
to_json();
</Exec>
</Input>| 1 | This statement adds 2 x 3600 seconds to the timestamp, i.e., 2 hours. |
The following JSON object shows the same log record after NXLog Agent processed it. The to_json() procedure transforms timestamps into local time by default.
{
"EventReceivedTime": "2024-02-19T13:45:28.530945+01:00",
"SourceModuleName": "system_log",
"SourceModuleType": "im_file",
"Hostname": "SERVER-1",
"SyslogFacilityValue": 3,
"SyslogFacility": "DAEMON",
"SyslogSeverityValue": 6,
"SyslogSeverity": "INFO",
"SeverityValue": 2,
"Severity": "INFO",
"EventTime": "2024-02-19T13:40:27.000000+01:00",
"SourceName": "sshd",
"ProcessID": 26459,
"Message": "Accepted publickey for john from 192.168.1.1 port 41193 ssh2"
}