Preserve timestamps between NXLog Agent instances
When transferring telemetry data between two or more NXLog Agent instances, you can preserve event fields using the Binary format. As a result, you only need to parse timestamps when the first instance processes the data.
The following is an example of configuring two NXLog Agent instances to send and receive telemetry data in Binary format.
The following event sample includes an ISO 8601 timestamp in UTC.
2024-02-22T16:14:15.003Z SERVER-1 An account failed to log on.
The first NXLog Agent instance reads events from a file and parses records to structured data with a regular expression. It then uses the parsedate() function to convert the event time field to datetime. Finally, it forwards the events to another NXLog Agent instance over TCP.
<Input auth_events>
Module im_file
File '/path/to/log/file'
<Exec>
if $raw_event =~ /^(.+)\t(.+)\t(.+)/
{
$EventTime = parsedate($1);
$Hostname = $2;
$Message = $3;
}
</Exec>
</Input>
<Output tcp>
Module om_tcp
Host 192.168.0.123:514
OutputType Binary (1)
</Output>
1 | Setting the OutputType directive to Binary outputs the data in an NXLog proprietary format that preserves fields and their values. |
The second NXLog Agent instance listens for data over TCP. Since the data is already parsed by the first NXLog Agent instance, it does not need to parse it again.
The configuration converts events to JSON for demonstration purposes. At this stage, the data is ready to be forwarded to its final destination, such as a SIEM.
<Extension json>
Module xm_json
</Extension>
<Input tcp_listen>
Module im_tcp
ListenAddr 0.0.0.0:1514
InputType Binary (1)
Exec to_json();
</Input>
1 | The InputType directive is set to Binary , which means it expects to receive data from another NXLog Agent instance. |
The following JSON object shows the same event after the two NXLog Agent instances processed it.
Notice that SourceModuleName
and SourceModuleType
contain values for the first instance since these fields are preserved when using the Binary
format.
The to_json() procedure transforms timestamps into local time by default.
{
"EventReceivedTime": "2024-02-22T17:15:16.950067+01:00",
"SourceModuleName": "auth_events",
"SourceModuleType": "im_file",
"Hostname": "SERVER-1",
"EventTime": "2024-02-22T17:14:15.003000+01:00",
"Message": "An account failed to log on.",
"MessageSourceAddress": "192.168.0.122"
}