Preserve timestamps between NXLog Agent instances

When transferring telemetry data between two or more NXLog Agent instances, you can preserve event fields using the Binary format. As a result, you only need to parse timestamps when the first instance processes the data.

The following is an example of configuring two NXLog Agent instances to send and receive telemetry data in Binary format.

Example 1. Transferring data between NXLog Agent instances in Binary format

The following event sample includes an ISO 8601 timestamp in UTC.

Input sample

2024-02-22T16:14:15.003Z	SERVER-1	An account failed to log on.

The first NXLog Agent instance reads events from a file and parses records to structured data with a regular expression. It then uses the parsedate() function to convert the event time field to datetime. Finally, it forwards the events to another NXLog Agent instance over TCP.

nxlog.conf

<Input auth_events>
    Module        im_file
    File          '/path/to/log/file'
    <Exec>
        if $raw_event =~ /^(.+)\t(.+)\t(.+)/
        {
            $EventTime = parsedate($1);
            $Hostname = $2;
            $Message = $3;
        }
    </Exec>
</Input>

<Output tcp>
    Module        om_tcp
    Host          192.168.0.123:514
    OutputType    Binary  (1)
</Output>

1	Setting the OutputType directive to `Binary` outputs the data in an NXLog proprietary format that preserves fields and their values.

The second NXLog Agent instance listens for data over TCP. Since the data is already parsed by the first NXLog Agent instance, it does not need to parse it again.

The configuration converts events to JSON for demonstration purposes. At this stage, the data is ready to be forwarded to its final destination, such as a SIEM.

nxlog.conf

<Extension json>
  Module          xm_json
</Extension>

<Input tcp_listen>
    Module        im_tcp
    ListenAddr    0.0.0.0:1514
    InputType     Binary  (1)
    Exec          to_json();
</Input>

1	The InputType directive is set to `Binary`, which means it expects to receive data from another NXLog Agent instance.

The following JSON object shows the same event after the two NXLog Agent instances processed it. Notice that SourceModuleName and SourceModuleType contain values for the first instance since these fields are preserved when using the Binary format. The to_json() procedure transforms timestamps into local time by default.

Output sample

{
  "EventReceivedTime": "2024-02-22T17:15:16.950067+01:00",
  "SourceModuleName": "auth_events",
  "SourceModuleType": "im_file",
  "Hostname": "SERVER-1",
  "EventTime": "2024-02-22T17:14:15.003000+01:00",
  "Message": "An account failed to log on.",
  "MessageSourceAddress": "192.168.0.122"
}