Preserve timestamps between NXLog Agent instances

When transferring logs between two or more NXLog Agent instances, you can preserve event fields using the Binary format. As a result, you only need to parse timestamps when the first instance processes the log event.

The following is an example of configuring two NXLog Agent instances to send and receive logs in Binary format.

Example 1. Transferring logs between NXLog Agent instances in Binary format

The following log sample includes an ISO 8601 timestamp in UTC.

Input sample

2024-02-22T16:14:15.003Z	SERVER-1	An account failed to log on.

The first NXLog Agent instance reads logs from a file and parses records to structured data with a regular expression. It then uses the parsedate() function to convert the event time field to datetime. Finally, it forwards the logs to another NXLog Agent instance over TCP.

nxlog.conf

<Input auth_events>
    Module        im_file
    File          '/path/to/log/file'
    <Exec>
        if $raw_event =~ /^(.+)\t(.+)\t(.+)/
        {
            $EventTime = parsedate($1);
            $Hostname = $2;
            $Message = $3;
        }
    </Exec>
</Input>

<Output tcp>
    Module        om_tcp
    Host          192.168.0.123:514
    OutputType    Binary  (1)
</Output>

1	The OutputType directive is set to `Binary` to output the data in an NXLog proprietary format that preserves fields and their values.

The second NXLog Agent instance listens for logs over TCP. Since the logs are already parsed by the first NXLog Agent instance, it does not need to parse them again.

The configuration converts logs to JSON for demonstration purposes. At this stage, the logs are ready to be forwarded to their final destination, such as a central log repository or SIEM.

nxlog.conf

<Extension json>
  Module          xm_json
</Extension>

<Input tcp_listen>
    Module        im_tcp
    ListenAddr    0.0.0.0:1514
    InputType     Binary  (1)
    Exec          to_json();
</Input>

1	The InputType directive is set to `Binary`, which means it expects to receive logs from another NXLog Agent instance.

The following JSON object shows the same log record after the two NXLog Agent instances processed it. Notice that SourceModuleName and SourceModuleType contain values for the first instance since these fields are preserved when using the Binary format. The to_json() procedure transforms timestamps into local time by default.

Output sample

{
  "EventReceivedTime": "2024-02-22T17:15:16.950067+01:00",
  "SourceModuleName": "auth_events",
  "SourceModuleType": "im_file",
  "Hostname": "SERVER-1",
  "EventTime": "2024-02-22T17:14:15.003000+01:00",
  "Message": "An account failed to log on.",
  "MessageSourceAddress": "192.168.0.122"
}