Configure load balancing

Several NXLog Agent modules include self-managed failover functionality. See NXLog Agent failover for a further explanation of failover.

Below, we provide examples to help you set up an NXLog Agent relay cluster with load balancing and automatic failover. Such a configuration distributes the data processing load between the cluster members, prevents idle nodes, and provides failover in case any nodes fail.

Create a load-balancing cluster with failover

The following diagram illustrates an NXLog Agent relay cluster receiving data from different sources. Each source sends data to a different primary node in the cluster with failover to the other nodes.

With this setup, data sources communicate with the NXLog Agent relay cluster in an active/passive model, but the cluster operates as a static active/active load balancer. This technique minimizes the chance of idle cluster nodes, ensuring the efficient use of each node and boosting performance.

NXLog Agent cluster with load balancing and failover

Figure 1. NXLog Agent load balancing and automatic failover mode

The following steps guide you in configuring an NXLog Agent cluster and forwarding logs to it as described above. See Configure an NXLog Agent relay for more configuration examples.

Configure the NXLog Agent cluster nodes

The first step is configuring each NXLog Agent node in the relay cluster to receive data from your sources and forward it to its destination.

Example 1. Configuring cluster nodes

This configuration uses an im_batchcompress module instance to receive telemetry data from other NXLog Agent instances. It then forwards the data to a SIEM over TCP using the om_tcp output module.

nxlog.conf

<Input nxlog_agent>
    Module        im_batchcompress
    ListenAddr    0.0.0.0:1514
</Input>

<Output siem>
    Module        om_tcp
    Host          siem.example.com:1514
</Output>

<Route relay>
    Path          nxlog_agent =>  siem
</Route>

Forward data to the cluster

Once your NXLog Agent cluster is up and running, you can configure your sources to forward data to it with the om_batchcompress output module.

Example 2. Forwarding data to an NXLog Agent cluster with failover

Configure your first source (the Linux and macOS nodes in the diagram above) to send data to the first NXLog Agent cluster node and failover to the other two nodes.

nxlog.conf

<Output agent_relay>
    Module    om_batchcompress

    # Active
    Host      192.168.1.51:1514
    
    # Passive
    Host      192.168.1.52:1514 
    Host      192.168.1.53:1514
</Output>

Configure your second source (the DNS Server nodes in the diagram above) to send data to the second NXLog Agent cluster node and failover to the other two nodes.

nxlog.conf

<Output agent_relay>
    Module  om_batchcompress

    # Active
    Host    192.168.1.52:1514 
    
    # Passive
    Host    192.168.1.53:1514 
    Host    192.168.1.51:1514
</Output>

Finally, configure your third source (the Sysmon node in the diagram above) to send data to the third NXLog Agent cluster node and failover to the other two nodes.

nxlog.conf

<Output agent_relay>
    Module  om_batchcompress

    # Active
    Host    192.168.1.53:1514 
    
    # Passive
    Host    192.168.1.51:1514
    Host    192.168.1.52:1514
</Output>