Control NXLog Agent bandwidth usage

In cases where NXLog Agent is running on a system hosting other services, you may need to ensure that no one service uses all the bandwidth, hindering the operation of the remaining services.

Below, we provide some options for limiting NXLog Agent’s bandwidth usage.

Rate limiting

Rate limiting allows you to cap the number of logs NXLog Agent processes within a given timeframe. Using the sleep() procedure, you can implement a rate limiting strategy. The rate may not be exact because the module instance can do additional processing that increases the processing time, but it is relatively close.

Example 1. Introducing a log processing delay

This configuration forwards logs over TCP with the om_tcp output module. It invokes a 500 microseconds sleep for every log record, forwarding 2,000 log events per second at most.

nxlog.conf

<Output siem>
    Module    om_tcp
    Host      192.168.0.123:1514
    Exec      sleep(500);
</Output>

We do not recommend using rate limiting when receiving logs over UDP.

Traffic shaping

Traffic shaping involves controlling the data flow over the network. Shaping NXLog Agent’s outgoing traffic ensures enough bandwidth remains for other applications and services.

An example script for shaping NXLog Agent’s outgoing traffic on Linux is available in our public Git repository. The script does not require any additional NXLog Agent configuration, but you must configure it to run when the operating system starts with a tool like crontab or rc.local.

For example, add the following @reboot task to /etc/crontab to run the traffic-shaping script on startup with crontab.

/etc/crontab

@reboot /usr/local/sbin/traffic-shaper.sh

Similarly, to run the traffic-shaping script with rc.local, add the script’s path to the /etc/rc.local file.

/etc/rc.local

/usr/local/sbin/traffic-shaper.sh

The traffic shaper ties to the destination port on the network level and can shape traffic according to priority. For example, a database server can have high priority, while a backup system might have low priority.

For more information on Linux traffic control, see the Traffic Control HOWTO on The Linux Documentation Project website.