Rate limiting and traffic shaping of logs

The application of rate limiting and traffic shaping for logs can improve utilization of services which are running in parallel with NXLog.

Rate limiting

Rate limiting restricts the number of logs that can be read by NXLog within a given time unit.

The poor man’s tool for rate limiting is the sleep() procedure.

Example 1. Rate limiting with the sleep() procedure

In the following example, sleep() is invoked with 500 microseconds. This means that the input module will be able to read at most 2000 messages per second.

nxlog.conf
<Input in>
    Module  im_tcp
    Host    0.0.0.0
    Port    1514
    Exec    sleep(500);
</Input>

This is not very precise because the module can do additional processing which can add to the total execution time, but it gets fairly close.

It is not recommended to use rate limiting on a route that reads logs over UDP.

Traffic shaping

Shaping the outgoing traffic of NXLog can guarantee that a required bandwidth remains for other running services.

The traffic shaping script can be downloaded from the nxlog-public/contrib repository.

The script does not require configuring NXLog, but it needs to be configured to run at startup with tools like crontab or rc.local.

To configure running the script with crontab, the @reboot task should be added to the /etc/crontab file.

/etc/crontab
@reboot /usr/local/sbin/traffic-shaper.sh

To configure running the script with rc.local, the script path should be added to the /etc/rc.local file.

/etc/rc.local
/usr/local/sbin/traffic-shaper.sh

The traffic shaper ties to the destination port on the network level and can shape traffic in accordance with priorities. For example, high priority can be configured for a database server and low priority for a backup system.

For more information about the Linux traffic control, see the Traffic Control HOWTO section on The Linux Documentation Project website.