Rate limiting and traffic shaping of logs
The application of rate limiting and traffic shaping for logs can improve utilization of services which are running in parallel with NXLog.
Rate limiting
Rate limiting restricts the number of logs that can be read by NXLog within a given time unit.
The poor man’s tool for rate limiting is the sleep() procedure.
In the following example, sleep() is invoked with 500 microseconds. This means that the input module will be able to read at most 2000 messages per second.
<Input in>
Module im_tcp
Host 0.0.0.0
Port 1514
Exec sleep(500);
</Input>This is not very precise because the module can do additional processing which can add to the total execution time, but it gets fairly close.
| It is not recommended to use rate limiting on a route that reads logs over UDP. |
Traffic shaping
Shaping the outgoing traffic of NXLog can guarantee that a required bandwidth remains for other running services.
The traffic shaping script can be downloaded from the nxlog-public/contrib repository.
The script does not require configuring NXLog, but it needs to be
configured to run at startup with tools like crontab or rc.local.
To configure running the script with crontab, the @reboot task should
be added to the /etc/crontab file.
@reboot /usr/local/sbin/traffic-shaper.shTo configure running the script with rc.local, the script path
should be added to the /etc/rc.local file.
/usr/local/sbin/traffic-shaper.shThe traffic shaper ties to the destination port on the network level and can shape traffic in accordance with priorities. For example, high priority can be configured for a database server and low priority for a backup system.
For more information about the Linux traffic control, see the Traffic Control HOWTO section on The Linux Documentation Project website.
