Controlling resource usage

NXLog is designed to collect and process logs as fast as possible. A side-effect of this may be high resource usage when there are many events to be processed. Operating system level tools can be used to limit resource consumption by NXLog, however this will also introduce a delay in log processing. Implementing a limit on resource usage is a good option when a log processing delay is not a problem, or to deal with intermittent high loads, since the delay will only occur for a short period of time. When dealing with a consistently high load and introducing a log processing delay is not desirable, other solutions may need to be considered such as splitting the load between multiple NXLog instances.

This section provides some options on how you can set up control methods that prevent NXLog from consuming more than the designated amount of RAM and CPU cycles. Different approaches and tools apply according to your operating system.

Linux-based systems

Using cpulimit

cpulimit is a command-line tool that can be used to limit the CPU usage of a process. A process manager like Supervisor can be used to start cpulimit automatically and execute it as a long running process. Both tools are available in most of the default repositories of Unix-like distributions.

Follow these steps to install and configure cpulimit and Supervisor:

Install both tools according to your platform.
- On RHEL/CentOS, enable the EPEL Repository:
  # yum install epel-release
  Then execute the following command to install cpulimit and Supervisor:
  # yum install cpulimit supervisor
- For Debian/Ubuntu, execute the following command:
  # apt install cpulimit supervisor
Open the Supervisor configuration file located at /etc/supervisor/supervisord.conf with a text editor.
Configure Supervisor to start cpulimit:
```
[program:cpulimit]
command=cpulimit --exe nxlog --limit 20
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/var/log/cpulimit.log
```
This configuration controls the resource usage of NXLog by limiting it to consume only 20% of processor time.

The autostart=TRUE option starts cpulimit automatically when Supervisor is started.

The autorestart=TRUE option automatically restarts cpulimit when it exits, regardless of its exit code.

Errors or messages from stdout and stderr will be written to file in /var/log/cpulimit.log. This log file provides information about process state changes, and can be useful for debugging the process if it’s not starting.

This configuration is only an example and it can be modified as needed.
Disable automatic startup of NXLog from systemd. This prevents it from colliding with the process started by cpulimit.
```
# systemctl disable nxlog
```
Execute the following commands to enable and start Supervisor:
```
# systemctl enable supervisord
# systemctl start supervisord
```

Using systemd

On operating systems that use systemd, resource usage can be controlled from the nxlog.service configuration file. Refer to the documentation of your operating system for more information on systemd usage.

The following instructions configure the NXLog service with a CPU usage limit of 10% and a maximum memory allocation of 512MB:

Open the nxlog.service file located at /lib/systemd/system with a text editor.

Add the following lines at the end of the [Service] section:

CPUQuota=10%
MemoryMax=512M

The complete section should look like:

[Service]
Type=simple
User=root
Group=root
PIDFile=/opt/nxlog/var/run/nxlog/nxlog.pid
ExecStartPre=/opt/nxlog/bin/nxlog -v
ExecStart=/opt/nxlog/bin/nxlog -f -q
ExecStop=/opt/nxlog/bin/nxlog -s
ExecReload=/opt/nxlog/bin/nxlog -r
KillMode=process
CPUQuota=10%
MemoryMax=512M

Execute the following command to apply the configuration:
```
# systemctl daemon-reload
```
If NXLog is already running, restart the service:
```
# systemctl restart nxlog
```

Using cgroups

Another way to limit RAM and CPU usage on Linux-based systems is through cgroups. cgroups allows you to allocate resources, such as CPU time, system memory, and network bandwidth, based on user-defined groups.

By default, NXLog runs as the nxlog User and Group. Follow these steps to install cgroups and configure CPU and memory limits for the nxlog group:

Install cgroups according to your platform.
- On RHEL/CentOS, execute the following command:
  # yum install libcgroup libcgroup-tools
- On Debian/Ubuntu, execute the following command:
  # apt install cgroup-tools
Open the cgroups configuration file located at /etc/cgconfig.conf with a text editor.
Add the following configuration at the end of the file:
cgconfig.conf
```
group nxlog {
        cpu {
                cpu.cfs_quota_us = 10000;
                cpu.cfs_period_us = 100000;
        }
        memory {
                memory.limit_in_bytes="1G";
                memory.memsw.limit_in_bytes="2G";
        }
}
```
The first part of this configuration limits CPU usage to 10%.

The second part of the configuration limits the amount of RAM that can be used to 1GB. If the memory limit is exceeded, NXLog can use an additional 1GB of swap space to compensate for the excess.
To apply this configuration to NXLog, the following line needs to be added to the /etc/cgrules.conf file:
cgrules.conf
```
@nxlog          cpu,memory      nxlog
```
This will set the memory and CPU limits defined in the cgroups configuration file for the NXLog group.

With the configuration in place, you can now run the following commands to start the cgroups processes:

# sudo systemctl enable cgconfig
# sudo systemctl start cgconfig
# sudo systemctl enable cgred
# sudo systemctl start cgred

Windows systems

Windows systems do not provide a native method to control the amount of memory or CPU a process can use. However, limiting resource consumption can be achieved with the use of third-party tools. This section provides an example of how to limit NXLog resource usage with Process Governor.

Using Process Governor

Process Governor, or procgov, is an open-source software that allows you to set constraints on a process on Windows operating systems.

Follow these steps to install and execute Process Governor:

Download procgov.zip, extract the archive to a folder, and run procgov.exe to install it.
There is no graphical interface for Process Governor, but the tool can be executed manually through the command line or via a script. The following example PowerShell script runs Process Governor with arguments that allow the NXLog process a maximum of 200MB memory, 1 CPU, and 20% CPU usage. If NXLog is already running, it will be restarted.
moderate-nxlog.ps1
```
$workingdir = "C:\Program Files\procgov"
restart-service nxlog
$nxlog = (get-process nxlog | select -expand id)
& "$workingdir\procgov64.exe" --maxmem 200M --cpu 1 --cpurate 20 --pid $nxlog
```
Assigning a low amount of memory for the NXLog process may cause it to crash. It is recommended to observe memory usage by NXLog under normal operation and assign a limit accordingly.

You may receive the following error message when attempting to manage resources for NXLog using Process Governor:

ERROR: Access is denied. (0x80040005)

This error indicates that Process Governor cannot restart the NXLog service. To resolve this issue, run Process Governor using the local SYSTEM account. You can do this with PsExec.

For example, execute the command as follows:

"$workingdir\psexec" -s -i "$workingdir\procgov64.exe" --maxmem 200M --cpu 1 --cpurate 20 --pid $nxlog

You may skip the -i (interactive) option if you do not need to see the Process Governor output.

Running Process Governor as a service

Running Process Governor from the command line or a script is adequate for troubleshooting, however it may not be viable as a long-term solution. An alternative is to run Process Governor as a service using a service manager such as NSSM.

An example script for installing NSSM and configuring Process Governor to run as a service is available in our public git repository. Download the contents of the procgov-resource-handling directory, and follow the README file to execute the script.

Disclaimer

While we endeavor to keep the information in this topic up to date and correct, NXLog makes no representations or warranties of any kind, express or implied about the completeness, accuracy, reliability, suitability, or availability of the content represented here. We update our screenshots and instructions on a best-effort basis.

The accurateness of the content was tested and proved to be working in our lab environment at the time of the last revision with the following software versions:

Debian 11 (Bullseye)
CentOS 7
Windows 10 Home
Windows Server 2012 R2
NXLog Enterprise Edition version 5.4.7313

Last revision: 10 December 2021