Debugging NXLog Agent

The NXLog Agent logs help troubleshoot general log processing and configuration issues. However, issues such as crashes and memory leaks require more in-depth investigation.

Below, we outline some techniques for debugging NXLog Agent.

Generating core dumps

The NXLog support and development teams rely on core dumps to debug issues. If you need help creating a core dump, please get in touch with support.

Core dumps on Linux

Prerequisites

You must install the NXLog Agent debug symbols package to generate a usable core dump.

Steps

Remove the User and Group directives from the NXLog Agent configuration. This step is required to run NXLog Agent as the root user. Otherwise, it won’t be able to create a core dump.
Execute the following command to remove the core file size limit:
```
$ sudo ulimit -c unlimited
```
Run NXLog Agent interactively to verify that it can create a core dump.
```
$ sudo /opt/nxlog/bin/nxlog -f
```

Terminate the NXLog Agent process by sending it the SIGABRT signal.

$ sudo kill -ABRT `ps aux | grep [/]opt/nxlog/bin/nxlog | awk '{print $2}'`

Verify that a core dump file was created in /opt/nxlog/var/spool/nxlog/core.

$ ls -l /opt/nxlog/var/spool/nxlog/
total 26708
-rw------- 1 root root 27348992 Oct 30 08:51 core

If the core dump file was created successfully, rerun NXLog Agent to capture the crash.
```
$ sudo /opt/nxlog/bin/nxlog -f
```

Core dumps on Windows

Microsoft Sysinternals' ProcDump can generate core dumps on Windows. It runs on Windows Vista and higher and Windows Server 2008 and higher.

For example, the following command writes a full dump of the nxlog process when its handle count exceeds 10,000:

> procdump -ma nxlog -p "\Process(nxlog)\Handle Count" 10000

Inspecting memory leaks

Inspecting a memory leak requires simultaneously observing the LogqueueSize and memory usage. The first step is to check the state of the log queue. While the log queue fills up, you may observe a monotonic upward trend in memory use, continuing until it plateaus at a maximum value. Sometimes, an OoM (Out of Memory) error may occur during this time, terminating the NXLog Agent process. However, this is not definitive proof of a memory leak, only over-allocation.

Once the log queue is full, memory consumption should stabilize with only minor fluctuations. If NXLog Agent never reaches an equilibrium, there is a legitimate reason to suspect a memory leak.

Inspecting memory leaks on Linux

We recommend using Valgrind to debug memory leaks on GNU/Linux.

Install the NXLog Agent debug symbols (-dbg) package (nxlog-dbg_6.3.9431_amd64.deb). The package is currently only available for Linux. It is not included with the NXLog Agent installation but we can provide it on request.
Install Valgrind.
Open the NXLog Agent configuration file and set the NoFreeOnExit directive to TRUE. This directive ensures that NXLog Agent does not unload modules when it stops, allowing Valgrind to resolve backtraces into modules.
Stop the NXLog Agent service if it’s running:
```
$ sudo systemctl stop nxlog
```

Start NXLog Agent under Valgrind using the following command:

$ sudo valgrind --tool=massif --pages-as-heap=yes --massif-out-file=/path/to/massif.out /opt/nxlog/bin/nxlog -f

$ su -c "valgrind --tool=massif --pages-as-heap=yes --massif-out-file=/path/to/massif.out /opt/nxlog/bin/nxlog -f"

Let NXLog Agent run until the Valgrind process shows a memory increase. Then, interrupt it with Ctrl+C. Valgrind writes the output to the path specified by the --massif-out-file argument.
Open a support ticket and attach the massif.out file.
Optionally, create a report from the massif.out.xxxx file with the ms_print command:
```
$ ms_print massif.out.xxxx
```
The output of the ms_print report contains an ASCII chart showing the increase in memory usage. The chart shows the sample number with the highest memory usage, marked with (peak). The peak is usually at the end of the chart, specifically the last sample. The backtrace from this sample indicates where most of the memory is allocated.

Inspecting memory leaks on Windows

Microsoft Sysinternals' Windows Process Explorer allows you to inspect the memory usage of all running programs.

If you detect a potential source of excessive memory consumption, you can inspect the NXLog Agent debugging output using DebugView.

Inspecting open file handles

Another method of debugging NXLog Agent is to investigate the open file handles associated with it. This information lets you determine whether it is accessing the correct configuration and log files.

Inspecting open file handles on Linux

Find the ID of the NXLog Agent process. You can do this by listing processes with the ps utility and filtering for the NXLog Agent process with grep.
```
$ ps -e | grep nxlog
12345 ?         0:00.00 nxlog
```
Search for all NXLog Agent open file handles with the lsof utility.
```
# lsof -p 12345
```
Send the output of lsof to the wc utility to count the number of open file handles.
```
# lsof -p 12345 | wc -l
```

Inspecting open file handles on Windows

The Microsoft Sysinternals' Handle utility allows you to inspect the number of open file handles on Windows.

Execute the following command to search for all NXLog Agent file handles:
```
> handle -p nxlog
```
Use the -s parameter to count the number of open file handles.
```
> handle -s -p nxlog
```