Debugging NXLog Agent
The NXLog Agent logs help troubleshoot general log processing and configuration issues. However, issues such as crashes and memory leaks require more in-depth investigation.
Below, we outline some techniques for debugging NXLog Agent.
Generating core dumps
The NXLog support and development teams rely on core dumps to debug issues. If you need help creating a core dump, please get in touch with support.
Core dumps on Linux
- Prerequisites
-
You must install the NXLog Agent debug symbols package to generate a usable core dump.
- Steps
-
-
Remove the User and Group directives from the NXLog Agent configuration. This step is required to run NXLog Agent as the root user. Otherwise, it won’t be able to create a core dump.
-
Execute the following command to remove the core file size limit:
$ sudo ulimit -c unlimited
-
Run NXLog Agent interactively to verify that it can create a core dump.
$ sudo /opt/nxlog/bin/nxlog -f
-
Terminate the NXLog Agent process by sending it the
SIGABRT
signal.$ sudo kill -ABRT `ps aux | grep [/]opt/nxlog/bin/nxlog | awk '{print $2}'`
-
Verify that a core dump file was created in
/opt/nxlog/var/spool/nxlog/core
.$ ls -l /opt/nxlog/var/spool/nxlog/ total 26708 -rw------- 1 root root 27348992 Oct 30 08:51 core
-
If the core dump file was created successfully, rerun NXLog Agent to capture the crash.
$ sudo /opt/nxlog/bin/nxlog -f
-
Core dumps on Windows
Microsoft Sysinternals' ProcDump can generate core dumps on Windows. It runs on Windows Vista and higher and Windows Server 2008 and higher.
For example, the following command writes a full dump of the nxlog
process when its handle count exceeds 10,000:
> procdump -ma nxlog -p "\Process(nxlog)\Handle Count" 10000
Inspecting memory leaks
Inspecting a memory leak requires simultaneously observing the LogqueueSize and memory usage. The first step is to check the state of the log queue. While the log queue fills up, you may observe a monotonic upward trend in memory use, continuing until it plateaus at a maximum value. Sometimes, an OoM (Out of Memory) error may occur during this time, terminating the NXLog Agent process. However, this is not definitive proof of a memory leak, only over-allocation.
Once the log queue is full, memory consumption should stabilize with only minor fluctuations. If NXLog Agent never reaches an equilibrium, there is a legitimate reason to suspect a memory leak.
Inspecting memory leaks on Linux
We recommend using Valgrind to debug memory leaks on GNU/Linux.
-
Install the NXLog Agent debug symbols (
-dbg
) package (nxlog-dbg_6.4.9634_amd64.deb
). The package is currently only available for Linux. It is not included with the NXLog Agent installation but we can provide it on request. -
Install Valgrind.
-
Open the NXLog Agent configuration file and set the NoFreeOnExit directive to
TRUE
. This directive ensures that NXLog Agent does not unload modules when it stops, allowing Valgrind to resolve backtraces into modules. -
Stop the NXLog Agent service if it’s running:
$ sudo systemctl stop nxlog
-
Start NXLog Agent under Valgrind using the following command:
$ sudo valgrind --tool=massif --pages-as-heap=yes --massif-out-file=/path/to/massif.out /opt/nxlog/bin/nxlog -f
or
$ su -c "valgrind --tool=massif --pages-as-heap=yes --massif-out-file=/path/to/massif.out /opt/nxlog/bin/nxlog -f"
-
Let NXLog Agent run until the Valgrind process shows a memory increase. Then, interrupt it with Ctrl+C. Valgrind writes the output to the path specified by the
--massif-out-file
argument. -
Open a support ticket and attach the
massif.out
file. -
Optionally, create a report from the
massif.out.xxxx
file with thems_print
command:$ ms_print massif.out.xxxx
The output of the
ms_print
report contains an ASCII chart showing the increase in memory usage. The chart shows the sample number with the highest memory usage, marked with(peak)
. The peak is usually at the end of the chart, specifically the last sample. The backtrace from this sample indicates where most of the memory is allocated.
Inspecting memory leaks on Windows
Microsoft Sysinternals' Windows Process Explorer allows you to inspect the memory usage of all running programs.
If you detect a potential source of excessive memory consumption, you can inspect the NXLog Agent debugging output using DebugView.
Inspecting open file handles
Another method of debugging NXLog Agent is to investigate the open file handles associated with it. This information lets you determine whether it is accessing the correct configuration and log files.
Inspecting open file handles on Linux
-
Find the ID of the NXLog Agent process. You can do this by listing processes with the
ps
utility and filtering for the NXLog Agent process withgrep
.$ ps -e | grep nxlog 12345 ? 0:00.00 nxlog
-
Search for all NXLog Agent open file handles with the
lsof
utility.# lsof -p 12345
-
Send the output of
lsof
to thewc
utility to count the number of open file handles.# lsof -p 12345 | wc -l
Inspecting open file handles on Windows
The Microsoft Sysinternals' Handle utility allows you to inspect the number of open file handles on Windows.
-
Execute the following command to search for all NXLog Agent file handles:
> handle -p nxlog
-
Use the
-s
parameter to count the number of open file handles.> handle -s -p nxlog