Debugging NXLog

When other troubleshooting fails to identify or resolve an issue, inspecting the NXLog agent itself can prove useful. Some techniques are outlined below.

Generate core dumps

Core dumps can act as a helpful resource for the NXLog development and support teams when debugging issues. Contact support to find out the level of work available for your installation.

Core dumps on Linux

It is necessary to install the NXLog debug symbols package in order to produce useful core dump files.
  1. Remove the User and Group directives from the configuration. NXLog needs to be running as root:root to produce a core dump.

  2. Use ulimit to remove the core file size limit.

    # ulimit -c unlimited
  3. Run NXLog manually to test that it can create a core dump.

    # /opt/nxlog/bin/nxlog -f
  4. Find the NXLog process and kill it with the SIGABRT signal.

    # kill -ABRT `ps aux | grep [/]opt/nxlog/bin/nxlog | awk '{print $2}'`
  5. Verify that a core dump file was created at /opt/nxlog/var/spool/nxlog/core.

    # ls -l /opt/nxlog/var/spool/nxlog/
    total 26708
    -rw------- 1 root root 27348992 Oct 30 08:51 core
  6. If the core dump file was created successfully, run NXLog again as root in order to catch the next crash.

    # /opt/nxlog/bin/nxlog -f

Core dumps on Microsoft Windows

Core dumps can be generated on Windows by using ProcDump from Microsoft Sysinternals.

ProcDump runs on Windows Vista and higher, and Windows Server 2008 and higher.

For example, run the following to write a full dump of the nxlog process when its handle count exceeds 10,000:

> procdump -ma nxlog -p "\Process(nxlog)\Handle Count" 10000

Inspect memory leaks

If NXLog’s memory usage exceeds 200 MB, there is likely a memory leak.

Inspecting memory leaks on Linux

We recommend using Valgrind on GNU/Linux to debug memory leaks.

  1. Install the debug symbols (-dbg) package (for example, nxlog-dbg_3.0.1759_amd64.deb).

    The NXLog debug symbols package is currently only available for Linux. This package is not included with NXLog by default, but can be provided on request.
  2. Install Valgrind.

  3. Set the NoFreeOnExit directive to TRUE in the NXLog configuration file. This directive ensures that modules are not unloaded when NXLog is stopped, which allows Valgrind to properly resolve backtraces into modules.

  4. Stop the NXLog service if it’s currently running:

    # systemctl stop nxlog
  5. Start NXLog under Valgrind with the following command:

    # sudo valgrind --tool=massif --pages-as-heap=yes --massif-out-file=/path/to/massif.out /opt/nxlog/bin/nxlog -f

    or

    # su -c "valgrind --tool=massif --pages-as-heap=yes --massif-out-file=/path/to/massif.out /opt/nxlog/bin/nxlog -f"
  6. Let NXLog run for a while until the Valgrind process shows the memory increase, then interrupt it with Ctrl+C. The output is written to the path specified by the --massif-out-file argument.

  7. Send the massif.out file with a bug report.

  8. Optionally, create a report from the massif.out.xxxx file with the ms_print command:

    # ms_print massif.out.xxxx

    The output of the ms_print report contains an ASCII chart at the top showing the increase in memory usage. The chart shows the sample number with the highest memory usage, marked with (peak). This is normally at the end of the chart (the last sample). The backtrace from this sample indicates where the most memory is allocated.

Inspecting memory leaks on Microsoft Windows

Windows Process Explorer from Microsoft Sysinternals can be used to inspect memory use of all running programs.

Once a potential source of excessive memory use has been determined, use DebugView from Microsoft Sysinternals to inspect the application’s debug output.