Internal logs

When issues arise while configuring or maintaining an NXLog instance, a stepwise troubleshooting approach (moving from the most likely and simple cases to the more complex and rare ones) generally yields favorable results.

By default, NXLog generates log messages about its operations. Inspecting these logs should be the first troubleshooting step when NXLog is not functioning as expected.

Log file

The LogFile global directive in the NXLog configuration file specifies the path to its log file, which must resolve to an existing directory. The directive defaults to C:\Program Files\nxlog\data\nxlog.log on Windows and /opt/nxlog/var/log/nxlog/nxlog.log on Linux-based systems. You can disable logging to file by removing the LogFile directive.

Some Windows applications, including WordPad, cannot open the log file while the NXLog process is running because of exclusive file locking. Instead, use a viewer that does not lock the file, such as Notepad.

systemd journal

On Linux-based systems, the NXLog service is controlled by systemd. The systemd journal captures the application standard output and standard error streams, which includes NXLog’s internal logging. Logs are available in the journal database even if logging to file is not enabled or the log file is deleted. You can use the journalctl tool to view the logs. The following are some useful journalctl commands for troubleshooting.

A user must be a member of the systemd-journal group to access the systemd journal logs.

This command displays all NXLog journal entries since it was installed, ordered by oldest first.

$ journalctl --unit nxlog

Use the -b option to only display log entries from the current boot.

$ journalctl --unit nxlog -b

Use the --since and --until options to narrow down the logging to a specific time-frame. The time is specified in local time.

$ journalctl --unit nxlog --since "2022-08-04 15:00:00" --until "2022-08-04 15:30:00"

You can redirect the output to file for easier processing.

$ journalctl --unit nxlog --since "2022-08-04 15:00:00" >nxlog.log

Logs are displayed in the syslog-style format by default. You can change the output format, for example, to JSON, with the -o option.

$ journalctl --unit nxlog --since "2022-08-04 15:00:00" -o json

Actively follow live logs with the -f option.

$ journalctl --unit nxlog -f

Change the logging level

The default logging level is INFO. You can set the log level to DEBUG to record more detailed information. However, this level produces a large amount of logging, so we only recommend it for troubleshooting.

To temporarily change the logging level for a running instance:

On Linux, send SIGUSR2:
```
# kill -SIGUSR2 $PID
```
On Windows, send service control command 201:
```
> sc control nxlog 201
```

For the change to persist between NXLog restarts, set the LogLevel directive in the configuration to DEBUG and restart NXLog.

Write customized log messages to the internal log

It may be helpful to add extra logging statements to your configuration using the log_info() procedure.

The generated messages will be visible:

in the file defined in the LogFile global directive
in the input from the im_internal module
on standard output when running NXLog in the foreground with the -f command line switch

In NXLog Enterprise Edition version 5, the log_info() procedure will truncate messages larger than 1024 bytes. In subsequent versions, messages longer than specified at LogSizeLimit will be truncated. If messages are being truncated by log_info(), you can save the messages to a file instead.

Example 1. Writing specific fields and values to the internal log

This configuration uses the log_info() procedure to send values to the internal log. Log messages are accepted over UDP on port 514. If keyword is found in the unparsed message, an INFO level message will be generated.

nxlog.conf

<Input in>
    Module  im_udp
    Port    514
    <Exec>
        if $raw_event =~ /keyword/
            log_info("FOUND KEYWORD IN MSG: [" + $raw_event + "]");
    </Exec>
</Input>

Send all fields to the internal log

Example 2. Send all fields to the internal log

In this configuration, the to_json() procedure from the xm_json module is used to send all the fields to the internal log.

nxlog.conf

<Extension _syslog>
    Module      xm_syslog
</Extension>

<Extension _json>
    Module      xm_json
</Extension>

<Input in>
    Module      im_tcp
    ListenAddr  0.0.0.0:1514
    <Exec>
        parse_syslog_bsd();

        # Dump $raw_event
        log_info("raw_event is: " + $raw_event);

        # Dump fields in JSON
        log_info("Other fields are: " + to_json());
    </Exec>
</Input>

Output sample

{
  "MessageSourceAddress": "127.0.0.1",
  "EventReceivedTime": "2012-05-18 13:11:35",
  "SourceModuleName": "in",
  "SourceModuleType": "im_tcp",
  "SyslogFacilityValue": 3,
  "SyslogFacility": "DAEMON",
  "SyslogSeverityValue": 3,
  "SyslogSeverity": "ERR",
  "SeverityValue": 4,
  "Severity": "ERROR",
  "Hostname": "host",
  "EventTime": "2010-10-12 12:49:06",
  "SourceName": "app",
  "ProcessID": "12345",
  "Message": "test message"
}

As mentioned above, in NXLog Enterprise Edition version 5, the log_info() procedure will truncate messages larger than 1024 bytes. In subsequent versions, messages longer than specified at LogSizeLimit will be truncated. If $raw_event is larger than 1024 bytes, you can write the data to a file instead.

Example 3. Save all fields to a file

This configuration shows how the om_file module is used to write all the fields to a file.

nxlog.conf

<Extension _json>
    Module      xm_json
</Extension>

<Output debug>
    Module      om_file
    Exec        $raw_event = $raw_event + " || " + to_json();
    File        "/tmp/debug.log"
</Output>

Send debug dump to the internal log

A simple way to quickly get a more complete picture of NXLog’s current status is to dump debug info into the internal log. This information can be helpful in determining, for example, why an input module is not sending to an output module. Normally, internal events are written to the log file configured with the LogFile directive.

On Linux, send SIGUSR1 to the application:

# kill -SIGUSR1 $PID

On Windows, send the service control command "200" to the application:

> sc control nxlog 200

Dumped debug info example

2017-03-29 10:05:19 INFO event queue has 2 events;jobgroup with priority 10;job
of module in/im_file, events: 0;job of module out/om_null, events: 0;non-module
job, events: 0;jobgroup with priority 99;non-module job, events: 0;[route 1]; -
in: type INPUT, status: RUNNING queuesize: 0; - out: type OUTPUT, status:
RUNNING queuesize: 0;

The status is the most important piece of information in the dumped log entries. A status of PAUSED means the input module is not able to send because the output module queue is full. In such a case the queuesize for the corresponding output(s) would be over 99. A status of STOPPED means the module is fully stopped, usually due to an error.

Send internal log to STDOUT

Run NXLog in the foreground to print internal logs to the standard output and standard error streams, which are both visible in the terminal. This is useful for NXLog agent interactive debugging.

Use nxlog -f to run NXLog in the foreground.

Log messages can be sent to STDOUT on Linux platforms by following the guidance on saving information to a local file. Modify the output file path of the file_write() procedure to point to /dev/stdout.

Alternatively, send information to STDOUT with the om_file module.

Example 4. Send log information to /dev/stdout

This configuration shows how to send log information to /dev/stdout using the om_file module.

nxlog.conf

<Output debug_to_stdout>
    Module      om_file
    File        "/dev/stdout"
</Output>

Windows has no direct equivalent with STDOUT. However, you can output log messages to the Powershell console.

Example 5. Send log information to Powershell

This configuration shows how to send log information to the Windows Powershell console using the om_exec module.

nxlog.conf

<Output debug_to_powershell>
    Module      om_exec
    Command     "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe"
    Arg         "-Command"
    Arg         "$Input | Select-Object"
</Output>

Write internal log messages to an existing route

NXLog can also write internal log data into a normal route using the im_internal module. Internal log messages can then be forwarded like any other log source.

Local logging is more fault-tolerant than routed logging, and is therefore recommended for troubleshooting.

It is not possible to use a log level higher than INFO with the im_internal module. DEBUG level messages can only be written to the local log file.

Saving information to a local file

Saving the internal log or other information to local files can also be useful while troubleshooting.

In this example, the file_write() procedure (from the xm_fileop module) is used to dump information to a local file.

nxlog.conf

<Extension _fileop>
    Module      xm_fileop
</Extension>

<Extension _syslog>
    Module      xm_syslog
</Extension>

<Input in>
    Module      im_tcp
    ListenAddr  0.0.0.0:1514
    <Exec>
        parse_syslog_bsd();

        # Debug $SyslogSeverity and $Hostname fields
        file_write("/tmp/debug.txt",
                   "Severity: " + $SyslogSeverity +
                   ", Hostname: " + $Hostname + "\n");
    </Exec>
</Input>

As an alternative, one could use the om_file module as illustrated in the Save all fields to a file example to accomplish the same thing.