General troubleshooting tips

This page provides tips for troubleshooting log processing issues, such as the output not being in the expected format or containing unexpected values. See also the NXLog Agent Reference Manual for more information about NXLog Agent logs.

Write to the NXLog Agent log file

When troubleshooting log processing issues, it is helpful to log messages at various stages of the log processing route. NXLog Agent provides the following procedures to write messages to the NXLog Agent log file:

In addition to the log file, the messages will appear as log records generated by an im_internal module instance and on the console when running NXLog Agent interactively.

NXLog Agent truncates messages longer than the LogSizeLimit. If you need to write longer messages, see the instructions for writing to a custom file.

Example 1. Logging messages to the NXLog Agent log file

This configuration listens for logs over UDP with the im_udp input module. This module populates the $raw_event core field. If the log record contains the word test, the configuration uses the log_info() procedure to write a message in the NXLog Agent log file.

nxlog.conf

<Input udp_listen>
    Module        im_udp
    ListenAddr    0.0.0.0:514
    <Exec>
        if $raw_event =~ /test/
        {
            log_info("FOUND TEST: [" + $raw_event + "]");
        }
    </Exec>
</Input>

Write to a custom file

Besides writing to the log file, you can configure NXLog Agent to write to any other file using the file_write() procedure of the xm_fileop module.

Example 2. Logging messages to a custom file

This configuration uses the file_write() procedure of xm_fileop to log the hostname when it encounters a warning, error, or critical message.

nxlog.conf

<Extension fileop>
    Module        xm_fileop
</Extension>

<Extension syslog>
    Module        xm_syslog
</Extension>

<Input tcp_listen>
    Module        im_tcp
    ListenAddr    0.0.0.0:1514
    <Exec>
        parse_syslog_bsd();  (1)
        if ($SeverityValue > 2)
        {
            file_write("/tmp/error_hosts.txt",
                      "Severity: " + $Severity +
                      ", Hostname: " + $Hostname + "\n");
        }
    </Exec>
</Input>

1	The parse_syslog() procedure parses syslog messages into structured data, creating fields such as `$Severity` and `$SeverityValue`.

Write to the standard output

On Linux, you can output custom messages to STDOUT by writing to /dev/stdout. You can write to a file from any module instance using the file_write() procedure of xm_fileop or an om_file module instance.

Example 3. Sending log messages to the standard output

This configuration uses the om_file module to output logs to /dev/stdout. This module writes the value of the $raw_event core field.

nxlog.conf

<Output stdout>
    Module    om_file
    File      '/dev/stdout'
</Output>

Windows does not have an equivalent direct method for writing to the standard output. However, you can output log messages to the PowerShell console.

Example 4. Send log messages to PowerShell

This configuration uses the om_exec module to output logs to the PowerShell console. This module writes the value of the $raw_event core field.

nxlog.conf

<Output powershell>
    Module     om_exec
    Command    "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe"
    Arg        "-Command"
    Arg        "$Input | Select-Object"
</Output>

Inspect event fields

Sometimes, you may need to troubleshoot issues with structured data. Inspecting log records helps you verify the event fields and their values, for example, when converting logs from one format to another or if the destination reports an incorrect field format. The to_json() procedure of the xm_json module is especially helpful for this purpose.

Example 5. Output event fields to a file

This configuration collects syslog messages from a file and uses the parse_syslog() procedure of the xm_syslog module to parse them into structured data. It then uses the to_json() procedure to convert the record to JSON and writes the output to a file.

nxlog.conf

<Extension syslog>
    Module        xm_syslog
</Extension>

<Extension json>
    Module        xm_json
    PrettyPrint   TRUE  (1)
</Extension>

<Input system_logs>
    Module        im_file
    File          '/var/log/syslog'
    Exec          parse_syslog();
</Input>

<Output debug>
    Module        om_file
    File          '/tmp/debug.log'
    Exec          to_json();
</Output>

1	Enables the PrettyPrint directive to write JSON output in a user-friendly format. By default, xm_json writes records in a single line.

Output sample

{
  "EventReceivedTime": "2024-07-12T16:47:00.854066+02:00",
  "SourceModuleName": "system_logs",
  "SourceModuleType": "im_file",
  "Hostname": "SERVER-1",
  "SyslogFacilityValue": 1,
  "SyslogFacility": "USER",
  "SyslogSeverityValue": 5,
  "SyslogSeverity": "NOTICE",
  "SeverityValue": 2,
  "Severity": "INFO",
  "EventTime": "2024-07-12T16:47:00.000000+02:00",
  "SourceName": "systemd",
  "ProcessID": 1,
  "Message": "Starting NXLog daemon..."
}