NXLog Legacy Documentation

Configuration overview

NXLog uses Apache style configuration files. The configuration file is loaded from its default location, or it can be explicitly specified with the -c command line argument.

The NXLog configuration file is comprised of blocks and directives. Blocks are similar to XML tags containing multiple directives. Directive names are not case-sensitive but arguments sometimes are.

Create your first configuration

Follow these steps to create your first NXLog configuration. With this simple example, you will read syslog messages from a file, parse the log records, and write them to another file in JSON format.

create first config 1 constants

Define constant values that can be used throughout the configuration. Typically used for directory paths, filenames, hostnames, or regular expressions.

define INSTALLDIR /opt/nxlog
define LOGDIR %INSTALLDIR%/var/log/nxlog
define MYLOGFILE %LOGDIR%/nxlog.log

create first config 2 global directives

Define global directives that control the overall behavior of NXLog. These include settings related to logging, batching, caching, and date format.

LogLevel INFO
LogFile  %MYLOGFILE%

create first config 3 extensions

Add extension modules required for processing logs, e.g., to parse or output log records in a specific format. Multiple instances of the same extension module can be added.

<Extension syslog>
    Module    xm_syslog
</Extension>

<Extension json>
    Module    xm_json
</Extension>

create first config 4 input

Add input module instances to collect or receive logs from your sources. Once a log record is read, you can parse it into structured data, add or remove fields, and transform it into the required output format.

<Input input_file>
    Module    im_file
    File      '/var/log/syslog'
    Exec      parse_syslog();
</Input>

create first config 5 output

Output modules send logs to their destination, such as a file, database, SIEM, or log analytics solution. You can also process logs in output module instances.

<Output output_file>
    Module    om_file
    File      '/tmp/logs/nxlog_output'
    Exec      to_json();
</Output>

create first config 6 route

Define the flow and processing order of logs. A route can include multiple input, processor, and output module instances. You can also define multiple routes.

<Route r1>
    Path    input_file => output_file
</Route>

The following is a sample of a log file containing syslog messages.

Mar 13 19:05:47 NXLog-Server-1 systemd[1]: systemd-hostnamed.service: Succeeded.
Mar 13 19:11:29 NXLog-Server-1 systemd[1260]: Started Application launched by gnome-shell.
Mar 13 19:11:30 NXLog-Server-1 systemd[1260]: Started VTE child process 12193 launched by gnome-terminal-server process 3008.
Mar 13 19:11:30 NXLog-Server-1 systemd[1260]: gnome-launched-org.gnome.Terminal.desktop-12183.scope: Succeeded.
Mar 13 19:11:44 NXLog-Server-1 systemd[1]: Starting NXLog daemon...
Mar 13 19:11:44 NXLog-Server-1 nxlog[12426]: INFO [CORE|main] configuration OK
Mar 13 19:11:44 NXLog-Server-1 systemd[1]: Started NXLog daemon.

This output sample depicts a log record in JSON format after it was processed by the NXLog configuration above.

{
  "EventReceivedTime": "2022-03-13T19:11:45.011927+01:00",
  "SourceModuleName": "input_file",
  "SourceModuleType": "im_file",
  "SyslogFacilityValue": 1,
  "SyslogFacility": "USER",
  "SyslogSeverityValue": 5,
  "SyslogSeverity": "NOTICE",
  "SeverityValue": 2,
  "Severity": "INFO",
  "Hostname": "NXLog-Server-1",
  "EventTime": "2022-03-13T19:11:44.000000+01:00",
  "SourceName": "systemd",
  "ProcessID": 1,
  "Message": "Started NXLog daemon."
}

Continue to the following sections to learn more about NXLog configuration components. Then, see the NXLog Enterprise Edition Reference Manual for more in-depth information on the NXLog language and module documentation.

Constant and macro definitions

A define is useful if there are many instances in the configuration where the same value must be used. Typically, defines are used for directories and hostnames. In such cases the value can be configured with a single definition. In addition to constants, other strings like code snippets or parser rules can be defined in this way.

An NXLog define works similar to the way it does in the C language, where the pre-processor substitutes the value for the macro wherever it is used. The NXLog configuration parser replaces all occurrences of the defined name with its value, and then after this substitution the configuration check occurs.

Example 1. Using defines

This example shows the use of two defines: BASEDIR and IGNORE_DEBUG. The first is a simple constant whose value is used in two File directives. The second is an NXLog language statement that is used in an Exec directive.

nxlog.conf

define BASEDIR /var/log
define IGNORE_DEBUG if $raw_event =~ /debug/ drop();

<Input messages>
    Module  im_file
    File    '%BASEDIR%/messages'
</Input>

<Input proftpd>
    Module  im_file
    File    '%BASEDIR%/proftpd.log'
    Exec    %IGNORE_DEBUG%
</Input>

The define directive can be used for statements as shown above, but multiple statements should be specified as a code block enclosed in curly braces ({}) to produce the desired results.

Example 2. Incorrect use of a define

The following example shows an incorrect use of the define directive. After substitution, the drop() procedure will always be executed; only the warning message will be logged conditionally.

nxlog.conf (incorrect)

define ACTION log_warning("dropping message"); drop();

<Input in>
    Module  im_file
    File    '/var/log/messages'
    Exec    if $raw_event =~ /dropme/ %ACTION%
</Input>

To avoid this problem, the action should be defined using a code block.

define ACTION { log_warning("dropping message"); drop(); }

Environment variables

The envvar directive works like define except that the value is retrieved from the environment. This makes it possible to reference the environment variable as if it were a define. This directive is only available in NXLog Enterprise Edition.

Example 3. Using environment variables

This is similar to the previous example using a define, but here the value is fetched from the environment.

nxlog.conf

envvar BASEDIR

<Input in>
    Module  im_file
    File    '%BASEDIR%/messages'
</Input>

Global directives

The global section contains directives that control the overall behavior of NXLog.

The LogFile directive sets a destination file for NXLog internal logs. If this directive is unset, the log file is disabled and internal NXLog logs are not written to file (unless configured via the im_internal module). See also Log rotation and retention.

With the User and Group directives set, NXLog will drop root privileges after starting and run as the specified user and group. If, for whatever reason, Linux capabilities need to be permanently set, the Capabilities directive can be used. Such Linux-specific directives are ignored if running on Windows.

After starting, NXLog will change its working directory to the directory specified by the SpoolDir directive.Any path specified in the configuration that is not an absolute path will be relative to this directory.

See the Reference Manual for a complete list of available global directives.

File inclusion

NXLog supports including static or dynamic configuration snippets from external files or via a script. When doing so, it is important to note that the SpoolDir directive only takes effect after NXLog loads the configuration. Therefore, the working directory before NXLog finishes loading the configuration is the folder of the NXLog executable.

We recommend using absolute paths whenever possible. However, you can also use one or more define directives to simulate a relative path. See the example for Using defines to include a configuration file below.

Including static configuration

You can use the include directive to load static configuration snippets from one or more files. See the examples below.

Example 4. Including a configuration file

This example loads the content of the /opt/nxlog/etc/syslog.conf file in the current configuration.

nxlog.conf

include /opt/nxlog/etc/syslog.conf

Example 5. Using defines to include a configuration file

This example uses two define directives to include a configuration file named eventlog.conf. It defines the path in two parts, i.e., the NXLog root directory as ROOT and the configuration folder relative to it as CONFDIR.

nxlog.conf

define  ROOT     C:\Program Files (x86)\nxlog
define  CONFDIR  %ROOT%\conf
include %CONFDIR%\eventlog.conf

The include directive also supports filenames containing the wildcard character *. For example, you could save multiple .conf files in the nxlog.d directory —or any other custom directory— and then automatically load them alphabetically from the main nxlog.conf file.

Each file might contain a small configuration snippet exclusive to a single log source, simplifying the maintenance of larger configurations. In addition, this modular approach allows you to update the configuration of specific log sources without modifying the main nxlog.conf file. For example, you can create OS-specific configuration snippets, such as windows.conf, ubuntu.conf, etc., or application-specific snippets, such as syslog.conf, httpd.conf, etc.

The include directive only supports loading configuration files from the specified directory. It does not support automatically loading files from subdirectories or wildcards in the directory name.

Example 6. Including a configuration directory on Linux

This example includes all .conf files under /opt/nxlog/etc/nxlog.d.

nxlog.conf

include /opt/nxlog/etc/nxlog.d/*.conf

You can also specify the folder path with the define directive.

nxlog.conf

define CONFDIR /opt/nxlog/etc/nxlog.d
include %CONFDIR%/*.conf

Example 7. Including a configuration directory on Windows

This example includes all .conf files from the nxlog.d folder on Windows.

nxlog.conf

include C:\Program Files\nxlog\conf\nxlog.d\*.conf

Alternatively, you can use the define directive to specify the folder path.

nxlog.conf

define CONFDIR C:\Program Files\nxlog\conf\nxlog.d
include %CONFDIR%\*.conf

Including dynamic configuration

You can use the include_stdout directive to load dynamic configuration snippets from a command or script. Use this directive like a command-line interface, for example, Terminal on Linux and Command Prompt on Windows.

There are several instances where you might need to load dynamic configuration, such as when you need to specify host-specific information for log normalization and data enrichment, retrieve application file paths, or validate a signed configuration.

Example 8. Loading NXLog configuration from a bash script

This configuration executes a bash script to define three constants for the operating system version, name, and architecture.

nxlog.conf

include_stdout /opt/nxlog/etc/host_info.sh

host_info.sh

#!/bin/bash

OSVERSION=$(cat /etc/os-release | grep VERSION= | grep -oP '"(.*)"' | sed s/\"//g)
OSNAME=$(cat /etc/os-release | grep -oP '^NAME=.*' | grep -oP '"(.*)"' | sed s/\"//g)
ARCHITECTURE=$(uname -m)

echo "define OSVERSION $OSVERSION"
echo "define OSNAME $OSNAME"
echo "define ARCHITECTURE $ARCHITECTURE"

The include_stdout directive does not support arguments or executing external scripts directly. However, a workaround for this is to encapsulate the script in a command file.

Example 9. Loading NXLog configuration from a PowerShell script

This configuration executes a batch script that, in turn, executes PowerShell commands to define two constants, OSNAME and ARCHITECTURE.

nxlog.conf

include_stdout C:\Program Files\nxlog\conf\host_info.cmd

host_info.cmd

@( Set "_= (
REM " ) <#
)
@Echo Off
SetLocal EnableExtensions DisableDelayedExpansion
set powershell=powershell.exe

%powershell% -ExecutionPolicy Bypass -NoProfile ^
-Command "iex ((gc '%~f0') -join [char]10)"
EndLocal & Exit /B %ErrorLevel%
#>

# PowerShell code starts here.
$osname = (Get-WmiObject -Class Win32_OperatingSystem).caption
$architecture = (Get-WmiObject CIM_OperatingSystem).OSArchitecture

# Anything written to standard output is used as configuration content
Write-Output "define OSNAME $osname"
Write-Output "define ARCHITECTURE $architecture"

Example 10. Loading NXLog configuration from a Python script

This configuration executes a bash script that, in turn, runs a Python script to define two constants, OSTYPE and ARCHITECTURE.

nxlog.conf

include_stdout /opt/nxlog/etc/run_script.sh

run_script.sh

#!/bin/bash

python3 /opt/nxlog/etc/scripts/host_infp.py

host_info.py

import platform
import os

print("define OSTYPE {}".format(platform.system()))
print("define ARCHITECTURE {}".format(platform.machine()))

See the following for more practical examples:

Normalizing data with NXLog in the Elastic Common Schema (ECS) integration guide.
Generating configuration with PowerShell in the Windows PowerShell integration guide.
Automatic retrieval of IIS site log locations in the Microsoft IIS integration guide.

Modules

NXLog will only load modules that are specified in the configuration file and used in an active route. A module instance is specified according to its corresponding module type (Extension, Input, Processor, or Output). Each module instance must have a unique name and a Module directive. The following is a skeleton configuration block for an input module.

Example 11. Modules

nxlog.conf

<Input instancename>
    Module  im_module
    ...
</Input>

For more details about module instance names, see Configuration in the Reference Manual.

Routes

Routes define the flow and processing order of the log messages. Each route instance must have a unique name and a Path.

Example 12. Route Block

This Route instance, named example, takes logs from Input module instances named in1 and in2, processes the logs with the proc Processor module instance, and sends the resulting logs to both Output module instances out1 and out2. These named module instances must be defined elsewhere in the configuration file.

nxlog.conf

<Route example>
    Path    in1, in2 => proc => out1, out2
</Route>

For more details about route instance names, see Configuration in the Reference Manual.

If no Route block is specified in the configuration, NXLog will automatically generate a route, with all the Input and Output instances specified in a single path.

Example 13. An automatic route block

As shown in this configuration, NXLog can function without the route being explicitly defined in a Route block.

nxlog.conf

<Input in1>
    Module  im_null
</Input>

<Input in2>
    Module  im_null
</Input>

<Output out1>
    Module  om_null
</Output>

<Output out2>
    Module  om_null
</Output>

The following Route block will be generated automatically.

nxlog.conf (generated route)

<Route r>
    Path    in1, in2 => out1, out2
</Route>

File paths

File paths are used by global and module directives, and should be enclosed in either single quotes (' ') or double quotes (" ").

Example 14. Using file paths

nxlog.conf

File    "/tmp/input"

Several modules, however, require file paths to be defined without any quotation marks.

nxlog.conf

DeviceFile    /dev/auditpipe

Quoting and escaping strings

String literals must be quoted using either single or double quotes. When a string is enclosed in single quotes, NXLog interprets each character as is. When a string is enclosed in double quotes, NXLog supports escape sequences starting with the backslash (\) character.

Table 1. Table of escape sequences supported by NXLog
Sequence	Description
`\\`	The backslash (`\`) character
`\"`	The double quote (`"`) character
`\n`	Line feed (LF)
`\r`	Carriage return (CR)
`\t`	Horizontal tab
`\b`	Audible bell
`\xXX`	A single byte in the form of a two digit hexadecimal number. For example the line-feed character can also be expressed as `\x0A`

When a string is enclosed in double quotes, NXLog will also interpret the question mark (?) and asterisk (*) as wildcard characters. This means that specifying "file*.log" in a file path will match file1.log, filea.log, etc. For more details, see the Configuration section of the im_file module documentation.

Extra care should be taken when specifying file paths within double quotes since the backslash (\) is used as the directory separator on Windows. For more information about the possible complications, see this note of the im_file module.

The examples below demonstrate how to specify various strings and their resulting values.

Example 15. Correct usage of single quotes

The statement below sets the $FilePath field to a path on Windows.

nxlog.conf

$FilePath = 'C:\AppLogs\logfile.txt'

No escaping is done when using single quotes, therefore the string is interpreted as is.

Output

C:\AppLogs\logfile.txt

Nested quotation is possible using double quotes. Nested quotation using single quotes is not supported.

nxlog.conf

$Message = 'Message: "This is a test"'

Output

Message: "This is a test"

Example 16. Correct usage of double quotes

The statement below sets the $FilePath field to a path on Windows. It uses escape sequences for the directory separator.

nxlog.conf

$FilePath = "C:\\AppLogs\\logfile.txt"

Since the string is enclosed in double quotes, \\ will be interpreted as an escape sequence and will result in a valid path.

Output

C:\AppLogs\logfile.txt

Both single and double quotes are supported for nested quotation. Double quotes need to be escaped with \".

nxlog.conf

$Message = "Message: \"This is a test\" using 'nested quotation'"

Output

Message: "This is a test" using 'nested quotation'

Example 17. Using wildcard characters

Wildcard characters are useful for reading data from multiple files based on their naming pattern. The File directive below uses the asterisk character (*) to read data from several files whose names start with log and have the .txt extension.

nxlog.conf

<Input from_file>
    Module    im_file
    File      "/tmp/log*.txt"
</Input>

Based on the naming pattern, both log1.txt and log_new.txt will be read.

Multiple lines

A directive and its argument must be specified on the same line. Values spanning multiple lines must have the newline escaped with a backslash (\) as shown below.

Example 18. Arguments spanning multiple lines

nxlog.conf

Fields    $Version, $Device_Vendor, $Device_Product, $Device_Version, \
          $Signature_ID, $Name, $Severity, $_Extension

A typical case of the backslash use is the Exec directive.

nxlog.conf

Exec    if ($QueryName == 'wpad') OR \
        ($QueryType != '1') drop();

Breaking regular expressions into multiple lines improves their readability. This sample of a define directive uses backslashes to continue the expression onto each new consecutive line.

nxlog.conf

define EVENT_REGEX /(?x)^(\d+.\d+.\d+)\s+(\d+:\d+:\d+).\d+\
                         \s+PID=(\d+)\s+TID=(\d+)\s+(\w+)\s+\
                         \S(\w+):\s+(.*)/

Comments

Blank lines in configuration files are ignored.

Lines starting with the hash mark (#) are also ignored and can be used as comments.

Example 19. Using comments

nxlog.conf

# This is the comment line

Each comment should start with a new line. Application of inline comments is impossible because each directive can only contain the directive definition and the argument.

Comments spanning multiple lines should start with the # mark on each line. Escaping multiline comments with a backslash (\) is not supported.

nxlog.conf

# This is a comment
# spanning multiple lines

Comments inside the QueryXML directive must be XML-style comments bounded by  as shown in the sample below.

nxlog.conf

<!--
XML-style comments can
span multiple lines in
QueryXML blocks like this.
-->