NXLog Documentation

Configuration overview

NXLog uses Apache style configuration files. The configuration file is loaded from its default location, or it can be explicitly specified with the -c command line argument.

The NXLog configuration file is comprised of blocks and directives. Blocks are similar to XML tags containing multiple directives. Directive names are not case-sensitive but arguments sometimes are.

Create your first configuration

Follow these steps to create your first NXLog configuration. With this simple example, you will read syslog messages from a file, parse the log records, and write them to another file in JSON format.

create first config 1 constants

Define constant values that can be used throughout the configuration. Typically used for directory paths, filenames, hostnames, or regular expressions.

define INSTALLDIR /opt/nxlog
define LOGDIR %INSTALLDIR%/var/log/nxlog
define MYLOGFILE %LOGDIR%/nxlog.log

create first config 2 global directives

Define global directives that control the overall behavior of NXLog. These include settings related to logging, batching, caching, and date format.

LogLevel INFO
LogFile  %MYLOGFILE%

create first config 3 extensions

Add extension modules required for processing logs, e.g., to parse or output log records in a specific format. Multiple instances of the same extension module can be added.

<Extension syslog>
    Module    xm_syslog
</Extension>

<Extension json>
    Module    xm_json
</Extension>

create first config 4 input

Add input module instances to collect or receive logs from your sources. Once a log record is read, you can parse it into structured data, add or remove fields, and transform it into the required output format.

<Input input_file>
    Module    im_file
    File      '/var/log/syslog'
    Exec      parse_syslog();
</Input>

create first config 5 output

Output modules send logs to their destination, such as a file, database, SIEM, or log analytics solution. You can also process logs in output module instances.

<Output output_file>
    Module    om_file
    File      '/tmp/logs/nxlog_output'
    Exec      to_json();
</Output>

create first config 6 route

Define the flow and processing order of logs. A route can include multiple input, processor, and output module instances. You can also define multiple routes.

<Route r1>
    Path    input_file => output_file
</Route>

The following is a sample of a log file containing syslog messages.

Mar 13 19:05:47 NXLog-Server-1 systemd[1]: systemd-hostnamed.service: Succeeded.
Mar 13 19:11:29 NXLog-Server-1 systemd[1260]: Started Application launched by gnome-shell.
Mar 13 19:11:30 NXLog-Server-1 systemd[1260]: Started VTE child process 12193 launched by gnome-terminal-server process 3008.
Mar 13 19:11:30 NXLog-Server-1 systemd[1260]: gnome-launched-org.gnome.Terminal.desktop-12183.scope: Succeeded.
Mar 13 19:11:44 NXLog-Server-1 systemd[1]: Starting NXLog daemon...
Mar 13 19:11:44 NXLog-Server-1 nxlog[12426]: INFO [CORE|main] configuration OK
Mar 13 19:11:44 NXLog-Server-1 systemd[1]: Started NXLog daemon.

This output sample depicts a log record in JSON format after it was processed by the NXLog configuration above.

{
  "EventReceivedTime": "2022-03-13T19:11:45.011927+01:00",
  "SourceModuleName": "input_file",
  "SourceModuleType": "im_file",
  "SyslogFacilityValue": 1,
  "SyslogFacility": "USER",
  "SyslogSeverityValue": 5,
  "SyslogSeverity": "NOTICE",
  "SeverityValue": 2,
  "Severity": "INFO",
  "Hostname": "NXLog-Server-1",
  "EventTime": "2022-03-13T19:11:44.000000+01:00",
  "SourceName": "systemd",
  "ProcessID": 1,
  "Message": "Started NXLog daemon."
}

Continue to the following sections to learn more about NXLog configuration components. Then, see the NXLog EE Reference Manual for more in-depth information on the NXLog language and module documentation.

Constant and macro definitions

A define is useful if there are many instances in the configuration where the same value must be used. Typically, defines are used for directories and hostnames. In such cases the value can be configured with a single definition. In addition to constants, other strings like code snippets or parser rules can be defined in this way.

An NXLog define works similar to the way it does in the C language, where the pre-processor substitutes the value for the macro wherever it is used. The NXLog configuration parser replaces all occurrences of the defined name with its value, and then after this substitution the configuration check occurs.

Example 1. Using defines

This example shows the use of two defines: BASEDIR and IGNORE_DEBUG. The first is a simple constant whose value is used in two File directives. The second is an NXLog language statement that is used in an Exec directive.

nxlog.conf
define BASEDIR /var/log
define IGNORE_DEBUG if $raw_event =~ /debug/ drop();

<Input messages>
    Module  im_file
    File    '%BASEDIR%/messages'
</Input>

<Input proftpd>
    Module  im_file
    File    '%BASEDIR%/proftpd.log'
    Exec    %IGNORE_DEBUG%
</Input>

The define directive can be used for statements as shown above, but multiple statements should be specified as a code block enclosed in curly braces ({}) to produce the desired results.

Example 2. Incorrect use of a define

The following example shows an incorrect use of the define directive. After substitution, the drop() procedure will always be executed; only the warning message will be logged conditionally.

nxlog.conf (incorrect)
define ACTION log_warning("dropping message"); drop();

<Input in>
    Module  im_file
    File    '/var/log/messages'
    Exec    if $raw_event =~ /dropme/ %ACTION%
</Input>

To avoid this problem, the action should be defined using a code block.

define ACTION { log_warning("dropping message"); drop(); }

Environment variables

The envvar directive works like define except that the value is retrieved from the environment. This makes it possible to reference the environment variable as if it were a define. This directive is only available in NXLog Enterprise Edition.

Example 3. Using environment variables

This is similar to the previous example using a define, but here the value is fetched from the environment.

nxlog.conf
envvar BASEDIR

<Input in>
    Module  im_file
    File    '%BASEDIR%/messages'
</Input>

Global directives

The global section contains directives that control the overall behavior of NXLog.

The LogFile directive sets a destination file for NXLog internal logs. If this directive is unset, the log file is disabled and internal NXLog logs are not written to file (unless configured via the im_internal module). See also Log rotation and retention.

With the User and Group directives set, NXLog will drop root privileges after starting and run as the specified user and group. If, for whatever reason, Linux capabilities need to be permanently set, the Capabilities directive can be used. Such Linux-specific directives are ignored if running on Windows.

After starting, NXLog will change its working directory to the directory specified by the SpoolDir directive.Any path specified in the configuration that is not an absolute path will be relative to this directory.

See the Reference Manual for a complete list of available global directives.

File inclusion

NXLog provides several features for including configuration directives and blocks from separate files or from executables.

The SpoolDir directive does not take effect until after the configuration has been parsed, so relative paths specified with these directives are relative to the working directory where NXLog was started from. Generally, it is recommended to use absolute paths. If desired, define directives can be used to simulate relative paths (see Using defines to include a configuration file).

With the include directive it is possible to specify a file, or a set of files, to be included in the current NXLog configuration.

Example 4. Including a configuration file

This example includes the contents of the /opt/nxlog/etc/syslog.conf file in the current configuration.

nxlog.conf
include /opt/nxlog/etc/syslog.conf
Example 5. Using defines to include a configuration file

In this example, two define directives are used to include an eventlog.conf configuration file on Windows by defining parts of this file’s path.

nxlog.conf
define ROOT C:\Program Files (x86)\nxlog
define CONFDIR %ROOT%\conf
include %CONFDIR%\eventlog.conf

The include directive also supports filenames containing the wildcard character (*). For example, multiple .conf files could be saved in the nxlog.d directory—​or some other custom configuration directory—​and then automatically included in the NXLog configuration in alphabetical order along with the nxlog.conf file.

Each included file might contain a small set of configuration information pertaining exclusively to a single log source. This essentially establishes a modular design for maintaining larger configurations. One benefit of this modular configuration approach is the ability to add/remove .conf files to/from such a directory for enabling/disabling specific log sources without ever needing to modify the main nxlog.conf configuration.

This solution could be used to specify OS-specific configuration snippets (like windows2003.conf) or application-specific snippets (such as syslog.conf).

Including subdirectories inside the configuration directory is not supported, neither are wildcarded directories.

Example 6. Including a configuration directory on Linux

This example includes all .conf files located under the /opt/nxlog/etc/nxlog.d path.

nxlog.conf
include /opt/nxlog/etc/nxlog.d/*.conf

The files can also be included using the define directive.

nxlog.conf
define CONFDIR /opt/nxlog/etc/nxlog.d
include %CONFDIR%/*.conf
Example 7. Including a configuration directory on Windows

This example includes all .conf files from the nxlog.d folder on Windows.

nxlog.conf
include C:\Program Files\nxlog\conf\nxlog.d\*.conf

The files can also be included using the define directive.

nxlog.conf
define CONFDIR C:\Program Files\nxlog\conf\nxlog.d
include %CONFDIR%\*.conf

With the include_stdout directive, an external command can be used to provide configuration content. There are many ways this could be used, including fetching, decrypting, and validating a signed configuration from a remote host, or generating configuration content dynamically.

Example 8. Using an executable to generate configuration

Here, a separate script is responsible for fetching the NXLog configuration.

nxlog.conf
include_stdout /opt/nxlog/etc/fetch_conf.sh

Modules

NXLog will only load modules that are specified in the configuration file and used in an active route. A module instance is specified according to its corresponding module type (Extension, Input, Processor, or Output). Each module instance must have a unique name and a Module directive. The following is a skeleton configuration block for an input module.

Example 9. Modules
nxlog.conf
<Input instancename>
    Module  im_module
    ...
</Input>

For more details about module instance names, see Configuration in the Reference Manual.

Routes

Routes define the flow and processing order of the log messages. Each route instance must have a unique name and a Path.

Example 10. Route Block

This Route instance, named example, takes logs from Input module instances named in1 and in2, processes the logs with the proc Processor module instance, and sends the resulting logs to both Output module instances out1 and out2. These named module instances must be defined elsewhere in the configuration file.

nxlog.conf
<Route example>
    Path    in1, in2 => proc => out1, out2
</Route>

For more details about route instance names, see Configuration in the Reference Manual.

If no Route block is specified in the configuration, NXLog will automatically generate a route, with all the Input and Output instances specified in a single path.

Example 11. An automatic route block

As shown in this configuration, NXLog can function without the route being explicitly defined in a Route block.

nxlog.conf
<Input in1>
    Module  im_null
</Input>

<Input in2>
    Module  im_null
</Input>

<Output out1>
    Module  om_null
</Output>

<Output out2>
    Module  om_null
</Output>

The following Route block will be generated automatically.

nxlog.conf (generated route)
<Route r>
    Path    in1, in2 => out1, out2
</Route>

File paths

File paths are used by global and module directives, and should be enclosed in either single quotes (' ') or double quotes (" ").

Example 12. Using file paths
nxlog.conf
File    "/tmp/input"

Several modules, however, require file paths to be defined without any quotation marks.

nxlog.conf
DeviceFile    /dev/auditpipe

Quoting and escaping strings

String literals must be quoted using either single or double quotes. When a string is enclosed in single quotes, NXLog interprets each character as is. When a string is enclosed in double quotes, NXLog supports escape sequences starting with the backslash (\) character.

Table 1. Table of escape sequences supported by NXLog
Sequence Description

\\

The backslash (\) character

\"

The double quote (") character

\n

Line feed (LF)

\r

Carriage return (CR)

\t

Horizontal tab

\b

Audible bell

\xXX

A single byte in the form of a two digit hexadecimal number.
For example the line-feed character can also be expressed as \x0A

When a string is enclosed in double quotes, NXLog will also interpret the question mark (?) and asterisk (*) as wildcard characters. This means that specifying "file*.log" in a file path will match file1.log, filea.log, etc. For more details, see the Configuration section of the im_file module documentation.

Extra care should be taken when specifying file paths within double quotes since the backslash (\) is used as the directory separator on Windows. For more information about the possible complications, see this note of the im_file module.

The examples below demonstrate how to specify various strings and their resulting values.

Example 13. Correct usage of single quotes

The statement below sets the $FilePath field to a path on Windows.

nxlog.conf
$FilePath = 'C:\AppLogs\logfile.txt'

No escaping is done when using single quotes, therefore the string is interpreted as is.

Output
C:\AppLogs\logfile.txt

Nested quotation is possible using double quotes. Nested quotation using single quotes is not supported.

nxlog.conf
$Message = 'Message: "This is a test"'
Output
Message: "This is a test"
Example 14. Correct usage of double quotes

The statement below sets the $FilePath field to a path on Windows. It uses escape sequences for the directory separator.

nxlog.conf
$FilePath = "C:\\AppLogs\\logfile.txt"

Since the string is enclosed in double quotes, \\ will be interpreted as an escape sequence and will result in a valid path.

Output
C:\AppLogs\logfile.txt

Both single and double quotes are supported for nested quotation. Double quotes need to be escaped with \".

nxlog.conf
$Message = "Message: \"This is a test\" using 'nested quotation'"
Output
Message: "This is a test" using 'nested quotation'
Example 15. Using wildcard characters

Wildcard characters are useful for reading data from multiple files based on their naming pattern. The File directive below uses the asterisk character (*) to read data from several files whose names start with log and have the .txt extension.

nxlog.conf
<Input from_file>
    Module    im_file
    File      "/tmp/log*.txt"
</Input>

Based on the naming pattern, both log1.txt and log_new.txt will be read.

Multiple lines

A directive and its argument must be specified on the same line. Values spanning multiple lines must have the newline escaped with a backslash (\) as shown below.

Example 16. Arguments spanning multiple lines
nxlog.conf
Fields    $Version, $Device_Vendor, $Device_Product, $Device_Version, \
          $Signature_ID, $Name, $Severity, $_Extension

A typical case of the backslash use is the Exec directive.

nxlog.conf
Exec    if ($QueryName == 'wpad') OR \
        ($QueryType != '1') drop();

Breaking regular expressions into multiple lines improves their readability. This sample of a define directive uses backslashes to continue the expression onto each new consecutive line.

nxlog.conf
define EVENT_REGEX /(?x)^(\d+.\d+.\d+)\s+(\d+:\d+:\d+).\d+\
                         \s+PID=(\d+)\s+TID=(\d+)\s+(\w+)\s+\
                         \S(\w+):\s+(.*)/

Comments

Blank lines in configuration files are ignored.

Lines starting with the hash mark (#) are also ignored and can be used as comments.

Example 17. Using comments
nxlog.conf
# This is the comment line

Each comment should start with a new line. Application of inline comments is impossible because each directive can only contain the directive definition and the argument.

Comments spanning multiple lines should start with the # mark on each line. Escaping multiline comments with a backslash (\) is not supported.

nxlog.conf
# This is a comment
# spanning multiple lines

Comments inside the QueryXML directive must be XML-style comments bounded by <!-- and --> as shown in the sample below.

nxlog.conf
<!--
XML-style comments can
span multiple lines in
QueryXML blocks like this.
-->