NXLog Docs

Configuration

An NXLog configuration consists of global directives, module instances, and routes. The following sections list the core NXLog directives provided. Additional directives are provided at the module level.

Logging configuration is valid without any module instances specified; however, for NXLog to process data, the configuration should contain at least one input module instance and at least one output module instance. If no route is specified, a route will be automatically generated; this route will connect all input module instances and all output module instances in a single path.

A module instance name may contain letters, digits, periods (.), and underscores (_). The first character in a module instance name must be a letter or an underscore. The corresponding regular expression is [a-zA-Z_][a-zA-Z0-9._]*.

A route instance name may contain letters, digits, periods (.), and underscores (_). The first character in a route instance name must be a letter, a digit, or an underscore. The corresponding regular expression is [a-zA-Z0-9_][a-zA-Z0-9._]*.

Any text written after a hash mark (#) is ignored and treated as a comment. To learn more configuration tips, read the Configuration Overview section of the User Guide.

General Directives

The following directives can be used throughout the configuration file. These directives are handled by the configuration parser, and substitutions occur before the configuration check.

define

Use this directive to configure a constant or macro to be used later. Refer to a define by surrounding the name with percent signs (%). Enclose a group of statements with curly braces ({}).

Example 1. Using the define Directive

This configuration shows three example defines: BASEDIR is a constant, IMPORTANT is a statement, and WARN_DROP is a group of statements.

nxlog.conf
define BASEDIR /var/log
define IMPORTANT if $raw_event =~ /important/ \
                    $Message = 'IMPORTANT ' + $raw_event;
define WARN_DROP { log_warning("dropping message"); drop(); }

<Input messages>
    Module  im_file
    File    '%BASEDIR%/messages'
</Input>

<Input proftpd>
    Module  im_file
    File    '%BASEDIR%/proftpd.log'
    <Exec>
        %IMPORTANT%
        if $raw_event =~ /dropme/ %WARN_DROP%
    </Exec>
</Input>
envvar

This directive works like define, except that the value is retrieved from the environment.

Example 2. Using the envvar Directive

This example is like the previous one, but BASEDIR is fetched from the environment instead.

nxlog.conf
envvar BASEDIR

<Input in>
    Module  im_file
    File    '%BASEDIR%/messages'
</Input>
include

This directive allows a specified file or files to be included in the current NXLog configuration. Wildcarded filenames are supported.

The SpoolDir directive only takes effect after the configuration is parsed, so relative paths specified with the include directive must be relative to the working directory NXLog was started from.

The examples below provide various ways of using the include directive.

Example 3. Using the include Directive

This example includes a file relative to the working directory.

nxlog.conf
include modules/module1.conf

In case multiple .conf files are to be defined, they can be saved in the nxlog.d directory and then automatically included in the NXLog configuration along with the nxlog.conf file. Adding .conf files in the nxlog.d directory extends the NXLog configuration, without needing to modify the main nxlog.conf file.

This solution could be useful to specify OS-specific configuration snippets (like windows2003.conf) or application-specific snippets (such as syslog.conf).

Inclusion of subdirectories inside the configuration directory is not supported.

Example 4. Including Files With Wildcarded Names

This example includes all matching files from the nxlog.d directory and uses absolute paths on Unix-like systems and Windows.

nxlog.conf
include /etc/nxlog.d/*.conf
nxlog.conf
include C:\Program Files\nxlog\conf\nxlog.d\*.conf
include_stdout

Use this directive to load dynamic configuration from the standard output of a command or script. include_stdout behaves like a command-line interface and accepts a path to a command or script file. It does not support arguments or executing external scripts, such as PowerShell, directly. However, you can encapsulate scripts in a command file.

Example 5. Using the include_stdout directive

This example executes a bash script to load dynamic NXLog configuration.

nxlog.conf
include_stdout /opt/nxlog/etc/fetch_conf.sh

See Including dynamic configuration in the User Guide for further examples.

setenv

This directive sets an environment variable for the nxlog process. The value must be in the format <VAR_NAME>=<VAR_VALUE>.

Example 6. Using the setenv directive

This example creates an environment variable named KRB5_KTNAME and assigns it the value C:\Program Files\nxlog\krb\krb5.keytab.

nxlog.conf
setenv KRB5_KTNAME=C:\Program Files\nxlog\krb\krb5.keytab

Global Directives

AbortOnDoubleSigterm

After receiving the first termination signal NXLog might not terminate right away due to its shutdown sequence. This boolean directive specifies whether NXLog should immediately terminate on second termination signal (SIGINT, SIGQUIT or SIGTERM). Default value is FALSE.

BatchSize

Modules accumulate multiple records together into a batch, and each batch is forwarded to the next module in the route. This directive specifies the maximum number of records to accumulate per batch. If not specified, it defaults to maximum 50 records per batch. The global batch size can be overridden per module, with the BatchSize module-level directive.

The use of this directive may produce high memory usage. For more details on the calculations that are used to determine the impact that LogqueueSize has on memory consumption, refer to BatchSize/LogqueueSize for memory usage.
BatchFlushInterval

Input and processor modules accumulate multiple records together into a batch, and each batch is forwarded to the next module in the route. This directive specifies the timeout, in seconds, before a batch of records will be forwarded to the next module in the route, even if it has accumulated fewer than the maximum number of records given by the BatchSize directive. If this directive is not specified, it defaults to 0.1 (100 milliseconds). The flush interval can be overriden per module, with the BatchFlushInterval module level directive.

This directive only applies to input and processor modules!
CacheDir

This directive specifies a directory where the cache file (configcache.dat) should be written. This directive has a compiled-in value which is used by default.

CacheFlushInterval

This directive specifies how often the in-memory position cache should be flushed to the cache file. A value of 0 indicates that the cache should only be flushed to file when the agent shuts down; if the server or agent crashes, the current position cache will be lost. A positive integer indicates the interval in seconds between flushes of the cache. The string always specifies that the cache should be flushed to file immediately when a module sets a value. If this directive is not specified the default value of 5 seconds is used. See also the CacheSync directive below.

CacheInvalidationTime

NXLog persists saved positions in cache that is written the disk. To prevent the cache growing indefinitely an invalidation period is used. This directive defines the invalidation period. If the last modification time of an entry exceeds the value set with this directive, the entry is discarded when the cache is read from disk. This directive accepts a positive Integer value. If the directive is not specified, the default value of 864000 (10 days) is used.

CacheSync

When the in-memory position cache is flushed to the cache file, the cache may not be immediately written to the disk due to file system buffering. When this directive is set to TRUE, the cache file is synced to disk immediately when it is written. The default is FALSE. CacheSync has no effect if CacheFlushInterval is set to FALSE or 0. Setting CacheSync to TRUE when CacheFlushInterval is set to always may reduce performance, yet it guarantees data integrity in the event of a crash.

Capabilities

This directive allows you to define Linux capabilities that NXLog must not drop after it has started.

For security reasons, NXLog drops nearly all capabilities during startup, and as a trade-off, loses the ability to use them later in its life cycle. In some rare scenarios this can cause a problem, for example when NXLog is started with an empty configuration, but configuration changes that require binding to a new privileged port are later added by NXLog Manager. In this case the binding will fail. Specifying the required capabilities upfront using this configuration directive eliminates this problem.

For usage examples, see the Setting Linux capabilities section in the NXLog User Guide.

DateFormat

This directive can be used to change the default date format as it appears in the LogFile, in the $raw_event generated by the modules, and when a datetime type value is converted to a string. The following values are accepted (corresponding to the formats accepted by the NXLog strftime() function):

  • YYYY-MM-DD hh:mm:ss (the default)

  • YYYY-MM-DDThh:mm:ssTZ

  • YYYY-MM-DDThh:mm:ss.sTZ

  • YYYY-MM-DD hh:mm:ssTZ

  • YYYY-MM-DD hh:mm:ss.sTZ

  • YYYY-MM-DDThh:mm:ssUTC

  • YYYY-MM-DDThh:mm:ss.sUTC

  • YYYY-MM-DD hh:mm:ssUTC

  • YYYY-MM-DD hh:mm:ss.sUTC

  • A format string accepted by the strftime() function in the C library

EscapeGlobPatterns

This boolean directive specifies whether the backslash (\) character in glob patterns or wildcarded entries should be enabled as an escape sequence. If set to TRUE, this directive implies that the backslash character (\) needs to be escaped by another backslash character (\\). File and directory patterns on Windows do not require escaping and are processed as non-escaped even if this directive is set to TRUE. The default is FALSE. This directive is used in the im_file, im_fim, and im_regmon modules.

FlowControl

This optional boolean directive specifies the flow control default for input and processor module instances. Output module instances do not inherit from this directive. By default, the global FlowControl value is TRUE. See the description of the module level FlowControl directive for more information.

FlowControlFIFO

This bolean directive, when set to TRUE, which is also the default, enables FIFO mode for modules that have flow control disabled. In this mode, when the log queue of a module is full, older records will be dropped in order to make room for newer ones. When set to FALSE, the old behavior is in effect: while the log queue is full, no records will be dropped, but new incoming records will be discarded instead.

GenerateDateInUTC

If set to TRUE, this boolean directive specifies that UTC should be used when generating dates in the format YYYY-MM-DD hh:mm:ss. If set to FALSE, local time will be used when generating dates in this format. The default is FALSE. See also ParseDateInUTC.

Group

Similar to User, NXLog will set the group ID to run under. The group can be specified by name or numeric ID. This directive has no effect when running on the Windows platform or with nxlog-processor(8).

IgnoreErrors

If set to FALSE, NXLog will stop when it encounters a problem with the configuration file (such as an invalid module directive) or if there is any other problem which would prevent all modules functioning correctly. If set to TRUE, NXLog will start after logging the problem. The default value is TRUE.

LogFile

NXLog will write its internal log to this file. If this directive is not specified, self logging is disabled. Note that the im_internal module can also be used to direct internal log messages to files or different output destinations, but this does not support log level below INFO. This LogFile directive is especially useful for debugging.

LogLevel

This directive has five possible values: CRITICAL, ERROR, WARNING, INFO, and DEBUG. It will set both the logging level used for LogFile and the standard output if NXLog is started in the foreground. The default LogLevel is INFO. This directive can also be used at the module level.

LogqueueDir

This directive specifies the directory where the files of the persistent queues are stored for processor and output module instances. Even if PersistLogqueue is set to FALSE, NXLog will persist in-memory queues to the LogqueueDir on shutdown. If not specified, the default is the value of CacheDir. This directive can also be used at the module level to specify a log queue directory for a specific module instance.

LogqueueSize

This directive controls the size of the log queue for all processor and output module instances. The default is 100 record batches. See the module-level LogqueueSize for more information.

The use of this directive may produce high memory usage. For more details on the calculations that are used to determine the impact that LogqueueSize has on memory consumption, refer to BatchSize/LogqueueSize for memory usage.
ManagedConfigFile

This directive controls which configuration file will be backed-up when a new version is written by the xm_admin PutFile command. The default is managed.conf.

ModuleDir

By default the NXLog binaries have a compiled-in value for the directory to search for loadable modules. This can be overridden with this directive. The module directory contains sub-directories for each module type (extension, input, output, and processor), and the module binaries are located in those.

NoCache

Some modules save data to a cache file which is persisted across a shutdown/restart. Modules such as im_file will save the file position in order to continue reading from the same position after a restart. This caching mechanism can be explicitly turned off with this directive. This is mostly useful with nxlog-processor(8) in offline mode. If this boolean directive is not specified, it defaults to FALSE (caching is enabled). Note that many input modules, such as im_file, provide a SavePos directive that can be used to disable the position cache for a specific module instance. SavePos has no effect if the cache is disabled globally with NoCache TRUE.

NoFreeOnExit

This directive is for debugging. When set to TRUE, NXLog will not free module resources on exit, allowing tools like Valgrind to show proper stack trace locations in module function calls. The default value is FALSE.

Panic

A panic condition is a critical state which usually indicates a bug. Assertions are used in NXLog code for checking conditions where the code will not work unless the asserted condition is satisfied, and for security. Failing assertions result in a panic and suggest a bug in the code. A typical case is checking for NULL pointers before pointer dereference. This directive can take three different values: HARD, SOFT, or OFF. HARD will cause an abort in case the assertion fails. This is how most C based programs work. SOFT will cause an exception to be thrown at the place of the panic/assertion. In case of NULL pointer checks this is identical to a NullPointerException in Java. It is possible that NXLog can recover from exceptions and can continue to process log messages, or at least the other modules can. In case of assertion failure the location and the condition is printed at CRITICAL log level in HARD mode and ERROR log level in SOFT mode. If Panic is set to OFF, the failing condition is printed in the logs but the execution will continue on the normal code path. Most of the time this will result in a segmentation fault or other undefined behavior, though in some cases turning off a buggy assertion or panic will solve the problems caused by it in HARD/SOFT mode. The default value for Panic is SOFT.

ParseDateInUTC

If set to TRUE, this boolean directive specifies that dates in the format YYYY-MM-DD hh:mm:ss should be parsed as UTC. If set to FALSE, dates in this format are assumed to be in local time. The default is FALSE. See also GenerateDateInUTC.

PersistLogqueue

This boolean directive specifies that log queues of processor and output module instances should be disk-based. See the module level PersistLogqueue directive for more information.

PidFile

Under Unix operating systems, NXLog writes a PID file as other system daemons do. The default PID file can be overridden with this directive in case multiple daemon instances need to be running. This directive has no effect when running on the Windows platform or with nxlog-processor(8).

ReadTimeout

This directive is specific to nxlog-processor(8). It controls the exit condition of nxlog-processor(8). Its value is a timeout in seconds. If nxlog-processor(8) gets no data to process during this period then it will stop waiting for more data and will exit. The default value is 0.05 s. For any non-negative value which is less than 0.05 this will be 0.05. In case nxlog-processor(8) is configured to read data from the network it is recommended to set this to a higher value.

RootDir

NXLog will set its root directory to the value specified with this directive. If SpoolDir is also set, this will be relative to the value of RootDir (chroot() is called first). This directive has no effect when running on the Windows platform or with the nxlog-processor(8).

SpoolDir

NXLog will change its working directory to the value specified with this directive. This is useful with files created through relative filenames (for example, with om_file) and in case of core dumps. This directive has no effect with the nxlog-processor(8).

StringLimit

To protect against memory exhaustion (and possibly a denial-of-service) caused by over-sized strings, there is a limit on the length of each string (in bytes). The default value is 5242880 bytes (strings will be truncated at 5 MiB).

SuppressRepeatingLogs

Under some circumstances it is possible for NXLog to generate an extreme amount of internal logs consisting of the same message due to an incorrect configuration or a software bug. In this case, the LogFile can quickly consume the available disk space. With this directive, NXLog will write at most 2 lines per second if the same message is generated successively, by logging "last message repeated n times" messages. If this boolean directive is not specified, it defaults to TRUE (suppression of repeating messages is enabled).

SyncLogqueue

When this directive is set to TRUE and PersistLogqueue is enabled, the disk-based queues of processor and output module instances will be immediately synced after each new entry is added to the queue. This greatly reduces performance but makes the queue more reliable and crash-safe. This directive can also be used at the module level.

Threads

This directive specifies the number of worker threads to use. The number of worker threads is calculated and set to an optimal value if this directive is not defined. Do not set this unless you know what you are doing.

User

NXLog will drop to the user specified with this directive. This is useful if NXLog needs privileged access to some system resources (such as kernel messages or to bind a port below 1024). On Linux systems NXLog will use capabilities to access these resources. In this case NXLog must be started as root. The user can be specified by name or numeric ID. This directive has no effect when running on the Windows platform or with nxlog-processor(8).

Common Module Directives

The following directives are common to all modules. The Module directive is mandatory.

Module

This mandatory directive specifies which binary should be loaded. The module binary has a .so extension on Unix and a .dll on Windows platforms and resides under the ModuleDir directory. Each module binary name is prefixed with im_, pm_, om_, or xm_ (for input, processor, output, and extension, respectively). It is possible for multiple instances to use the same loadable binary. In this case the binary is only loaded once but instantiated multiple times. Different module instances may have different configurations.


BatchSize

This directive specifies how many records will be collected by the module and forwarded together as a batch to the next module in the route. For configuring the batch size for all modules globally, read the global BatchSize directive description.

The use of this directive may produce high memory usage. For more details on the calculations that are used to determine the impact that LogqueueSize has on memory consumption, refer to BatchSize/LogqueueSize for memory usage.
BatchFlushInterval

This directive can only be used with input and processor modules. It specifies the timeout before a record-batch is forwarded to the next module in the route. See the description of the global BatchFlushInterval directive for more information.

BufferSize

This directive specifies the read/write buffer size for input and output modules in bytes. The default buffer size is 65000 bytes. Only stream-oriented modules use this directive, i.e., those connecting over the network (UDP, TCP, and HTTP) or reading and writing to files. Each module instance can have multiple buffers. For example, an im_file instance collecting logs from multiple files will have a buffer for each open file. Similarly, an im_tcp instance will have a buffer for each open connection. The value set in this directive applies to each buffer and is not cumulative; therefore, it does not limit how much memory a module instance can use.

The buffer size you specify should be large enough to hold the biggest chunk of data you expect an input module to receive or an output module to write. Hence, the appropriate buffer size is use-case dependent, and no general ideal buffer size exists.

It is important to note that the buffer referred to here is a user-space buffer holding a chunk of data the module is processing, not the OS kernel buffer.

Setting a very large buffer size is unlikely to improve performance and will increase the probability of data loss if NXLog terminates unexpectedly.
FlowControl

This optional boolean directive specifies whether the module instance should use flow control. FlowControl is only valid for input, processor, and output modules. For input and processor modules, the FlowControl default is inherited from the global FlowControl directive (which defaults to TRUE). To maintain backward compatibility, the FlowControl default for output modules is TRUE regardless of the global FlowControl value.

Under normal conditions, when all module instances are operating normally and buffers are not full, flow control has no effect. However, if a module becomes blocked and is unable to process events, flow control will automatically suspend module instances as necessary to prevent events from being lost. For example, consider a route with im_file and om_tcp. If a network error blocks the output, NXLog will stop reading events from file until the network error is resolved. If flow control is disabled, NXLog will continue reading from file and processing events; the events will be discarded when passed to the output module.

In most cases, flow control should be left enabled, but it may be necessary to disable it in certain scenarios. It is recommended to leave flow control enabled globally and only specify the FlowControl directive in two cases. First, set FlowControl FALSE on any input module instance that should never be suspended. Second, if a route contains multiple output instances, it may be desirable to continue sending events to other outputs even if one output becomes blocked—set FlowControl FALSE on the output(s) where events can be discarded to prevent the route from being blocked.

Internally, flow control takes effect when the log queue of the next module instance in the route becomes full. If flow control is enabled, the instance is suspended. If flow control is disabled, events are discarded. If the next module in the route is an output module, both FlowControl values are consulted—flow control is enabled only if both are TRUE.

Suspending an im_linuxaudit instance could cause an Audit backlog, blocking processes that generate Audit messages. Suspending an im_udp instance is ineffective, because UDP provides no receipt acknowledgement. Suspending an im_uds instance when collecting local syslog messages from the /dev/log Unix domain socket will cause the syslog() system call to block in any programs trying to write to the system log. It is generally recommended to disable flow control in these cases.
InputType

This directive specifies the name of the registered input reader function to be used for parsing raw events from input data. Names are treated case insensitively. This directive is only available for stream oriented input modules: im_file, im_exec, im_ssl, im_tcp, im_udp, im_uds, and im_pipe. These modules work by filling an input buffer with data read from the source. If the read operation was successful and data is available from the source, the module calls the specified callback function. If this is not explicitly specified, the module default will be used. Note that im_udp may only work properly if log messages do not span multiple packets and are within the UDP message size limit. Otherwise the loss of a packet may lead to parsing errors.

Modules may provide custom input reader functions. Once they are registered into the NXLog core, the modules listed above will be able to use these functions. This makes it easier to implement custom protocols because they can be developed without concern for the transport layer.

The following input reader functions are provided by the NXLog core:

Binary

The input is parsed in the NXLog binary format, which preserves the parsed fields of the event records. The LineBased reader will automatically detect event records in the NXLog binary format, so it is only recommended to configure InputType to Binary if compatibility with other logging software is not required.

Dgram

Once the buffer is filled with data, it is considered to be one event record. This is the default for the im_udp input module, since UDP Syslog messages arrive in separate packets.

LineBased

The input is assumed to contain event records separated by newlines. It can handle both CRLF (Windows) and LF (Unix) line-breaks. Thus if an LF (\n) or CRLF (\r\n) is found, the function assumes that it has reached the end of the event record. If the input begins with the UTF-8 byte order mark (BOM) sequence (0xEF,0xBB,0xBF), that sequence is automatically skipped.

Example 7. TCP input assuming NXLog format

This configuration explicitly specifies the Binary InputType.

nxlog.conf
<Input tcp>
    Module      im_tcp
    Port        2345
    InputType   Binary
</Input>

With the im_file module, this directive also supports one or more data converters to process input data before reading.

Data converters are processed sequentially from left to right, thus the order they are specified in is important. The following example shows the syntax of the InputType directive with two data converters and the input reader function:

InputType    <InstanceName>.<DataConverter1>, \
             <InstanceName>.<DataConverter2>, \
             <InputReaderFunction>

Where <InstanceName> is the given name of an extension module instance and <DataConverter> is the name of the data converter being invoked. Currently, data converters are available in the xm_crypto and xm_zlib modules. The InputType directive can contain multiple data conversion operations from the same extension module.

For more details, see the documentation of the xm_crypto and xm_zlib modules.

Example 8. Decompression and decryption of data

This configuration contains one instance of the xm_zlib module to decompress the input data and one instance of the xm_crypto module to decrypt it. Since the InputType directive of the im_file module instance does not specify an input reader function, the default LineBased will be used.

nxlog.conf
<Extension cryptography>
    Module            xm_crypto
    UseSalt           TRUE
    PasswordFile      passwordfile
</Extension>

<Extension gzip>
    Module            xm_zlib
    Format            gzip
    CompressionLevel  9
    CompBufSize       16384
    DecompBufsize     16384
</Extension>

<Input from_file>
    Module            im_file
    File              '/tmp/input'
    InputType         cryptography.aes_decrypt, gzip.decompress
</Input>
LogLevel

This directive can be used to override the value of the global LogLevel. This can be useful for debugging purposes when DEBUG is set at the module level, or a noisy module can be silenced when set to CRITICAL or ERROR.

Example 9. Using LogLevel at module level
<Input fim>
    Module      im_fim
    LogLevel    Debug
    ...
</Input>
LogqueueDir

This directive specifies the directory where the files of the persistent queue are stored. LogqueueDir is only valid for processor and output module instances. See the description of the global LogqueueDir for more information.

LogqueueSize

Every processor and output instance has an input log queue for events waiting to be processed by that module. The size of the queue is measured in batches of event records, and can be set with this directive—the default is 100 batches. When the log queue of a module instance is full and FlowControl is enabled for the preceding module, the preceding module will be suspended. If flow control is not enabled for the preceding module, events will be discarded. This directive is only valid for processor and output module instances. This directive can be used at the global level to affect all modules.

The use of this directive may produce high memory usage. For more details on the calculations that are used to determine the impact that LogqueueSize has on memory consumption, refer to BatchSize/LogqueueSize for memory usage.
OutputType

This directive specifies the name of the registered output writer function to be used for formatting raw events when storing or forwarding output. Names are treated case insensitively. This directive is only available for stream oriented output modules: om_exec, om_file, om_pipe, om_redis, om_ssl, om_tcp, om_udp, om_udpspoof, om_uds, and om_zmq. These modules work by filling the output buffer with data to be written to the destination. The specified callback function is called before the write operation. If this is not explicitly specified, the module default will be used.

Modules may provide custom output formatter functions. Once they are registered into the NXLog core, the modules listed above will be able to use these functions. This makes it easier to implement custom protocols because they can be developed without concern for the transport layer.

The following output writer functions are provided by the NXLog core:

Binary

The output is written in the NXLog binary format which preserves parsed fields of the event records.

Dgram

Once the buffer is filled with data, it is considered to be one event record. This is the default for the om_udp, om_udpspoof, om_redis, and om_zmq output modules.

LineBased

The output will contain event records separated by newlines. The record terminator is CRLF (\r\n) on Windows and LF (\n) on Unix.

LineBased_CRLF

The output will contain event records separated by Windows style newlines where the record terminator is CRLF (\r\n).

LineBased_LF

The output will contain event records separated by Unix style newlines where the record terminator is LF (\n).

Example 10. TCP output sending messages in NXLog format

This configuration explicitly specifies the Binary OutputType.

nxlog.conf
<Output tcp>
    Module      om_tcp
    Port        2345
    Host        localhost
    OutputType  Binary
</Output>

With the om_file module, this directive also supports one or more data converters to process output data after writing.

Data converters are processed sequentially from left to right, thus the order they are specified in is important. The following example shows the syntax of the OutputType directive with two data converters and the output writer function:

OutputType    <OutputWriterFunction>, \
              <InstanceName>.<DataConverter1>, \
              <InstanceName>.<DataConverter2>

Where <InstanceName> is the given name of an extension module instance and <DataConverter> is the name of the data converter being invoked. Currently, data converters are available in the xm_crypto and xm_zlib modules. The OutputType directive can contain multiple data conversion operations from the same extension module.

Rotation of files is done automatically when encrypting log data with the xm_crypto module.

For more details, see the the documentation of the xm_crypto and xm_zlib modules.

Example 11. Compression and encryption of data

This configuration contains one instance of the xm_zlib module to compress the output file and one instance of the xm_crypto module to encrypt it. Since the OutputType directive of the om_file module instance does not specify an output writer function, the default LineBased will be used.

nxlog.conf
<Extension gzip>
    Module            xm_zlib
    Format            gzip
    CompressionLevel  9
    CompBufSize       16384
    DecompBufsize     16384
</Extension>

<Extension cryptography>
    Module            xm_crypto
    UseSalt           TRUE
    PasswordFile      passwordfile
</Extension>

<Input from_tcp>
    Module            im_tcp
    Host              192.168.31.11
    Port              10500
</Input>

<Output to_file>
    Module            om_file
    File              '/tmp/output'
    OutputType        gzip.compress, cryptography.aes_encrypt
</Output>
PersistLogqueue

When a module passes an event to the next module along the route, it puts it into the next module’s queue. This queue can be either a memory-based or disk-based (persistent) queue. When this directive is set to TRUE, the module will use a persistent (disk-based) queue. With the default value of FALSE, the module’s incoming log queue will be memory-based; however, in-memory log queues will still be persisted to disk on shutdown. This directive is only valid for processor and output module instances. This directive can also be used at the global level.

SyncLogqueue

When this directive is set to TRUE and PersistLogqueue is enabled, the disk-based queue will be immediately synced after each new entry is added to the queue. This greatly reduces performance but makes the queue more reliable and crash-safe. This directive is only valid for processor and output module instances. This directive can be used at the global level to affect all modules.

Exec

The Exec directive/block contains statements in the NXLog language which are executed when a module receives a log message. This directive is available in all input, processor, and output modules. It is not available in most extension modules because these do not handle log messages directly (the xm_multiline and xm_rewrite modules do provide Exec directives).

Example 12. Simple Exec Statement

This statement assigns a value to the $Hostname field in the event record.

nxlog.conf
Exec    $Hostname = 'myhost';

Each directive must be on one line unless it contains a trailing backslash (\) character.

Example 13. Exec statement spanning multiple lines

This if statement uses line continuation to span multiple lines.

nxlog.conf
Exec    if $Message =~ /something interesting/       \
            log_info("found something interesting"); \
        else                                         \
            log_debug("found nothing interesting");

More than one Exec directive or block may be specified. They are executed in the order of appearance. Each Exec directive must contain a full statement. Therefore it is not possible to split the lines in the previous example into multiple Exec directives. It is only possible to split the Exec directive if it contains multiple statements.

Example 14. Equivalent Use of Statements in Exec

This example shows two equivalent uses of the Exec directive.

nxlog.conf
Exec    log_info("first"); \
        log_info("second");

This produces identical behavior:

nxlog.conf
Exec    log_info("first");
Exec    log_info("second");

The Exec directive can also be used as a block. To use multiple statements spanning more than one line, it is recommended to use the <Exec> block instead. When using a block, it is not necessary to use the backslash (\) character for line continuation.

Example 15. Using the Exec block

This example shows two equivalent uses of Exec, first as a directive, then as a block.

nxlog.conf
Exec    log_info("first"); \
        log_info("second");

The following Exec block is equivalent. Notice the backslash (\) is omitted.

nxlog.conf
<Exec>
    log_info("first");
    log_info("second");
</Exec>

Schedule

The Schedule block can be used to execute periodic jobs, such as log rotation or any other task. Scheduled jobs have the same priority as the module. The Schedule block has the following directives:

Every

In addition to the crontab format it is possible to schedule execution at periodic intervals. With the crontab format it is not possible to run a job every five days for example, but this directive enables it in a simple way. It takes a positive integer value with an optional unit. The unit can be one of the following: sec, min, hour, day, or week. If the unit is not specified, the value is assumed to be in seconds. The Every directive cannot be used in combination with RunCount 1.

Exec

The mandatory Exec directive takes one or more NXLog statements. This is the code which is actually being scheduled. Multiple Exec directives can be specified within one Schedule block. See the module-level Exec directive, this behaves the same. Note that it is not possible to use fields in statements here because execution is not triggered by log messages.

First

This directive sets the first execution time. If the value is in the past, the next execution time is calculated as if NXLog has been running since and jobs will not be run to make up for missed events in the past. The directive takes a datetime literal value.

RunCount

This optional directive can be used to specify a maximum number of times that the corresponding Exec statement(s) should be executed. For example, with RunCount 1 the statement(s) will only be executed once.

When

This directive takes a value similar to a crontab entry: five space-separated definitions for minute, hour, day, month, and weekday. See the crontab(5) manual for the field definitions. It supports lists as comma separated values and/or ranges. Step values are also supported with the slash. Month and week days are not supported, these must be defined with numeric values. The following extensions are also supported:

@startup       Run once when NXLog starts.
@reboot        (Same as @startup)
@yearly        Run once a year, "0 0 1 1 *".
@annually      (Same as @yearly)
@monthly       Run once a month, "0 0 1 * *".
@weekly        Run once a week, "0 0 * * 0".
@daily         Run once a day, "0 0 * * *".
@midnight      (Same as @daily)
@hourly        Run once an hour, "0 * * * *".
Example 16. Scheduled Exec Statements

This example shows two scheduled Exec statements in a im_tcp module instance. The first is executed every second, while the second uses a crontab(5) style value.

nxlog.conf
<Input in>
    Module  im_tcp
    Port    2345

    <Schedule>
        Every   1 sec
        First   2010-12-17 00:19:06
        Exec    log_info("scheduled execution at " + now());
    </Schedule>

    <Schedule>
        When    1 */2 2-4 * *
        Exec    log_info("scheduled execution at " + now());
    </Schedule>
</Input>

Route Directives

The following directives can be used in Route blocks. The Path directive is mandatory.

Path

The data flow is defined by the Path directive. First, the instance names of input modules are specified. If the route contains more than one input source, then these must be separated by a comma. The list of input modules is followed by an arrow (=>). Either processor modules or output modules follow. Processor modules must be separated by an arrow, not a comma, because they operate in series, unlike input and output modules, which work in parallel. Output modules are separated by a comma. The Path must specify at least one input and one output. The syntax is as follows:

Path INPUT1[, INPUT2...] => [PROCESSOR1 [=> PROCESSOR2...] =>] OUTPUT1[, OUTPUT2...]

Example 17. Specifying routes

The following configuration shows the log router with modules being used in two routes.

nxlog.conf
<Input in1>
    Module  im_null
</Input>

<Input in2>
    Module  im_null
</Input>

<Processor p1>
    Module  pm_null
</Processor>

<Processor p2>
    Module  pm_null
</Processor>

<Output out1>
    Module  om_null
</Output>

<Output out2>
    Module  om_null
</Output>

<Route 1>
    # Basic route
    Path    in1 => out1
</Route>

<Route 2>
    # Complex route with multiple input/output/processor modules
    Path    in1, in2 => p1 => p2 => out1, out2
</Route>

Priority

This directive takes an integer value in the range of 1-100 as a parameter, and the default is 10. Log messages in routes with a lower Priority value will be processed before others. Internally, this value is assigned to each module part of the route. The events of the modules are processed in priority order by the NXLog engine. Modules of a route with a lower Priority value (higher priority) will process log messages first.

Example 18. Prioritized processing

This configuration prioritizes the UDP route over the TCP route in order to minimize the possibility of losing logs sent over UDP when the system is busy.

nxlog.conf
<Input tcpin>
    Module      im_tcp
    Host        localhost
    Port        514
</Input>

<Input udpin>
    Module      im_udp
    Host        localhost
    Port        514
</Input>

<Output tcpfile>
    Module      om_file
    File        "/var/log/tcp.log"
</Output>

<Output udpfile>
    Module      om_file
    File        "/var/log/udp.log"
</Output>

<Route udp>
    Priority    1
    Path        udpin => udpfile
</Route>

<Route tcp>
    Priority    2
    Path        tcpin => tcpfile
</Route>