Common module settings

A module instance name may contain letters, digits, periods (.), and underscores (_). The first character of a module instance name must be a letter or an underscore. NXLog Agent uses the following regular expression to validate the name: [a-zA-Z_][a-zA-Z0-9._]*.

The following directives are common to all modules.

Required directives

Module: This mandatory directive specifies which binary should be loaded. The module binary has a .so extension on Unix and a .dll on Windows platforms and resides under the ModuleDir directory. Each module binary name is prefixed with im_, pm_, om_, or xm_ (for input, processor, output, and extension, respectively). Multiple instances can use the same loadable binary. In this case, the binary is only loaded once but instantiated multiple times. Different module instances may have different configurations.

Optional directives

BatchSize: This directive specifies how many records will be collected by the module and forwarded together as a batch to the next module in the route. For configuring the batch size for all modules globally, read the global BatchSize directive description.

BatchFlushInterval: This directive can only be used with input and processor modules. It specifies the timeout before a batch of records is forwarded to the next module in the route. See the description of the global BatchFlushInterval directive for more information.

BufferSize

This directive specifies the read/write buffer size for input and output modules in bytes. The default buffer size is 65000 bytes. Only stream-oriented modules use this directive, i.e., those connecting over the network (UDP, TCP, and HTTP) or reading and writing to files. Each module instance can have multiple buffers. For example, an im_file instance collecting logs from multiple files will have a buffer for each open file. Similarly, an im_tcp instance will have a buffer for each open connection. The value set in this directive applies to each buffer and is not cumulative; therefore, it does not limit how much memory a module instance can use.

The buffer size you specify should be large enough to hold the biggest chunk of data you expect an input module to receive or an output module to write. Hence, the appropriate buffer size is use-case-dependent, and no general ideal buffer size exists.

It is important to note that the buffer referred to here is a user-space buffer holding a chunk of data the module is processing, not the OS kernel buffer.

Setting a very large buffer size is unlikely to improve performance and will increase the probability of data loss if NXLog Agent terminates unexpectedly.

FlowControl

This optional boolean directive specifies whether the module instance should use flow control. FlowControl is only valid for input, processor, and output modules. For input and processor modules, the FlowControl default is inherited from the global FlowControl directive (which defaults to TRUE). To maintain backward compatibility, the FlowControl default for output modules is TRUE regardless of the global FlowControl value.

Under normal conditions, when all module instances are operating normally and buffers are not full, flow control has no effect. However, if a module becomes blocked and is unable to process events, flow control will automatically suspend module instances as necessary to prevent events from being lost. For example, consider a route with im_file and om_tcp. If a network error blocks the output, NXLog Agent will stop reading events from the file until the network error is resolved. If flow control is disabled, NXLog Agent will continue reading from file and processing events; the events will be discarded when passed to the output module.

In most cases, flow control should be enabled, but it may be necessary to disable it in certain scenarios. It is recommended to leave flow control enabled globally and only specify the FlowControl directive in two cases. First, set FlowControl FALSE on any input module instance that should never be suspended. Second, if a route contains multiple output instances, it may be desirable to continue sending events to other outputs even if one output becomes blocked—set FlowControl FALSE on the output(s) where events can be discarded to prevent the route from being blocked.

Internally, flow control takes effect when the log queue of the next module instance in the route becomes full. If flow control is enabled, the instance is suspended. If flow control is disabled, events are discarded. If the next module in the route is an output module, both FlowControl values are consulted—flow control is enabled only if both are TRUE.

Suspending an im_linuxaudit instance could cause an Audit backlog, blocking processes that generate Audit messages. Suspending an im_udp instance is ineffective because UDP provides no receipt acknowledgment. Suspending an im_uds instance when collecting local syslog messages from the /dev/log Unix domain socket will cause the syslog() system call to block in any programs trying to write to the system log. It is generally recommended to disable flow control in these cases.

InputType

This directive specifies the name of the registered input reader function to be used for parsing raw events from input data. Names are not case-sensitive. This directive is only available for stream-oriented input modules: im_file, im_exec, im_ssl, im_tcp, im_udp, im_uds, and im_pipe. These modules work by filling an input buffer with data read from the source. If the read operation is successful and data is available from the source, the module calls the specified callback function. If this is not explicitly specified, the module default will be used. Note that im_udp may only work properly if log messages do not span multiple packets and are within the UDP message size limit. Otherwise, the loss of a packet may lead to parsing errors.

Modules may provide custom input reader functions. Once they are registered into the NXLog Agent core, the modules listed above will be able to use these functions. This makes it easier to implement custom protocols because they can be developed without concern for the transport layer.

The following input reader functions are provided by the NXLog Agent core:

Binary: The input is parsed in the NXLog Agent binary format, which preserves the parsed fields of the event records. The LineBased reader will automatically detect event records in the NXLog Agent binary format, so it is only recommended to configure InputType to Binary if compatibility with other logging software is not required.

Dgram: Once the buffer is filled with data, it is considered to be one event record. This is the default for the im_udp input module, since UDP Syslog messages arrive in separate packets.

LineBased: The input is assumed to contain event records separated by newlines. It can handle both CRLF (Windows) and LF (Unix) line-breaks. Thus if an LF (\n) or CRLF (\r\n) is found, the function assumes that it has reached the end of the event record. If the input begins with the UTF-8 byte order mark (BOM) sequence (0xEF,0xBB,0xBF), that sequence is automatically skipped.

Example 1. TCP input assuming NXLog Agent format

This configuration explicitly specifies the Binary InputType.

nxlog.conf

<Input tcp_listen>
    Module        im_tcp
    ListenAddr    191.168.1.123:1514
    InputType     Binary
</Input>

With the im_file module, this directive also supports one or more data converters to process input data before reading.

Data converters are processed sequentially from left to right, thus the order they are specified is important. The following example shows the syntax of the InputType directive with two data converters and the input reader function:

InputType    <InstanceName>.<DataConverter1>, \
             <InstanceName>.<DataConverter2>, \
             <InputReaderFunction>

Where <InstanceName> is the given name of an extension module instance and <DataConverter> is the name of the data converter being invoked. Currently, data converters are available in the xm_charconv, xm_crypto and xm_zlib modules. The InputType directive can contain multiple data conversion operations from the same extension module.

For more details, see the documentation of the xm_charconv, xm_crypto and xm_zlib modules.

Example 2. Decompression, decryption and conversion of data

This configuration contains one instance of the xm_zlib module to decompress the input data, one instance of the xm_crypto module to decrypt it and one instance of xm_charconv to convert the input data to UTF-8 encoding. Since the InputType directive of the im_file module instance does not specify an input reader function, the default LineBased will be used.

nxlog.conf

<Extension crypto>
    Module            xm_crypto
    UseSalt           TRUE
    Password          secret
</Extension>

<Extension gzip>
    Module            xm_zlib
    Format            gzip
    CompressionLevel  9
    CompBufSize       16384
    DecompBufsize     16384
</Extension>

<Extension charset>
    Module            xm_charconv
    InputEncoding     UCS-2LE
</Extension>

<Input file>
    Module            im_file
    File              '/tmp/input'
    InputType         crypto.aes_decrypt, gzip.decompress, charset.convert
</Input>

LogLevel

This directive can be used to override the value of the global LogLevel. This can be useful for debugging purposes when DEBUG is set at the module level, or a noisy module can be silenced when set to CRITICAL or ERROR.

Example 3. Using LogLevel at module level

<Input fim>
    Module      im_fim
    LogLevel    Debug
    ...
</Input>

LogqueueDir: This directive specifies the directory where the files of the persistent queue are stored. LogqueueDir is only valid for processor and output module instances. See the description of the global LogqueueDir for more information.

LogqueueSize

Every processor and output instance has an input log queue for events waiting to be processed by that module. The maximum size of the module queue in bytes — the default is 2 MiB (2097152 bytes). The lowest possible value is 512 KiB (524288 bytes) which will be enforced in case of a smaller value. When the log queue of a module instance is full and FlowControl is enabled for the preceding module, the preceding module will be suspended. If flow control is not enabled for the preceding module, events will be discarded. This directive is only valid for processor and output module instances. This directive can be used at the global level to affect all modules.

The use of this directive may produce high memory usage. Also, note that the actual queue size depends on the number of event records (and their sizes) in the module queue. The limit, which is set by this directive, is not a hard one as the sizes of the events are based on approximations and the queue can store an extra batch of records even after reaching the limited size.

OutputType

This directive specifies the name of the registered output writer function to be used for formatting raw events when storing or forwarding output. Names are not case-sensitive. This directive is only available for stream-oriented output modules: om_exec, om_file, om_pipe, om_redis, om_ssl, om_tcp, om_udp, om_udpspoof, om_uds, and om_zmq. These modules work by filling the output buffer with data to be written to the destination. The specified callback function is called before the write operation. If this is not explicitly specified, the module default will be used.

Modules may provide custom output formatting functions. Once they are registered into the NXLog Agent core, the modules listed above will be able to use these functions. This makes it easier to implement custom protocols because they can be developed without concern for the transport layer.

The following output writer functions are provided by the NXLog Agent core:

Binary: The output is written in the NXLog Agent binary format which preserves parsed fields of the event records.

Dgram: Once the buffer is filled with data, it is considered to be one event record. This is the default for the om_udp, om_udpspoof, om_redis, and om_zmq output modules.

LineBased: The output will contain event records separated by newlines. The record terminator is CRLF (\r\n) on Windows and LF (\n) on Unix.

LineBased_CRLF: The output will contain event records separated by Windows-style newlines where the record terminator is CRLF (\r\n).

LineBased_LF: The output will contain event records separated by Unix style newlines where the record terminator is LF (\n).

Example 4. TCP output sending messages in NXLog Agent format

This configuration explicitly specifies the Binary OutputType.

nxlog.conf

<Output tcp>
    Module        om_tcp
    Host          192.168.1.123:1514
    OutputType    Binary
</Output>

With the om_file module, this directive also supports one or more data converters to process output data after writing.

Data converters are processed sequentially from left to right, thus the order they are specified is important. The following example shows the syntax of the OutputType directive with two data converters and the output writer function:

OutputType    <OutputWriterFunction>, \
              <InstanceName>.<DataConverter1>, \
              <InstanceName>.<DataConverter2>

Where <InstanceName> is the given name of an extension module instance and <DataConverter> is the name of the data converter being invoked. Currently, data converters are available in the xm_charconv, xm_crypto and xm_zlib modules. The OutputType directive can contain multiple data conversion operations from the same extension module.

Rotation of files is done automatically when encrypting log data with the xm_crypto module.

For more details, see the documentation of the xm_charconv, xm_crypto and xm_zlib modules.

Example 5. Conversion, compression and encryption of data

This configuration contains one instance of the xm_charconv module to convert the output data to UTF-32LE, one instance of xm_zlib module to compress the output file and one instance of the xm_crypto module to encrypt it. Since the OutputType directive of the om_file module instance does not specify an output writer function, the default LineBased will be used.

nxlog.conf

<Extension charset>
    Module            xm_charconv
    OutputEncoding    UTF-32LE
</Extension>

<Extension gzip>
    Module            xm_zlib
    Format            gzip
    CompressionLevel  9
    CompBufSize       16384
    DecompBufsize     16384
</Extension>

<Extension crypto>
    Module            xm_crypto
    UseSalt           TRUE
    Password          secret
</Extension>

<Input tcp_listen>
    Module            im_tcp
    Host              192.168.31.11:1514
</Input>

<Output file>
    Module            om_file
    File              '/tmp/output'
    OutputType        charset.convert, gzip.compress, crypto.aes_encrypt
</Output>

PersistLogqueue: When a module passes an event to the next module along the route, it puts it into the next module’s queue. This queue can be either a memory-based or disk-based (persistent) queue. When this directive is set to TRUE, the module will use a persistent (disk-based) queue. With the default value of FALSE, the module’s incoming log queue will be memory-based; however, in-memory log queues will still be persisted to disk on shutdown. This directive is only valid for processor and output module instances. This directive can also be used at the global level.

SyncLogqueue: When this directive is set to TRUE and PersistLogqueue is enabled, the disk-based queue will be immediately synced after each new entry is added to the queue. This greatly reduces performance but makes the queue more reliable and crash-safe. This directive is only valid for processor and output module instances. This directive can be used at the global level to affect all modules.

Exec

The Exec directive/block contains statements in the NXLog language which are executed when a module receives a log message. This directive is available in all input, processor, and output modules. It is not available in most extension modules because these do not handle log messages directly (the xm_multiline and xm_rewrite modules do provide Exec directives).

Example 6. Simple Exec Statement

This statement assigns a value to the $Hostname field in the event record.

nxlog.conf

Exec    $Hostname = 'myhost';

Each directive must be on one line unless it contains a trailing backslash (\) character.

Example 7. Exec statement spanning multiple lines

This if statement uses line continuation to span multiple lines.

nxlog.conf

Exec    if $Message =~ /something interesting/       \
            log_info("found something interesting"); \
        else                                         \
            log_debug("found nothing interesting");

More than one Exec directive or block may be specified. They are executed in the order of appearance. Each Exec directive must contain a full statement. Therefore it is not possible to split the lines in the previous example into multiple Exec directives. It is only possible to split the Exec directive if it contains multiple statements.

Example 8. Equivalent Use of Statements in Exec

This example shows two equivalent uses of the Exec directive.

nxlog.conf

Exec    log_info("first"); \
        log_info("second");

This produces identical behavior:

nxlog.conf

Exec    log_info("first");
Exec    log_info("second");

The Exec directive can also be used as a block. To use multiple statements spanning more than one line, it is recommended to use the <Exec> block instead. When using a block, it is not necessary to use the backslash (\) character for line continuation.

Example 9. Using the Exec block

This example shows two equivalent uses of Exec, first as a directive, then as a block.

nxlog.conf

Exec    log_info("first"); \
        log_info("second");

The following Exec block is equivalent. Notice the backslash (\) is omitted.

nxlog.conf

<Exec>
    log_info("first");
    log_info("second");
</Exec>

Schedule

The Schedule block can be used to execute periodic jobs, such as log rotation or any other task. Scheduled jobs have the same priority as the module. The Schedule block has the following directives:

Every: In addition to the crontab format it is possible to schedule execution at periodic intervals. With the crontab format it is not possible to run a job every five days for example, but this directive enables it in a simple way. It takes a positive integer value with an optional unit. The unit can be one of the following: sec, min, hour, day, or week. If the unit is not specified, the value is assumed to be in seconds. The Every directive cannot be used in combination with RunCount 1.

Exec: The mandatory Exec directive takes one or more NXLog Agent statements. This is the code being scheduled. Multiple Exec directives can be specified within one Schedule block. See the module-level Exec directive, this behaves the same. Note that it is not possible to use fields in statements here because execution is not triggered by log messages.

First: This directive sets the first execution time. If the value is in the past, the next execution time is calculated as if NXLog Agent has been running since and jobs will not be run to make up for missed events in the past. The directive takes a datetime literal value.

RunCount: This optional directive can be used to specify a maximum number of times that the corresponding Exec statement(s) should be executed. For example, with RunCount 1 the statement(s) will only be executed once.

When

This directive takes a value similar to a crontab entry: five space-separated definitions for minute, hour, day, month, and weekday. See the crontab(5) manual for the field definitions. It supports lists as comma-separated values and/or ranges. Step values are also supported with the slash. Month and weekdays are not supported, these must be defined with numeric values. The following extensions are also supported:

@startup       Run once when NXLog Agent starts.
@reboot        (Same as @startup)
@yearly        Run once a year, "0 0 1 1 *".
@annually      (Same as @yearly)
@monthly       Run once a month, "0 0 1 * *".
@weekly        Run once a week, "0 0 * * 0".
@daily         Run once a day, "0 0 * * *".
@midnight      (Same as @daily)
@hourly        Run once an hour, "0 * * * *".

Example 10. Scheduled Exec Statements

This example shows two scheduled Exec statements in an im_tcp module instance. The first is executed every second, while the second uses a crontab(5) style value.

nxlog.conf

<Input tcp_listen>
    Module        im_tcp
    ListenAddr    0.0.0.0:1514

    <Schedule>
        Every    1 sec
        First    2024-02-27 00:19:06
        Exec     log_info("scheduled execution at " + now());
    </Schedule>

    <Schedule>
        When     1 */2 2-4 * *
        Exec     log_info("scheduled execution at " + now());
    </Schedule>
</Input>