Configuration
An NXLog configuration consists of global directives, module instances, and routes. The following sections list the core NXLog directives provided. Additional directives are provided at the module level.
Logging configuration is valid without any module instances specified; however, for NXLog to process data, the configuration should contain at least one input module instance and at least one output module instance. If no route is specified, a route will be automatically generated; this route will connect all input module instances and all output module instances in a single path.
A module instance name may contain letters, digits, periods (.
), and
underscores (_
). The first character in a module instance name must
be a letter or an underscore. The corresponding regular expression is
[a-zA-Z_][a-zA-Z0-9._]*
.
A route instance name may contain letters, digits, periods (.
), and
underscores (_
). The first character in a route instance name must be
a letter, a digit, or an underscore. The corresponding regular expression is
[a-zA-Z0-9_][a-zA-Z0-9._]*
.
Any text written after a hash mark (#
) is ignored and treated as a comment.
To learn more configuration tips, read the Configuration Overview section of the User Guide.
General Directives
The following directives can be used throughout the configuration file. These directives are handled by the configuration parser, and substitutions occur before the configuration check.
- define
-
Use this directive to configure a constant or macro to be used later. Refer to a
define
by surrounding the name with percent signs (%
). Enclose a group of statements with curly braces ({}
).Example 1. Using the define directiveThis configuration shows three example defines:
BASEDIR
is a constant,IMPORTANT
is a statement, andWARN_DROP
is a group of statements.nxlog.confdefine BASEDIR /var/log define IMPORTANT if $raw_event =~ /important/ \ $Message = 'IMPORTANT ' + $raw_event; define WARN_DROP { log_warning("dropping message"); drop(); } <Input messages> Module im_file File '%BASEDIR%/messages' </Input> <Input proftpd> Module im_file File '%BASEDIR%/proftpd.log' <Exec> %IMPORTANT% if $raw_event =~ /dropme/ %WARN_DROP% </Exec> </Input>
- envvar
-
This directive works like define, except that the value is retrieved from the environment.
Example 2. Using the envvar directiveThis example is like the previous one, but
BASEDIR
is fetched from the environment instead.nxlog.confenvvar BASEDIR <Input file> Module im_file File '%BASEDIR%/messages' </Input>
- include
-
This directive allows a specified file or files to be included in the current NXLog configuration. Wildcarded filenames are supported.
The SpoolDir directive only takes effect after the configuration is parsed, so relative paths specified with the include directive must be relative to the working directory NXLog was started from. The examples below provide various ways of using the include directive.
Example 3. Using the include directiveThis example includes a file relative to the working directory.
nxlog.confinclude modules/module1.conf
In case multiple
.conf
files are to be defined, they can be saved in thenxlog.d
directory and then automatically included in the NXLog configuration along with thenxlog.conf
file. Adding.conf
files in thenxlog.d
directory extends the NXLog configuration, without needing to modify the mainnxlog.conf
file.This solution could be useful to specify OS-specific configuration snippets (like windows2003.conf
) or application-specific snippets (such assyslog.conf
).Inclusion of subdirectories inside the configuration directory is not supported.
Example 4. Including Files With Wildcarded NamesThis example includes all matching files from the
nxlog.d
directory and uses absolute paths on Unix-like systems and Windows.nxlog.confinclude /etc/nxlog.d/*.conf
nxlog.confinclude C:\Program Files\nxlog\conf\nxlog.d\*.conf
- include_stdout
-
Use this directive to load dynamic configuration from the standard output of a command or script. include_stdout behaves like a command-line interface and accepts a path to a command or script file. It does not support arguments or executing external scripts, such as PowerShell, directly. However, you can encapsulate scripts in a command file.
Example 5. Using the include_stdout directiveThis example executes a bash script to load dynamic NXLog configuration.
nxlog.confinclude_stdout /opt/nxlog/etc/fetch_conf.sh
See Including dynamic configuration in the NXLog User Guide for further examples.
- setenv
-
This directive sets an environment variable for the nxlog process. The value must be in the format
<VAR_NAME>=<VAR_VALUE>
.Example 6. Using the setenv directiveThis example creates an environment variable named
KRB5_KTNAME
and assigns it the valueC:\Program Files\nxlog\krb\krb5.keytab
.nxlog.confsetenv KRB5_KTNAME=C:\Program Files\nxlog\krb\krb5.keytab
Global Directives
- AbortOnDoubleSigterm
-
After receiving the first termination signal NXLog might not terminate right away due to its shutdown sequence. This boolean directive specifies whether NXLog should immediately terminate on second termination signal (
SIGINT
,SIGQUIT
orSIGTERM
). Default value isFALSE
.
- BatchSize
-
Modules accumulate multiple records together into a batch, and each batch is forwarded to the next module in the route. This directive specifies the maximum number of records to accumulate per batch. If not specified, it defaults to maximum 50 records per batch. The global batch size can be overridden per module with the BatchSize module-level directive.
The use of this directive may produce high memory usage. For more details on the calculations that are used to determine the impact that LogqueueSize
has on memory consumption, refer to BatchSize/LogqueueSize for memory usage.
- BatchFlushInterval
-
Input and processor modules accumulate multiple records together into a batch, and each batch is forwarded to the next module in the route. This directive specifies the timeout, in seconds, before a batch of records will be forwarded to the next module in the route, even if it has accumulated fewer than the maximum number of records given by the BatchSize directive. If this directive is not specified, it defaults to
0.1
(100 milliseconds). The flush interval can be overriden per module, with the BatchFlushInterval module level directive.This directive only applies to input and processor modules!
- CacheDir
-
This directive specifies a directory where the cache file (
configcache.dat
) should be written. This directive has a compiled-in value which is used by default.
- AgentUidDir
-
This directive specifies a directory where the agent files (
.nxlog.id
) and (.nxlog.id.sig
) should be written. This directive has a compiled-in value which is used by default.
- CacheFlushInterval
-
This directive specifies how often the in-memory position cache should be flushed to the cache file. A value of 0 indicates that the cache should only be flushed to file when the agent shuts down; if the server or agent crashes, the current position cache will be lost. A positive integer indicates the interval in seconds between flushes of the cache. The string
always
specifies that the cache should be flushed to file immediately when a module sets a value. If this directive is not specified the default value of 5 seconds is used. See also the CacheSync directive below.
- CacheInvalidationTime
-
NXLog persists saved positions in cache that is written the disk. To prevent the cache growing indefinitely an invalidation period is used. This directive defines the invalidation period. If the last modification time of an entry exceeds the value set with this directive, the entry is discarded when the cache is read from disk. This directive accepts a positive Integer value. If the directive is not specified, the default value of 864000 (10 days) is used.
- CacheSync
-
When the in-memory position cache is flushed to the cache file, the cache may not be immediately written to the disk due to file system buffering. When this directive is set to
TRUE
, the cache file is synced to disk immediately when it is written. The default isFALSE
. CacheSync has no effect if CacheFlushInterval is set toFALSE
or 0. Setting CacheSync toTRUE
when CacheFlushInterval is set toalways
may reduce performance, yet it guarantees data integrity in the event of a crash.
- Capabilities
-
This directive allows you to define Linux capabilities that NXLog must not drop after it has started.
For security reasons, NXLog drops nearly all capabilities during startup, and as a trade-off, loses the ability to use them later in its life cycle. In some rare scenarios this can cause a problem, for example when NXLog is started with an empty configuration, but configuration changes that require binding to a new privileged port are later added by NXLog Manager. In this case the binding will fail. Specifying the required capabilities upfront using this configuration directive eliminates this problem.
For usage examples, see the Setting Linux capabilities section in the NXLog User Guide.
- ConnectionIdleTimeout
-
This optional directive defines the maximum time in seconds before NXLog closes TCP connections without traffic. The minimum timeout value is 15 seconds. If this directive is not specified, NXLog does not close idle TCP connections.
The ConnectionIdleTimeout can also be defined at the module level. If both global and module directives are defined, the module directive takes precedence over the global directive.
- DateFormat
-
This directive can be used to change the default date format as it appears in the LogFile, in the
$raw_event
generated by the modules, and when a datetime type value is converted to a string. The following values are accepted (corresponding to the formats accepted by the NXLog strftime() function):-
YYYY-MM-DD hh:mm:ss
(the default) -
YYYY-MM-DDThh:mm:ssTZ
-
YYYY-MM-DDThh:mm:ss.sTZ
-
YYYY-MM-DD hh:mm:ssTZ
-
YYYY-MM-DD hh:mm:ss.sTZ
-
YYYY-MM-DDThh:mm:ssUTC
-
YYYY-MM-DDThh:mm:ss.sUTC
-
YYYY-MM-DD hh:mm:ssUTC
-
YYYY-MM-DD hh:mm:ss.sUTC
-
A format string accepted by the strftime() function in the C library
-
- EscapeGlobPatterns
-
This boolean directive specifies whether the backslash (
\
) character in glob patterns or wildcarded entries should be enabled as an escape sequence. If set toTRUE
, this directive implies that the backslash character (\
) needs to be escaped by another backslash character (\\
). File and directory patterns on Windows do not require escaping and are processed as non-escaped even if this directive is set toTRUE
. The default isFALSE
. This directive is used in the im_file, im_fim, and im_regmon modules.
- FlowControl
-
This optional boolean directive specifies the flow control default for input and processor module instances. Output module instances do not inherit from this directive. By default, the global FlowControl value is
TRUE
. See the description of the module level FlowControl directive for more information.
- FlowControlFIFO
-
This bolean directive, when set to
TRUE
, which is also the default, enables FIFO mode for modules that have flow control disabled. In this mode, when the log queue of a module is full, older records will be dropped in order to make room for newer ones. When set toFALSE
, the old behavior is in effect: while the log queue is full, no records will be dropped, but new incoming records will be discarded instead.
- GenerateDateInUTC
-
If set to
TRUE
, this boolean directive specifies that UTC should be used when generating dates in the formatYYYY-MM-DD hh:mm:ss
. If set toFALSE
, local time will be used when generating dates in this format. The default isFALSE
. See also ParseDateInUTC.
- Group
-
Similar to User, NXLog will set the group ID to run under. The group can be specified by name or numeric ID. This directive has no effect when running on the Windows platform or with nxlog-processor(8).
- IgnoreErrors
-
If set to
FALSE
, NXLog will stop when it encounters a problem with the configuration file (such as an invalid module directive) or if there is any other problem which would prevent all modules functioning correctly. If set toTRUE
, NXLog will start after logging the problem. The default value isTRUE
.
- LogFile
-
NXLog will write its internal log to this file. If this directive is not specified, self logging is disabled. Note that the im_internal module can also be used to direct internal log messages to files or different output destinations, but this does not support log level below
INFO
. This LogFile directive is especially useful for debugging.
- LogLevel
-
This directive has five possible values:
CRITICAL
,ERROR
,WARNING
,INFO
, andDEBUG
. It will set both the logging level used for LogFile and the standard output if NXLog is started in the foreground. The default LogLevel isINFO
. This directive can also be used at the module level.
- LogSizeLimit
-
This directive specifies the maximum size of the log lines in bytes, created by NXLog’s
log_*()
procedures. The default is 1024 bytes. The minimum allowed value is 128, while the maximum is 65536 bytes. The closest limit value (128 or 65536) will be used if the value is below or over the allowed limits. Lines that exceed this limit will be truncated. The actual output will always be at least two bytes shorter than the specified value.
- LogqueueDir
-
This directive specifies the directory where the files of the persistent queues are stored for processor and output module instances. Even if PersistLogqueue is set to
FALSE
, NXLog will persist in-memory queues to the LogqueueDir on shutdown. If not specified, the default is the value of CacheDir. This directive can also be used at the module level to specify a log queue directory for a specific module instance.
- LogqueueSize
-
Maximum size of the log queue in bytes. The log queue is an input log queue for events waiting to be processed by that module instance. The default is 2097152 bytes (2 MiB). The minimum allowed value is 524288 bytes (512 KiB), which NXLog enforces if you enter a smaller value. This directive is only valid for processor and output module instances.
When the log queue of a module instance is full and FlowControl is enabled for the preceding module, the preceding module will be suspended. If flow control is not enabled for the preceding module, events will be discarded. This directive can be used at the module level to affect a single module instance.
The use of this directive may produce high memory usage. Also, note that the actual queue size depends on the number of event records (and their size) in the module queue. The limit, which is set by this directive, is not a hard one as the sizes of the events are based on approximations and the queue can store an extra batch of records even after reaching the size limit. See Buffering and flow control for a detailed explanation of log queues.
- ManagedConfigFile
-
This directive controls which configuration file will be backed-up when a new version is written by the xm_admin PutFile command. The default is
managed.conf
.
- ModuleDir
-
By default the NXLog binaries have a compiled-in value for the directory to search for loadable modules. This can be overridden with this directive. The module directory contains sub-directories for each module type (extension, input, output, and processor), and the module binaries are located in those.
- NoCache
-
Some modules save data to a cache file which is persisted across a shutdown/restart. Modules such as im_file will save the file position in order to continue reading from the same position after a restart. This caching mechanism can be explicitly turned off with this directive. This is mostly useful with nxlog-processor(8) in offline mode. If this boolean directive is not specified, it defaults to
FALSE
(caching is enabled). Note that many input modules, such as im_file, provide a SavePos directive that can be used to disable the position cache for a specific module instance. SavePos has no effect if the cache is disabled globally withNoCache TRUE
.
- NoFreeOnExit
-
This directive is for debugging. When set to
TRUE
, NXLog will not free module resources on exit, allowing tools like Valgrind to show proper stack trace locations in module function calls. The default value isFALSE
.
- Panic
-
A panic condition is a critical state which usually indicates a bug. Assertions are used in NXLog code for checking conditions where the code will not work unless the asserted condition is satisfied, and for security. Failing assertions result in a panic and suggest a bug in the code. A typical case is checking for NULL pointers before pointer dereference. This directive can take three different values:
HARD
,SOFT
, orOFF
.HARD
will cause an abort in case the assertion fails. This is how most C based programs work.SOFT
will cause an exception to be thrown at the place of the panic/assertion. In case of NULL pointer checks this is identical to a NullPointerException in Java. It is possible that NXLog can recover from exceptions and can continue to process log messages, or at least the other modules can. In case of assertion failure the location and the condition is printed atCRITICAL
log level inHARD
mode andERROR
log level inSOFT
mode. If Panic is set toOFF
, the failing condition is printed in the logs but the execution will continue on the normal code path. Most of the time this will result in a segmentation fault or other undefined behavior, though in some cases turning off a buggy assertion or panic will solve the problems caused by it inHARD
/SOFT
mode. The default value for Panic isSOFT
.
- ParseDateInUTC
-
If set to
TRUE
, this boolean directive specifies that dates in the formatYYYY-MM-DD hh:mm:ss
should be parsed as UTC. If set toFALSE
, dates in this format are assumed to be in local time. The default isFALSE
. See also GenerateDateInUTC.
- PersistLogqueue
-
This boolean directive specifies that log queues of processor and output module instances should be disk-based. See the module level PersistLogqueue directive for more information.
See Buffering and flow control for a detailed explanation of log queues.
- PidFile
-
Under Unix operating systems, NXLog writes a PID file as other system daemons do. The default PID file can be overridden with this directive in case multiple daemon instances need to be running. This directive has no effect when running on the Windows platform or with nxlog-processor(8).
- ReadTimeout
-
This directive is specific to nxlog-processor(8). It controls the exit condition of nxlog-processor(8). Its value is a timeout in seconds. If nxlog-processor(8) gets no data to process during this period then it will stop waiting for more data and will exit. The default value is 0.05 s. For any non-negative value which is less than 0.05 this will be 0.05. In case nxlog-processor(8) is configured to read data from the network it is recommended to set this to a higher value.
- RootDir
-
NXLog will set its root directory to the value specified with this directive. If SpoolDir is also set, this will be relative to the value of RootDir (chroot() is called first). This directive has no effect when running on the Windows platform or with the nxlog-processor(8).
- SpoolDir
-
NXLog will change its working directory to the value specified with this directive. This is useful with files created through relative filenames (for example, with om_file) and in case of core dumps. This directive has no effect with the nxlog-processor(8).
- StringLimit
-
To protect against memory exhaustion (and possibly a denial-of-service) caused by over-sized strings, there is a limit on the length of each string (in bytes). The default value is 5242880 bytes (strings will be truncated at 5 MiB).
- SuppressRepeatingLogs
-
Under some circumstances it is possible for NXLog to generate an extreme amount of internal logs consisting of the same message due to an incorrect configuration or a software bug. In this case, the LogFile can quickly consume the available disk space. With this directive, NXLog will write at most 2 lines per second if the same message is generated successively, by logging "last message repeated n times" messages. If this boolean directive is not specified, it defaults to
TRUE
(suppression of repeating messages is enabled).
- SyncLogqueue
-
When this directive is set to
TRUE
and PersistLogqueue is enabled, the disk-based queues of processor and output module instances will be immediately synced after each new entry is added to the queue. This greatly reduces performance but makes the queue more reliable and crash-safe. If this boolean directive is not specified, it defaults toFALSE
(immediate syncing is disabled). This directive can also be used at the module level.See Buffering and flow control for a detailed explanation of log queues.
- Threads
-
This directive specifies the number of worker threads to use. The number of worker threads is calculated and set to an optimal value if this directive is not defined. Do not set this unless you know what you are doing.
- UseCNGCertificates
-
You must set this directive to
TRUE
when the certificates you plan to use are nonexportable; this includes software crypto storage and TPM modules.When UseCNGCertificates is
TRUE
, the behavior is the following:-
All types of certificates are loaded through the CNG API.
When UseCNGCertificates is
FALSE
(default), the behavior is the following:-
The certificates are loaded through the older Windows API and OpenSSL functions.
-
Exportable certificates work normally.
-
However, nonexportable certificates fail to load as the CNG API engine is not used.
-
- User
-
NXLog will drop to the user specified with this directive. This is useful if NXLog needs privileged access to some system resources (such as kernel messages or to bind a port below 1024). On Linux systems NXLog will use capabilities to access these resources. In this case NXLog must be started as root. The user can be specified by name or numeric ID. This directive has no effect when running on the Windows platform or with nxlog-processor(8).
Common Module Directives
The following directives are common to all modules. The Module directive is mandatory.
- Module
-
This mandatory directive specifies which binary should be loaded. The module binary has a
.so
extension on Unix and a.dll
on Windows platforms and resides under the ModuleDir directory. Each module binary name is prefixed withim_
,pm_
,om_
, orxm_
(for input, processor, output, and extension, respectively). It is possible for multiple instances to use the same loadable binary. In this case the binary is only loaded once but instantiated multiple times. Different module instances may have different configurations.
- BatchSize
-
This directive specifies how many records will be collected by the module and forwarded together as a batch to the next module in the route. For configuring the batch size for all modules globally, read the global BatchSize directive description.
The use of this directive may produce high memory usage. For more details on the calculations that are used to determine the impact that LogqueueSize
has on memory consumption, refer to BatchSize/LogqueueSize for memory usage.
- BatchFlushInterval
-
This directive can only be used with input and processor modules. It specifies the timeout before a record-batch is forwarded to the next module in the route. See the description of the global BatchFlushInterval directive for more information.
- BufferSize
-
This directive specifies the read/write buffer size for input and output modules in bytes. The default buffer size is 65000 bytes. Only stream-oriented modules use this directive, i.e., those connecting over the network (UDP, TCP, and HTTP) or reading and writing to files. Each module instance can have multiple buffers. For example, an im_file instance collecting logs from multiple files will have a buffer for each open file. Similarly, an im_tcp instance will have a buffer for each open connection. The value set in this directive applies to each buffer and is not cumulative; therefore, it does not limit how much memory a module instance can use.
The buffer size you specify should be large enough to hold the biggest chunk of data you expect an input module to receive or an output module to write. Hence, the appropriate buffer size is use-case dependent, and no general ideal buffer size exists.
It is important to note that the buffer referred to here is a user-space buffer holding a chunk of data the module is processing, not the OS kernel buffer.
Setting a very large buffer size is unlikely to improve performance and will increase the probability of data loss if NXLog terminates unexpectedly.
- FlowControl
-
This optional boolean directive specifies whether the module instance should use flow control. FlowControl is only valid for input, processor, and output modules. For input and processor modules, the FlowControl default is inherited from the global FlowControl directive (which defaults to
TRUE
). To maintain backward compatibility, the FlowControl default for output modules isTRUE
regardless of the global FlowControl value.Under normal conditions, when all module instances are operating normally and buffers are not full, flow control has no effect. However, if a module becomes blocked and is unable to process events, flow control will automatically suspend module instances as necessary to prevent events from being lost. For example, consider a route with im_file and om_tcp. If a network error blocks the output, NXLog will stop reading events from file until the network error is resolved. If flow control is disabled, NXLog will continue reading from file and processing events; the events will be discarded when passed to the output module.
In most cases, flow control should be left enabled, but it may be necessary to disable it in certain scenarios. It is recommended to leave flow control enabled globally and only specify the FlowControl directive in two cases. First, set
FlowControl FALSE
on any input module instance that should never be suspended. Second, if a route contains multiple output instances, it may be desirable to continue sending events to other outputs even if one output becomes blocked—setFlowControl FALSE
on the output(s) where events can be discarded to prevent the route from being blocked.Internally, flow control takes effect when the log queue of the next module instance in the route becomes full. If flow control is enabled, the instance is suspended. If flow control is disabled, events are discarded. If the next module in the route is an output module, both FlowControl values are consulted—flow control is enabled only if both are
TRUE
.Suspending an im_linuxaudit instance could cause an Audit backlog, blocking processes that generate Audit messages. Suspending an im_udp instance is ineffective, because UDP provides no receipt acknowledgement. Suspending an im_uds instance when collecting local syslog messages from the /dev/log Unix domain socket will cause the syslog() system call to block in any programs trying to write to the system log. It is generally recommended to disable flow control in these cases.
- InputType
-
This directive specifies the name of the registered input reader function to be used for parsing raw events from input data. Names are treated case insensitively. This directive is only available for stream oriented input modules: im_file, im_exec, im_ssl, im_tcp, im_udp, im_uds, and im_pipe. These modules work by filling an input buffer with data read from the source. If the read operation was successful and data is available from the source, the module calls the specified callback function. If this is not explicitly specified, the module default will be used. Note that im_udp may only work properly if log messages do not span multiple packets and are within the UDP message size limit. Otherwise the loss of a packet may lead to parsing errors.
Modules may provide custom input reader functions. Once they are registered into the NXLog core, the modules listed above will be able to use these functions. This makes it easier to implement custom protocols because they can be developed without concern for the transport layer.
The following input reader functions are provided by the NXLog core:
- Binary
-
The input is parsed in the NXLog binary format, which preserves the parsed fields of the event records. The LineBased reader will automatically detect event records in the NXLog binary format, so it is only recommended to configure InputType to Binary if compatibility with other logging software is not required. Other than that, Binary is the recommended format for sending logs between NXLog instances.
If you need to use another format, carefully consider its characteristics to make sure it will preserve the data that you need. For example, JSON presents datetime values as string type but it is more compliant to standards and a text-based, human-readable format.
- Dgram
-
Once the buffer is filled with data, it is considered to be one event record. This is the default for the im_udp input module, since UDP Syslog messages arrive in separate packets.
- LineBased
-
The input is assumed to contain event records separated by newlines. It can handle both CRLF (Windows) and LF (Unix) line-breaks. Thus if an LF (
\n
) or CRLF (\r\n
) is found, the function assumes that it has reached the end of the event record. If the input begins with the UTF-8 byte order mark (BOM) sequence (0xEF,0xBB,0xBF
), that sequence is automatically skipped.
Example 7. TCP input assuming NXLog formatThis configuration explicitly specifies the Binary InputType.
nxlog.conf<Input tcp_listen> Module im_tcp ListenAddr 191.168.1.123:1514 InputType Binary </Input>
With the im_file module, this directive also supports one or more data converters to process input data before reading.
Data converters are processed sequentially from left to right, thus the order they are specified in is important. The following example shows the syntax of the InputType directive with two data converters and the input reader function:
InputType <InstanceName>.<DataConverter1>, \ <InstanceName>.<DataConverter2>, \ <InputReaderFunction>
Where
<InstanceName>
is the given name of an extension module instance and<DataConverter>
is the name of the data converter being invoked. Currently, data converters are available in the xm_charconv, xm_crypto and xm_zlib modules. The InputType directive can contain multiple data conversion operations from the same extension module.For more details, see the documentation of the xm_charconv, xm_crypto and xm_zlib modules.
Example 8. Decompression, decryption and conversion of dataThis configuration contains one instance of the xm_zlib module to decompress the input data, one instance of the xm_crypto module to decrypt it and one instance of xm_charconv to convert the input data to UTF-8 encoding. Since the InputType directive of the im_file module instance does not specify an input reader function, the default LineBased will be used.
nxlog.conf<Extension crypto> Module xm_crypto UseSalt TRUE Password secret </Extension> <Extension gzip> Module xm_zlib Format gzip CompressionLevel 9 CompBufSize 16384 DecompBufsize 16384 </Extension> <Extension charset> Module xm_charconv InputEncoding UCS-2LE </Extension> <Input file> Module im_file File '/tmp/input' InputType crypto.aes_decrypt, gzip.decompress, charset.convert </Input>
- LogLevel
-
This directive can be used to override the value of the global LogLevel. This can be useful for debugging purposes when
DEBUG
is set at the module level, or a noisy module can be silenced when set toCRITICAL
orERROR
.Example 9. Using LogLevel at module level<Input fim> Module im_fim LogLevel Debug ... </Input>
- LogqueueDir
-
This directive specifies the directory where the files of the persistent queue are stored. LogqueueDir is only valid for processor and output module instances. See the description of the global LogqueueDir for more information.
- LogqueueSize
-
Maximum size of the log queue in bytes. The log queue is an input log queue for events waiting to be processed by that module instance. The default is 2097152 bytes (2 MiB). The minimum allowed value is 524288 bytes (512 KiB), which NXLog enforces if you enter a smaller value. This directive is only valid for processor and output module instances.
When the log queue of a module instance is full and FlowControl is enabled for the preceding module, the preceding module will be suspended. If flow control is not enabled for the preceding module, events will be discarded. This directive can be used at the global level to affect all processor and output module instances.
The use of this directive may produce high memory usage. Also, note that the actual queue size depends on the number of event records (and their size) in the module queue. The limit, which is set by this directive, is not a hard one as the sizes of the events are based on approximations and the queue can store an extra batch of records even after reaching the size limit. See Buffering and flow control for a detailed explanation of log queues.
- OutputType
-
This directive specifies the name of the registered output writer function to be used for formatting raw events when storing or forwarding output. Names are treated case insensitively. This directive is only available for stream oriented output modules: om_exec, om_file, om_pipe, om_redis, om_ssl, om_tcp, om_udp, om_udpspoof, om_uds, and om_zmq. These modules work by filling the output buffer with data to be written to the destination. The specified callback function is called before the write operation. If this is not explicitly specified, the module default will be used.
Modules may provide custom output formatter functions. Once they are registered into the NXLog core, the modules listed above will be able to use these functions. This makes it easier to implement custom protocols because they can be developed without concern for the transport layer.
The following output writer functions are provided by the NXLog core:
- Binary
-
The output is written in the NXLog binary format which preserves parsed fields of the event records. This format is the recommended format when sending data between NXLog instances.
If you need to use another format, carefully consider its characteristics to make sure it will preserve the data that you need. For example, JSON presents datetime values as string type but it is more compliant to standards and a text-based, human-readable format.
- Dgram
-
Once the buffer is filled with data, it is considered to be one event record. This is the default for the om_udp, om_udpspoof, om_redis, and om_zmq output modules.
- LineBased
-
The output will contain event records separated by newlines. The record terminator is CRLF (
\r\n
) on Windows and LF (\n
) on Unix.
- LineBased_CRLF
-
The output will contain event records separated by Windows style newlines where the record terminator is CRLF (
\r\n
).
- LineBased_LF
-
The output will contain event records separated by Unix style newlines where the record terminator is LF (
\n
).
Example 10. TCP output sending messages in NXLog formatThis configuration explicitly specifies the Binary OutputType.
nxlog.conf<Output tcp> Module om_tcp Host 192.168.1.123:1514 OutputType Binary </Output>
With the om_file module, this directive also supports one or more data converters to process output data after writing.
Data converters are processed sequentially from left to right, thus the order they are specified in is important. The following example shows the syntax of the OutputType directive with two data converters and the output writer function:
OutputType <OutputWriterFunction>, \ <InstanceName>.<DataConverter1>, \ <InstanceName>.<DataConverter2>
Where
<InstanceName>
is the given name of an extension module instance and<DataConverter>
is the name of the data converter being invoked. Currently, data converters are available in the xm_charconv, xm_crypto and xm_zlib modules. The OutputType directive can contain multiple data conversion operations from the same extension module.Rotation of files is done automatically when encrypting log data with the xm_crypto module. For more details, see the documentation of the xm_charconv, xm_crypto and xm_zlib modules.
Example 11. Conversion, compression and encryption of dataThis configuration contains one instance of the xm_charconv module to convert the output data to UTF-32LE, one instance of xm_zlib module to compress the output file and one instance of the xm_crypto module to encrypt it. Since the OutputType directive of the om_file module instance does not specify an output writer function, the default LineBased will be used.
nxlog.conf<Extension charset> Module xm_charconv OutputEncoding UTF-32LE </Extension> <Extension gzip> Module xm_zlib Format gzip CompressionLevel 9 CompBufSize 16384 DecompBufsize 16384 </Extension> <Extension crypto> Module xm_crypto UseSalt TRUE Password secret </Extension> <Input tcp_listen> Module im_tcp ListenAddr 0.0.0.0:1514 </Input> <Output file> Module om_file File '/tmp/output' OutputType charset.convert, gzip.compress, crypto.aes_encrypt </Output>
- PersistLogqueue
-
When a module passes an event to the next module along the route, it puts it into the next module’s queue. This queue can be either a memory-based or disk-based (persistent) queue. When this directive is set to
TRUE
, the module will use a persistent (disk-based) queue. With the default value ofFALSE
, the module’s incoming log queue will be memory-based; however, in-memory log queues will still be persisted to disk on shutdown. This directive is only valid for processor and output module instances. This directive can also be used at the global level.See Buffering and flow control for a detailed explanation of log queues.
- SyncLogqueue
-
When this directive is set to
TRUE
and PersistLogqueue is enabled, the disk-based queue will be immediately synced after each new entry is added to the queue. This greatly reduces performance but makes the queue more reliable and crash-safe. If this boolean directive is not specified, it defaults toFALSE
(immediate syncing is disabled). This directive is only valid for processor and output module instances. This directive can be used at the global level to affect all modules.See Buffering and flow control for a detailed explanation of log queues.
- UseCngCertificates
-
This directive is available for modules that support TLS/SSL on the Windows platform. When set to TRUE, the module uses the Windows Cryptography API: Next Generation (CNG) to access the private keys associated with certificates identified by a thumbprint. This functionality is only available on Windows platforms.
Exec
The Exec directive/block contains statements in the NXLog language which are executed when a module receives a log message. This directive is available in all input, processor, and output modules. It is not available in most extension modules because these do not handle log messages directly (the xm_multiline and xm_rewrite modules do provide Exec directives).
This statement assigns a value to the $Hostname
field in the event
record.
Exec $Hostname = 'myhost';
Each directive must be on one line unless it contains a trailing
backslash (\
) character.
This if statement uses line continuation to span multiple lines.
Exec if $Message =~ /something interesting/ \
log_info("found something interesting"); \
else \
log_debug("found nothing interesting");
More than one Exec directive or block may be specified. They are executed in the order of appearance. Each Exec directive must contain a full statement. Therefore it is not possible to split the lines in the previous example into multiple Exec directives. It is only possible to split the Exec directive if it contains multiple statements.
This example shows two equivalent uses of the Exec directive.
Exec log_info("first"); \
log_info("second");
This produces identical behavior:
Exec log_info("first");
Exec log_info("second");
The Exec directive can also be used as a block. To use multiple
statements spanning more than one line, it is recommended to use the
<Exec>
block instead. When using a block, it is not necessary to use
the backslash (\
) character for line continuation.
This example shows two equivalent uses of Exec, first as a directive, then as a block.
Exec log_info("first"); \
log_info("second");
The following Exec block is equivalent. Notice the backslash (\
)
is omitted.
<Exec>
log_info("first");
log_info("second");
</Exec>
Schedule
The Schedule block can be used to execute periodic jobs, such as log rotation or any other task. Scheduled jobs have the same priority as the module. The Schedule block has the following directives:
- Every
-
In addition to the crontab format it is possible to schedule execution at periodic intervals. With the crontab format it is not possible to run a job every five days for example, but this directive enables it in a simple way. It takes a positive integer value with an optional unit. The unit can be one of the following:
sec
,min
,hour
,day
, orweek
. If the unit is not specified, the value is assumed to be in seconds. The Every directive cannot be used in combination withRunCount 1
.
- Exec
-
The mandatory Exec directive takes one or more NXLog statements. This is the code which is actually being scheduled. Multiple Exec directives can be specified within one Schedule block. See the module-level Exec directive, this behaves the same. Note that it is not possible to use fields in statements here because execution is not triggered by log messages.
- First
-
This directive sets the first execution time. If the value is in the past, the next execution time is calculated as if NXLog has been running since and jobs will not be run to make up for missed events in the past. The directive takes a datetime literal value.
- RunCount
-
This optional directive can be used to specify a maximum number of times that the corresponding Exec statement(s) should be executed. For example, with
RunCount 1
the statement(s) will only be executed once.
- When
-
This directive takes a value similar to a crontab entry: five space-separated definitions for minute, hour, day, month, and weekday. See the crontab(5) manual for the field definitions. It supports lists as comma separated values and/or ranges. Step values are also supported with the slash. Month and week days are not supported, these must be defined with numeric values. The following extensions are also supported:
@startup Run once when NXLog starts. @reboot (Same as @startup) @yearly Run once a year, "0 0 1 1 *". @annually (Same as @yearly) @monthly Run once a month, "0 0 1 * *". @weekly Run once a week, "0 0 * * 0". @daily Run once a day, "0 0 * * *". @midnight (Same as @daily) @hourly Run once an hour, "0 * * * *".
This example shows two scheduled Exec statements in a im_tcp module instance. The first is executed every second, while the second uses a crontab(5) style value.
<Input tcp_listen>
Module im_tcp
ListenAddr 0.0.0.0:1514
<Schedule>
Every 1 sec
First 2024-02-27 00:19:06
Exec log_info("scheduled execution at " + now());
</Schedule>
<Schedule>
When 1 */2 2-4 * *
Exec log_info("scheduled execution at " + now());
</Schedule>
</Input>
Route Directives
The following directives can be used in Route blocks. The Path directive is mandatory.
- Path
-
The data flow is defined by the Path directive. First, the instance names of input modules are specified. If the route contains more than one input source, then these must be separated by a comma. The list of input modules is followed by an arrow (
=>
). Either processor modules or output modules follow. Processor modules must be separated by an arrow, not a comma, because they operate in series, unlike input and output modules, which work in parallel. Output modules are separated by a comma. The Path must specify at least one input and one output. The syntax is as follows:Path INPUT1[, INPUT2...] => [PROCESSOR1 [=> PROCESSOR2...] =>] OUTPUT1[, OUTPUT2...]
Example 17. Specifying routesThe following configuration shows the log router with modules being used in two routes.
nxlog.conf<Input in1> Module im_null </Input> <Input in2> Module im_null </Input> <Processor p1> Module pm_null </Processor> <Processor p2> Module pm_null </Processor> <Output out1> Module om_null </Output> <Output out2> Module om_null </Output> <Route 1> # Basic route Path in1 => out1 </Route> <Route 2> # Complex route with multiple input, output, and processor modules Path in1, in2 => p1 => p2 => out1, out2 </Route>
- Priority
-
This directive takes an integer value in the range of 1-100 as a parameter, and the default is
10
. Log messages in routes with a lower Priority value will be processed before others. Internally, this value is assigned to each module part of the route. The events of the modules are processed in priority order by the NXLog engine. Modules of a route with a lower Priority value (higher priority) will process log messages first.Example 18. Prioritized processingThis configuration prioritizes the UDP route over the TCP route in order to minimize the possibility of losing logs sent over UDP when the system is busy.
nxlog.conf<Input tcp_listen> Module im_tcp ListenAddr 0.0.0.0:1514 </Input> <Input udp_listen> Module im_udp ListenAddr 0.0.0.0:514 </Input> <Output tcp_output> Module om_file File '/var/log/tcp.log' </Output> <Output udp_output> Module om_file File '/var/log/udp.log' </Output> <Route udp> Priority 1 Path udp_listen => udp_output </Route> <Route tcp> Priority 2 Path tcp_listen => tcp_output </Route>