Modules and routes

While the NXLog core is responsible for managing the overall log processing operation, NXLog’s architecture utilizes loadable modules that provide input, output, and parsing functionality. Different modules can be used together to create log data routes that meet the requirements of the logging environment. Each route accepts log messages in a particular format, processes or transforms them, and then outputs them in one of the supported formats.

Files and sockets are added to the core by the various modules, and the core delegates events when necessary. Modules also dispatch log events to the core, which passes each one to the appropriate module. In this way, the core can centrally control all events and the order of their execution making prioritized processing possible. Each event belonging to the same module instance is executed in sequential order, not concurrently. This ensures that message order is kept and allows modules to be written without concern for concurrency. Yet because the modules and routes run concurrently, the global log processing flow remains parallelized.

Modules

A module is a foo.so or foo.dll that can be loaded by the NXLog core and provides a particular capability. A module instance is a configured module that can be used in the configured data flow. For example, the configuration block for an input module instance begins with <Input instancename>. See the Instance examples below. A single module can be used in multiple instances. With regard to configuration, a module instance is often referred to as simply a module.

There are four types of modules.

Input

Functionality for accepting or retrieving log data is provided by input modules. An input module instance is a source or producer. It accepts log data from a source and produces event records.

An input module instance

<Input foo_in>
    Module im_foo
    ...
</Input>

Output

Output modules provide functionality for sending log data to a local or remote destination. An output module instance is a sink, destination, or consumer. It is responsible for consuming event records produced by one or more input module instances.

An output module instance

<Output foo_out>
    Module om_foo
    ...
</Output>

Extension

The NXLog language can be extended with extension modules. Extension module instances do not process log data directly. Instead, they provide features (usually functions and procedures) that can be used from other parts of the processing pipeline. Many extension module instances require no directives other than the Module directive.

Example 1. Using an extension module

In this example, the xm_syslog module is loaded by the Extension block. This module provides the parse_syslog() procedure, in addition to other functions and procedures. In the following Input instance, the Exec directive calls parse_syslog() to parse the Syslog-formatted event.

nxlog.conf

<Extension _syslog>
    Module  xm_syslog
</Extension>

<Input in>
    Module  im_file
    File    '/var/log/messages'
    Exec    parse_syslog();
</Input>

Processor

Processor modules offer features for transforming, filtering, or converting log messages. One or more processor module instances can be used in a route between input and output module instances.

A processor module instance

<Processor foo>
    Module pm_foo
    ...
</Processor>

Many processing functions and procedures are available through the NXLog language and can be accessed through the Exec directive in an Input or Output block without using a separate processor module instance. However, a separate processor module (pm_null, perhaps) will use a separate worker thread, providing additional processing parallelization.

For a list of available modules, see Available modules.

Routes

Most log processing solutions are built around the same concept. The input is read from a source, log messages are processed, and then log data is written to a destination. In NXLog, this path is called a "route" and is configured with a Route block.

Routes are made up of one or more inputs, zero or more processors, and one or more outputs.

Example 2. A simple route

This route accepts input with the in module and sends it to the out module. This is the simplest functional route.

nxlog.conf

<Route r1>
    Path    in => out
</Route>

Example 3. A route with a processor

This route extends the previous example by adding an intermediate processing module proc.

nxlog.conf

<Route r2>
    Path    in => proc => out
</Route>

Example 4. Advanced route with multiple input/output modules

This route uses two input modules and two output modules. Input from in1 and in2 will be combined and sent to both out1 and out2.

nxlog.conf

<Route r3>
    Path    in1, in2 => out1, out2
</Route>

Example 5. Branching: two routes using one input module

A module can be used by multiple routes simultaneously, as in this example. The in1 module instance is only declared once, but is used by both routes.

nxlog.conf

<Route r1>
    Path    in => out1
</Route>

<Route r2>
    Path    in => proc => out2
</Route>