Modules and routes
While the NXLog core is responsible for managing the overall log processing operation, NXLog’s architecture utilizes loadable modules that provide input, output, and parsing functionality. Different modules can be used together to create log data routes that meet the requirements of the logging environment. Each route accepts log messages in a particular format, processes or transforms them, and then outputs them in one of the supported formats.
Files and sockets are added to the core by the various modules, and the core delegates events when necessary. Modules also dispatch log events to the core, which passes each one to the appropriate module. In this way, the core can centrally control all events and the order of their execution making prioritized processing possible. Each event belonging to the same module instance is executed in sequential order, not concurrently. This ensures that message order is kept and allows modules to be written without concern for concurrency. Yet because the modules and routes run concurrently, the global log processing flow remains parallelized.
A module is a
foo.dll that can be loaded by the
NXLog core and provides a particular capability.
A module instance is a configured module that can be used in the configured
flow. For example, the configuration block for an input module
instance begins with
<Input instancename>. See the Instance
examples below. A single module can be used in multiple
instances. With regard to configuration, a module instance is often
referred to as simply a module.
There are four types of modules.
Functionality for accepting or retrieving log data is provided by input modules. An input module instance is a source or producer. It accepts log data from a source and produces event records.An input module instance
<Input foo_in> Module im_foo ... </Input>
Output modules provide functionality for sending log data to a local or remote destination. An output module instance is a sink, destination, or consumer. It is responsible for consuming event records produced by one or more input module instances.An output module instance
<Output foo_out> Module om_foo ... </Output>
The NXLog language can be extended with extension modules. Extension module instances do not process log data directly. Instead, they provide features (usually functions and procedures) that can be used from other parts of the processing pipeline. Many extension module instances require no directives other than the Module directive.Example 1. Using an extension module
In this example, the xm_syslog module is loaded by the
Extensionblock. This module provides the parse_syslog() procedure, in addition to other functions and procedures. In the following
Inputinstance, the Exec directive calls
parse_syslog()to parse the Syslog-formatted event.nxlog.conf
<Extension _syslog> Module xm_syslog </Extension> <Input in> Module im_file File '/var/log/messages' Exec parse_syslog(); </Input>
Processor modules offer features for transforming, filtering, or converting log messages. One or more processor module instances can be used in a route between input and output module instances.A processor module instance
<Processor foo> Module pm_foo ... </Processor>
Many processing functions and procedures are available through the NXLog language and can be accessed through the Exec directive in an Input or Output block without using a separate processor module instance. However, a separate processor module (pm_null, perhaps) will use a separate worker thread, providing additional processing parallelization.
For a list of available modules, see Available modules.
Most log processing solutions are built around the same concept. The input is read from a source, log messages are processed, and then log data is written to a destination. In NXLog, this path is called a "route" and is configured with a Route block.
Routes are made up of one or more inputs, zero or more processors, and one or more outputs.
This route accepts input with the
in module and sends it to the
out module. This is the simplest functional route.
<Route r1> Path in => out </Route>
This route extends the previous example by adding an intermediate
<Route r2> Path in => proc => out </Route>
This route uses two input modules and two output modules. Input from
in2 will be combined and sent to both
<Route r3> Path in1, in2 => out1, out2 </Route>
A module can be used by multiple routes simultaneously, as in this
in1 module instance is only declared once, but is used
by both routes.
<Route r1> Path in => out1 </Route> <Route r2> Path in => proc => out2 </Route>