NXLog Agent modules and routes

NXLog Agent provides loadable modules for configuring data collection, processing, and forwarding. You then use routes to define the data processing sequence according to your requirements. Each route must contain at least one input and one output module instance. Optionally, you can add processor module instances to the route.

Each input module instance processes records in the order it reads them. The NXLog Agent core delegates records between the modules according to the routes you configure, making it possible to prioritize records. However, modules and routes run synchronously to maintain a parallel global data processing flow.

Modules

A module is a .so or .dll file providing specialized functionality. NXLog Agent only loads these modules if your configuration uses them. You can add several instances of the same module in your configuration, e.g., using multiple im_file instances to collect distinct file-based logs or multiple xm_xml extension instances to parse different XML structures. We often refer to module instances simply as modules when describing configuration.

Input modules

Input modules collect or receive telemetry data. Consider input module instances as a source or producer. They collect data from a source and produce records.

<Input foo_in>
    Module    im_foo
</Input>

Output modules

Output modules forward or write telemetry data to a local or remote destination. Consider output module instances as sinks, destinations, or consumers. They are responsible for consuming records produced by the input module instances.

<Output foo_out>
    Module    om_foo
</Output>

Extensions

Extension module instances provide functionality for processing data, usually in the form of functions and procedures you can call from your input and output module instances. Many extension modules do not require any further configuration.

For example, the configuration below loads the xm_syslog module, which provides the parse_syslog() procedure to parse syslog messages.

<Extension syslog>
    Module    xm_syslog
</Extension>

<Input file>
    Module    im_file
    File      '/var/log/messages'
    Exec      parse_syslog();
</Input>

Processor modules

Processor modules offer features for transforming, filtering, or converting data. You can specify one or more processor module instances in a route between the input and output module instances.

<Processor foo>
    Module    pm_foo
</Processor>

The NXLog language provides many functions and procedures you can invoke from any input or output module instance with the Exec directive. However, a processor module instance uses a separate worker thread for process parallelization, e.g., using a pm_null module instance to distribute the processing load.

Refer to the NXLog Agent Reference Manual for more information on specific modules.

Routes

NXLog Agent collects telemetry data from a source, processes it, and forwards it to its destination. In NXLog Agent configuration terms, this sequence is called a Route. The path in a route block must include one or more inputs, optional processors, and at least one output.

Simple routes

The most basic functional route consists of one input module instance sending data to a single output module instance.

Figure 1. A simple route

For example, the following configuration routes logs from the in input instance to the out output instance.

<Route r1>
    Path    in => out
</Route>

Routes with processors

Routes can have one or more intermediate processors between the input and output module instances.

Figure 2. A route with a processor

For example, the following configuration collects data with the in input instance, processes it with the proc processor module instance, and forwards it with the out output instance.

<Route r2>
    Path    in => proc => out
</Route>

Advanced routes

A single route can have multiple input and output instances. Such a configuration is useful when you need to store the same data in different locations, e.g., a SIEM and on disk for long-term archival.

An NXLog Agent route with multiple inputs and outputs

Figure 3. A route with multiple inputs and outputs

For example, the following configuration collects data from two sources and forwards all records to two separate destinations.

<Route r3>
    Path    in1, in2 => out1, out2
</Route>

Route branching

The same module instance can be used in multiple routes simultaneously. Route branching can be useful when you need to store the same data in various locations but want to filter or format it differently according to the destination.

An NXLog Agent route with two inputs and one output

Figure 4. Two routes with the same input

For example, the following configuration defines the in input instance in two routes. The first route forwards the raw data to the destination, while the second processes the records with a processor module instance before forwarding it.

<Route r1>
    Path    in => out1
</Route>

<Route r2>
    Path    in => proc => out2
</Route>