NXLog Agent modules and routes

NXLog Agent provides loadable modules for configuring log collection, processing, and forwarding. You then use routes to define the log processing sequence according to your requirements. Each route must contain at least one input and one output module instance. Optionally, you can add processor module instances to the route.

Each input module instance processes event records in the order it reads them. The NXLog Agent core delegates events between the modules according to the routes you configure, making it possible to prioritize logs. However, modules and routes run synchronously to maintain a parallel global log processing flow.

Modules

A module is a foo.so or foo.dll providing specialized functionality. NXLog Agent only loads these modules if your configuration uses them. You can add several instances of the same module in your configuration, e.g., using multiple im_file instances to collect distinct file-based logs or multiple xm_xml extension instances to parse different XML structures. We often refer to module instances simply as modules when describing configuration.

Input modules

Input modules collect or receive log data. Consider input module instances as a source or producer. They collect logs from a source and produce log records.

<Input foo_in>
    Module    im_foo
</Input>

Output modules

Output modules forward or write log data to a local or remote destination. Consider output module instances as sinks, destinations, or consumers. They are responsible for consuming log records produced by the input module instances.

<Output foo_out>
    Module    om_foo
</Output>

Extensions

Extension module instances provide functionality for processing logs, usually in the form of functions and procedures you can call from your input and output module instances. Many extension modules do not require any further configuration.

For example, the configuration below loads the xm_syslog module, which provides the parse_syslog() procedure to parse syslog messages.

<Extension syslog>
    Module    xm_syslog
</Extension>

<Input file>
    Module    im_file
    File      '/var/log/messages'
    Exec      parse_syslog();
</Input>

Processor modules

Processor modules offer features for transforming, filtering, or converting logs. You can specify one or more processor module instances in a route between the input and output module instances.

<Processor foo>
    Module    pm_foo
</Processor>

The NXLog language provides many functions and procedures you can invoke from any input or output module instance with the Exec directive. However, a processor module instance uses a separate worker thread for process parallelization, e.g., using a pm_null module instance to distribute the log processing load.

Refer to the NXLog Agent Reference Manual for more information on specific modules.

Routes

Like most log collectors, NXLog Agent collects log events from a source, processes them, and forwards them to their destination. In NXLog Agent configuration terms, this sequence is called a Route. The path in a route block must include one or more inputs, optional processors, and at least one output.

Simple routes

The most basic functional route consists of one input module instance sending logs to a single output module instance.

Figure 1. A simple route

For example, the following configuration routes logs from the in input instance to the out output instance.

<Route r1>
    Path    in => out
</Route>

Routes with processors

Routes can have one or more intermediate processors between the input and output module instances.

Figure 2. A route with a processor

For example, the following configuration collects log events with the in input instance, processes them with the proc processor module instance, and forwards them with the out output instance.

<Route r2>
    Path    in => proc => out
</Route>

Advanced routes

A single route can have multiple input and output instances. Such a configuration is useful when you need to store the same logs in different locations, e.g., a SIEM and on disk for long-term archival.

Figure 3. A route with multiple inputs and outputs

For example, the following configuration collects log events from two sources and forwards all logs to two separate destinations.

<Route r3>
    Path    in1, in2 => out1, out2
</Route>

Route branching

The same module instance can be used in multiple routes simultaneously. Route branching can be useful when you need to store the same logs in various locations but want to filter or format logs differently according to the destination.

Figure 4. Two routes with the same input

For example, the following configuration defines the in input instance in two routes. The first route forwards the raw logs to their destination, while the second processes the log records with a processor module instance before forwarding them.

<Route r1>
    Path    in => out1
</Route>

<Route r2>
    Path    in => proc => out2
</Route>