NXLog Agent log processing overview

NXLog Agent uses a modular, file-based configuration system that allows you to tailor your configuration according to your requirements. It is worth taking a few minutes to understand how NXLog Agent processes logs. Once you have a good grasp of the concept, you can easily create configurations to collect and process logs from diverse sources.

Architecture

NXLog Agent processes logs via input, processor, and output modules. In addition, extension modules provide support for processing specific log formats.

With this modular architecture, you can choose which modules to load and separate the input from the output, so you can route a single input to multiple outputs and multiple inputs to a single output.

Figure 1. Processing logs with NXLog Agent

Input

Collect events from your log sources. Inputs can include local or remote file-based logs, Windows events, syslog messages received over TCP or UDP, or any other source supported by NXLog Agent input modules.

NXLog Agent treats each event as a log record consisting of fields. In most cases, it saves the collected log message in the $raw_event field and creates the following core fields:

$EventReceivedTime
$Hostname
$SourceModuleName
$SourceModuleType

Depending on the input module, other fields may be created automatically.

Parse

Parse log messages into structured data to enable further processing of the record. Various extension modules support parsing standard log formats or provide functions to extend log processing capabilities through external scripts. Additionally, the NXLog language supports regular expressions to parse custom log formats. Some input modules automatically parse log messages into fields, removing the need for further parsing.

Process / Normalize

Once a log message is parsed and additional fields are created, you can decide what to do with the log record. For example, you can drop records based on the event type or severity, truncate data, or enrich the record with additional data. You can also normalize data according to your SIEM requirements at this stage. See Normalize logs.

Format

Convert the log record to the required output format. Most output modules use the value of $raw_event, so you must ensure that the data in this field is in the required format. Extension modules support converting records to standard formats, such as JSON, CSV, and XML.

Output

Forward the data to the destination, such as a centralized file repository, database, SIEM, log analytics solution, or any destination supported by NXLog Agent output modules.

The documentation of individual modules provides details on configuration settings, fields created by default parsing, and functions and procedures made available by the module.