NXLog Agent log collection modes

NXLog Agent can process logs in three modes. Each mode has different characteristics, and you can use any combination of modes within your logging infrastructure.

Agent-based log collection: NXLog Agent runs on the system that generates the log data.
Agentless log collection: Applications or devices send their logs to NXLog Agent over the network.
Offline log processing: Use the nxlog-processor(8) tool to process logs manually or via a script.

Agent-based log collection

In agent-based log collection, you install NXLog Agent on the log source and configure it to collect, process, and forward the logs.

Agent-based log collection is especially suitable if you need to transform logs before forwarding them to their destination and for secure and reliable log collection and transfer. We recommend this mode for most use cases.

Figure 1. Agent-based log collection

Agent-based log collection offers significant advantages over agentless collection, some of which are:

You have more log collection options. Log sources often provide multiple output methods and formats you can choose from according to your requirements. For example, you can collect logs from a file rather than being tied to using an unreliable logging process to send logs over the network.
You can filter, normalize, and enrich log records before forwarding them to their destination. NXLog Agent has a comprehensive list of log processing capabilities, including transforming logs to a different format, such as JSON, XML, or CSV.
You have complete control over how you transfer logs. NXLog Agent supports several network protocols, including TLS/SSL over TCP and HTTPS for secure data transfer. You can also compress logs and implement buffering if necessary.
Implement reliable and secure log collection. NXLog Platform includes delivery guarantees and flow control systems to ensure your logs reach their destination. You can also monitor the health of NXLog Agent instances from NXLog Platform to maintain operational integrity.

Although agent-based log collection has its benefits, there are instances where installing an agent on the log source is not possible, including:

Many network devices and embedded systems, such as routers and firewalls, do not support installing third-party software.
Compliance or regulatory mandates may prohibit you from installing third-party software on certain systems.

Agentless log collection

In agentless log collection, you configure a central NXLog Agent instance to receive and process logs from remote sources. You then configure applications or devices to send log data to this NXLog Agent instance over the network.

We only recommend agentless log collection for log sources where you cannot install third-party software, such as network devices and legacy or embedded systems.

Figure 2. Agentless log collection

Agentless log collection can be advantageous because you do not need to install additional software on the log source, and applications and devices that support log forwarding over the network generally only require minimal configuration.

However, it also has some disadvantages that are worth considering, including:

Agentless log collection may be slower than agent-based collection. For example, on Windows, the Windows Management Instrumentation (WMI) process used to forward logs can consume a considerable amount of system resources compared to the NXLog Agent.
Logs may not be transferred reliably and securely. For example, most syslog forwarders use UDP to transfer logs over the network, which is neither reliable nor secure. In addition, it is unlikely that you’ll be able to monitor the health of the log-forwarding process, resulting in potential data loss if the process or communication breaks down.

Offline log processing

While the other two modes process logs in real-time, you can process logs offline with the nxlog-processor(8) tool. The tool is similar to the main NXLog Agent service and uses the same configuration system but runs in the foreground and exits once it processes all the logs.

There are several reasons why you may need to process logs offline, such as:

Transferring logs from files to a database.
Converting logs to a different format.
Testing patterns.
Correlating events.
Checking HMAC message integrity.