Log processing modes

NXLog can process logs in three modes. Each mode has different characteristics, and you can use any combination of modes for your overall logging infrastructure.

Agent-based collection: NXLog runs on the system that is generating the log data.
Agentless collection: Hosts or devices generate log data and send it over the network to NXLog.
Offline log processing: The nxlog-processor(8) tool performs batch log processing.

Agent-based collection

With agent-based collection, NXLog runs as an agent on the system that is generating the log data. It collects, processes, and outputs the log data based on its configuration.

We recommend agent-based log collection for most use cases. In particular, we recommend this mode if you need strong security and reliability or need to transform log data before it leaves the system on which it was generated.

Agent-based log collection offers several important advantages over agentless collection.

Log data can be collected from more sources. For example, you can collect logs directly from files, instead of relying on a logging process to send log data across the network.
NXLog has a complete list of processing features available. You can filter, normalize, and rewrite log data before sending it to a destination, whether an NXLog instance or a log aggregation system. This includes the ability to send structured log data, such as JSON and key-value pairs.
You have full control over the transfer of the log data. Messages can be sent using a variety of protocols, including over TLS/SSL encrypted connections for security. Log data can be sent in compressed batches and can be buffered if necessary.
Log collection in this mode is more reliable. NXLog includes delivery guarantees and flow control systems that ensure your log data reaches its destination. You can monitor the health of the NXLog agent to verify its operational integrity.

Although agent-based collection has many compelling advantages, there are use cases where using agents is not feasible.

Many network-based and embedded systems, such as routers and firewalls, do not support installing third-party software. In this case, it is impossible to install the NXLog agent.
Compliance or regulatory mandates may prohibit you from installing third-party software on certain systems.

Agentless collection

With this mode of log collection, a server or device sends log data to an NXLog instance over the network, using its native protocols. NXLog collects and processes the information that it receives.

We only recommend agentless log collection in cases where agent-based log collection is not feasible, for example from legacy or embedded systems that do not support installing the NXLog agent.

Agentless log collection has the following advantages.

It is not necessary to install an NXLog agent application on the target system to collect log data from it.
Generally, a device or system requires only minimal configuration to send log data over the network to an NXLog instance in its native format.

Agentless log collection has some disadvantages that should be taken into consideration.

Agentless log collection may provide lower performance than agent-based collection. On Windows systems, the Windows Management Instrumentation process can consume more system resources than the NXLog agent.
Reliability is also a potential issue. Since most Syslog log forwarders use UDP to transfer log data, some data could be lost if the server restarts or becomes unreachable over the network. Unlike agent-based log collection, you often cannot monitor the health of the logging source.
Data transfers are less secure when using agentless collection since most Syslog sources transfer data over unencrypted UDP.

Agentless collection is commonly used with the following protocols.

BSD Syslog (RFC 3164) and IETF Syslog (RFC 5424) sources (see Collecting and Parsing Syslog)
Windows Event Log sources (with NXLog Enterprise Edition):
- The MSRPC protocol, using the im_msvistalog module (see Remote collection with im_msvistalog)
- Windows Event Forwarding, using the im_wseventing module (see Remote collection with im_wseventing)

Offline log processing

While the other modes process log data in real-time, NXLog can also be used to perform batch log processing. This is provided by the nxlog-processor(8) tool, which is similar to the NXLog logging daemon and uses the same configuration file. However, it runs in the foreground and exits after all input log data has been processed.

Common input sources are files and databases. This tool is useful for many log processing tasks.

Loading a group of files into a database.
Converting between different formats.
Testing patterns.
Correlating offline events.
Checking HMAC message integrity.