Centralized log collection

Centralized log collection, log aggregation, or log centralization means sending logs from different sources to a dedicated server or service for storage, search facility, and analytics.

The following diagram depicts a centralized log collection architecture.

Figure 1. Centralized log collection

There are several benefits to storing logs on a centralized system, including:

You can access the logs if the originating server is offline, compromised, or decommissioned.
You can analyze and correlate logs from multiple systems.
Backing up logs on another server makes it more difficult for malicious actors to remove evidence of their activity.
Investigating incidents and auditing is more straightforward when all event data is in a single location.
Implementing scalable, high-availability, and redundancy solutions on a single, centralized server is easier.
You will only need to manage compliance with log storage and retention standards from one point.

NXLog Platform supports collecting logs from a maximum of 1,000 NXLog Agent instances. If your scenario requires collecting logs from more agents, use NXLog Agent relays to aggregate logs before sending them to NXLog Platform.

Using NXLog Agent as a relay

Depending on your network topology and the location of your centralized log server, you may need to introduce one or more servers to act as a relay.

The purpose of a relay is to gather logs from multiple sources and forward them to the final destination, such as NXLog Platform. Optionally, you can configure a relay to perform additional log processing, such as filtering and data conversion. Relays are especially useful when you’re transferring logs between different networks and want to restrict access between the networks. See Configure an NXLog Agent relay for for configuration examples.

Figure 2. Using NXLog Agent as a relay

Considerations

Your log collection strategy depends on your logging requirements. When deciding how to set up centralized log collection, consider the following:

Reliability: If data loss is unacceptable, avoid transferring logs over UDP since it does not guarantee message delivery. Use TCP instead. In addition, NXLog Agent provides application-level, guaranteed delivery.
Structured data: Correlating data from multiple log sources requires parsing event data into a standard schema. You can configure NXLog Agent to parse log events at any point in the log processing path. We recommend normalizing logs at the source whenever possible to reduce the load on relays and the centralized log server. See NXLog Agent log records and fields for more information.
Encryption: Use TLS when transferring logs across the network to maintain the confidentiality and integrity of your logs. See TLS/SSL log transfer for more information.
Compression: If bandwidth is a concern, consider compressing logs. Most log data is highly compressible, significantly reducing bandwidth demands. The im_batchcompress and om_batchcompress modules efficiently transport compressed logs between NXLog Agent instances.