Centralized telemetry data collection

Centralized telemetry data collection, aggregation, or centralization means sending telemetry data from different sources to a dedicated server or service for storage, search facility, and analytics.

The following diagram depicts a centralized telemetry data collection architecture.

Figure 1. Centralized telemetry data collection

There are several benefits to storing telemetry data on a centralized system, including:

You can access the data if the originating server is offline, compromised, or decommissioned.
You can analyze and correlate events from multiple systems.
Backing up logs on another server makes it more difficult for malicious actors to remove evidence of their activity.
Investigating incidents and auditing is more straightforward when all the data is in a single location.
Implementing scalable, high-availability, and redundancy solutions on a single, centralized server is easier.
You will only need to manage compliance with log storage and retention standards from one point.

NXLog Platform supports collecting data from a maximum of 500 NXLog Agent instances. If your scenario requires collecting data from more agents, use NXLog Agent relays to aggregate the data before sending it to NXLog Platform.

Using NXLog Agent as a relay

Depending on your network topology and the location of your centralized data server, you may need to introduce one or more servers to act as a relay.

The purpose of a relay is to gather telemetry data from multiple sources and forward it to the final destination, such as NXLog Platform. Optionally, you can configure a relay to perform additional data processing, such as filtering and data conversion. Relays are especially useful when you’re transferring data between different networks and want to restrict access between the networks. See Configure an NXLog Agent relay for configuration examples.

Figure 2. Using NXLog Agent as a relay

Considerations

Your telemetry data collection strategy depends on your requirements. When deciding how to set up centralized data collection, consider the following:

Reliability: If data loss is unacceptable, avoid transferring data over UDP since it does not guarantee message delivery. Use TCP instead. In addition, NXLog Agent provides application-level, guaranteed delivery. See Protect against data loss for more information.
Structured data: Correlating events from multiple sources requires parsing the data into a standard schema. You can configure NXLog Agent to parse events at any point in the data processing path. We recommend normalizing events at the source whenever possible to reduce the load on relays and the centralized data server. See NXLog Agent records and fields for more information.
Encryption: Use TLS when transferring data across the network to maintain the confidentiality and integrity of your data. See TLS/SSL data transfer for more information.
Compression: If bandwidth is a concern, consider compressing data. Most log data is highly compressible, significantly reducing bandwidth demands. The im_batchcompress and om_batchcompress modules efficiently transport compressed data between NXLog Agent instances.