Protect against data loss

Regulatory compliance usually requires that the telemetry data pipeline functions in an ultra-reliable manner. You can configure NXLog Agent to guarantee you do not lose data in case of a crash or network outage.

See Buffering and flow control and Batch processing for more information.

Use a persistent queue and cache

Various reasons could cause a crash, including power failure, kernel panic, and software bugs. To protect against data loss in these situations, NXLog Agent uses the following techniques:

It stores data in memory buffers. However, data can still be lost in the case of a crash. To mitigate data loss, you can enable persistent message queues to store data on disk instead of in memory. When using a persistent queue, NXLog Agent only removes data from the queue once it’s delivered successfully. See the PersistLogqueue and SyncLogqueue global settings and the PersistLogqueue and SyncLogqueue module settings.

Processor modules remove data from the queue before it is delivered, which can result in data loss in case of a crash. We do not recommend processor modules if high reliability is essential.
Several input modules, such as File (im_file), save the last record they processed in a cache file. By default, the file is saved to disk every 5 seconds. Depending on the last saved value, some records may be duplicated or lost in a crash. You can configure NXLog Agent to flush and sync the cache file to disk more frequently with the CacheFlushInterval and CacheSync global settings.

Example 1. Using a persistent queue and cache

This configuration uses disk-based queues for all processor and output module instances and syncs the queue with every record it processes. In addition, it configures the cache file containing information about the last record read by input modules to be synced every time there is an updated value.

This configuration may slow down data processing but ensures crash-safe operation and data integrity.

nxlog.conf

PersistLogqueue       TRUE    (1)
SyncLogqueue          TRUE    (2)
CacheFlushInterval    always  (3)
CacheSync             TRUE    (4)

<Input file>
    Module            im_file
    File              '/path/to/input.log'
</Input>

<Output tcp>
    Module            om_tcp
    Host              192.168.1.123:1514
</Output>

1	Enables disk-based queues.
2	Syncs the queue with every record the processor or output module instance processes.
3	Writes the last position to the cache file every time an input module instance sets a new value.
4	Ensures the cache file on disk is updated as soon as a new value is available, avoiding delays due to file system buffering.