Protect against data loss

Regulatory compliance usually requires that the logging infrastructure function in an ultra-reliable manner. You can configure NXLog Agent to guarantee you do not lose logs in case of a crash or network outage.

See Buffering and flow control and Batch processing for more information.

Use a persistent log queue and cache

Various reasons could cause a crash, including power failure, kernel panic, and software bugs. To protect against data loss in these situations, NXLog Agent uses the following techniques:

  • It stores log events in memory buffers. However, logs can still be lost in the case of a crash. To mitigate data loss, you can enable persistent message queues to store logs on disk instead of in memory. When using a persistent log queue, NXLog Agent only removes log events from the queue once they’re delivered successfully. See the PersistLogqueue and SyncLogqueue global settings and the PersistLogqueue and SyncLogqueue module settings.

    Processor modules remove log events from the queue before they are delivered, which can result in data loss in case of a crash. We do not recommend processor modules if high reliability is essential.
  • Several input modules, such as File (im_file), save the last log event they processed in a cache file. By default, the file is saved to disk every 5 seconds. Depending on the last saved value, some log events may be duplicated or lost in a crash. You can configure NXLog Agent to flush and sync the cache file to disk more frequently with the CacheFlushInterval and CacheSync global settings.

Example 1. Using a persistent log queue and cache

This configuration uses disk-based log queues for all processor and output module instances and syncs the log queue with every log event it processes. In addition, it configures the cache file containing information about the last event read by input modules to be synced every time there is an updated value.

This configuration may slow down log processing but ensures crash-safe operation and data integrity.

nxlog.conf
PersistLogqueue       TRUE    (1)
SyncLogqueue          TRUE    (2)
CacheFlushInterval    always  (3)
CacheSync             TRUE    (4)

<Input file>
    Module            im_file
    File              '/path/to/input.log'
</Input>

<Output tcp>
    Module            om_tcp
    Host              192.168.1.123:1514
</Output>
1 Enables disk-based log queues.
2 Syncs the log queue with every log event the processor or output module instance processes.
3 Writes the last position to the cache file every time an input module instance sets a new value.
4 Ensures the cache file on disk is updated as soon as a new value is available, avoiding delays due to file system buffering.