Configure NXLog Agent log buffering
NXLog Agent implements several log buffering features that temporarily store log events during processing. See NXLog Agent buffering and flow control for a further explanation of log buffering.
Below, we provide examples that help you optimize the flow of log events and reduce the risk of data loss.
Increase a module’s log buffer
The simplest way to increase a module’s read or write buffer is to use the BufferSize directive. However, this directive only works for stream-oriented modules, i.e., modules connecting over the network (UDP, TCP, and HTTP) or reading and writing to files.
This configuration collects file-based logs with the im_file input module and forwards them to a remote server with the om_tcp output module. The BufferSize directive in both module instances increases the buffer size to 130,000 from the default 65,000 bytes.
<Input file>
Module im_file
File '/tmp/input.log'
BufferSize 130000 (1)
</Input>
<Output tcp>
Module om_tcp
Host 192.168.1.1
BufferSize 130000 (2)
</Output>
<Route r1>
Path file => tcp
</Route>| 1 | Sets the om_file instance read buffer size in bytes. |
| 2 | Sets the om_file instace write buffer size in bytes. |
The following diagram illustrates the read and write buffers in the configuration above.
Out of all stream-oriented modules, UDP is unique because of the nature of the protocol. Since UDP is connectionless, the receiving end must process packets immediately in order not to lose data. Therefore, it is essential to have an appropriate buffer in case the receiver can’t keep up with the data flow. The UDP input and output modules provide the SockBufSize directive to configure the sending or receiving socket buffer space.
This configuration listens for logs with the im_udp input module and forwards them over the network with the om_udp output module. The SockBufSize directive in both module instances sets the socket buffer size to 150,000 bytes. In addition, it assigns the route the highest Priority to ensure that NXLog Agent processes logs received over UDP first, minimizing the risk of data loss in a multi-route configuration.
<Input udp_listen>
Module im_udp
SockBufSize 150000000 (1)
</Input>
<Output udp_output>
Module om_udp
Host 192.168.1.100:514
SockBufSize 150000000 (2)
</Output>
<Route r1>
Path udp_listen => udp_output
Priority 1
</Route>| 1 | Sets the im_udp instance SockBufSize to 150 MB. |
| 2 | Sets the om_udp instance SockBufSize to 150 MB. |
Increase a module’s log queue size
Every processor and output module instance has an input log queue containing the events it needs to process. The previous instance in the route writes logs in this queue until it reaches the size limit. Once the log queue is full, the instance stops processing further incoming logs. If there is a chance that the output destination cannot keep up with the demand or becomes unavailable, you can increase the log queue size globally or per module to reduce the risk of losing data.
This configuration collects logs from a file and forwards them over the network using the om_tcp output module. It doubles the LogqueueSize of the output instance to 4 MiB.
<Input file>
Module im_file
File '/tmp/in.log'
</Input>
<Output tcp>
Module om_tcp
Host 192.168.1.100:514
LogqueueSize 4194304 (1)
</Output>| 1 | Sets the om_tcp instace log queue size in bytes. |
The following diagram illustrates log processing with the configuration above. If the TCP destination is unreachable, the om_tcp instance is blocked. The im_file instance will continue to process logs until the log queue reaches the size limit, at which stage it will pause.
By default, NXLog Agent uses memory-based log queues. However, you can configure it to persist log queues to disk globally or per module. Optionally, you can sync disk-based log queues with every new event to mitigate data loss further at the expense of reduced performance.
| NXLog Agent writes any events in the log queue to disk when it stops, regardless of the PersistLogqueue setting. |
The om_elasticsearch instance in this configuration uses a disk-based log queue synced with every event. Therefore, the instance will write every event to disk before it processes the next event. It also uses a 4 MiB log queue, double the default size.
<Input acct>
Module im_acct
</Input>
<Output elasticsearch>
Module om_elasticsearch
URL http://192.168.1.100:9200/_bulk
LogqueueSize 4194304 (1)
PersistLogqueue TRUE (2)
SyncLogqueue TRUE (3)
</Output>
<Route r1>
Path acct => elasticsearch
</Route>| 1 | Sets the LogqueueSize in bytes. |
| 2 | Enables the PersistLogqueue directive to use a disk-based log queue. |
| 3 | Enables the SyncLogqueue directive to persist data to disk with every event. |
The following diagram illustrates log processing with the configuration above.
Implement additional buffering
In addition to the default buffering functionality, the pm_buffer module allows you to configure custom memory or disk-based buffers.
This configuration receives logs over UDP and forwards them over the network via HTTP. To prevent data loss in case the destination is unreachable, it uses a pm_buffer module instance to save log events in memory temporarily.
<Input udp>
Module im_udp
</Input>
<Processor buffer>
Module pm_buffer
Type Mem (1)
MaxSize 5120 (2)
WarnLimit 2048 (3)
</Processor>
<Output http>
Module om_http
URL http://192.168.1.100:8080/
</Output>
<Route r1>
Path udp => buffer => http
</Route>| 1 | Sets the Type to a memory-based buffer. |
| 2 | Sets the MaxSize to 5120 KB. |
| 3 | Sets a WarnLimit to log a warning in the NXLog Agent log file when the buffer size reaches 2048 KB. |
The following diagram illustrates log processing with the configuration above in a scenario where the HTTP destination is unreachable. When the om_http instance is blocked, its log queue may reach the size limit. If this happens, the input instance will continue processing logs and writing them to the intermediate buffer.
A disk-based buffer might be more suitable if you expect logs to be in the buffer for a long time. For example, if you want NXLog Agent to collect logs but schedule forwarding them over the network to a specific timeframe.
This configuration receives logs over UDP and forwards them to a remote NXLog Agent via TCP. However, it uses the pm_blocker module to forward logs only on Saturdays and Sundays. NXLog Agent saves logs to disk during the rest of the week with a pm_buffer instance.
<Input udp>
Module im_udp
</Input>
<Processor buffer>
Module pm_buffer
Type Disk (1)
MaxSize 512000 (2)
WarnLimit 409600 (3)
</Processor>
<Processor schedule>
Module pm_blocker
<Schedule> (4)
When 0 0 * * 1
Exec schedule->block(TRUE);
</Schedule>
<Schedule> (5)
When 0 0 * * 6
Exec schedule->block(FALSE);
</Schedule>
</Processor>
<Output batch>
Module om_batchcompress
Host 192.168.1.100:2514
</Output>
<Route r1>
Path udp => buffer => schedule => batch
</Route>| 1 | Sets the Type to a disk-based buffer. |
| 2 | Sets the MaxSize to 512000 KB. |
| 3 | Sets a WarnLimit to log a warning in the NXLog Agent log file when the buffer size reaches 409600 KB. |
| 4 | Blocks log forwarding on Monday morning |
| 5 | Resumes log forwarding on Saturday morning |
The following diagram illustrates log processing with the configuration above in a scenario where log forwarding is blocked. When the pm_blocker instance is blocked, its log queue will reach the size limit. Once this happens, the input instance will continue processing logs and writing them to the intermediate disk-based buffer.
Implementing measures to reduce the risk of data loss, such as enabling the PersistLogQueue directive or using disk-based buffers with the pm_buffer module, can negatively affect performance since they substantially increase disk read and write operations. By default, NXLog Agent uses log queues and buffers with a flow control mechanism to mitigate data loss. Therefore, you only need to implement additional measures if reliability is critical. In most cases, you need to find a balance between log processing speed and data integrity.
This configuration uses the im_tcp input module to listen for connections on port 1514. It first stores received logs in a memory-based buffer and then in a larger disk-based buffer until they are saved to a file.
<Input tcp_listen>
Module im_tcp
ListenAddr 0.0.0.0:1514
</Input>
<Processor memory_buffer>
Module pm_buffer
MaxSize 1024 (1)
Type Mem
</Processor>
<Processor disk_buffer>
Module pm_buffer
MaxSize 512000 (2)
Type Disk
WarnLimit 409600 (3)
</Processor>
<Output output_file>
Module om_file
File '/path/to/output/file'
</Output>
<Route r1>
Path tcp_listen => memory_buffer => disk_buffer => output_file
</Route>
Configure flow control
Flow control is an automatic mechanism that suspends log processing when a route is blocked. However, there are situations where you should turn off flow control and configure buffering separately.
For example, when collecting syslog messages with the im_uds module, a suspended instance will block the syslog() system call, preventing other programs from logging messages.
You can turn off flow control globally or per module to avoid this.
This configuration collects syslog messages with the im_uds input module. The module instance has flow control turned off. To prevent NXLog Agent from discarding events if the route is blocked and the log queues reach their size limit, it increases the log queue size of the output instance to 4 MiB, double the default size.
<Extension syslog>
Module xm_syslog
</Extension>
<Input dev_log>
Module im_uds
UDS /dev/log
Exec parse_syslog();
FlowControl FALSE (1)
</Input>
<Output elasticsearch>
Module om_elasticsearch
URL http://192.168.2.1:9022/_bulk
LogqueueSize 4194304 (2)
</Output>
<Route r1>
Path dev_log => elasticsearch
</Route>| 1 | Turning off FlowControl ensures that the instance is never suspended. |
| 2 | Sets the LogQueuesize in bytes. |
The following diagram illustrates log processing with the configuration above in a scenario where log forwarding is blocked. When the om_elasticsearch instance is blocked, its log queue may reach the size limit. If this happens, the input instance will discard incoming logs until the route is operable.
You can also configure flow control for output modules. For example, when routing logs to multiple destinations, you can define how a blocked output instance affects log processing.
This configuration collects Windows logs with the im_msvistalog input module. It then forwards log events to a remote host with the om_tcp module and saves a copy to file with the om_file module.
The om_tcp instance has FlowControl switched off. Therefore, NXLog Agent will continue to process logs even if the remote destination is unreachable or the network cannot handle the events quickly enough.
<Input eventlog>
Module im_msvistalog
</Input>
<Output tcp>
Module om_tcp
Host 192.168.1.100
FlowControl FALSE
</Output>
<Output file>
Module om_file
File 'C:\temp\output.log'
</Output>
<Route r>
Path eventlog => tcp, file
</Route>If the om_tcp instance cannot forward logs and its log queue reaches the size limit, the im_msvistalog instance continues to collect and deliver logs to both output instances.
The file instance will continue to write events to the output file.
However, the tcp instance will discard events until it recovers.
On the other hand, if the om_file instance cannot process logs fast enough and its log queue reaches the size limit, the im_msvistalog instance will stop collecting logs until the file instance recovers.
