Configure NXLog Agent log buffering

NXLog Agent implements several log buffering features that temporarily store log events during processing. See NXLog Agent buffering and flow control for a further explanation of log buffering.

Below, we provide examples that help you optimize the flow of log events and reduce the risk of data loss.

Increase a module’s log buffer

The simplest way to increase a module’s read or write buffer is to use the BufferSize directive. However, this directive only works for stream-oriented modules, i.e., modules connecting over the network (UDP, TCP, and HTTP) or reading and writing to files.

Example 1. Increasing the buffer size of stream-oriented modules

This configuration collects file-based logs with the im_file input module and forwards them to a remote server with the om_tcp output module. The BufferSize directive in both module instances increases the buffer size to 130,000 from the default 65,000 bytes.

nxlog.conf
<Input file>
    Module      im_file
    File        '/tmp/input.log'
    BufferSize  130000 (1)
</Input>

<Output tcp>
    Module      om_tcp
    Host        192.168.1.1
    BufferSize  130000 (2)
</Output>

<Route r1>
    Path        file => tcp
</Route>
1 Sets the om_file instance read buffer size in bytes.
2 Sets the om_file instace write buffer size in bytes.

The following diagram illustrates the read and write buffers in the configuration above.

gv configure log buffering increase buffer size

Out of all stream-oriented modules, UDP is unique because of the nature of the protocol. Since UDP is connectionless, the receiving end must process packets immediately in order not to lose data. Therefore, it is essential to have an appropriate buffer in case the receiver can’t keep up with the data flow. The UDP input and output modules provide the SockBufSize directive to configure the sending or receiving socket buffer space.

Example 2. Increasing the UDP buffer to prevent message loss

This configuration listens for logs with the im_udp input module and forwards them over the network with the om_udp output module. The SockBufSize directive in both module instances sets the socket buffer size to 150,000 bytes. In addition, it assigns the route the highest Priority to ensure that NXLog Agent processes logs received over UDP first, minimizing the risk of data loss in a multi-route configuration.

nxlog.conf
<Input udp_listen>
    Module          im_udp
    SockBufSize     150000000 (1)
</Input>

<Output udp_output>
    Module          om_udp
    Host            192.168.1.100:514
    SockBufSize     150000000  (2)
</Output>

<Route r1>
    Path            udp_listen => udp_output
    Priority        1 
</Route>
1 Sets the im_udp instance SockBufSize to 150 MB.
2 Sets the om_udp instance SockBufSize to 150 MB.

Increase a module’s log queue size

Every processor and output module instance has an input log queue containing the events it needs to process. The previous instance in the route writes logs in this queue until it reaches the size limit. Once the log queue is full, the instance stops processing further incoming logs. If there is a chance that the output destination cannot keep up with the demand or becomes unavailable, you can increase the log queue size globally or per module to reduce the risk of losing data.

Example 3. Increasing the log queue size

This configuration collects logs from a file and forwards them over the network using the om_tcp output module. It doubles the LogqueueSize of the output instance to 4 MiB.

nxlog.conf
<Input file>
    Module          im_file
    File            '/tmp/in.log'
</Input>

<Output tcp>
    Module          om_tcp
    Host            192.168.1.100:514
    LogqueueSize    4194304 (1)
</Output>
1 Sets the om_tcp instace log queue size in bytes.

The following diagram illustrates log processing with the configuration above. If the TCP destination is unreachable, the om_tcp instance is blocked. The im_file instance will continue to process logs until the log queue reaches the size limit, at which stage it will pause.

gv configure log buffering tcp logqueuesize

By default, NXLog Agent uses memory-based log queues. However, you can configure it to persist log queues to disk globally or per module. Optionally, you can sync disk-based log queues with every new event to mitigate data loss further at the expense of reduced performance.

NXLog Agent writes any events in the log queue to disk when it stops, regardless of the PersistLogqueue setting.
Example 4. Using a persistent log queue

The om_elasticsearch instance in this configuration uses a disk-based log queue synced with every event. Therefore, the instance will write every event to disk before it processes the next event. It also uses a 4 MiB log queue, double the default size.

nxlog.conf
<Input acct>
    Module             im_acct
</Input>

<Output elasticsearch>
    Module             om_elasticsearch
    URL                http://192.168.1.100:9200/_bulk
    LogqueueSize       4194304  (1)
    PersistLogqueue    TRUE (2)
    SyncLogqueue       TRUE (3)
</Output>

<Route r1>
    Path               acct => elasticsearch
</Route>
1 Sets the LogqueueSize in bytes.
2 Enables the PersistLogqueue directive to use a disk-based log queue.
3 Enables the SyncLogqueue directive to persist data to disk with every event.

The following diagram illustrates log processing with the configuration above.

gv configure log buffering persistent queue

Implement additional buffering

In addition to the default buffering functionality, the pm_buffer module allows you to configure custom memory or disk-based buffers.

Example 5. Using a memory-based buffer

This configuration receives logs over UDP and forwards them over the network via HTTP. To prevent data loss in case the destination is unreachable, it uses a pm_buffer module instance to save log events in memory temporarily.

nxlog.conf
<Input udp>
    Module         im_udp
</Input>

<Processor buffer>
    Module         pm_buffer
    Type           Mem       (1)
    MaxSize        5120      (2)
    WarnLimit      2048      (3)
</Processor>

<Output http>
    Module         om_http
    URL            http://192.168.1.100:8080/
</Output>

<Route r1>
    Path           udp => buffer => http
</Route>
1 Sets the Type to a memory-based buffer.
2 Sets the MaxSize to 5120 KB.
3 Sets a WarnLimit to log a warning in the NXLog Agent log file when the buffer size reaches 2048 KB.

The following diagram illustrates log processing with the configuration above in a scenario where the HTTP destination is unreachable. When the om_http instance is blocked, its log queue may reach the size limit. If this happens, the input instance will continue processing logs and writing them to the intermediate buffer.

gv configure log buffering memory based buffer

A disk-based buffer might be more suitable if you expect logs to be in the buffer for a long time. For example, if you want NXLog Agent to collect logs but schedule forwarding them over the network to a specific timeframe.

Example 6. Using a disk-based buffer with a log-forwarding schedule

This configuration receives logs over UDP and forwards them to a remote NXLog Agent via TCP. However, it uses the pm_blocker module to forward logs only on Saturdays and Sundays. NXLog Agent saves logs to disk during the rest of the week with a pm_buffer instance.

nxlog.conf
<Input udp>
    Module       im_udp
</Input>

<Processor buffer>
    Module       pm_buffer
    Type         Disk   (1)
    MaxSize      512000 (2)
    WarnLimit    409600 (3)
</Processor>

<Processor schedule>
    Module       pm_blocker
    <Schedule> (4)
        When    0 0 * * 1
        Exec    schedule->block(TRUE);
    </Schedule>
    <Schedule> (5)
        When    0 0 * * 6
        Exec    schedule->block(FALSE);
    </Schedule>
</Processor>

<Output batch>
    Module       om_batchcompress
    Host         192.168.1.100:2514
</Output>

<Route r1>
    Path         udp => buffer => schedule => batch
</Route>
1 Sets the Type to a disk-based buffer.
2 Sets the MaxSize to 512000 KB.
3 Sets a WarnLimit to log a warning in the NXLog Agent log file when the buffer size reaches 409600 KB.
4 Blocks log forwarding on Monday morning
5 Resumes log forwarding on Saturday morning

The following diagram illustrates log processing with the configuration above in a scenario where log forwarding is blocked. When the pm_blocker instance is blocked, its log queue will reach the size limit. Once this happens, the input instance will continue processing logs and writing them to the intermediate disk-based buffer.

gv configure log buffering scheduled forwarding

Configure flow control

Flow control is an automatic mechanism that suspends log processing when a route is blocked. However, there are situations where you should turn off flow control and configure buffering separately.

For example, when collecting syslog messages with the im_uds module, a suspended instance will block the syslog() system call, preventing other programs from logging messages. You can turn off flow control globally or per module to avoid this.

Example 7. Disabling flow control

This configuration collects syslog messages with the im_uds input module. The module instance has flow control turned off. To prevent NXLog Agent from discarding events if the route is blocked and the log queues reach their size limit, it increases the log queue size of the output instance to 4 MiB, double the default size.

nxlog.conf
<Extension syslog>
    Module          xm_syslog
</Extension>

<Input dev_log>
    Module          im_uds
    UDS             /dev/log
    Exec            parse_syslog();
    FlowControl     FALSE    (1)
</Input>

<Output elasticsearch>
    Module          om_elasticsearch
    URL             http://192.168.2.1:9022/_bulk
    LogqueueSize    4194304  (2)
</Output>

<Route r1>
    Path            dev_log => elasticsearch
</Route>
1 Turning off FlowControl ensures that the instance is never suspended.
2 Sets the LogQueuesize in bytes.

The following diagram illustrates log processing with the configuration above in a scenario where log forwarding is blocked. When the om_elasticsearch instance is blocked, its log queue may reach the size limit. If this happens, the input instance will discard incoming logs until the route is operable.

gv configure log buffering disable flow control

You can also configure flow control for output modules. For example, when routing logs to multiple destinations, you can define how a blocked output instance affects log processing.

Example 8. Flow control with multiple outputs

This configuration collects Windows logs with the im_msvistalog input module. It then forwards log events to a remote host with the om_tcp module and saves a copy to file with the om_file module.

The om_tcp instance has FlowControl switched off. Therefore, NXLog Agent will continue to process logs even if the remote destination is unreachable or the network cannot handle the events quickly enough.

nxlog.conf
<Input eventlog>
    Module         im_msvistalog
</Input>

<Output tcp>
    Module         om_tcp
    Host           192.168.1.100
    FlowControl    FALSE
</Output>

<Output file>
    Module         om_file
    File           'C:\temp\output.log'
</Output>

<Route r>
    Path           eventlog => tcp, file
</Route>

If the om_tcp instance cannot forward logs and its log queue reaches the size limit, the im_msvistalog instance continues to collect and deliver logs to both output instances. The file instance will continue to write events to the output file. However, the tcp instance will discard events until it recovers.

gv configure log buffering flow control dual output

On the other hand, if the om_file instance cannot process logs fast enough and its log queue reaches the size limit, the im_msvistalog instance will stop collecting logs until the file instance recovers.

gv configure log buffering flow control dual output paused