Buffer (pm_buffer)

Messages received over UDP may be dropped by the operating system if packets are not read from the socket buffer fast enough. Some logging subsystems using a small circular buffer can overwrite old logs in the buffer if it is not read, also resulting in loss of log data. Buffering can help in such situations.

The pm_buffer module supports disk- and memory-based log message buffering. If both are required, multiple pm_buffer instances can be used with different settings. Because a memory buffer can be faster, though its size is limited, combining memory and disk-based buffering can be a good idea if buffering is frequently used.

The disk-based buffering mode stores the log message data in chunks. When all the data is successfully forwarded from a chunk, it is then deleted to save disk space.

Using pm_buffer is only recommended when there is a chance of message loss. The built-in flow control in NXLog Agent ensures that messages will not be read by the input module until the output side can send, store, or forward. When reading from files (with im_file) or the Windows EventLog (with im_mseventlog or im_msvistalog) it is rarely necessary to use the pm_buffer module unless log rotation is used. During a rotation, there is a possibility of dropping some data while the output module (im_tcp, for example) is being blocked.

Configuration

The pm_buffer module accepts the following directives in addition to the common module directives. The MaxSize and Type directives are required.

Required directives

The following directives are required for the module to start.

MaxSize

This mandatory directive specifies the size of the buffer in kilobytes.

Type

This directive can be set to either Mem or Disk to select memory- or disk-based buffering.

Optional directives

CreateDir

If set to TRUE, this optional boolean directive instructs the module to create the output directory before opening the file for writing if it does not exist. The default is FALSE.

Directory

This directory will be used to store the disk buffer file chunks. This is only valid if Type is set to Disk.

WarnLimit

This directive specifies an optional limit, smaller than MaxSize, which will trigger a warning message when reached. The log message will not be generated again until the buffer size drops to half of WarnLimit and reaches it again to protect against a warning message flood.

Functions

The following functions are exported by pm_buffer.

integer buffer_count(): Return the number of log messages held in the memory buffer.

integer buffer_size(): Return the size of the memory buffer in bytes.

Examples

Example 1. Using a Memory Buffer to Protect Against UDP Message Loss

This configuration accepts log messages via UDP and forwards them via TCP. An intermediate memory-based buffer allows the im_udp module instance to continue accepting messages even if the om_tcp output stops working (caused by downtime of the remote host or network issues, for example).

nxlog.conf

<Input udp>
    Module      im_udp
    ListenAddr  0.0.0.0:514
</Input>

<Processor buffer>
    Module      pm_buffer
    # 1 MB buffer
    MaxSize     1024
    Type        Mem
    # warn at 512k
    WarnLimit   512
</Processor>

<Output tcp>
    Module      om_tcp
    Host        192.168.1.1:1514
</Output>

<Route udp_to_tcp>
    Path        udp => buffer => tcp
</Route>