De-Duplicator (pm_norepeat)

This module can be used to filter out repeating messages. Like Syslog daemons, this module checks the previous message against the current. If they match, the current message is dropped. The module waits one second for duplicated messages to arrive. If duplicates are detected, the first message is forwarded, the rest are dropped, and a message containing "last message repeated n times" is sent instead.

This module is being phased out and will be removed in a future release. You can implement the same functionality using Variables.

Configuration

The pm_norepeat module accepts the following directives in addition to the common module directives.

Optional directives

CheckFields

This optional directive takes a comma-separated list of field names which are used to compare log messages. Only the fields listed here are compared, the others are ignored. For example, the $EventTime field will be different in repeating messages, so this field should not be used in the comparison. If this directive is not specified, the default field to be checked is $Message.

Fields

The following fields are used by pm_norepeat.

$raw_event (type: string): A string containing the last message repeated n times message.

$EventTime (type: datetime): The time of the last event or the current time if EventTime was not present in the last event.

$Message (type: string): The same value as $raw_event.

$ProcessID (type: integer): The process ID of the NXLog Agent process.

$Severity (type: string): The severity name: INFO.

$SeverityValue (type: integer): The INFO severity level value: 2.

$SourceName (type: string): Set to nxlog.

Examples

Example 1. Filtering Out Duplicated Messages

This configuration reads log messages from the socket. The $Hostname, $SourceName, and $Message fields are used to detect duplicates. Then the messages are written to a file.

nxlog.conf

<Input uds>
    Module      im_uds
    UDS         /dev/log
</Input>

<Processor norepeat>
    Module      pm_norepeat
    CheckFields Hostname, SourceName, Message
</Processor>

<Output file>
    Module      om_file
    File        "/var/log/messages"
</Output>


<Route uds_to_file>
    Path        uds => norepeat => file
</Route>