Filter logs with NXLog Agent

NXLog Agent can reduce the size of log data by filtering out unnecessary or duplicate events. Filtering logs at the source means you transfer and store less data, reducing the data size during all subsequent log processing stages. This helps to reduce bandwidth usage, storage requirements, and licensing costs for commercial SIEMs that bill by data volume.

You can configure NXLog Agent to detect events using any available fields or a regular expression. Depending on your logging requirements, you can then trim, normalize, or discard log events.

See these resources for other log filtering and size-reduction strategies:

Drop noise events

The most common approach to reduce noise events is to filter log events by their severity. In most cases, you can safely ignore lower severity or debugging events.

Example 1. Dropping unnecessary events

This configuration uses the im_udp input module to listen for syslog messages and parses records with the xm_syslog module. It then discards events that have a SeverityValue that is less than 3 (warning).

nxlog.conf

<Extension syslog>
    Module        xm_syslog
</Extension>

<Input udp_listen>
    Module        im_udp
    ListenAddr    0.0.0.0:514
    <Exec>
        parse_syslog(); 
        if $SeverityValue < 3 
        {
            drop();
        }
    </Exec>
</Input>

Drop duplicate events

NXLog Agent’s pm_norepeat module can detect and discard duplicate events. In their place, it generates a single event with a last message repeated n times message.

Example 2. Dropping duplicate events

This configuration collects syslog messages with the im_udp input module and parses them with the xm_syslog parse_syslog() procedure. It then routes records through a pm_norepeat module instance, which checks the $Hostname, $Message, and $SourceName fields to detect duplicate messages.

nxlog.conf

<Extension syslog>
    Module         xm_syslog
</Extension>

<Input udp_listen>
    Module         im_udp
    ListenAddr     0.0.0.0:514
    Exec           parse_syslog();
</Input>

<Processor norepeat>
    Module         pm_norepeat
    CheckFields    Hostname, Message, SourceName
</Processor>

<Output agent_relay>
    Module         om_batchcompress
    Host           192.168.0.123:2514
</Output>

<Route r>
    Path           udp_listen => norepeat => agent_relay
</Route>

Output logs to different destinations

Another option for dropping noise events is archiving them to local or external storage while sending important events to your SIEM. That way, you retain a copy of the log events and can retrieve them if needed.

Example 3. Filtering and outputting logs to different destinations

This configuration collects syslog messages with the im_udp input module. It then filters log records and sends warnings, errors, and critical messages to NXLog Platform but saves informational and debug messages to a file.

nxlog.conf

<Extension syslog>
    Module        xm_syslog
</Extension>

<Input udp_listen>
    Module        im_udp
    ListenAddr    0.0.0.0:514
    Exec          parse_syslog(); (1)
</Input>

<Output nxlog_platform>
    Module        om_batchcompress
    Host          relay.nxlog.example.com:5514
    SocketType    SSL
    CAFile        %CERTDIR%/agent-ca.pem
    CertFile      %CERTDIR%/agent-cert.pem
    CertKeyFile   %CERTDIR%/agent-key.pem
    <Exec>
        if not (defined($SourceName) and $SeverityValue > 2)  (2)
        {
            drop();
        }
    </Exec>
</Output>

<Output file>
    Module        om_file
    File          '/tmp/logs/info'
    <Exec>
        if not (defined($SourceName) and $SeverityValue < 3)  (3)
        {
            drop();
        }
    </Exec>
</Output>

<Route r1>
    Path          udp_listen => nxlog_platform, file
</Route>

1	Parses log records into fields with the xm_syslog parse_syslog() procedure.
2	Drops records that do not have a $SourceName or have an information or debug $SeverityValue, so the remaining ones are forwarded to NXLog Platform.
3	Drops records that do not have a $SourceName or have a warning, error, or critical $SeverityValue, so the remaining ones are written to `/tmp/logs/info`.