Filter events with NXLog Agent
NXLog Agent can reduce the size of telemetry data by filtering out unnecessary or duplicate events. Filtering events at the source means you transfer and store less data, reducing the data size during all subsequent stages of your telemetry pipeline. This helps to reduce bandwidth usage, storage requirements, and licensing costs for commercial SIEMs that bill by data volume.
You can configure NXLog Agent to detect events using any available fields or a regular expression. Depending on your requirements, you can then trim, normalize, or discard events.
See these resources for other event filtering and size-reduction strategies: |
Drop noise events
The most common approach to reduce noise events is to filter log events by their severity. In most cases, you can safely ignore lower severity or debugging events.
This configuration uses the im_udp input module to listen for syslog messages and parses events with the xm_syslog module. It then discards events that have a SeverityValue that is less than 3 (warning).
<Extension syslog>
Module xm_syslog
</Extension>
<Input udp_listen>
Module im_udp
ListenAddr 0.0.0.0:514
<Exec>
parse_syslog();
if $SeverityValue < 3
{
drop();
}
</Exec>
</Input>
Drop duplicate events
NXLog Agent’s pm_norepeat module can detect and discard duplicate events.
In their place, it generates a single event with a last message repeated n times
message.
This configuration collects syslog messages with the im_udp input module and parses them with the xm_syslog parse_syslog() procedure.
It then routes events through a pm_norepeat module instance, which checks the $Hostname
, $Message
, and $SourceName
fields to detect duplicate messages.
<Extension syslog>
Module xm_syslog
</Extension>
<Input udp_listen>
Module im_udp
ListenAddr 0.0.0.0:514
Exec parse_syslog();
</Input>
<Processor norepeat>
Module pm_norepeat
CheckFields Hostname, Message, SourceName
</Processor>
<Output agent_relay>
Module om_batchcompress
Host 192.168.0.123:2514
</Output>
<Route r>
Path udp_listen => norepeat => agent_relay
</Route>
Output events to different destinations
Another option for dropping noise events is archiving them to local or external storage while sending important events to your SIEM. That way, you retain a copy of the events and can retrieve them if needed.
This configuration collects syslog messages with the im_udp input module. It then filters records and sends warnings, errors, and critical messages to NXLog Platform but saves informational and debug messages to a file.
<Extension syslog>
Module xm_syslog
</Extension>
<Input udp_listen>
Module im_udp
ListenAddr 0.0.0.0:514
Exec parse_syslog(); (1)
</Input>
<Output nxlog_platform>
Module om_batchcompress
Host relay.nxlog.example.com:5514
UseSSL TRUE
CAFile %CERTDIR%/agent-ca.pem
CertFile %CERTDIR%/agent-cert.pem
CertKeyFile %CERTDIR%/agent-key.pem
<Exec>
if not (defined($SourceName) and $SeverityValue > 2) (2)
{
drop();
}
</Exec>
</Output>
<Output file>
Module om_file
File '/tmp/logs/info'
<Exec>
if not (defined($SourceName) and $SeverityValue < 3) (3)
{
drop();
}
</Exec>
</Output>
<Route r1>
Path udp_listen => nxlog_platform, file
</Route>
1 | Parses events into fields with the xm_syslog parse_syslog() procedure. |
2 | Drops events that do not have a $SourceName or have an information or debug $SeverityValue, so the remaining ones are forwarded to NXLog Platform. |
3 | Drops events that do not have a $SourceName or have a warning, error, or critical $SeverityValue, so the remaining ones are written to /tmp/logs/info . |