Filtering logs

Log filtering is a process where only some of the received log messages are kept. Filtering is possible using regular expressions or other operators using any of the fields. See the NXLog language section for complete details on expressions.

Using the drop() procedure

Use the drop() procedure in an Exec directive to conditionally discard logs.

Example 1. Using drop() to discard unmatched messages

In this example, any line that matches neither of the two regular expressions will be discarded with the drop() procedure. Only lines that match at least one of the regular expressions will be kept.

nxlog.conf

<Input file>
    Module    im_file
    File      "/var/log/myapp/*.log"
    Exec      if not ($raw_event =~ /failed/ or $raw_event =~ /error/) drop();
</Input>

Example 2. Using drop() with $SourceName and $Message to isolate authentication errors

In this example, events collected from multiple hosts and multiple sources by a centralized log server are contained in an input file. By defining a list of targeted $SourceName values along with the presence of certain keywords in the $Message field as criteria for authentication failures, the drop() procedure will discard all non-matching events.

nxlog.conf

define AUTHSOURCES "su", "sudo", "sshd", "unix_chkpwd"

<Input combined>
    Module    im_file
    File      "tmp/central-logging"
    <Exec>
        if not (
               defined($SourceName)
           and $SourceName IN (%AUTHSOURCES%)
           and (
                $Message =~ /fail/
             or $Message =~ /error/
             or $Message =~ /illegal/
             or $Message =~ /invalid/
           )
        ) drop();
    </Exec>
</Input>

Example 3. Using drop() with $SourceName and $EventID to collect all DNS logs

In this example events are to be collected from all DNS sources. Three of the four sources contain only DNS-specific events which can be matched by their $SourceName value alone against the defined list, but the Sysmon source can contain other non-DNS events as well. However, all Sysmon events with an Event ID of 22 are DNS log events. The conditional statement drops all events that do not have a $SourceName in the defined list as well as those that match the Sysmon $SourceName but do not have a value of 22 for their $EventID.

nxlog.conf

define DNSSOURCES  "Microsoft-Windows-DNSServer",   \
                   "Microsoft-Windows-DNS-Client",  \
                   "systemd-resolved"

<Input combined>
    Module im_file
    File   "tmp/central-logging"
    <Exec>
        if not (defined($SourceName)
           and ($SourceName IN (%DNSSOURCES%)
                 or ($SourceName == "Microsoft-Windows-Sysmon"
                     and $EventID ==  22)))
           drop();
    </Exec>
</Input>

Example 4. Filtering during the output phase to create multiple event logs from a single input

This example uses the same centralized log server events from the previous examples above as an input to three outputs. Separate categories based on a single $SourceName are created and written to three separate files. Each output instance defines a range of values for $EventID, the criteria for the categorization into two groups: DNS Server Audit or DNS Server Analytical. The conditional statement in the second instance uses $SeverityValue to keep only those audit logs having a value greater than 2 (warnings or errors).

nxlog.conf

<Input combined>
    Module    im_file
    File      "tmp/central-logging"
</Input>

<Output DNS_Audit>
    Module    om_file
    File      "tmp/DNS-Server-Audit"
    <Exec>
        if not (
               defined($SourceName)
           and $SourceName == "Microsoft-Windows-DNSServer"
           and $EventID >= 513
           and $EventID <= 582
        ) drop();
    </Exec>
</Output>

<Output DNS_Audit_Action_Required>
    Module    om_file
    File      "tmp/DNS-Server-Audit-Action-Required"
    <Exec>
        if not (
               defined($SourceName)
           and $SourceName == "Microsoft-Windows-DNSServer"
           and $EventID >= 513
           and $EventID <= 582
           and $SeverityValue > 2  # Severity higher than INFO
        ) drop();
    </Exec>
</Output>

<Output DNS_Analytical>
    Module    om_file
    File      "tmp/DNS-Server-Analytical"
    <Exec>
        if not (
               defined($SourceName)
           and $SourceName == "Microsoft-Windows-DNSServer"
           and $EventID >= 257
           and $EventID <= 280
        ) drop();
    </Exec>
</Output>

<Route combined_to_dns_audit_and_analytical>
    Path combined  =>  DNS_Audit,  DNS_Audit_Action_Required, DNS_Analytical
</Route>

Other options for filtering

The NXLog language also supports embedded XML queries in two input modules: Windows 2008/Vista and Later (im_msvistalog) and Windows Event Collector (im_wseventing). For more detailed information about filtering events from Collecting logs from Windows Event Log see the Filtering events section.