Log classification

Pattern matching is commonly used for log classification. When certain strings are detected in a log message, the log message gets tagged with classifiers. Thus it is possible to query or take action based on the classifiers only. There are several ways to classify logs based on patterns.

See also Extracting data, a closely related topic, for more examples of classification.

Simple matching on fields

Often, log classification can be performed during parsing. However, if the required fields have already been parsed or the input module provides structured data, then it is only necessary to match the relevant fields and set classifiers.

Example 1. Classifying Windows Security events based on field values

This example classifies Windows Security login failure events with Event ID 4625 (controlled by the "Audit logon events" audit policy setting). If a received event has that ID, it is classified as a failed authentication attempt and the $AccountName field is set to the value of $TargetUserName.

Table 1. Sample event via im_msvistalog (excerpt)
Field Value

$EventType

AUDIT_FAILURE

$EventID

4625

$SourceName

Microsoft-Windows-Security-Auditing

$Channel

Security

$Category

Logon

$TargetUserSid

S-1-0-0

$TargetUserName

linda

$TargetDomainName

WINHOST

$Status

0xc000006d

$FailureReason

%%2313

$SubStatus

0xc000006a

$LogonType

2

nxlog.conf
<Input in>
    Module  im_msvistalog
    <Exec>
        if ($EventID == 4625) and
           ($SourceName == 'Microsoft-Windows-Security-Auditing')
        {
            $TaxonomyAction = 'Authenticate';
            $TaxonomyStatus = 'Failure';
            $AccountName = $TargetUserName;
        }
    </Exec>
</Input>
Table 2. Fields added to the event record
Field Value

$TaxonomyAction

Authenticate

$TaxonomyStatus

Failure

$AccountName

linda

Regular expressions via the Exec directive

The =~ operator can be used for regular expression matching in an Exec directive.

Example 2. Log classification with regular expression

When the contents of the $Message field match against the regular expression, the $AccountName and $AccountID fields are filled with the appropriate values from the referenced captured sub-strings. Additionally the value LoginEvent is stored in the $Action field.

if $Message =~ /(?x)^pam_unix\(sshd:session\):\ session\ opened\ for\ user\ (\S+)
                \ by\ \(uid=(\d+)\)/
{
    $AccountName = $1;
    $AccountID = integer($2);
    $Action = 'LoginEvent';
}

Using pm_pattern

When there are a lot of patterns, writing them all in the configuration file is inefficient. Instead, the pm_pattern module can be used.

Example 3. Classifying with pm_pattern

The above pattern matching rule can be defined in the pm_pattern modules’s XML format in the following way, which will accomplish the same result.

Pattern database (patterndb.xml)
<pattern>
   <id>42</id>
   <name>ssh_pam_session_opened</name>
   <description>ssh pam session opened</description>
   <matchfield>
      <name>Message</name>
      <type>REGEXP</type>
      <value>
      ^pam_unix\(sshd:session\): session opened for user (\S+) by \(uid=(\d+)\)
      </value>
      <capturedfield>
         <name>AccountName</name>
         <type>STRING</type>
      </capturedfield>
      <capturedfield>
         <name>AccountID</name>
         <type>INTEGER</type>
      </capturedfield>
   </matchfield>
   <set>
      <field>
          <name>Action</name>
          <type>STRING</type>
          <value>LoginEvent</value>
      </field>
   </set>
</pattern>
nxlog.conf
<Extension _syslog>
    Module      xm_syslog
</Extension>

<Input uds>
    Module      im_uds
    UDS         /dev/log
    Exec        parse_syslog_bsd();
</Input>

<Processor pattern>
    Module      pm_pattern
    PatternFile /var/lib/nxlog/patterndb.xml
</Processor>

<Output file>
    Module      om_file
    File        "/var/log/messages"
    Exec        to_syslog_bsd();
</Output>

<Route uds_to_file>
    Path        uds => pattern => file
</Route>