Log classification

Pattern matching is commonly used for log classification. When certain strings are detected in a log message, the log message gets tagged with classifiers. Thus it is possible to query or take action based on the classifiers only. There are several ways to classify logs based on patterns.

See also Extracting data, a closely related topic, for more examples of classification.

Simple matching on fields

Often, log classification can be performed during parsing. However, if the required fields have already been parsed or the input module provides structured data, then it is only necessary to match the relevant fields and set classifiers.

Example 1. Classifying Windows Security events based on field values

This example classifies Windows Security login failure events with Event ID 4625 (controlled by the "Audit logon events" audit policy setting). If a received event has that ID, it is classified as a failed authentication attempt and the $AccountName field is set to the value of $TargetUserName.

Table 1. Sample event via im_msvistalog (excerpt)
Field	Value
$EventType	AUDIT_FAILURE
$EventID	4625
$SourceName	Microsoft-Windows-Security-Auditing
$Channel	Security
$Category	Logon
$TargetUserSid	S-1-0-0
$TargetUserName	linda
$TargetDomainName	WINHOST
$Status	0xc000006d
$FailureReason	%%2313
$SubStatus	0xc000006a
$LogonType	2

nxlog.conf

<Input in>
    Module  im_msvistalog
    <Exec>
        if ($EventID == 4625) and
           ($SourceName == 'Microsoft-Windows-Security-Auditing')
        {
            $TaxonomyAction = 'Authenticate';
            $TaxonomyStatus = 'Failure';
            $AccountName = $TargetUserName;
        }
    </Exec>
</Input>

Table 2. Fields added to the event record
Field	Value
$TaxonomyAction	Authenticate
$TaxonomyStatus	Failure
$AccountName	linda

Regular expressions via the Exec directive

The =~ operator can be used for regular expression matching in an Exec directive.

Example 2. Log classification with regular expression

When the contents of the $Message field match against the regular expression, the $AccountName and $AccountID fields are filled with the appropriate values from the referenced captured sub-strings. Additionally the value LoginEvent is stored in the $Action field.

if $Message =~ /(?x)^pam_unix\(sshd:session\):\ session\ opened\ for\ user\ (\S+)
                \ by\ \(uid=(\d+)\)/
{
    $AccountName = $1;
    $AccountID = integer($2);
    $Action = 'LoginEvent';
}

Using pm_pattern

NXLog Enterprise Edition exclusive feature

When there are a lot of patterns, writing them all in the configuration file is inefficient. Instead, the pm_pattern module can be used.

Example 3. Classifying with pm_pattern

The above pattern matching rule can be defined in the pm_pattern modules’s XML format in the following way, which will accomplish the same result.

Pattern database (patterndb.xml)

<pattern>
   <id>42</id>
   <name>ssh_pam_session_opened</name>
   <description>ssh pam session opened</description>
   <matchfield>
      <name>Message</name>
      <type>REGEXP</type>
      <value>
      ^pam_unix\(sshd:session\): session opened for user (\S+) by \(uid=(\d+)\)
      </value>
      <capturedfield>
         <name>AccountName</name>
         <type>STRING</type>
      </capturedfield>
      <capturedfield>
         <name>AccountID</name>
         <type>INTEGER</type>
      </capturedfield>
   </matchfield>
   <set>
      <field>
          <name>Action</name>
          <type>STRING</type>
          <value>LoginEvent</value>
      </field>
   </set>
</pattern>

nxlog.conf

<Extension _syslog>
    Module      xm_syslog
</Extension>

<Input uds>
    Module      im_uds
    UDS         /dev/log
    Exec        parse_syslog_bsd();
</Input>

<Processor pattern>
    Module      pm_pattern
    PatternFile /var/lib/nxlog/patterndb.xml
</Processor>

<Output file>
    Module      om_file
    File        "/var/log/messages"
    Exec        to_syslog_bsd();
</Output>

<Route uds_to_file>
    Path        uds => pattern => file
</Route>