Pattern Matcher (xm_pattern)

This module parses unstructured data using an NXLog pattern definition. xm_pattern is designed for efficient, non-linear pattern-matching and is more efficient than processing logs against regular expressions defined in an Exec directive. In addition, the module automatically reprioritizes patterns according to your logs to further improve processing speed.

To examine the supported platforms, see the list of installation packages.

NXLog patterns are defined in XML format and support matching multiple fields. Therefore, unlike traditional pattern-matching techniques, you do not need to convert log records into a single string. It also supports pattern groups, where you can define multiple patterns in a block. If a log record matches the group criteria, the module continues to process the patterns in the group. Otherwise, it skips the entire group. See the examples below for a practical application of pattern groups.

When a log record matches a pattern, xm_pattern adds two fields: $PatternID and $PatternName. You can use these fields for conditional processing and correlation further down the pipeline, for example, to process with pm_evcorr.

xm_pattern stops processing patterns after the first match. Therefore, a log record can only match one pattern. As a result, we recommend avoiding defining patterns that match the same subset of logs.

For example, if you define two regular expression patterns, ^\d+ and ^\d\d, a log record will match either one or the other, but not both. There are no rules about which pattern it will match because xm_pattern reorders the priority of patterns according to the highest match count. If you require linear pattern matching, you are better off using regular expressions in an Exec directive.

The pattern database XML Schema Definition (XSD) is available in our public git repository.

Configuration

The xm_pattern module accepts the following directives in addition to the common module directives.

Required directives

The following directives are required for the module to start.

PatternFile

Specify the path to the pattern database file.

Functions

The following functions are exported by xm_pattern.

type: boolean match_pattern(): Execute the match_pattern() procedure. If the event is successfully matched, return TRUE, otherwise FALSE.

Procedures

The following procedures are exported by xm_pattern.

match_pattern();: Attempt to match the current event according to the PatternFile. Execute statements and add fields as specified.

Fields

The following fields are used by xm_pattern.

$PatternID (type: integer): The ID of the pattern that matched the event.
$PatternName (type: string): The name of the pattern that matched the event.

Pattern database schema

The following table lists the XML elements used to create the pattern database. Click on each element for details.

Element	Description
capturedfield	Defines a field captured from a regular expression.
capturedvalue	Defines a captured field and value.
created	Defines the creation date of the pattern database.
description	Used to add comments to the schema.
exec	Used to execute commands in the NXLog language.
field	Defines a field name and value.
group	Defines a group of related patterns.
id	The unique identifier of the parent element.
matchfield	Used to define matching criteria for a field.
matchtype	Used to define the field-matching type.
name	The name of the parent element or a field.
pattern	Used to define matching criteria.
patterndb	The top-level element of the pattern database.
set	Defines fields and values to be set if the event matches the pattern.
testcase	Defines a field and value to match in a pattern.
type	Used to define the data type of a field.
value	Used to define the value of a field.
version	Defines the version of the pattern database.

Element

Description

capturedfield

Defines a field captured from a regular expression.

capturedvalue

Defines a captured field and value.

created

Defines the creation date of the pattern database.

description

Used to add comments to the schema.

exec

Used to execute commands in the NXLog language.

field

Defines a field name and value.

group

Defines a group of related patterns.

The unique identifier of the parent element.

matchfield

Used to define matching criteria for a field.

matchtype

Used to define the field-matching type.

name

The name of the parent element or a field.

pattern

Used to define matching criteria.

patterndb

The top-level element of the pattern database.

set

Defines fields and values to be set if the event matches the pattern.

testcase

Defines a field and value to match in a pattern.

type

Used to define the data type of a field.

value

Used to define the value of a field.

version

Defines the version of the pattern database.

Examples

Example 1. Using exact matching

The pattern file below defines a single pattern, which matches Windows security events by ID.

pattern.xml

<?xml version='1.0' encoding='UTF-8'?>
<patterndb>
  <created>2025-10-09 01:02:03</created>
    <version>1</version>
    <!-- First pattern group in this file -->
    <group>
      <name>Security</name>
      <description></description>
      <id>1</id>
      <!-- Only apply this pattern group if the $Channel field is "Security" -->
      <matchfield>
        <name>Channel</name>
        <type>exact</type>
        <value>Security</value>
      </matchfield>
      <!-- First pattern in this pattern group -->
      <pattern>
        <id>1</id>
        <name>Security Event 4624</name>
        <description></description>
        <matchfield>
          <name>EventID</name>
          <type>exact</type>
          <value>4624</value>
        </matchfield>
        <!-- If the pattern matches, remove the event summary -->
        <exec>
          $Message =~ s/\s*This event is generated.*$//s;
        </exec>
      </pattern>
      <!-- Second pattern in this pattern group -->
      <pattern>
        <id>2</id>
        <name>Security Event 4625</name>
        <description></description>
        <matchfield>
          <name>EventID</name>
          <type>exact</type>
          <value>4625</value>
        </matchfield>
        <!-- If the pattern matches, remove the event summary -->
        <exec>
          $Message =~ s/\s*This event is generated.*$//s;
        </exec>
      </pattern>
    </group>
</patterndb>

This configuration collects Windows events using the im_msvistalog input module. This module automatically parses events into structured data and populates the $Channel, $EventID, and $Message fields used by the above pattern.

It then calls the match_pattern() function to process the event against the pattern file. If the event does not match any pattern, it drops the event. Finally, it forwards the record over TCP in JSON format.

nxlog.conf

<Extension pattern>
    Module       xm_pattern (1)
    PatternFile  'C:\Program File\nxlog\conf\pattern.xml' (2)
</Extension>

<Extension json>
    Module       xm_json (3)
</Extension>

<Input windows_events>
    Module       im_msvistalog
    <Exec>
        if not match_pattern() {
            drop();
        }
    </Exec>
</Input>

<Output siem>
    Module        om_tcp
    Host          192.168.1.123:1514
    Exec          to_json(); (4)
</Output>

<Route r1>
    Path         windows_events  => siem
</Route>

1	The xm_pattern module provides the match_pattern() function.
2	The PatternFile directive specifies the path to the pattern file.
3	The xm_json module provides the to_json() procedure.
4	Converts the record to JSON and writes it to the `$raw_event` core field.

The following is an example of a Windows security event with ID 4625 pertaining to a failed login attempt.

Input sample

Log Name:      Security
Source:        Microsoft-Windows-Security-Auditing
Date:          09/10/2025 10:23:07
Event ID:      4625
Task Category: Logon
Level:         Information
Keywords:      Audit Failure
User:          N/A
Computer:      SRV01
Description:
An account failed to log on.

Subject:
	Security ID:		SYSTEM
	Account Name:		SRV01$
	Account Domain:		WORKGROUP
	Logon ID:		0x3E7

Logon Type:			2

Account For Which Logon Failed:
	Security ID:		NULL SID
	Account Name:		-
	Account Domain:		-

Failure Information:
	Failure Reason:		An Error occured during Logon.
	Status:			0xC000006D
	Sub Status:		0xC0000380

Process Information:
	Caller Process ID:	0x9f0
	Caller Process Name:	C:\Windows\System32\svchost.exe

Network Information:
	Workstation Name:	-
	Source Network Address:	127.0.0.1
	Source Port:		0

Detailed Authentication Information:
	Logon Process:		User32
	Authentication Package:	Negotiate
	Transited Services:	-
	Package Name (NTLM only):	-
	Key Length:		0

This event is generated when a logon request fails. It is generated on the computer where access was attempted.

The Subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.

The Logon Type field indicates the kind of logon that was requested. The most common types are 2 (interactive) and 3 (network).

The Process Information fields indicate which account and process on the system requested the logon.

The Network Information fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.

The authentication information fields provide detailed information about this specific logon request.
	- Transited services indicate which intermediate services have participated in this logon request.
	- Package name indicates which sub-protocol was used among the NTLM protocols.
	- Key length indicates the length of the generated session key. This will be 0 if no session key was requested.

When the NXLog Agent configuration above processes this event, it transforms it into the following JSON object.

Output sample

{
  "EventTime": "2025-10-09T10:23:07.277606+02:00",
  "Hostname": "SRV01",
  "Keywords": "0x8010000000000000",
  "LevelValue": 0,
  "EventType": "AUDIT_FAILURE",
  "SeverityValue": 4,
  "Severity": "ERROR",
  "EventID": 4625,
  "SourceName": "Microsoft-Windows-Security-Auditing",
  "ProviderGuid": "{54849625-5478-4994-A5BA-3E3B0328C30D}",
  "Version": 0,
  "TaskValue": 12544,
  "OpcodeValue": 0,
  "RecordNumber": 1115568,
  "ActivityID": "{8FC1CC2E-22AB-0002-CCCC-C18FAB22DC01}",
  "ExecutionProcessID": 1340,
  "ExecutionThreadID": 3336,
  "Channel": "Security",
  "Message": "An account failed to log on.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-1-5-18\r\n\tAccount Name:\t\tSRV01$\r\n\tAccount Domain:\t\tWORKGROUP\r\n\tLogon ID:\t\t0x3e7\r\n\r\nLogon Type:\t\t\t2\r\n\r\nAccount For Which Logon Failed:\r\n\tSecurity ID:\t\tS-1-0-0\r\n\tAccount Name:\t\t-\r\n\tAccount Domain:\t\t-\r\n\r\nFailure Information:\r\n\tFailure Reason:\t\t%%2304\r\n\tStatus:\t\t\t0xc000006d\r\n\tSub Status:\t\t0xc0000380\r\n\r\nProcess Information:\r\n\tCaller Process ID:\t0x9f0\r\n\tCaller Process Name:\tC:\\Windows\\System32\\svchost.exe\r\n\r\nNetwork Information:\r\n\tWorkstation Name:\t-\r\n\tSource Network Address:\t127.0.0.1\r\n\tSource Port:\t\t0\r\n\r\nDetailed Authentication Information:\r\n\tLogon Process:\t\tUser32 \r\n\tAuthentication Package:\tNegotiate\r\n\tTransited Services:\t-\r\n\tPackage Name (NTLM only):\t-\r\n\tKey Length:\t\t0",
  "Category": "User Account Management",
  "Opcode": "Info",
  "Level": "Information",
  "SubjectUserSid": "S-1-5-18",
  "SubjectUserName": "SRV01$",
  "SubjectDomainName": "WORKGROUP",
  "SubjectLogonId": "0x3e7",
  "TargetUserSid": "S-1-0-0",
  "TargetUserName": "-",
  "TargetDomainName": "-",
  "Status": "0xc000006d",
  "FailureReason": "%%2304",
  "FailureReasonResolved": "An Error occured during Logon.",
  "SubStatus": "0xc0000380",
  "LogonType": 2,
  "LogonProcessName": "User32 ",
  "AuthenticationPackageName": "Negotiate",
  "WorkstationName": "-",
  "TransmittedServices": "-",
  "LmPackageName": "-",
  "KeyLength": 0,
  "ProcessId": "0x9f0",
  "ProcessName": "C:\\Windows\\System32\\svchost.exe",
  "IpAddress": "127.0.0.1",
  "IpPort": "0",
  "EventReceivedTime": "2025-10-09T10:23:08.710835+02:00",
  "SourceModuleName": "windows_events",
  "SourceModuleType": "im_msvistalog",
  "PatternID": 2,
  "PatternName": "Security Event 4625"
}

Example 2. Using regular expression matching

The pattern file below defines a single pattern, which uses a regular expression to detect and enrich successful SSH authentication events.

pattern.xml

<?xml version='1.0' encoding='UTF-8'?>
<patterndb>
  <created>2025-10-09 01:02:03</created>
  <version>1</version>
  <!-- First pattern group in this file -->
  <group>
    <id>1</id>
    <name>ssh</name>
    <!-- Only apply this pattern group if the $SourceName field is "sshd" -->
    <matchfield>
      <name>SourceName</name>
      <type>exact</type>
      <value>sshd</value>
    </matchfield>
    <!-- First pattern in this pattern group -->
    <pattern>
      <id>1</id>
      <name>ssh auth success</name>
      <!-- Apply a regex pattern to the $Message field -->
      <matchfield>
        <name>Message</name>
        <type>regexp</type>
        <value>^Accepted (\S+) for (\S+) from (\S+) port \d+ ssh2</value>
        <!-- If the pattern matches, add fields from the captured strings -->
        <capturedfield>
          <name>AuthMethod</name>
          <type>STRING</type>
        </capturedfield>
        <capturedfield>
          <name>AccountName</name>
          <type>STRING</type>
        </capturedfield>
        <capturedfield>
          <name>SourceIPAddress</name>
          <type>IPADDR</type>
        </capturedfield>
      </matchfield>
      <!-- If the pattern matches, set additional fields -->
      <set>
        <field>
          <name>Status</name>
          <type>STRING</type>
          <value>success</value>
        </field>
        <field>
          <name>Action</name>
          <type>STRING</type>
          <value>authenticate</value>
        </field>
      </set>
	</pattern>
  </group>
</patterndb>

This configuration collects syslog messages from a file and parses them into structured data using the parse_syslog() procedure. This procedure populates the $SourceName and $Message fields used by the above pattern.

It then calls the match_pattern() procedure to process the record against the pattern file. Finally, it forwards the record over TCP in JSON format.

nxlog.conf

<Extension syslog>
    Module       xm_syslog (1)
</Extension>

<Extension pattern>
    Module       xm_pattern (2)
    PatternFile  '/opt/nxlog/etc/pattern.xml' (3)
</Extension>

<Extension json>
    Module       xm_json (4)
</Extension>

<Input system_logs>
    Module       im_file
    File         '/var/log/auth.log'
    <Exec>
        parse_syslog();
        match_pattern();
    </Exec>
</Input>

<Output siem>
    Module        om_tcp
    Host          192.168.1.123:1514
    Exec          to_json(); (5)
</Output>

<Route r1>
    Path          system_logs  => siem
</Route>

1	The xm_syslog module provides the parse_syslog() procedure.
2	The xm_pattern module provides the match_pattern() procedure.
3	The PatternFile directive specifies the path to the pattern file.
4	The xm_json module provides the to_json() procedure.
5	Converts the record to JSON and writes it to the `$raw_event` core field.

The following is a syslog message for a successful SSH login.

Input sample

<30>Oct 9 13:40:27 SRV01 sshd[26459]: Accepted publickey for john from 192.168.1.1 port 41193 ssh2

When the NXLog Agent configuration above processes this event, it transforms it into the following JSON object.

Output sample

{
  "EventReceivedTime": "2025-10-09T13:40:56.746802+02:00",
  "SourceModuleName": "system_logs",
  "SourceModuleType": "im_file",
  "Hostname": "SRV01",
  "SyslogFacilityValue": 3,
  "SyslogFacility": "DAEMON",
  "SyslogSeverityValue": 6,
  "SyslogSeverity": "INFO",
  "SeverityValue": 2,
  "Severity": "INFO",
  "EventTime": "2025-10-09T13:40:27.000000+02:00",
  "SourceName": "sshd",
  "ProcessID": 26459,
  "Message": "Accepted publickey for john from 192.168.1.1 port 41193 ssh2",
  "AuthMethod": "publickey",
  "AccountName": "john",
  "SourceIPAddress": "192.168.1.1",
  "PatternID": 1,
  "PatternName": "ssh auth success",
  "Status": "success",
  "Action": "authenticate"
}