Classify logs with NXLog Agent

Classifying logs enables you to efficiently troubleshoot issues, monitor system health, and identify security threats. In other words, it helps to make sense of your logs. SIEM solutions usually use a standard taxonomy, or you can create your own taxonomy to classify log events. Standard taxonomies might include error, warning, information, security, and more.

Below, we provide examples of classifying log records with NXLog Agent. See also Normalize logs, a closely related topic, for more examples of log classification.

Map log classification fields

In many instances, you can classify logs during parsing. However, if the log records are already structured, or the input module parses logs automatically, you only need to match and map the relevant fields.

Example 1. Classifying Windows Security events

Below is an example of some fields included in Windows Security event 4624: An account was successfully logged on.

Table 1. Windows successful logon event
Field	Value
$EventType	AUDIT_SUCCESS
$EventID	4624
$SourceName	Microsoft-Windows-Security-Auditing
$Channel	Security
$Category	Logon
$TargetUserSid	S-1-5-18
$TargetUserName	linda
$TargetDomainName	WINHOST
$Status	0xc000006d
$LogonType	5

This configuration collects Windows events with the im_msvistalog input module, which automatically parses the events into structured data. It filters for successful logon events and enriches matching event records with $EventAction, $EventStatus, and $AccountName fields.

Finally, it converts records to JSON format using the to_json() procedure of the xm_json module.

nxlog.conf

<Extension json>
    Module    xm_json
</Extension>

<Input windows_security>
    Module    im_msvistalog
    <Exec>
        if ($EventID == 4624) and
           ($SourceName == 'Microsoft-Windows-Security-Auditing')
        {
            $EventAction = 'Authenticate';
            $EventStatus = 'Success';
            $AccountName = $TargetUserName;
        }
        to_json();
    </Exec>
</Input>

The following JSON object shows the same log record after NXLog Agent processed it. You can see the last three fields added by the configuration above.

Output sample

{
  "EventTime": "2023-10-31 05:49:42",
  "Hostname": "PC-1",
  "Keywords": "9232379236109516800",
  "EventType": "AUDIT_SUCCESS",
  "SeverityValue": 2,
  "Severity": "INFO",
  "EventID": 4624,
  "SourceName": "Microsoft-Windows-Security-Auditing",
  "ProviderGuid": "{54849625-5478-4994-A5BA-3E3B0328C30D}",
  "Version": 2,
  "TaskValue": 12544,
  "OpcodeValue": 0,
  "RecordNumber": 2747,
  "ActivityID": "{D3963786-6526-0000-8939-96D32665D601}",
  "ExecutionProcessID": 576,
  "ExecutionThreadID": 2816,
  "Channel": "Security",
  "Category": "Logon",
  "Opcode": "Info",
  "SubjectUserSid": "S-1-5-18",
  "SubjectUserName": "PC-1$",
  "SubjectDomainName": "WORKGROUP",
  "SubjectLogonId": "0x3e7",
  "TargetUserSid": "S-1-0-0",
  "TargetUserName": "linda",
  "TargetDomainName": "WINHOST",
  "TargetLogonId": "0x3e7",
  "LogonType": "5",
  "LogonProcessName": "Advapi ",
  "AuthenticationPackageName": "Negotiate",
  "WorkstationName": "-",
  "LogonGuid": "{00000000-0000-0000-0000-000000000000}",
  "TransmittedServices": "-",
  "LmPackageName": "-",
  "KeyLength": "0",
  "ProcessId": "0x238",
  "ProcessName": "C:\\Windows\\System32\\services.exe",
  "IpAddress": "-",
  "IpPort": "-",
  "ImpersonationLevel": "%%1833",
  "RestrictedAdminMode": "-",
  "TargetOutboundUserName": "-",
  "TargetOutboundDomainName": "-",
  "VirtualAccount": "%%1843",
  "TargetLinkedLogonId": "0x0",
  "ElevatedToken": "%%1842",
  "EventReceivedTime": "2023-10-31 05:50:47",
  "SourceModuleName": "windows_security",
  "SourceModuleType": "im_msvistalog",
  "EventAction": "Authenticate",
  "EventStatus": "Success",
  "AccountName": "linda"
}

Parse logs with a regular expression

If your logs are unstructured, you must first parse them into fields. A common way to do this is with a regular expression.

Example 2. Classifying logs with a regular expression

Below is an example of an SSH login event from the Linux auth.log file.

SSH login event

Oct 31 15:02:27 SERVER-1 sshd[261799]: pam_unix(sshd:session): session opened for user root by (uid=0)

This configuration collects logs from the Linux authentication log file with the im_file input module and parses them into structured data using the parse_syslog_bsd() procedure of the xm_syslog module. This procedure adds the $Message field to the event record.

It then uses a regular expression to filter for SSH login events and enriches matching event records with $AccountName, $AccountID, and $EventAction fields. Finally, it converts records to JSON format using the to_json() procedure of the xm_json module.

nxlog.conf

<Extension syslog>
    Module    xm_syslog
</Extension>

<Extension json>
    Module    xm_json
</Extension>

<Input linux_auth>
    Module    im_file
    File      '/var/log/auth.log'
    <Exec>
        parse_syslog_bsd();
        if $Message =~ /(?x)^pam_unix\(sshd:session\):\ session\ opened\ for\ user\ (\S+)
                       \ by\ \(uid=(\d+)\)/ 
        {
            $AccountName = $1; 
            $AccountID = integer($2);
            $EventAction = 'Authenticate';
        }
        to_json();
    </Exec>
</Input>

The following JSON object shows the same log record after NXLog Agent processed it. You can see the last three fields added by the configuration above.

Output sample

{
  "EventReceivedTime": "2023-10-31T15:03:14.666435+01:00",
  "SourceModuleName": "linux_auth",
  "SourceModuleType": "im_file",
  "Hostname": "SERVER-1",
  "SyslogFacilityValue": 1,
  "SyslogFacility": "USER",
  "SyslogSeverityValue": 5,
  "SyslogSeverity": "NOTICE",
  "SeverityValue": 2,
  "Severity": "INFO",
  "EventTime": "2023-10-31T15:02:27.000000+01:00",
  "SourceName": "sshd",
  "ProcessID": 261799,
  "Message": "pam_unix(sshd:session): session opened for user root by (uid=0)",
  "AccountName": "root",
  "AccountID": 0,
  "EventAction": "Authenticate"
}

Parse logs with a pattern file

If your configuration includes many patterns, evaluating them linearly can slow down log processing. A more efficient approach is to use the xm_pattern module, which prioritizes pattern-matching by popularity.

Example 3. Classifying logs with xm_pattern

The first step is to define the patterns to parse your logs. In this example, we define a pattern for SSH login events.

SSH login event

Oct 31 15:02:27 SERVER-1 sshd[261799]: pam_unix(sshd:session): session opened for user root by (uid=0)

The following NXLog pattern file defines an ssh group that uses the $SourceName field. If an event matches the source, it parses the $Message field with a regular expression that matches SSH login events. See the pattern database schema in the NXLog Agent Reference Manual for details of the XML schema.

patterndb.xml

<?xml version='1.0' encoding='UTF-8'?>
<patterndb>
    <created>2023-10-16 10:21:24</created>
    <version>1</version>
    <!-- First and only pattern group in this file -->
    <group>
        <name>ssh</name>
        <id>40</id>
        <!-- Only try to match this group if $SourceName == "sshd" -->
        <matchfield>
            <name>SourceName</name>
            <type>exact</type>
            <value>sshd</value>
        </matchfield>
        <!-- First and only pattern in this pattern group -->
        <pattern>
            <id>2</id>
            <name>ssh pam session opened</name>
            <!-- Match regular expression on $Message field -->
            <matchfield>
                <name>Message</name>
                <type>regexp</type>
                <value>^pam_unix\(sshd:session\): session opened for user (\S+) by \(uid=(\d+)\)</value>
                <!-- Set fields from the 2 capturing groups -->
                <capturedfield>
                    <name>AccountName</name>
                    <type>string</type>
                </capturedfield>
                <capturedfield>
                    <name>AccountID</name>
                    <type>string</type>
                </capturedfield>
            </matchfield>
            <!-- Set addition $EventAction field -->
            <set>
                <field>
                    <name>EventAction</name>
                    <type>string</type>
                    <value>Authenticate</value>
                </field>
            </set>
        </pattern>
    </group>
</patterndb>

You can now use the pattern file with NXLog Agent. This configuration collects logs from the Linux authentication log file with the im_file input module and parses records into structured data using the parse_syslog_bsd() procedure of the xm_syslog module. This procedure adds the $SourceName and $Message fields to the event record.

It then uses the match_pattern() procedure of the xm_pattern module to process the record against the patterns defined in the pattern database. Finally, it converts records to JSON format using the to_json() procedure of the xm_json module.

nxlog.conf

<Extension json>
    Module         xm_json
</Extension>

<Extension syslog>
    Module         xm_syslog
</Extension>

<Extension pattern>
    Module         xm_pattern
    PatternFile    '/path/to/patterndb.xml'
</Extension>

<Input linux_auth>
    Module         im_file
    File           '/var/log/auth.log' 
    <Exec>
        parse_syslog_bsd();
        match_pattern();
        to_json();
    </Exec>	
</Input>

The following JSON object shows the same log record after NXLog Agent processed it. You can see the last five fields added by the configuration above.

Output sample

{
  "EventReceivedTime":"2023-10-31T15:03:14.499824+01:00",
  "SourceModuleName":"linux_auth",
  "SourceModuleType":"im_file",
  "Hostname":"SERVER-1",
  "SyslogFacilityValue":1,
  "SyslogFacility":"USER",
  "SyslogSeverityValue":5,
  "SyslogSeverity":"NOTICE",
  "SeverityValue":2,
  "Severity":"INFO",
  "EventTime":"2023-10-31T15:02:27.000000+01:00",
  "SourceName":"sshd",
  "ProcessID":261799,
  "Message":"pam_unix(sshd:session): session opened for user root by (uid=0)",
  "AccountName":"root",
  "AccountID":0,
  "PatternID":42,
  "PatternName":"ssh pam session opened",
  "EventAction":"Authenticate"
}