Common Event Format (xm_cef)

This module provides functions for generating and parsing data in the ArcSight Common Event Format (CEF). For more information about the format, see Implementing ArcSight Common Event Format (CEF).

CEF uses syslog as a transport. For this reason, the xm_syslog module must be used in conjunction with xm_cef to parse or generate the additional syslog header unless the CEF data is used without syslog. See examples for both cases below.

Configuration

The xm_cef module accepts the following directive in addition to the common module directives.

Optional directives

IncludeHiddenFields

This boolean directive specifies that the to_cef() function or the to_cef() procedure should inlude fields having a leading dot (.) or underscore (_) in their names. The default is TRUE. If IncludeHiddenFields is set to TRUE, the generated CEF text will contain these otherwise excluded fields as extension fields.

Functions

The following functions are exported by xm_cef.

string to_cef()

Convert the specified fields to a single CEF formatted string.

Note that directive IncludeHiddenFields has an effect on extension fields in the output.

Procedures

The following procedures are exported by xm_cef.

parse_cef();

Parse the $raw_event field as CEF input.

parse_cef(string source);

Parse the given string as CEF format.

to_cef();

Format the specified fields as CEF and put this into the $raw_event field. The CEF header fields can be overridden by values contained in the following NXLog fields: $CEFVersion, $CEFDeviceVendor, $CEFDeviceProduct, $CEFDeviceVersion, $CEFSignatureID, $CEFName, and $CEFSeverity.

Note that directive IncludeHiddenFields has an effect on extension fields in the output.

Fields

The following fields are used by xm_cef.

In addition to the fields listed below, the parse_cef() procedure will create a field for every key-value pair contained in the Extension CEF field, such as $act, $cnt, $dhost, etc.

$CEFDeviceProduct (type: string)

The name of the software or appliance that sent the CEF-formatted event log. This field takes the value of the Device Product CEF header field.

$CEFDeviceVendor (type: string)

The vendor or manufacturer of the device that sent the CEF-formatted event log. This field takes the value of the Device Vendor CEF header field.

$CEFDeviceVersion (type: string)

The version of the software or appliance that sent the CEF-formatted event log. This field takes the value of the Device Version CEF header field.

$CEFName (type: string)

A human-readable description of the event. This field takes the value of the Name CEF header field.

$CEFSeverity (type: integer)

A numeric value between 1 and 10 that indicates the severity of the event, where:

  • 1 is the lowest event severity,

  • 10 is the highest event severity.

This field takes the value of the Severity CEF header field.

$CEFSignatureID (type: string)

A unique identifier (unique per event type) used to determine the type of the reported event. This field takes the value of the Signature ID CEF header field.

$CEFVersion (type: integer)

The version of the CEF format. This field takes the value of the Version CEF header field.

Mapping of Windows Event Log data

NXLog Agent automatically assigns Windows Event Log data to the ArcSight Common Event Format fields. These are internal fields used by the xm_cef module that are not available as part of the log record, i.e., they cannot be renamed or referenced. If you want to use the value of one of these fields, you need to reference the mapped field according to the table.

Several CEF fields are mapped to multiple Windows Event Log fields. In this case, if the first Windows Event Log field is missing, the second field value will be applied, and so on.

Table 1. Mapping of Windows Event Log to CEF
Common Event Format field Windows Event Log field

app

Protocol

cat

Category

cn1

LogonType
PreAuthType

cn2

KeyLength

cs1

AccessMask

cs4

Status

cs5

AuthenticationPackageName
LmPackageName
member_domain
PackageName

cs6

member_user

destinationServiceName

ServiceName

deviceFacility

Channel

dhost

TargetServerName

dntdom

TargetDomainName

dpriv

PrivilegeList

dproc

ProcessName

dpt

DestPort

dst

DestAddress

duid

RemoteUserID
TargetSid
TargetUserSid

duser

TargetUserName

dvchost

Hostname

externalId

EventID

fileId

HandleId

filePath

RelativeTargetName

filePermission

Accesses

fileType

ObjectType

fname

ObjectName
ShareName

msg

Message

outcome

EventType

reason

AccessReason
FailureReason

rt

EventTime
EventReceivedTime

If both fields are present, two rt fields will
be created to contain their values.

shost

Workstation
WorkstationName

sntdom

SubjectDomainName

sproc

Application
LogonProcessName

spt

IpPort
SourcePort

src

IpAddress
SourceAddress

suid

SubjectLogonId
SubjectUserSid

suser

SubjectUserName

See the configuration example below for how CEF fields can be manipulated.

Examples

Example 1. Sending Windows Event Log events formatted in CEF over UDP

This configuration collects both Windows Event Log and NXLog Agent internal messages, converts them to CEF with Syslog headers, and forwards the logs via UDP.

nxlog.conf
<Extension cef>
    Module  xm_cef
</Extension>

<Extension syslog>
    Module  xm_syslog
</Extension>

<Input internal>
    Module  im_internal
</Input>

<Input eventlog>
    Module  im_msvistalog
</Input>

<Output udp_output>
    Module  om_udp
    Host    192.168.168.2:1514
    Exec    $Message = to_cef(); to_syslog_bsd();
</Output>
Example 2. Parsing CEF

The following configuration receives CEF logs over UDP and converts the parsed data into JSON.

nxlog.conf
<Extension cef>
    Module      xm_cef
</Extension>

<Extension syslog>
    Module      xm_syslog
</Extension>

<Extension json>
    Module      xm_json
</Extension>

<Input udp_input>
    Module      im_udp
    ListenAddr  0.0.0.0:1514
    Exec        parse_syslog(); parse_cef($Message);
</Input>

<Output file_output>
    Module      om_file
    File        "cef2json.log"
    Exec        to_json();
</Output>
Example 3. Converting Windows Event Log messages to CEF using field mapping

This example collects Windows Event Log messages with Event ID 4776. It demonstrates how to modify the values of mapped CEF fields before converting the record to ArcSight CEF format.

Windows Event Log record sample
Log Name:      Security
Source:        Microsoft-Windows-Security-Auditing
Date:          8/27/2021 1:18:16 AM
Event ID:      4776
Task Category: Credential Validation
Level:         Information
Keywords:      Audit Success
User:          N/A
Computer:      WIN-QML5T5DJJIA
Description:
The computer attempted to validate the credentials for an account.

Authentication Package: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0
Logon Account:  Administrator
Source Workstation: WIN-QML5T5DJJIA
Error Code: 0x0
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-Security-Auditing" Guid="{54849625-5478-4994-A5BA-3E3B0328C30D}" />
    <EventID>4776</EventID>
    <Version>0</Version>
    <Level>0</Level>
    <Task>14336</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8020000000000000</Keywords>
    <TimeCreated SystemTime="2021-08-27T08:18:16.961005700Z" />
    <EventRecordID>659</EventRecordID>
    <Correlation ActivityID="{DB66F5B6-9B6F-0000-F4F6-66DB6F9BD701}" />
    <Execution ProcessID="544" ThreadID="588" />
    <Channel>Security</Channel>
    <Computer>WIN-QML5T5DJJIA</Computer>
    <Security />
  </System>
  <EventData>
    <Data Name="PackageName">MICROSOFT_AUTHENTICATION_PACKAGE_V1_0</Data>
    <Data Name="TargetUserName">Administrator</Data>
    <Data Name="Workstation">WIN-QML5T5DJJIA</Data>
    <Data Name="Status">0x0</Data>
  </EventData>
</Event>

This configuration uses the im_msvistalog module to read Windows Event Log messages with Event ID 4776.

The Exec block in the input module instance performs the following actions:

  • Sets the CEF header fields to custom values

  • Assigns values for CEF fields which cannot be automatically mapped

  • Removes unnecessary fields to shorten the output

All other Windows Event Log fields are automatically mapped to CEF fields as described in the Mapping of Windows Event Log data section. The result is converted to CEF using the to_cef() function.

nxlog.conf
define CEFNAME          The domain controller attempted to validate the
define CEFSIGNATUREID   Microsoft-Windows-Security-Auditing:4776

<Extension cef>
    Module              xm_cef
</Extension>

<Input from_eventlog>
    Module              im_msvistalog
    <QueryXML>
        <QueryList>
            <Query Id="0">
                <Select Path="Security">
                    *[System[Level=0 and (EventID=4776)]]
                </Select>
            </Query>
        </QueryList>
    </QueryXML>
    <Exec>
        # Overriding the default values
        $CEFDeviceVendor = 'Microsoft';
        $CEFDeviceProduct = 'Microsoft Windows';
        $CEFDeviceVersion = ' ';
        $CEFName = '%CEFNAME% credentials for an account';
        $CEFSignatureID = '%CEFSIGNATUREID%';

        # $TargetServerName is not available for Event ID 4776, $Hostname is used
        # instead. Without it, $dhost will not be available in the output
        $dhost = $Hostname;

        # Deleting the $Message field to shorten the output
        delete($Message);

        # Converting to CEF
        $raw_event = to_cef();
    </Exec>
</Input>
Output sample in CEF format
CEF:0|Microsoft|Microsoft Windows| |Microsoft-Windows-Security-Auditing:4776|The domain controller attempted to validate the credentials for an account|7|rt=1630052296961 dvchost=WIN-QML5T5DJJIA Keywords=9232379236109516800 outcome=AUDIT_SUCCESS SeverityValue=2 Severity=INFO externalId=4776 SourceName=Microsoft-Windows-Security-Auditing ProviderGuid={54849625-5478-4994-A5BA-3E3B0328C30D} Version=0 TaskValue=14336 OpcodeValue=0 RecordNumber=659 ActivityID={DB66F5B6-9B6F-0000-F4F6-66DB6F9BD701} ExecutionProcessID=544 ExecutionThreadID=588 deviceFacility=Security cat=Credential Validation Opcode=Info cs5=MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 duser=Administrator shost=WIN-QML5T5DJJIA cs4=0x0 rt=1630052402607 SourceModuleName=from_eventlog SourceModuleType=im_msvistalog MessageSourceAddress=192.168.31.109 dhost=WIN-QML5T5DJJIA