Common Event Format (xm_cef)

This module provides functions for generating and parsing data in the ArcSight Common Event Format (CEF). For more information about the format, see Implementing ArcSight Common Event Format (CEF).

CEF uses syslog as a transport. For this reason, the xm_syslog module must be used in conjunction with xm_cef to parse or generate the additional syslog header unless the CEF data is used without syslog. See examples for both cases below.

Configuration

The xm_cef module accepts the following directive in addition to the common module directives.

Optional directives

IncludeHiddenFields

This boolean directive specifies that the to_cef() function or the to_cef() procedure should inlude fields having a leading dot (.) or underscore (_) in their names. The default is TRUE. If IncludeHiddenFields is set to TRUE, the generated CEF text will contain these otherwise excluded fields as extension fields.

Functions

The following functions are exported by xm_cef.

string to_cef(): Convert the specified fields to a single CEF formatted string.

Note that directive IncludeHiddenFields has an effect on extension fields in the output.

Procedures

The following procedures are exported by xm_cef.

parse_cef();: Parse the $raw_event field as CEF input.

parse_cef(string source);: Parse the given string as CEF format.

to_cef();: Format the specified fields as CEF and put this into the $raw_event field. The CEF header fields can be overridden by values contained in the following NXLog fields: $CEFVersion, $CEFDeviceVendor, $CEFDeviceProduct, $CEFDeviceVersion, $CEFSignatureID, $CEFName, and $CEFSeverity.

Note that directive IncludeHiddenFields has an effect on extension fields in the output.

Fields

The following fields are used by xm_cef.

In addition to the fields listed below, the parse_cef() procedure will create a field for every key-value pair contained in the Extension CEF field, such as $act, $cnt, $dhost, etc.

$CEFDeviceProduct (type: string): The name of the software or appliance that sent the CEF-formatted event log. This field takes the value of the Device Product CEF header field.

$CEFDeviceVendor (type: string): The vendor or manufacturer of the device that sent the CEF-formatted event log. This field takes the value of the Device Vendor CEF header field.

$CEFDeviceVersion (type: string): The version of the software or appliance that sent the CEF-formatted event log. This field takes the value of the Device Version CEF header field.

$CEFName (type: string): A human-readable description of the event. This field takes the value of the Name CEF header field.

$CEFSeverity (type: integer)

A numeric value between 1 and 10 that indicates the severity of the event, where:

1 is the lowest event severity,
10 is the highest event severity.

This field takes the value of the Severity CEF header field.

$CEFSignatureID (type: string): A unique identifier (unique per event type) used to determine the type of the reported event. This field takes the value of the Signature ID CEF header field.

$CEFVersion (type: integer): The version of the CEF format. This field takes the value of the Version CEF header field.

Mapping of Windows Event Log data

NXLog Agent automatically assigns Windows Event Log data to the ArcSight Common Event Format fields. These are internal fields used by the xm_cef module that are not available as part of the log record, i.e., they cannot be renamed or referenced. If you want to use the value of one of these fields, you need to reference the mapped field according to the table.

Several CEF fields are mapped to multiple Windows Event Log fields. In this case, if the first Windows Event Log field is missing, the second field value will be applied, and so on.

Table 1. Mapping of Windows Event Log to CEF
Common Event Format field	Windows Event Log field
app	Protocol
cat	Category
cn1	LogonType PreAuthType
cn2	KeyLength
cs1	AccessMask
cs4	Status
cs5	AuthenticationPackageName LmPackageName member_domain PackageName
cs6	member_user
destinationServiceName	ServiceName
deviceFacility	Channel
dhost	TargetServerName
dntdom	TargetDomainName
dpriv	PrivilegeList
dproc	ProcessName
dpt	DestPort
dst	DestAddress
duid	RemoteUserID TargetSid TargetUserSid
duser	TargetUserName
dvchost	Hostname
externalId	EventID
fileId	HandleId
filePath	RelativeTargetName
filePermission	Accesses
fileType	ObjectType
fname	ObjectName ShareName
msg	Message
outcome	EventType
reason	AccessReason FailureReason
rt	EventTime EventReceivedTime If both fields are present, two `rt` fields will be created to contain their values.
shost	Workstation WorkstationName
sntdom	SubjectDomainName
sproc	Application LogonProcessName
spt	IpPort SourcePort
src	IpAddress SourceAddress
suid	SubjectLogonId SubjectUserSid
suser	SubjectUserName

See the configuration example below for how CEF fields can be manipulated.

Examples

Example 1. Sending Windows Event Log events formatted in CEF over UDP

This configuration collects both Windows Event Log and NXLog Agent internal messages, converts them to CEF with Syslog headers, and forwards the logs via UDP.

nxlog.conf

<Extension cef>
    Module  xm_cef
</Extension>

<Extension syslog>
    Module  xm_syslog
</Extension>

<Input internal>
    Module  im_internal
</Input>

<Input eventlog>
    Module  im_msvistalog
</Input>

<Output udp_output>
    Module  om_udp
    Host    192.168.168.2:1514
    Exec    $Message = to_cef(); to_syslog_bsd();
</Output>

Example 2. Parsing CEF

The following configuration receives CEF logs over UDP and converts the parsed data into JSON.

nxlog.conf

<Extension cef>
    Module      xm_cef
</Extension>

<Extension syslog>
    Module      xm_syslog
</Extension>

<Extension json>
    Module      xm_json
</Extension>

<Input udp_input>
    Module      im_udp
    ListenAddr  0.0.0.0:1514
    Exec        parse_syslog(); parse_cef($Message);
</Input>

<Output file_output>
    Module      om_file
    File        "cef2json.log"
    Exec        to_json();
</Output>

Example 3. Converting Windows Event Log messages to CEF using field mapping

This example collects Windows Event Log messages with Event ID 4776. It demonstrates how to modify the values of mapped CEF fields before converting the record to ArcSight CEF format.

Windows Event Log record sample

Log Name:      Security
Source:        Microsoft-Windows-Security-Auditing
Date:          8/27/2021 1:18:16 AM
Event ID:      4776
Task Category: Credential Validation
Level:         Information
Keywords:      Audit Success
User:          N/A
Computer:      WIN-QML5T5DJJIA
Description:
The computer attempted to validate the credentials for an account.

Authentication Package: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0
Logon Account:  Administrator
Source Workstation: WIN-QML5T5DJJIA
Error Code: 0x0
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-Security-Auditing" Guid="{54849625-5478-4994-A5BA-3E3B0328C30D}" />
    <EventID>4776</EventID>
    <Version>0</Version>
    <Level>0</Level>
    <Task>14336</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8020000000000000</Keywords>
    <TimeCreated SystemTime="2021-08-27T08:18:16.961005700Z" />
    <EventRecordID>659</EventRecordID>
    <Correlation ActivityID="{DB66F5B6-9B6F-0000-F4F6-66DB6F9BD701}" />
    <Execution ProcessID="544" ThreadID="588" />
    <Channel>Security</Channel>
    <Computer>WIN-QML5T5DJJIA</Computer>
    <Security />
  </System>
  <EventData>
    <Data Name="PackageName">MICROSOFT_AUTHENTICATION_PACKAGE_V1_0</Data>
    <Data Name="TargetUserName">Administrator</Data>
    <Data Name="Workstation">WIN-QML5T5DJJIA</Data>
    <Data Name="Status">0x0</Data>
  </EventData>
</Event>

This configuration uses the im_msvistalog module to read Windows Event Log messages with Event ID 4776.

The Exec block in the input module instance performs the following actions:

Sets the CEF header fields to custom values
Assigns values for CEF fields which cannot be automatically mapped
Removes unnecessary fields to shorten the output

All other Windows Event Log fields are automatically mapped to CEF fields as described in the Mapping of Windows Event Log data section. The result is converted to CEF using the to_cef() function.

nxlog.conf

define CEFNAME          The domain controller attempted to validate the
define CEFSIGNATUREID   Microsoft-Windows-Security-Auditing:4776

<Extension cef>
    Module              xm_cef
</Extension>

<Input from_eventlog>
    Module              im_msvistalog
    <QueryXML>
        <QueryList>
            <Query Id="0">
                <Select Path="Security">
                    *[System[Level=0 and (EventID=4776)]]
                </Select>
            </Query>
        </QueryList>
    </QueryXML>
    <Exec>
        # Overriding the default values
        $CEFDeviceVendor = 'Microsoft';
        $CEFDeviceProduct = 'Microsoft Windows';
        $CEFDeviceVersion = ' ';
        $CEFName = '%CEFNAME% credentials for an account';
        $CEFSignatureID = '%CEFSIGNATUREID%';

        # $TargetServerName is not available for Event ID 4776, $Hostname is used
        # instead. Without it, $dhost will not be available in the output
        $dhost = $Hostname;

        # Deleting the $Message field to shorten the output
        delete($Message);

        # Converting to CEF
        $raw_event = to_cef();
    </Exec>
</Input>

Output sample in CEF format

CEF:0|Microsoft|Microsoft Windows| |Microsoft-Windows-Security-Auditing:4776|The domain controller attempted to validate the credentials for an account|7|rt=1630052296961 dvchost=WIN-QML5T5DJJIA Keywords=9232379236109516800 outcome=AUDIT_SUCCESS SeverityValue=2 Severity=INFO externalId=4776 SourceName=Microsoft-Windows-Security-Auditing ProviderGuid={54849625-5478-4994-A5BA-3E3B0328C30D} Version=0 TaskValue=14336 OpcodeValue=0 RecordNumber=659 ActivityID={DB66F5B6-9B6F-0000-F4F6-66DB6F9BD701} ExecutionProcessID=544 ExecutionThreadID=588 deviceFacility=Security cat=Credential Validation Opcode=Info cs5=MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 duser=Administrator shost=WIN-QML5T5DJJIA cs4=0x0 rt=1630052402607 SourceModuleName=from_eventlog SourceModuleType=im_msvistalog MessageSourceAddress=192.168.31.109 dhost=WIN-QML5T5DJJIA