Common Event Format (xm_cef)
This module provides functions for generating and parsing data in the ArcSight Common Event Format (CEF). For more information about the format, see Implementing ArcSight Common Event Format (CEF).
CEF uses syslog as a transport. For this reason, the xm_syslog module must be used in conjunction with xm_cef to parse or generate the additional syslog header unless the CEF data is used without syslog. See examples for both cases below. |
Configuration
The xm_cef module accepts the following directive in addition to the common module directives.
Optional directives
|
This boolean directive specifies that the to_cef() function or the to_cef() procedure should include fields having a leading dot ( |
Functions
The following functions are exported by xm_cef.
- string
to_cef()
-
Convert the specified fields to a single CEF formatted string.
Note that directive IncludeHiddenFields has an effect on extension fields in the output.
Procedures
The following procedures are exported by xm_cef.
parse_cef();
-
Parse the
$raw_event
field as CEF input.
parse_cef(string source);
-
Parse the given string as CEF format.
to_cef();
-
Format the specified fields as CEF and put this into the
$raw_event
field. The CEF header fields can be overridden by values contained in the following NXLog fields: $CEFVersion, $CEFDeviceVendor, $CEFDeviceProduct, $CEFDeviceVersion, $CEFSignatureID, $CEFName, and $CEFSeverity.Note that directive IncludeHiddenFields has an effect on extension fields in the output.
Fields
The following fields are used by xm_cef.
In addition to the fields listed below, the
parse_cef()
procedure will create a field for every key-value pair contained in the
Extension
CEF field, such as $act
, $cnt
, $dhost
, etc.
$CEFDeviceProduct
(type: string)-
The name of the software or appliance that sent the CEF-formatted event log. This field takes the value of the
Device Product
CEF header field.
$CEFDeviceVendor
(type: string)-
The vendor or manufacturer of the device that sent the CEF-formatted event log. This field takes the value of the
Device Vendor
CEF header field.
$CEFDeviceVersion
(type: string)-
The version of the software or appliance that sent the CEF-formatted event log. This field takes the value of the
Device Version
CEF header field.
$CEFName
(type: string)-
A human-readable description of the event. This field takes the value of the
Name
CEF header field.
$CEFSeverity
(type: integer)-
A numeric value between 1 and 10 that indicates the severity of the event, where:
-
1 is the lowest event severity,
-
10 is the highest event severity.
This field takes the value of the
Severity
CEF header field. -
$CEFSignatureID
(type: string)-
A unique identifier (unique per event type) used to determine the type of the reported event. This field takes the value of the
Signature ID
CEF header field.
$CEFVersion
(type: integer)-
The version of the CEF format. This field takes the value of the
Version
CEF header field.
Mapping of Windows Event Log data
NXLog Agent automatically assigns Windows Event Log data to the ArcSight Common Event Format fields. These are internal fields used by the xm_cef module that are not available as part of the log record, i.e., they cannot be renamed or referenced. If you want to use the value of one of these fields, you need to reference the mapped field according to the table.
Several CEF fields are mapped to multiple Windows Event Log fields. In this case, if the first Windows Event Log field is missing, the second field value will be applied, and so on.
Common Event Format field | Windows Event Log field |
---|---|
app |
Protocol |
cat |
Category |
cn1 |
LogonType |
cn2 |
KeyLength |
cs1 |
AccessMask |
cs4 |
Status |
cs5 |
AuthenticationPackageName |
cs6 |
member_user |
destinationServiceName |
ServiceName |
deviceFacility |
Channel |
dhost |
TargetServerName |
dntdom |
TargetDomainName |
dpriv |
PrivilegeList |
dproc |
ProcessName |
dpt |
DestPort |
dst |
DestAddress |
duid |
RemoteUserID |
duser |
TargetUserName |
dvchost |
Hostname |
externalId |
EventID |
fileId |
HandleId |
filePath |
RelativeTargetName |
filePermission |
Accesses |
fileType |
ObjectType |
fname |
ObjectName |
msg |
Message |
outcome |
EventType |
reason |
AccessReason |
rt |
EventTime If both fields are present, two |
shost |
Workstation |
sntdom |
SubjectDomainName |
sproc |
Application |
spt |
IpPort |
src |
IpAddress |
suid |
SubjectLogonId |
suser |
SubjectUserName |
See the configuration example below for how CEF fields can be manipulated.
Examples
This configuration collects both Windows Event Log and NXLog Agent internal messages, converts them to CEF with Syslog headers, and forwards the logs via UDP.
<Extension cef>
Module xm_cef
</Extension>
<Extension syslog>
Module xm_syslog
</Extension>
<Input internal>
Module im_internal
</Input>
<Input eventlog>
Module im_msvistalog
</Input>
<Output udp_output>
Module om_udp
Host 192.168.168.2:1514
Exec $Message = to_cef(); to_syslog_bsd();
</Output>
The following configuration receives CEF logs over UDP and converts the parsed data into JSON.
<Extension cef>
Module xm_cef
</Extension>
<Extension syslog>
Module xm_syslog
</Extension>
<Extension json>
Module xm_json
</Extension>
<Input udp_input>
Module im_udp
ListenAddr 0.0.0.0:1514
Exec parse_syslog(); parse_cef($Message);
</Input>
<Output file_output>
Module om_file
File "cef2json.log"
Exec to_json();
</Output>
This example collects Windows Event Log messages with Event ID 4776. It demonstrates how to modify the values of mapped CEF fields before converting the record to ArcSight CEF format.
Log Name: Security
Source: Microsoft-Windows-Security-Auditing
Date: 8/27/2021 1:18:16 AM
Event ID: 4776
Task Category: Credential Validation
Level: Information
Keywords: Audit Success
User: N/A
Computer: WIN-QML5T5DJJIA
Description:
The computer attempted to validate the credentials for an account.
Authentication Package: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0
Logon Account: Administrator
Source Workstation: WIN-QML5T5DJJIA
Error Code: 0x0
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Microsoft-Windows-Security-Auditing" Guid="{54849625-5478-4994-A5BA-3E3B0328C30D}" />
<EventID>4776</EventID>
<Version>0</Version>
<Level>0</Level>
<Task>14336</Task>
<Opcode>0</Opcode>
<Keywords>0x8020000000000000</Keywords>
<TimeCreated SystemTime="2021-08-27T08:18:16.961005700Z" />
<EventRecordID>659</EventRecordID>
<Correlation ActivityID="{DB66F5B6-9B6F-0000-F4F6-66DB6F9BD701}" />
<Execution ProcessID="544" ThreadID="588" />
<Channel>Security</Channel>
<Computer>WIN-QML5T5DJJIA</Computer>
<Security />
</System>
<EventData>
<Data Name="PackageName">MICROSOFT_AUTHENTICATION_PACKAGE_V1_0</Data>
<Data Name="TargetUserName">Administrator</Data>
<Data Name="Workstation">WIN-QML5T5DJJIA</Data>
<Data Name="Status">0x0</Data>
</EventData>
</Event>
This configuration uses the im_msvistalog module to read Windows Event Log messages with Event ID 4776.
The Exec block in the input module instance performs the following actions:
-
Sets the CEF header fields to custom values
-
Assigns values for CEF fields which cannot be automatically mapped
-
Removes unnecessary fields to shorten the output
All other Windows Event Log fields are automatically mapped to CEF fields as described in the Mapping of Windows Event Log data section. The result is converted to CEF using the to_cef() function.
define CEFNAME The domain controller attempted to validate the
define CEFSIGNATUREID Microsoft-Windows-Security-Auditing:4776
<Extension cef>
Module xm_cef
</Extension>
<Input from_eventlog>
Module im_msvistalog
<QueryXML>
<QueryList>
<Query Id="0">
<Select Path="Security">
*[System[Level=0 and (EventID=4776)]]
</Select>
</Query>
</QueryList>
</QueryXML>
<Exec>
# Overriding the default values
$CEFDeviceVendor = 'Microsoft';
$CEFDeviceProduct = 'Microsoft Windows';
$CEFDeviceVersion = ' ';
$CEFName = '%CEFNAME% credentials for an account';
$CEFSignatureID = '%CEFSIGNATUREID%';
# $TargetServerName is not available for Event ID 4776, $Hostname is used
# instead. Without it, $dhost will not be available in the output
$dhost = $Hostname;
# Deleting the $Message field to shorten the output
delete($Message);
# Converting to CEF
$raw_event = to_cef();
</Exec>
</Input>
CEF:0|Microsoft|Microsoft Windows| |Microsoft-Windows-Security-Auditing:4776|The domain controller attempted to validate the credentials for an account|7|rt=1630052296961 dvchost=WIN-QML5T5DJJIA Keywords=9232379236109516800 outcome=AUDIT_SUCCESS SeverityValue=2 Severity=INFO externalId=4776 SourceName=Microsoft-Windows-Security-Auditing ProviderGuid={54849625-5478-4994-A5BA-3E3B0328C30D} Version=0 TaskValue=14336 OpcodeValue=0 RecordNumber=659 ActivityID={DB66F5B6-9B6F-0000-F4F6-66DB6F9BD701} ExecutionProcessID=544 ExecutionThreadID=588 deviceFacility=Security cat=Credential Validation Opcode=Info cs5=MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 duser=Administrator shost=WIN-QML5T5DJJIA cs4=0x0 rt=1630052402607 SourceModuleName=from_eventlog SourceModuleType=im_msvistalog MessageSourceAddress=192.168.31.109 dhost=WIN-QML5T5DJJIA