Process ArcSight Common Event Format (CEF) logs
CEF is a text-based log format developed by ArcSight™ and used by HP ArcSight™ products. It uses syslog as transport. The full format includes a syslog header or "prefix", a CEF "header", and a CEF "extension". The extension contains a list of key-value pairs. Standard key names are provided, and user-defined extensions can be used for additional key names. In some cases, the CEF format is used with the syslog header omitted.
NXLog can be configured to collect or forward logs in Common Event Format (CEF). NXLog has a dedicated xm_cef module for collecting and parsing CEF logs. NXLog can also be configured to generate and forward CEF logs using the same module. Additionally, NXLog’s xm_csv and xm_kvp modules can be used as an alternative to xm_cef for the same purpose of handling CEF logs.
Jan 11 10:25:39 host CEF:Version|Device Vendor|Device Product|Device Version|Device Event Class ID|Name|Severity|[Extension]Oct 12 04:16:11 localhost CEF:0|nxlog.org|nxlog|2.7.1243|Executable Code was Detected|Advanced exploit detected|100|src=192.168.255.110 spt=46117 dst=172.25.212.204 dpt=80Collecting and parsing CEF logs
NXLog Enterprise Edition can be configured to collect and parse CEF logs with the xm_cef module.
The ArcSight™ Logger can be configured to send CEF logs via TCP with the following steps:
-
Log in to the Logger control panel.
-
Browse to .
-
Click Add to create a new Forwarder:
-
Name:
nxlog -
Type:
TCP Forwarder -
Type of Filter:
Unified Query
-
-
Click Next to proceed to edit the new Forwarder:
-
Query: (define as required)
-
IP/Host: (enter the IP address or hostname of the system running NXLog)
-
Port:
1514
-
-
Click Save.
With this configuration, NXLog will collect CEF logs via TCP, convert them to plain JSON format, and save them to file.
<Extension cef>
Module xm_cef
</Extension>
<Extension json>
Module xm_json
</Extension>
<Extension syslog>
Module xm_syslog
</Extension>
<Input logger_tcp>
Module im_tcp
Host 0.0.0.0
Port 1514
Exec parse_syslog(); parse_cef($Message);
</Input>
<Output json_file>
Module om_file
File '/var/log/json'
Exec to_json();
</Output>Generating and forwarding CEF logs
NXLog Enterprise Edition can be configured to generate and forward CEF logs with the xm_cef module.
The ArcSight™ Logger can be configured to receive CEF logs via TCP with the following steps:
-
Log in to the Logger control panel.
-
Browse to in the navigation menu.
-
Click Add to create a new Receiver:
-
Name:
nxlog -
Type:
CEF TCP Receiver
-
-
Click Next to proceed to edit the new Receiver:
-
Port:
574 -
Encoding:
UTF-8 -
Source Type:
CEF
-
-
Click Save.
With this configuration, NXLog will read syslog logs from file, convert them to CEF, and forward them to the ArcSight Logger via TCP. Default values will be used for the CEF header unless corresponding fields are defined in the event record (see the to_cef() procedure in the NXLog Reference Manual for a list of fields).
<Extension cef>
Module xm_cef
</Extension>
<Extension syslog>
Module xm_syslog
</Extension>
<Input messages_file>
Module im_file
File '/var/log/messages'
Exec parse_syslog();
</Input>
<Output logger_tcp>
Module om_tcp
Host 192.168.1.1
Port 574
Exec $Message = to_cef(); to_syslog_bsd();
</Output>Using xm_csv and xm_kvp modules to handle CEF logs
Because NXLog Community Edition does not include the xm_cef module, the xm_csv and xm_kvp modules may be used instead to handle CEF logs.
Here, the xm_csv module is used to parse the pipe-delimited CEF header, while the xm_kvp module is used to parse the space-delimited key-value pairs in the CEF extension. The required extension configurations are shown below.
<Extension cef_header>
Module xm_csv
Fields $Version, $Device_Vendor, $Device_Product, $Device_Version, \
$Signature_ID, $Name, $Severity, $_Extension
Delimiter |
QuoteMethod None
</Extension>
<Extension cef_extension>
Module xm_kvp
KVDelimiter '='
KVPDelimiter ' '
QuoteMethod None
</Extension>
<Extension syslog>
Module xm_syslog
</Extension>For CEF input, use an input instance like this one:
<Input in>
Module im_tcp
Host 0.0.0.0
Port 1514
<Exec>
parse_syslog();
cef_header->parse_csv($Message);
cef_extension->parse_kvp($_Extension);
</Exec>
</Input>For CEF output, use an output instance like this one:
<Output out>
Module om_tcp
Host 192.168.1.1
Port 574
<Exec>
$_Extension = cef_extension->to_kvp();
$Version = 'CEF:0';
$Device_Vendor = 'NXLog';
$Device_Product = 'NXLog';
$Device_Version = '';
$Signature_ID = '0';
$Name = '-';
$Severity = '';
$Message = cef_header->to_csv();
to_syslog_bsd();
</Exec>
</Output>
