Micro Focus ArcSight Logger
ArcSight Logger is a log management solution that provides secure storage, efficient search, reporting, and analysis of log data. NXLog Agent can integrate with ArcSight Logger by sending log data to it in Common Event Format (CEF) over UDP or TCP. NXLog Agent also supports receiving logs from an ArcSight Logger Forwarder.
The instructions and examples in this guide were tested with the software-based solution of ArcSight Logger. In some cases, these may vary from the appliance-based solution. Consult the ArcSight documentation for instructions related to your version. |
Sending logs to ArcSight Logger
For ArcSight Logger to receive logs over UDP or TCP, a receiver must be created from the Logger user interface. Follow these steps to create a CEF TCP Receiver for use with NXLog Agent.
-
Log in to the Logger user interface with an account that has permission to modify the configuration.
-
From the left menu, navigate to Configuration > Data > Receivers.
-
Click Add to create a new receiver:
-
Provide a unique name for the receiver.
-
Select CEF TCP Receiver for Type.
-
Click Next to continue with the configuration.
-
Choose the IP/Host network connection that will be used to listen for incoming data.
-
Enter the Port. The examples in this guide use port 6514.
-
Select UTF-8 character encoding.
-
The Source Type will be set to CEF by default, this is the only option for this type of receiver.
-
Ensure that the Enable option is checked and click Save.
-
Sending CEF logs over TCP
NXLog Agent supports log collection from various sources and can transform the data to CEF using the xm_cef extension. The examples below demonstrate how you can collect logs from Windows Event Log or Linux syslog and forward them to ArcSight Logger over TCP.
This configuration uses the im_msvistalog input module to collect Application, Security, and System logs from Windows Event Log. Logs are converted to CEF using the to_cef() procedure of the xm_cef module, and then forwarded to ArcSight Logger over TCP using the om_tcp output module.
Log records are enriched with additional CEF fields to facilitate searching
in ArcSight Logger. The include_stdout
directive is used to execute a PowerShell script that retrieves the Windows
OS version, and injects this value in the NXLog Agent configuration as the
%OSNAME%
constant. This constant is used to set the $CEFDeviceVersion
field.
include_stdout C:\Program Files\nxlog\conf\windows-os-version.cmd
<Extension cef>
Module xm_cef
</Extension>
<Input eventlog>
Module im_msvistalog
<QueryXML>
<QueryList>
<Query Id='0'>
<Select Path='Application'>*</Select>
<Select Path='Security'>*</Select>
<Select Path='System'>*</Select>
</Query>
</QueryList>
</QueryXML>
<Exec>
$CEFDeviceVendor = "Microsoft";
$CEFDeviceProduct = "Microsoft Windows";
# This value is obtained through executing a command file.
# See the include_stdout directive above.
$CEFDeviceVersion = '%OSNAME%';
# This is required for some of the included searches in
# ArcSight Logger to work.
$CEFSignatureID = $SourceName+":"+$EventID;
to_cef();
</Exec>
</Input>
<Output arcsight_logger>
Module om_tcp
Host <arcsight_logger_ip>:6514
</Output>
@( Set "_= (
REM " ) <#
)
@Echo Off
SetLocal EnableExtensions DisableDelayedExpansion
set powershell=powershell.exe
REM Use this if you need 64-bit PowerShell (has no effect on 32-bit systems).
REM if defined PROCESSOR_ARCHITEW6432 (
REM set powershell=%SystemRoot%\SysNative\WindowsPowerShell\v1.0\powershell.exe
REM )
REM Use this if you need 32-bit PowerShell.
REM if NOT %PROCESSOR_ARCHITECTURE% == x86 (
REM set powershell=%SystemRoot%\SysWOW64\WindowsPowerShell\v1.0\powershell.exe
REM )
%powershell% -ExecutionPolicy Bypass -NoProfile ^
-Command "iex ((gc '%~f0') -join [char]10)"
EndLocal & Exit /B %ErrorLevel%
#>
# PowerShell code starts here.
# To make NXLog return an error, write to standard error and exit 1
if ($false) {
[Console]::Error.WriteLine("This is an error")
exit 1
}
else {
# Anything written to standard output is used as configuration content
$osname = (Get-WmiObject -Class Win32_OperatingSystem).caption
Write-Output "define OSNAME $osname"
}
The cmd file needs to be saved in the path specified by the
include_stdout directive. The user running NXLog Agent must have
permission to execute the file.
|
CEF:0|Microsoft|Microsoft Windows|Microsoft Windows Server 2016 Standard|SceCli:1704|-|7|rt=1611907510135 dvchost=WIN2016.example.com Keywords=36028797018963968 outcome=INFO SeverityValue=2 Severity=INFO externalId=1704 SourceName=SceCli TaskValue=0 RecordNumber=1217 ExecutionProcessID=0 ExecutionThreadID=0 deviceFacility=Application msg=Security policy in the Group policy objects has been applied successfully. rt=1632145308412 SourceModuleName=eventlog SourceModuleType=im_msvistalog
This configuration uses the im_file input module to collect system and authentication logs from a Linux host. Logs are parsed into structured data using the parse_syslog() procedure of the xm_syslog module, and converted to CEF using the to_cef() procedure of the xm_cef module.
Log records are enriched with additional CEF fields to facilitate searching
in ArcSight Logger. The include_stdout
directive is used to execute a bash script that retrieves the Linux
OS version, and injects this value in the NXLog Agent configuration as the
%OSNAME%
constant. This constant is used to set the $CEFDeviceVersion
field.
Finally, logs are forwarded to ArcSight Logger over TCP using the om_tcp output module..
include_stdout /opt/nxlog/etc/linux-os-version.sh
<Extension syslog>
Module xm_syslog
</Extension>
<Extension cef>
Module xm_cef
</Extension>
<Input auth>
Module im_file
File '/var/log/auth.log'
<Exec>
if not defined ($Hostname)
{
$Hostname = hostname();
}
$ipaddress = host_ip();
</Exec>
</Input>
<Input messages>
Module im_file
File '/var/log/messages'
Exec $ipaddress = host_ip();
</Input>
<Output arcsight_logger>
Module om_tcp
Host <arcsight_logger_ip>:6514
<Exec>
parse_syslog();
# Changing this to Linux helps us better identify Linux
# events instead of searching for specific events.
$CEFDeviceVendor = "Linux";
# This value is obtained through executing a shell script.
# See the include_stdout directive above.
$CEFDeviceVersion = '%OSNAME%';
to_cef();
</Exec>
</Output>
#!/bin/bash
OSHOST=$(cat /etc/os-release | grep -oP '^NAME=.*' | grep -oP '"(.*)"' | sed s/\"//g)
OSVERSION=$(cat /etc/os-release | grep VERSION= | grep -oP '"(.*)"' | sed s/\"//g)
OSNAME="$OSHOST $OSVERSION"
echo "define OSNAME $OSNAME"
The script needs to be saved in the path specified by the
include_stdout directive. The user running NXLog Agent must have
permission to execute the file.
|
CEF:0|Linux|NXLog|Ubuntu 20.04.2 LTS (Focal Fossa)|0|-|7|rt=1631698896137 SourceModuleName=messages SourceModuleType=im_file ipaddress=192.168.0.103 SyslogFacilityValue=1 SyslogFacility=USER SyslogSeverityValue=5 SyslogSeverity=NOTICE SeverityValue=2 Severity=INFO dvchost=NXLog-Ubuntu-1 rt=1631698896000 SourceName=systemd ProcessID=1 msg=Started NXLog daemon.
The configuration above does not explicitly set the $CEFDeviceProduct
field. This results in the to_cef() procedure
using the default value for this field, which is NXLog Agent.
|
Receiving logs from ArcSight Logger
An ArcSight Logger forwarder can be configured to forward events to a remote server over UDP or TCP. NXLog Agent supports receiving logs over both these protocols. Follow these steps to create a TCP Forwarder and configure it to forward events to NXLog Agent.
-
Log in to the Logger user interface with an account that has permission to modify the configuration.
-
From the left menu, navigate to Configuration > Data > Forwarders.
-
Click Add to create a new forwarder:
-
Provide a unique name for the forwarder.
-
Select TCP Forwarder for Type and Unified Query for Type of Filter.
-
On the next screen you will be given the option to define the query that will be used to select which logs to forward. In this example we used:
deviceVendor = "Microsoft Windows" AND deviceEventClassId = "SceCli:1704"
-
For IP/Host, specify the IP of the NXLog Agent host.
-
Enter the Port that NXLog Agent will be listening on.
-
Click Save.
-
The TCP Forwarder sends data in the same format that it was ingested by ArcSight Logger. In this example, NXLog Agent expects to receive logs in CEF.
The configuration below uses the im_tcp input module to listen for incoming connections on port 6514. It parses log records into structured data using the parse_cef() procedure of the xm_cef module, and then converts them to JSON using the to_json() procedure of the xm_json module.
<Extension json>
Module xm_json
</Extension>
<Extension cef>
Module xm_cef
</Extension>
<Input arcsight_logger>
Module im_tcp
ListenAddr 0.0.0.0:6514
Exec parse_cef(); to_json();
</Input>
CEF:0|Microsoft|Microsoft Windows|Microsoft Windows Server 2016 Standard|SceCli:1704|-|7|rt=1611907510135 dvchost=WIN2016.example.com Keywords=36028797018963968 outcome=INFO SeverityValue=2 Severity=INFO externalId=1704 SourceName=SceCli TaskValue=0 RecordNumber=1217 ExecutionProcessID=0 ExecutionThreadID=0 deviceFacility=Application msg=Security policy in the Group policy objects has been applied successfully.
{
"EventReceivedTime": "2021-09-20T08:57:37.335273-05:00",
"SourceModuleName": "arcsight_logger",
"SourceModuleType": "im_tcp",
"CEFVersion": 0,
"CEFDeviceVendor": "Microsoft",
"CEFDeviceProduct": "Microsoft Windows",
"CEFDeviceVersion": "Microsoft Windows Server 2016 Standard",
"CEFSignatureID": "SceCli:1704",
"CEFName": "-",
"CEFSeverity": 7,
"SeverityValue": "2",
"Severity": "INFO",
"rt": "2021-01-29T02:05:10.135000-06:00",
"dvchost": "WIN2016.example.com",
"Keywords": "36028797018963968",
"outcome": "INFO",
"externalId": "1704",
"SourceName": "SceCli",
"TaskValue": "0",
"RecordNumber": "1217",
"ExecutionProcessID": "0",
"ExecutionThreadID": "0",
"deviceFacility": "Application",
"msg": "Security policy in the Group policy objects has been applied successfully."
}
Verifying data in ArcSight Logger
Reception of log data can be verified using the ArcSight Logger user interface. One way to do this is to search for logs by fields. Navigate to Analyze > Search, enter your query under Enter search query…, and click the Go! button. By default, only events that occurred in the last 10 minutes are displayed. You may need to modify this setting from the time range selector.
The image below shows a query to filter by deviceVendor and deviceEventClassId.
The next image shows the result of the query above.
Click on an event to display the full event details.