Micro Focus ArcSight Logger

ArcSight Logger is a log management solution that provides secure storage, efficient search, reporting, and analysis of log data. NXLog can integrate with ArcSight Logger by sending log data to it in Common Event Format (CEF) over UDP or TCP. NXLog also supports receiving logs from an ArcSight Logger Forwarder.

The instructions and examples in this guide were tested with the software-based solution of ArcSight Logger. In some cases, these may vary from the appliance-based solution. Consult the ArcSight documentation for instructions related to your version.

Sending logs to ArcSight Logger

For ArcSight Logger to receive logs over UDP or TCP, a receiver must be created from the Logger user interface. Follow these steps to create a CEF TCP Receiver for use with NXLog.

  1. Log in to the Logger user interface with an account that has permission to modify the configuration.

  2. From the left menu, navigate to Configuration > Data > Receivers.

  3. Click Add to create a new receiver:

    1. Provide a unique name for the receiver.

    2. Select CEF TCP Receiver for Type.

    3. Click Next to continue with the configuration.

    4. Choose the IP/Host network connection that will be used to listen for incoming data.

    5. Enter the Port. The examples in this guide use port 6514.

    6. Select UTF-8 character encoding.

    7. The Source Type will be set to CEF by default, this is the only option for this type of receiver.

    8. Ensure that the Enable option is checked and click Save.

Sending CEF logs over TCP

NXLog supports log collection from various sources and can transform the data to CEF using the xm_cef extension. The examples below demonstrate how you can collect logs from Windows Event Log or Linux syslog and forward them to ArcSight Logger over TCP.

Example 1. Sending Windows Event Log events

This configuration uses the im_msvistalog input module to collect Application, Security, and System logs from Windows Event Log. Logs are converted to CEF using the to_cef() procedure of the xm_cef module, and then forwarded to ArcSight Logger over TCP using the om_tcp output module.

Log records are enriched with additional CEF fields to facilitate searching in ArcSight Logger. The include_stdout directive is used to execute a PowerShell script that retrieves the Windows OS version, and injects this value in the NXLog configuration as the %OSNAME% constant. This constant is used to set the $CEFDeviceVersion field.

nxlog.conf
include_stdout     C:\Program Files\nxlog\conf\windows-os-version.cmd

<Extension cef>
    Module         xm_cef
</Extension>

<Input eventlog>
    Module         im_msvistalog
    <QueryXML>
        <QueryList>
            <Query Id='0'>
                <Select Path='Application'>*</Select>
                <Select Path='Security'>*</Select>
                <Select Path='System'>*</Select>
            </Query>
        </QueryList>
    </QueryXML>
    <Exec>
      $CEFDeviceVendor = "Microsoft";
      $CEFDeviceProduct = "Microsoft Windows";

      # This value is obtained through executing a command file.
      # See the include_stdout directive above.
      $CEFDeviceVersion = '%OSNAME%';

      # This is required for some of the included searches in
      # ArcSight Logger to work.
      $CEFSignatureID = $SourceName+":"+$EventID;

      to_cef();
    </Exec>
</Input>

<Output arcsight_logger>
    Module         om_tcp
    Host           <arcsight_logger_ip>:6514
</Output>
windows-os-version.cmd
# To make NXLog return an error, write to standard error and exit 1
if ($false) {
    [Console]::Error.WriteLine("This is an error")
    exit 1
}
else {
# Anything written to standard output is used as configuration content
	$osname = (Get-WmiObject -Class Win32_OperatingSystem).caption
	Write-Output "define OSNAME $osname"
}
The cmd file needs to be saved in the path specified by the include_stdout directive. The user running NXLog needs to have permission to execute the file.
Output sample in CEF
CEF:0|Microsoft|Microsoft Windows|Microsoft Windows Server 2016 Standard|SceCli:1704|-|7|rt=1611907510135 dvchost=WIN2016.example.com Keywords=36028797018963968 outcome=INFO SeverityValue=2 Severity=INFO externalId=1704 SourceName=SceCli TaskValue=0 RecordNumber=1217 ExecutionProcessID=0 ExecutionThreadID=0 deviceFacility=Application msg=Security policy in the Group policy objects has been applied successfully. rt=1632145308412 SourceModuleName=eventlog SourceModuleType=im_msvistalog
Example 2. Sending Linux syslog messages

This configuration uses the im_file input module to collect system and authentication logs from a Linux host. Logs are parsed into structured data using the parse_syslog() procedure of the xm_syslog module, and converted to CEF using the to_cef() procedure of the xm_cef module.

Log records are enriched with additional CEF fields to facilitate searching in ArcSight Logger. The include_stdout directive is used to execute a bash script that retrieves the Linux OS version, and injects this value in the NXLog configuration as the %OSNAME% constant. This constant is used to set the $CEFDeviceVersion field.

Finally, logs are forwarded to ArcSight Logger over TCP using the om_tcp output module..

nxlog.conf
include_stdout     /opt/nxlog/etc/linux-os-version.sh

<Extension syslog>
    Module         xm_syslog
</Extension>

<Extension cef>
    Module         xm_cef
</Extension>

<Input auth>
    Module         im_file
    File           '/var/log/auth.log'
    <Exec>
        if not defined ($Hostname)
        {
             $Hostname = hostname();
        }
        $ipaddress = host_ip();
    </Exec>
</Input>

<Input messages>
    Module         im_file
    File           '/var/log/messages'
    Exec           $ipaddress = host_ip();
</Input>

<Output arcsight_logger>
    Module         om_tcp
    Host           <arcsight_logger_ip>:6514
    <Exec>
        parse_syslog();

        # Changing this to Linux helps us better identify Linux
        # events instead of searching for specific events.
        $CEFDeviceVendor = "Linux";

        # This value is obtained through executing a shell script.
        # See the include_stdout directive above.
        $CEFDeviceVersion = '%OSNAME%';
        
        to_cef();
    </Exec>
</Output>
linux-os-version.sh
#!/bin/bash
OSHOST=$(cat /etc/os-release | grep -oP '^NAME=.*' | grep -oP '"(.*)"' | sed s/\"//g)
OSVERSION=$(cat /etc/os-release | grep VERSION= | grep -oP '"(.*)"' | sed s/\"//g)
OSNAME="$OSHOST $OSVERSION"
echo "define OSNAME $OSNAME"
The script needs to be saved in the path specified by the include_stdout directive. The user running NXLog needs to have permission to execute the file.
Output sample in CEF
CEF:0|Linux|NXLog|Ubuntu 20.04.2 LTS (Focal Fossa)|0|-|7|rt=1631698896137 SourceModuleName=messages SourceModuleType=im_file ipaddress=192.168.0.103 SyslogFacilityValue=1 SyslogFacility=USER SyslogSeverityValue=5 SyslogSeverity=NOTICE SeverityValue=2 Severity=INFO dvchost=NXLog-Ubuntu-1 rt=1631698896000 SourceName=systemd ProcessID=1 msg=Started NXLog daemon.
The configuration above does not explicitly set the $CEFDeviceProduct field. This results in the to_cef() procedure using the default value for this field, which is NXLog.

Receiving logs from ArcSight Logger

An ArcSight Logger forwarder can be configured to forward events to a remote server over UDP or TCP. NXLog supports receiving logs over both these protocols. Follow these steps to create a TCP Forwarder and configure it to forward events to NXLog.

  1. Log in to the Logger user interface with an account that has permission to modify the configuration.

  2. From the left menu, navigate to Configuration > Data > Forwarders.

  3. Click Add to create a new forwarder:

    1. Provide a unique name for the forwarder.

    2. Select TCP Forwarder for Type and Unified Query for Type of Filter.

    3. On the next screen you will be given the option to define the query that will be used to select which logs to forward. In this example we used:

      deviceVendor = "Microsoft Windows" AND deviceEventClassId = "SceCli:1704"
    4. For IP/Host, specify the IP of the NXLog host.

    5. Enter the Port that NXLog will be listening on.

    6. Click Save.

Example 3. Receiving CEF logs

The TCP Forwarder sends data in the same format that it was ingested by ArcSight Logger. In this example, NXLog expects to receive logs in CEF.

The configuration below uses the im_tcp input module to listen for incoming connections on port 6514. It parses log records into structured data using the parse_cef() procedure of the xm_cef module, and then converts them to JSON using the to_json() procedure of the xm_json module.

nxlog.conf
<Extension json>
    Module        xm_json
</Extension>

<Extension cef>
    Module        xm_cef
</Extension>

<Input arcsight_logger>
    Module        im_tcp
    ListenAddr    0.0.0.0:6514
    Exec          parse_cef(); to_json();
</Input>
Input sample in CEF
CEF:0|Microsoft|Microsoft Windows|Microsoft Windows Server 2016 Standard|SceCli:1704|-|7|rt=1611907510135 dvchost=WIN2016.example.com Keywords=36028797018963968 outcome=INFO SeverityValue=2 Severity=INFO externalId=1704 SourceName=SceCli TaskValue=0 RecordNumber=1217 ExecutionProcessID=0 ExecutionThreadID=0 deviceFacility=Application msg=Security policy in the Group policy objects has been applied successfully.
Output sample in JSON format
{
  "EventReceivedTime": "2021-09-20T08:57:37.335273-05:00",
  "SourceModuleName": "arcsight_logger",
  "SourceModuleType": "im_tcp",
  "CEFVersion": 0,
  "CEFDeviceVendor": "Microsoft",
  "CEFDeviceProduct": "Microsoft Windows",
  "CEFDeviceVersion": "Microsoft Windows Server 2016 Standard",
  "CEFSignatureID": "SceCli:1704",
  "CEFName": "-",
  "CEFSeverity": 7,
  "SeverityValue": "2",
  "Severity": "INFO",
  "rt": "2021-01-29T02:05:10.135000-06:00",
  "dvchost": "WIN2016.example.com",
  "Keywords": "36028797018963968",
  "outcome": "INFO",
  "externalId": "1704",
  "SourceName": "SceCli",
  "TaskValue": "0",
  "RecordNumber": "1217",
  "ExecutionProcessID": "0",
  "ExecutionThreadID": "0",
  "deviceFacility": "Application",
  "msg": "Security policy in the Group policy objects has been applied successfully."
}

Verifying data in ArcSight Logger

Reception of log data can be verified using the ArcSight Logger user interface. One way to do this is to search for logs by fields. Navigate to Analyze > Search, enter your query under Enter search query…​, and click the Go! button. By default, only events that occurred in the last 10 minutes are displayed. You may need to modify this setting from the time range selector.

The image below shows a query to filter by deviceVendor and deviceEventClassId.

ArcSight Logger search query

The next image shows the result of the query above.

ArcSight Logger search result

Click on an event to display the full event details.

ArcSight Logger event details
Disclaimer

While we endeavor to keep the information in this topic up to date and correct, NXLog makes no representations or warranties of any kind, express or implied about the completeness, accuracy, reliability, suitability, or availability of the content represented here.

The accurateness of the content was tested and proved to be working in our lab environment at the time of the last revision with the following software versions:

ArcSight Logger version 7.1.2.8354.0
NXLog Enterprise Edition version 5.3.6985

Last revision: 16 September 2021