Send logs to Splunk

Splunk is a log collection, indexing, analysis, and visualization platform. It is available in two editions, the on-premises Splunk Enterprise, and the cloud-based Splunk Cloud Platform.

NXLog Agent can send logs to Splunk via UDP, TCP with TLS, and HTTP(S). You can collect logs with NXLog Agent from diverse log sources, including Windows, Linux, and macOS, and send them directly to Splunk. Alternatively, you can use NXLog Agent as a relay, receiving logs from different network sources and forwarding them to Splunk from a single instance or an NXLog Agent cluster. NXLog Agent’s advanced log collection, processing, and forwarding capabilities make it the ideal log collector for Splunk SIEM.

A Splunk Universal Forwarder alternative

The Splunk Universal Forwarder is a Splunk agent commonly used in a similar role as NXLog Agent. However, NXLog Agent offers some significant advantages over the Splunk forwarder, including full-featured log parsing and filtering before forwarding, which results in faster indexing by Splunk. In controlled tests, Splunk indexers processed events forwarded by NXLog Agent over ten times faster than the same Windows events forwarded by the Splunk Universal Forwarder, despite the overhead of transforming the events to emulate Splunk’s proprietary format.

When planning a migration to NXLog Agent, you should first evaluate the log formats you’re collecting with the Splunk Universal Forwarder. Depending on the type of logs, it could be as simple as creating a new Splunk TCP data input and following some of the examples in this guide, such as forwarding IETF syslog or JSON formats. You do not need to create any special configuration as long as the logs are in a standard format that Splunk can parse and index.

Configure Splunk Enterprise to receive logs

Splunk Enterprise accepts logs via UDP, TCP, and HTTP(S). The following sections describe how to prepare Splunk for receiving logs from NXLog Agent.

Splunk UDP and TCP data inputs

You can configure UDP and TCP inputs from Splunk Web. However, you can only configure TLS encryption by editing Splunk configuration files.

Add a new data input.
1. Open Splunk Web and navigate to Settings > Data inputs.
2. In the Local inputs section, click Add new for the TCP (or UDP) input type.
3. Enter the Port on which to listen for log data (for example, port 514).
4. Fill in the remaining values if required and click Next.
Configure the input settings.
1. Select the appropriate log Source type. See the examples in Configure NXLog Agent to collect and process logs for more information.
2. Choose an App context; for example, Search & Reporting (search).
3. Adjust the remaining default values, if required, and click Review.
Review the pending changes and click Submit.

Splunk HTTP Event Collector (HEC)

The HTTP Event Collector is a stateless, high-performance solution that is easy to scale with a load balancer. Furthermore, it offers token-based authentication. For more information about configuring and using Splunk HEC, see Setup and use HTTP Event Collector in Splunk Web, Format events for HTTP Event Collector, and Input endpoint descriptions in the Splunk documentation.

By default, HEC is disabled on Splunk Enterprise installations. Follow these steps to enable it:

Navigate to Settings > Data inputs and click on the HTTP Event Collector type.
Click the Global Settings button (in the upper-right corner).
For All Tokens, click the Enabled button.
Optionally, set the Default Source Type, Default Index, and Default Output Group settings.
Check Enable SSL to enforce secure data transfer (recommended). See Enable TLS on Splunk Enterprise.
Change the HTTP Port Number if required.
Click Save.

Once you enable HEC, create a new token as follows:

If not already on the HTTP Event Collector page, open Settings > Data inputs and click on the HTTP Event Collector type.
Click New Token.
Enter a name for the token and modify any other settings if required; then click Next.
For the Source type, choose Automatic. The source type will be included with the event metadata, as shown in the examples below.
Adjust the remaining default values if required and click Review.
Verify the information on the summary page and click Submit. Take note of the HEC token. You will need this to configure NXLog Agent.
Use the following command to verify your new HEC input. Replace <host> with your Splunk hostname or IP and <token> with your HEC token.
$ curl -k https://<host>:8088/services/collector \ -H 'Authorization: Splunk <token>' -d '{"event":"test"}'
If configured correctly, Splunk will return a successful response.
{"text":"Success","code":0}

Enable TLS on Splunk Enterprise

Splunk Enterprise 6.6 and newer installations include default self-signed certificates. Follow these steps to configure your Splunk inputs to use TLS. You will need:

The Splunk CA certificate (cacert.pem)
The Splunk server certificate (server.pem)
The Splunk server certificate key password (password by default)

If you are using the default Splunk certificates, you can locate these files in $SPLUNK_HOME/etc/auth.

Add the sslRootCAPath parameter to the [sslConfig] stanza in the $SPLUNK_HOME/etc/system/local/server.conf file as follows:
server.conf
```
[sslConfig]
sslPassword = <Automatically generated>
sslRootCAPath = /opt/splunk/etc/auth/cacert.pem
```
Add [tcp-ssl] and [SSL] stanzas to the inputs.conf file of the relevant Splunk app(s). For example, the Search & Reporting app path is $SPLUNK_HOME/etc/apps/search/local.
inputs.conf
```
[tcp-ssl://10514]
disabled = 0
sourcetype = <optional>

[SSL]
serverCert = /opt/splunk/etc/auth/server.pem
sslPassword = password
requireClientCert = 0
```
Splunk HEC inherits SSL settings from the global server.conf. Therefore, you do not need to set SSL configuration for HEC inputs.
Finally, restart Splunk to apply the new configuration.
```
$ sudo $SPLUNK_HOME/bin/splunk restart
```
Test your configuration with netstat or a similar command. If configured correctly, the command should list the TCP port you specified in step 2.
```
$ netstat -an | grep :10514
tcp   0   0 0.0.0.0:10514   0.0.0.0:*   LISTEN
```
Copy cacert.pem from $SPLUNK_HOME/etc/auth to the NXLog Agent certificates directory.

Configure Splunk Cloud Platform to receive logs

Splunk Cloud Platform accepts logs via HTTP Event Collector (HEC) inputs. HEC is enabled by default with a 1 MB content length limit. Follow these steps to verify that HEC is enabled for your instance:

Navigate to Settings > Data inputs and click on the HTTP Event Collector type.
Click the Global Settings button (in the upper-right corner).
For All Tokens, click the Enabled button.
Optionally, set the Default Source Type and Default Index settings.
Check Enable SSL to require secure data transfer (recommended).
Change the HTTP Port Number if needed.
Click Save.

You cannot modify HEC SSL and port options on Splunk Cloud Platform trial instances.

Export the Splunk Cloud Platform CA certificate

Exporting the CA certificate using OpenSSL

Execute the following command:
```
$ openssl s_client -showcerts -connect <your_instance_url>:<your_hec_port> < /dev/null
```
Replace <your_instance_url> with your Splunk Cloud Platform instance URL and <your_hec_port> with the HTTP port number you specified when you created the HEC input. For example:
```
$ openssl s_client -showcerts -connect prd-p-5tzek.splunkcloud.com:8088 < /dev/null
```
The information returned will contain the complete certificate chain. Copy the last certificate shown, including the -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- lines, to a text file. If the response contains more than two certificates, copy all the certificates except the first one to the same text file.
Save the file to a directory accessible by NXLog Agent.

Exporting the CA certificate from Chrome on Windows

Open Chrome, and enter your Splunk Cloud Platform instance URL, including the HEC port, in the address bar. For example, https://prd-p-5tzek.splunkcloud.com:8088/.
Click on the padlock icon next to the URL in the browser address bar. If the browser shows that your connection is not private, click on the Not secure warning instead.
Click Certificate is valid or Certificate is not valid.
Select the topmost certificate on the Details tab and click the Export button. If more than two certificates are displayed, select the one before the last.
Select the Base-64 encoded ASCII, certificate chain (*.pem;*.crt) option.
Select the directory where to save the certificate and click Save.

The procedure to export certificates differs according to the browser and operating system. Refer to your browser documentation for the appropriate steps.

Configure NXLog Agent to collect and process logs

Forward generic logs in JSON format

For Splunk to parse JSON logs, you simply need to set the data input source type to _json. See how to configure Splunk Enterprise and Splunk Cloud Platform above.

Furthermore, configure the data input or source type with NXLog Agent’s integer timestamp format to ensure that Splunk parses the event timestamp correctly. To modify the _json source type from Splunk Web:

Navigate to Settings > Source types.
Find the _json source type and click Edit.
Open the Advanced tab and add the following configuration value:

Name Value

TIME_PREFIX

"time":"

TIME_FORMAT

%s.%6N

Name	Value
`TIME_PREFIX`	`"time":"`
`TIME_FORMAT`	`%s.%6N`

Example 1. Collecting file-based logs and converting them to JSON

This configuration collects syslog messages from a file with the im_file input module. First, it parses the log records to structured data using the parse_sylog() of the xm_syslog module. It then creates a $time field containing the event’s integer timestamp. Finally, it converts the log record to JSON using the to_json() procedure of the xm_json module.

nxlog.conf

<Extension json>
    Module    xm_json
</Extension>

<Extension syslog>
    Module    xm_syslog
</Extension>

<Input syslog_file>
    Module    im_file
    File      '/var/log/syslog'
    <Exec>
        # Parse the log record into fields
        parse_syslog();

        # Convert $EventTime to decimal seconds since epoch UTC
        $time = string(integer($EventTime));
        $time =~ /^(?<sec>\d+)(?<ms>\d{6})$/;
        $time = $sec + "." + $ms;
        delete($sec);
        delete($ms);

        # Convert to JSON
        to_json();
    </Exec>
</Input>

Output sample

{
  "EventReceivedTime": "2023-02-01T05:36:59.448973+00:00",
  "SourceModuleName": "syslog",
  "SourceModuleType": "im_file",
  "SyslogFacility": "USER",
  "vendor_severity": "INFO",
  "severity_id": 2,
  "EventTime": "2023-02-01T05:36:58.190689+00:00",
  "Hostname": "NXLog-Ubuntu-1",
  "ProcessID": 14533,
  "SourceName": "sshd",
  "Message": "Failed password for invalid user",
  "time": "1675229818.190689"
}

Forward Windows logs in JSON format

This section assumes that you will manage any pre-existing Windows logs in Splunk separately until they age out of the system. This is because Splunk alters some of the original Windows field names. However, if you need to maintain Splunk-specific field names for Windows logs, see Forward Windows logs in the Splunk Universal Forwarder format below.

Example 2. Collecting Windows DNS Server logs and converting them to JSON

This configuration uses the im_etw input module to collect logs from the Microsoft-Windows-DNSServer ETW provider. It then converts log records to JSON format using the to_json() procedure of the xm_json module.

nxlog.conf

<Extension json>
    Module      xm_json
</Extension>

<Input dns_server>
    Module      im_etw
    Provider    Microsoft-Windows-DNSServer
    Exec        to_json();
</Input>

Output sample

{
  "SourceName": "Microsoft-Windows-DNSServer",
  "ProviderGuid": "{EB79061A-A566-4698-9119-3ED2807060E7}",
  "EventID": 515,
  "Version": 0,
  "ChannelID": 17,
  "OpcodeValue": 0,
  "TaskValue": 5,
  "Keywords": "4611686018428436480",
  "EventTime": "2020-05-19T10:42:06.313322-05:00",
  "ExecutionProcessID": 1536,
  "ExecutionThreadID": 3896,
  "EventType": "INFO",
  "SeverityValue": 2,
  "Severity": "INFO",
  "Domain": "WIN-R4QHULN6KLH",
  "AccountName": "Administrator",
  "UserID": "S-1-5-21-915329490-2962477901-227355065-500",
  "AccountType": "User",
  "Flags": "EXTENDED_INFO|IS_64_BIT_HEADER|PROCESSOR_INDEX (577)",
  "Type": "5",
  "NAME": "www.example.com",
  "TTL": "3600",
  "BufferSize": "17",
  "RDATA": "0x106E73312E6578616D706C652E636F6D2E",
  "Zone": "example.com",
  "ZoneScope": "Default",
  "VirtualizationID": ".",
  "EventReceivedTime": "2020-05-19T10:42:07.313482-05:00",
  "SourceModuleName": "dns_server",
  "SourceModuleType": "im_etw",
  "MessageSourceAddress": "192.168.1.61"
}

You can send JSON-formatted logs to a data input configured with automatic or _json source type, and Splunk will parse and index them. For example, the screenshot below shows a query in Splunk Web that uses the spath command to retrieve the DNS Server logs forwarded by NXLog Agent.

Forward Windows logs in XML format

The Splunk Add-on for Microsoft Windows provides log source types for parsing Windows logs in XML format. Follow these steps to configure a Splunk data input to parse Windows XML logs:

Install the Splunk Add-on for Microsoft Windows. See About installing Splunk add-ons in the Splunk documentation for more details.
Set the log source type of your data input to XmlWinEventLog.
Optionally, configure the data input or event source to use the record’s SystemTime value for Splunk’s event _time field during indexing. Otherwise, Splunk will default to using the ingestion time. To modify the XmlWinEventLog source type from Splunk Web:
1. Navigate to Settings > Source types.
2. Find the XmlWinEventLog source type (uncheck Show only popular option) and click Edit.
3. Open the Advanced tab and add the following configuration value:
  
  Name Value
  
  EVAL-_time
  
  strptime(SystemTime, "'%Y-%m-%dT%H:%M:%S.%9N%Z'")

Name	Value
`EVAL-_time`	`strptime(SystemTime, "'%Y-%m-%dT%H:%M:%S.%9N%Z'")`

Example 3. Collecting Windows Security logs and converting them to XML

This configuration uses the im_msvistalog input module with the CaptureEventXML directive enabled to collect Windows Security logs. It then sets the $raw_event field to the value of the $EventXML field.

nxlog.conf

<Input windows_security>
    Module             im_msvistalog
    Channel            Security
    CaptureEventXML    TRUE
    Exec               $raw_event = $EventXML;
</Input>

Splunk will automatically parse and index the logs, as shown in the screenshot below.

Forward Windows logs in the Splunk Universal Forwarder format

The Splunk Universal Forwarder renames at least four fields and adds several new ones to Windows logs. Therefore specific NXLog Agent configuration is required to emulate this event schema. See the comparison table below.

Table 1. Comparison of field names for Windows events
Windows	NXLog Agent	Splunk
Channel	Channel	Logname
Computer	Hostname *	ComputerName
EventID	EventID	EventCode
Execution_ProcessID	ExecutionProcessID	—
Execution_ThreadID	ExecutionThreadID	—
ProviderGuid	ProviderGuid	—
UserID	UserID	Sid
—	—	Type
—	—	idType

* NXLog Agent normalizes this field name across all modules and log sources.

It should be emphasized that NXLog Agent can forward Windows or any other kind of structured logs to Splunk without any need to emulate the Splunk Universal Forwarder format or schema. There is no technical requirement or advantage in using Splunk’s proprietary format to forward logs to Splunk, especially for new Splunk deployments with no existing corpus of Windows event logs.

The only purpose of emulating the Splunk Universal Forwarder format is to maintain continuity with previously indexed Windows events. Therefore, the preferred method is to forward Windows logs to Splunk in JSON format.

If it is essential to retain the Splunk Universal Forwarder format after migrating to NXLog Agent, then adhering to the following procedures is imperative for Splunk to ingest the emulated logs correctly.

There are a few principles you need to observe when creating an NXLog Agent configuration that maintains backward compatibility with events collected by the Splunk Universal Forwarder:

When creating a new TCP data input in Splunk, you must choose the correct Source Type.
In the NXLog Agent configuration, rename fields to what Splunk associates with that Source Type.
In the NXLog Agent configuration, ensure the data matches the format shown in Splunk as closely as possible unless Splunk fails to parse specific fields.
In the NXLog Agent configuration, manually parse embedded structured data as new, full-fledged fields. When using this technique, a common cause of parsing failure is fields containing strings of embedded subfields.

Follow these steps for each log source type you are forwarding:

Examine the events in Splunk and note the sourcetype value listed below each event. The Splunk Universal Forwarder may assign different sourcetype values for logs from the same source. Try to determine which one is the best fit.
In Splunk, create a new TCP Data Input port for each log source type to be forwarded. Set the Source Type to the same value assigned to events received from the Splunk Universal Forwarder.
Note which fields Splunk parses after indexing the logs.
Create an NXLog Agent configuration that collects the logs, renames the fields to those associated with the relevant Source Type, and rewrites the records to match the Splunk Universal Forwarder format.

The Splunk Universal Forwarder format, called "cooked data," includes a binary header component and a footer. A log record starts with a line containing the event’s timestamp. The timestamp is followed by a set of unquoted key-value pairs separated by an equals sign (=), one key-value pair per line. The header and footer are not required to forward logs to a TCP Data Input. Instead, you only need to send the first line containing the event’s timestamp and the subsequent lines containing the key-value pairs.

You can forward Windows logs from NXLog Agent so that Splunk parses and indexes them as if the Splunk Universal Forwarder sent them. There are only three criteria you must meet:

Install the Splunk Add-on for Microsoft Windows on the Splunk instance receiving the logs. See About installing Splunk add-ons in the Splunk documentation for more details.
Configure NXLog Agent to rewrite log records to match the field names expected by the corresponding log source in the Splunk Add-on for Microsoft Windows.
Create a unique TCP Data Input port for each type of Windows log provider by following the procedure in Splunk UDP and TCP data inputs above. It is imperative to specify the correct Source type following this naming convention: WinEventLog:Provider/Channel.

When adding a new TCP Data Input, the desired Source type for Windows might not be available in the Select Source Type dropdown menu. If so, select or manually enter WinEventLog. Once you create the TCP Data Input, go back to edit it, select Manual for Set source type, and enter the correct value in the Source type field.

Example 4. Emulating the Splunk Universal Forwarder format for Windows DNS Server audit logs

This example illustrates the method for emulating the Splunk Universal Forwarder sending Windows DNS Server audit logs to Splunk. The first step is to create a new data input with a Source type of WinEventLog:Microsoft-Windows-DNSServer/Audit.

TCP input for Windows DNS Server audit logs

This configuration uses the im_msvistalog module to collect and parse the logs. Since there is no need for filtering in this example, a simple File directive defines the location of the log source. Otherwise, use the Channel directive with a QueryXML block to specify filters. Next, the Exec block contains the logic for converting log records to the Splunk Universal Forwarder format. Finally, the configuration sets the $raw_event field to the multi-line string containing the event data and uses the xm_rewrite to delete the original fields.

nxlog.conf

<Extension drop_fields>
    Module    xm_rewrite
    Keep      # Remove all
</Extension>

<Input dns_server_audit>
    Module    im_msvistalog
    File      %SystemRoot%\System32\Winevt\Logs\Microsoft-Windows-DNSServer%4Audit.evtx
    <Exec>
        # Create a header variable for storing the Splunk datetime string
        create_var('timestamp_header');
        create_var('event');            # The Splunk equivalent of a $raw_event
        create_var('message');          # For preserving the $Message field
        create_var('vip_fields');       # Message subfields converted to fields

        # Format the datetime string for the Header Line
        $dts = strftime($EventTime,'YYYY-MM-DD hh:mm:ss.sTZ');
        $hr = "";  # Hours, 2-digit
        $ap = "";  # For either "AM" or "PM";
        if ($dts =~ /(\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2})/ ) {
            if (hour($EventTime) < 12) {
                $ap = "AM";
                $hr = $4;
                if (hour($EventTime) == 0) $hr = "12";
            }
            if (hour($EventTime) > 11) {
                $ap = "PM";
                if (hour($EventTime) == 12) $hr = $4;;
                if (hour($EventTime) > 12) {
                    $hr = hour($EventTime) - 12;
                    if (hour($EventTime) < 22)  $hr = "0" + $hr;
                }
            }
            $dts = $2 +"/"+ $3 +"/"+ $1 +" "+ \
                   $hr +":"+ $5 +":"+ $6 +" "+ $ap + "\n";
        }
        set_var('timestamp_header', $dts);

        # Convert $EventType to the Splunk equivalent
        $EventType = ($EventType == "INFO" ? 4 : $EventType);

        # Some really important DNS fields that Splunk doesn't parse
        $vipFields = "";

        if (defined($NAME))
            $vipFields = $vipFields + 'NAME=' + $NAME + "\r\n";
        if (defined($Severity))
            $vipFields = $vipFields + 'Severity=' + $Severity + "\r\n";
        if (defined($TTL))
            $vipFields = $vipFields + 'TTL=' + $TTL + "\r\n";
        if (defined($BufferSize))
            $vipFields = $vipFields + 'BufferSize=' + $BufferSize + "\r\n";
        if (defined($RDATA))
            $vipFields = $vipFields + 'RDATA=' + $RDATA + "\r\n";
        if (defined($Zone))
            $vipFields = $vipFields + 'Zone=' + $Zone + "\r\n";
        if (defined($ZoneScope))
            $vipFields = $vipFields + 'ZoneScope=' + $ZoneScope + "\r\n";
        set_var('vip_fields', $vipFields);

        # Store the original $Message field at the end of the list
        # just in case Splunk parses it correctly
        set_var('message', $Message);

        # Set the event data in the Splunk format
        set_var('event', \
            'LogName='      + $Channel         +"\r\n"+ \
            'SourceName='   + $SourceName      +"\r\n"+ \
            'EventCode='    + $EventID         +"\r\n"+ \
            'EventType='    + $EventType       +"\r\n"+ \
            'Type='         + 'Information'    +"\r\n"+ \
            'ComputerName=' + $Hostname        +"\r\n"+ \
            'User='         + 'NOT_TRANSLATED' +"\r\n"+ \
            'Sid='          + $UserID          +"\r\n"+ \
            'SidType='      + '0'              +"\r\n"+ \
            'TaskCategory=' + $Category        +"\r\n"+ \
            'OpCode='       + $OpCode          +"\r\n"+ \
            'RecordNumber=' + $RecordNumber    +"\r\n"+ \
            'Keywords='     + $Keywords        +"\r\n" \
        );

        # Remove all NXLog fields.
        drop_fields->process();

        # Add the Splunk datetime string as a "header" line
        $raw_event = get_var('timestamp_header') + get_var('event') + \
            get_var('vip_fields') + 'Message=' + get_var('message') +"\n";
    </Exec>
</Input>

The following is a sample DNS Server audit log record ingested by Splunk.

Splunk will automatically parse and index the logs, as shown in the screenshot below.

Parsed DNS Server audit log record in Splunk

Example 5. Emulating the Splunk Universal Forwarder format for Sysmon DNS Query logs

This example illustrates the method for emulating the Splunk Universal Forwarder sending Windows Sysmon DNS Query logs to Splunk. The first step is to create a new data input with a Source type of WinEventLog:Microsoft-Windows-Sysmon/Operational.

This configuration uses the im_msvistalog module to collect and parse the logs. The QueryXML block specifies the Sysmon provider and filters for DNS Query logs. Next, the Exec block contains the logic for converting log records to the Splunk Universal Forwarder format. Finally, the configuration sets the $raw_event field to the multi-line string containing the event data and uses the xm_rewrite to delete the original fields.

nxlog.conf

<Extension drop_fields>
    Module    xm_rewrite
    Keep      # Remove all
</Extension>

<Input dns_sysmon>
    Module    im_msvistalog
    <QueryXML>
        <QueryList>
            <Query Id="0">
                <Select Path="Microsoft-Windows-Sysmon/Operational">
                    *[System[(EventID=22)]]
                </Select>
            </Query>
        </QueryList>
    </QueryXML>
    <Exec>
        # Create a header variable for storing the Splunk datetime string
        create_var('timestamp_header');
        create_var('event');            # The Splunk equivalent of a $raw_event
        create_var('message');          # For preserving the $Message field
        create_var('message_fields');   # Message subfields converted to fields

        # Format the datetime string for the Header Line
        $dts = strftime($EventTime,'YYYY-MM-DD hh:mm:ss.sTZ');
        $hr = "";  # Hours, 2-digit
        $ap = "";  # For either "AM" or "PM";
        if ($dts =~ /(\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2})/ ) {
            if (hour($EventTime) < 12) {
                $ap = "AM";
                $hr = $4;
                if (hour($EventTime) == 0) $hr = "12";
            }
            if (hour($EventTime) > 11) {
                $ap = "PM";
                if (hour($EventTime) == 12) $hr = $4;;
                if (hour($EventTime) > 12) {
                    $hr = hour($EventTime) - 12;
                    if (hour($EventTime) < 22)  $hr = "0" + $hr;
                }
            }
            $dts = $2 +"/"+ $3 +"/"+ $1 +" "+ \
                   $hr +":"+ $5 +":"+ $6 +" "+ $ap + "\n";
        }
        set_var('timestamp_header', $dts);

        # Convert $EventType to the Splunk equivalent
        $EventType = ($EventType == "INFO" ? 4 : $EventType);

        # Since Splunk often fails to parse the sub-fields of the Sysmon
        # $Message field, create them as individual fields
        $Subfields = "";
        if (defined($UtsTime))
            $Subfields = $Subfields + 'UtcTime=' + $UtcTime + "\r\n";
        if (defined($ProcessGuid))
            $Subfields = $Subfields + 'ProcessGuid=' + $ProcessGuid + "\r\n";
        if (defined($ProcessId))
            $Subfields = $Subfields + 'ProcessId=' + $ProcessId + "\r\n";
        if (defined($QueryName))
            $Subfields = $Subfields + 'QueryName=' + $QueryName + "\r\n";
        if (defined($QueryStatus))
            $Subfields = $Subfields + 'QueryStatus=' + $QueryStatus + "\r\n";
        if (defined($QueryResults))
            $Subfields = $Subfields + 'QueryResults=' + $QueryResults + "\r\n";
        if (defined($Image))
            $Subfields = $Subfields + 'Image=' + $Image + "\r\n";
        set_var('message_fields', $Subfields);

        # Store the original $Message field at the end of the list
        # just in case Splunk parses it correctly
        set_var('message', $Message);

        # Set the event data in the Splunk format
        set_var('event', \
            'LogName='      + $Channel         +"\r\n"+ \
            'SourceName='   + $SourceName      +"\r\n"+ \
            'EventCode='    + $EventID         +"\r\n"+ \
            'EventType='    + $EventType       +"\r\n"+ \
            'Type='         + 'Information'    +"\r\n"+ \
            'ComputerName=' + $Hostname        +"\r\n"+ \
            'User='         + 'NOT_TRANSLATED' +"\r\n"+ \
            'Sid='          + $UserID          +"\r\n"+ \
            'SidType='      + '0'              +"\r\n"+ \
            'TaskCategory=' + $Category        +"\r\n"+ \
            'OpCode='       + $OpCode          +"\r\n"+ \
            'RecordNumber=' + $RecordNumber    +"\r\n"+ \
            'Keywords='     + $Keywords        +"\r\n" \
        );

        # Remove all NXLog fields.
        Drop_Fields->process();

        # Add the Splunk datetime string as a "header" line
        $raw_event = get_var('timestamp_header') + get_var('event') + \
            get_var('message_fields') + 'Message=' + get_var('message') +"\n";
    </Exec>
</Input>

The following is a sample Sysmon DNS query log record ingested by Splunk:

Splunk will automatically parse and index the logs, as shown in the screenshot below.

Parsed Sysmon DNS queryt log record in Splunk

Forward IETF syslog

For Splunk to parse IETF syslog messages, you simply need to set the data input source type to syslog. See Configure Splunk Enterprise to receive logs above.

Example 6. Forwarding IETF syslog messages to Splunk

This configuration uses the im_systemd input module to collect systemd journal logs. It then uses the to_syslog_ietf() procedure of the xm_syslog module to convert records to IETF syslog format.

nxlog.conf

<Extension syslog>
    Module    xm_syslog
</Extension>

<Input systemd>
    Module    im_systemd
    Exec      to_syslog_ietf();  
</Input>

Configure NXLog Agent as a relay

NXLog Agent implements its protocol for optimized log forwarding between NXLog Agent nodes. You can leverage this feature to reduce the number of open network connections to a Splunk Enterprise server or the Splunk Cloud Platform.

Example 7. Centralizing log forwarding to Splunk with NXLog Agent

In the following configuration, a central NXLog Agent instance receives logs from different sources, caches them to disk, and then forwards all records to a Splunk TCP data input configured to receive logs in JSON format.

nxlog.conf

<Extension json>
    Module        xm_json
</Extension>

<Extension syslog>
    Module        xm_syslog
</Extension>

# Receive syslog messages from any device
<Input syslog_in>
    Module        im_tcp
    ListenAddr    0.0.0.0:1514
</Input>

# Receive DNS audit logs from another NXLog agent
<Input dns_audit_in>
    Module        im_tcp
    ListenAddr    0.0.0.0:1515
    InputType     Binary
</Input>

# Cache the logs to disk in case Splunk is unavailabe
<Output syslog_cache>
    Module        om_file
    File          '/opt/nxlog/var/log/cached/syslog.bin'
    OutputType    Binary
</Output>

<Output dns_audit_cache>
    Module        om_file
    File          '/opt/nxlog/var/log/cached/dns-audit.bin'
    OutputType    Binary
</Output>

# Read the cached logs from disk
<Input syslog_bin>
    Module        im_file
    File          '/opt/nxlog/var/log/cached/syslog.bin'
    Exec          parse_syslog();
</Input>

<Input dns_audit_bin>
    Module        im_file
    File          '/opt/nxlog/var/log/cached/dns-audit.bin'
</Input>

#Forward cached logs to Splunk in JSON format
<Output splunk_syslog>
    Module        om_tcp
    Host          192.168.1.71:1524
    Exec          to_json();
</Output>

<Output splunk_dns_audit>
    Module        om_tcp
    Host          192.168.1.71:1525
    Exec          to_json();
</Output>

# Routes: TCP received to local files
<Route syslog_tcp_to_cache>
    Path          syslog_in  =>  syslog_cache
</Route>

<Route dns_audit_tcp_to_cache>
    Path          dns_audit_in  =>  dns_audit_cache
</Route>

# Routes: Local files to Splunk
<Route syslog_bin_to_splunk>
    Path          syslog_bin  =>  splunk_syslog
</Route>

<Route dns_audit_bin_to_splunk>
    Path          dns_audit_bin  =>  splunk_dns_audit
</Route>

Configure NXLog Agent to send logs to Splunk

Send logs to Splunk via TCP/TLS

Below, you can find NXLog Agent configuration examples for sending logs to Splunk over plain TCP or securely with TLS.

Example 8. Sending logs to Splunk via TCP

This configuration uses the om_tcp output module to forward logs to a Splunk TCP data input configured to receive data on port 1514. See Splunk UDP and TCP data inputs for more information.

nxlog.conf

<Output splunk_tcp>
    Module    om_tcp
    Host      192.168.1.71:1514
</Output>

Example 9. Sending logs to Splunk via TCP with TLS

This configuration uses the om_ssl output module to forward logs to a Splunk tcp-ssl input. The CAFile directive specifies the path to the Splunk CA certificate. See Enable TLS on Splunk Enterprise for more information.

nxlog.conf

<Output splunk_tls>
    Module    om_ssl
    Host      192.168.1.71:10514
    CAFile    %CERTDIR%/cacert.pem
</Output>

Send logs to Splunk via HTTP(S)

Splunk accepts data over HTTP(S) via the HTTP Event Collector. Refer to the setup instructions for Splunk Enterprise and Splunk Cloud Platform above.

Splunk HEC provides two endpoints:

/services/collector/event requires event data to be in a specific JSON format
/services/collector/raw for raw text

The Splunk JSON standard consists of the following metadata fields:

time (epoch time)
host
source
sourcetype
index
fields
event

The event field must be a JSON object when using the event endpoint or a string when using the raw endpoint. See Event metadata and Event data in the Splunk documentation for more information.

Since each log record contains the source type, you do not need to configure different Splunk data inputs for each source. Therefore, you can configure multiple NXLog Agent input instances gathering logs from various sources and forwarding them to Splunk from a single output instance.

Example 10. Sending logs to a Splunk HEC input

This configuration uses the om_http output module to forward logs to a Splunk HEC input and specifies the following directives:

URL

Sends data to the Splunk HEC event endpoint. See Send data to HTTP Event Collector on Splunk Cloud Platform in the Splunk documentation for the required URI for your instance.

During our testing, http-inputs-<host>.splunkcloud.com did not work with a Splunk Cloud Platform free trial instance. However, inputs-<host>.splunkcloud.com and <host>.splunkcloud.com did.

AddHeader

Specifies the HEC token value. See Splunk HTTP Event Collector (HEC) above for instructions to create a token.

HTTPSCAFile

The HTTPSCAFile directive specifies the path to the Splunk CA certificate. See Enable TLS on Splunk Enterprise for more information.

BatchMode

Specifies multiline to send batched logs in a single HTTP request, one record per line.

Compression

Uses gzip HTTP compression to send data. HEC accepts SSL compression by default. See the allowSslCompression setting in the http: (HTTP Event Collector) specification.

nxlog.conf

<Extension json>
    module         xm_json
</Extension>

<Extension clean_splunk_fields>
    Module         xm_rewrite
    Keep           time, host, source, sourcetype, event
    Rename         Source, source
</Extension>

<Output splunk_hec>
    Module         om_http
    URL            https://192.168.1.71:8088/services/collector/event
    AddHeader      Authorization: Splunk c6580856-29e8-4abf-8bcb-ee07f06c80b3
    HTTPSCAFile    %CERTDIR%/cacert.pem
    BatchMode      multiline
    Compression    gzip
    <Exec>
        # Rename event fields to Splunk taxonomy
        if $Severity rename_field($Severity, $vendor_severity);
        if $SeverityValue rename_field($SeverityValue, $severity_id);

        # Convert all fields to JSON and write to $event field
        $event = to_json();

        # Convert $EventTime to decimal seconds since epoch UTC
        $time = string(integer($EventTime));
        $time =~ /^(?<sec>\d+)(?<ms>\d{6})$/;
        $time = $sec + "." + $ms;

        # Set the log source type
        $sourcetype = "_json";

        # Add HEC metadata fields if available
        if $Hostname $host = $Hostname;
        if $SourceName $source = $SourceName;

        # Remove unnecessary fields
        clean_splunk_fields->process();

        # Convert to JSON
        to_json();
    </Exec>
</Output>

Output sample

{
  "event": {
    "EventReceivedTime": "2019-10-18 19:58:19",
    "SourceModuleName": "in",
    "SourceModuleType": "im_file",
    "SyslogFacility": "USER",
    "vendor_severity": "INFO",
    "severity_id": 2,
    "EventTime": "2019-10-18 19:58:02",
    "Hostname": "myserver2",
    "ProcessID": 14533,
    "SourceName": "sshd",
    "Message": "Failed password for invalid user"
  },
  "time": "1571428682.218749",
  "sourcetype": "_json",
  "host": "myserver2",
  "source": "sshd"
}

Send and receive logs to and from the Splunk Universal Forwarder

The Splunk Universal Forwarder sends data in a so-called cooked format using the Splunk-to-Splunk (S2S) protocol. However, the cooked format can be disabled for communication with third-party solutions.

Example 11. Sending data to a Splunk Universal Forwarder

You must configure the Splunk indexer to receive data from Splunk Universal Forwarder. See Deploy the Universal Forwarder in the Splunk documentation for more information.

For example, execute the following command to configure a Splunk Universal Forwarder to send data to an indexer on port 9997:

$ sudo splunk add forward-server 192.168.1.71:9997

The following NXLog Agent configuration reads syslog messages from /var/log/messages and forwards them to a Splunk Universal Forwarder instance in JSON format.

nxlog.conf

<Extension syslog>
    Module    xm_syslog
</Extension>

<Extension json>
    Module    xm_json
</Extension>

<Input var_log_messages>
    Module    im_file
    File      '/var/log/messages'
    Exec      parse_syslog();
</Input>

<Output splunk_universal_forwarder>
    Module    om_tcp
    Host      192.168.1.81:1538
    <Exec>
        $Forwarder = TRUE;
        to_json();
    </Exec>
</Output>

Example 12. Receiving logs from the Splunk Universal Forwarder

First, configure the Splunk Universal Forwarder to send logs to NXLog Agent by opening the configuration file at $SPLUNK_HOME/etc/system/local/outputs.conf and adding the following settings. Replace <nxlog-server> with NXLog Agent IP or hostname.

outputs.conf

[indexAndForward]
index = false

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
disabled = false
server = <nxlog-server>:9997

[tcpout-server://<nxlog-server>:9997]
sendCookedData = false

Then configure NXLog Agent to receive logs from the Splunk Universal Forwarder. The following configuration uses the im_tcp input module to listen for connections on all available interfaces.

nxlog.conf

<Input tcp_listen>
    Module        im_tcp
    ListenAddr    0.0.0.0:9997
    InputType     Dgram (1)
</Input>

1	The InputType common module directive needs to be set to `Dgram` to treat each packet as an event.

Disclaimer

While we endeavor to keep the information in our guides up to date and correct, NXLog makes no representations or warranties of any kind, express or implied about the completeness, accuracy, reliability, suitability, or availability of the content represented here. We update our screenshots and instructions on a best-effort basis.

The accurateness of the content was tested and proved to be working in our lab environment at the time of the last revision with the following software versions:

NXLog Agent version 5.7.7898
Splunk Enterprise version 9.0.3
Splunk Cloud Platform 9.0.2209.2
Splunk Add-on for Microsoft Windows version 8.5.0

Last revision: 10 February 2023