NXLog Legacy Documentation

Rapid7 InsightIDR SIEM

Rapid7 InsightIDR is an intruder analytics suite that helps detect and investigate security incidents. It works with data collected from network logs, authentication logs, and other log sources from endpoint devices.

NXLog can be configured to collect and forward event logs to Rapid7 SIEM. It can also be used to rewrite event fields to meet the log field name requirements of InsightIDR’s Universal Event Format (UEF).

Configuring InsightIDR for log collection

This topic provides information about setting up log sources for InsightIDR. This will need to be done once for each log source, making sure that the correct details are provided for each log type collected from that source. In addition to this guide, please see the Rapid7 InsightIDR documentation.

Create, deploy and activate an InsightIDR Collector. A Collector is required before adding any data sources to InsightIDR.

Read more about the requirements in Rapid7’s InsightIDR Collector Requirements documentation before you install and deploy the InsightIDR Collector.
To confirm that the Collector is running, select Data Collection in the left side panel, then under the Data Collection Management pane, select the Collectors tab.

Here you can check the state of the Collectors. If the Collector is not running, review the Collector Troubleshooting page in the Rapid7’s Collector Troubleshooting documentation.
To add a new Data Source, in the Data Collection Management pane, select the Event Sources tab, then in the Product Types list, adjacent to Rapid7, click Add.

The Add Event source wizard opens.
To configure the Event Source, select the name of the Collector and the Event Source from the corresponding dropdown lists, optionally enter the Display Name, and then select the Timezone from the dropdown list.

For the Event Source, select either Rapid 7 Raw Data (if using JSON) or Rapid7 Generic Syslog (if using Syslog-formatted logs).
For the Collection Method, select the Listen For Syslog button, in the Port field enter the port number, then from the drop down list select a Protocol.
If TCP was selected for the Protocol, optionally select Encrypted, then click Save.

The newly created Event Source is visible under the Event Sources tab of the Data Collection Management Pane.

Configuring NXLog for log processing

This section shows example configurations to send event logs to InsightIDR.

Sending generic structured logs

NXLog can be used to send structured logs (as JSON or other KVP), and generic log formats like Snare or Syslog to InsightIDR. To illustrate this, the following examples show configurations collecting and sending event logs from common Windows log sources.

Example 1. Logs collected from Event Tracing for Windows (ETW)

NXLog Enterprise Edition exclusive feature

This configuration uses the im_etw module to collect Windows DNS Server log data and send it to InsightIDR as JSON.

nxlog.conf

<Input etw_in>
    Module      im_etw
    Provider    Microsoft-Windows-DNSServer
    Exec        to_json();
</Input>

Structured JSON event sample in InsightIDR

{
  "SourceName": "Microsoft-Windows-DNSServer",
  "ProviderGuid": "{EB79061A-A566-4698-9119-3ED2807060E7}",
  "EventID": 256,
  "EventTime": "2019-02-07T11:03:18.320983+00:00",
  "Domain": "NT AUTHORITY",
  "AccountName": "SYSTEM",
  "UserID": "S-1-5-18",
  "AccountType": "User",
  "Source": "172.31.33.197",
  "QNAME": "1.0.0.127.in-addr.arpa.",
  "QTYPE": "12",
  "XID": "2767",
  "EventReceivedTime": "2019-02-07T11:03:19.330496+00:00",
  "SourceModuleName": "etw_in",
  "SourceModuleType": "im_etw"
}

Example 2. Sending Windows Event Log security logs

This example sends Windows Event Log collected from the Security Channel using the im_msvistalog module. The events are sent to InsightIDR in Snare format. When sending Windows Event Log security logs, create a data source with the type Rapid7 Generic Windows Event Log.

nxlog.conf

<Input eventlog_in>
    Module    im_msvistalog
    <QueryXML>
        <QueryList>
            <Query Id='0'>
            <Select Path='Security'>*</Select>
            </Query>
        </QueryList>
    </QueryXML>
    <Exec>
        $Message = replace($Message, "\t", " "); $Message = replace \
        ($Message, "\n", " "); $Message = replace($Message, "\r", " ");
        $raw_event = $Message;
        to_syslog_snare();
    </Exec>
</Input>

Output sample ("Snare Over Syslog")

<14>Dec 13 18:12:48 DC71.nxlog.internal MSWinEventLog	1	Security	1	Fri Dec 13 18:12:48 2019	4634	Microsoft-Windows-Security-Auditing	N/A	N/A	Success Audit	DC71.nxlog.internal	Logoff		An account was logged off.    Subject:   Security ID:  S-1-5-18   Account Name:  DC01$   Account Domain:  NXLOG   Logon ID:  0x4DD51    Logon Type:   3   25885

Example 3. Sending other Windows Event Log events

In this configuration, the im_msvistalog module is configured to collect Windows DHCP logs and send them as JSON, but other types of Windows events can be collected too. In this case, the Rapid7 log source is set as Rapid7 Generic Syslog, so the logs are indexed and parsed.

nxlog.conf

<Input dhcp_server_eventlog>
    Module    im_msvistalog
    <QueryXML>
        <QueryList>
            <Query Id="0">
                <Select Path="DhcpAdminEvents">*</Select>
                <Select Path="Microsoft-Windows-Dhcp-Server/FilterNotifications"> \
                *</Select>
                <Select Path="Microsoft-Windows-Dhcp-Server/Operational">*</Select>
            </Query>
        </QueryList>
    </QueryXML>
    Exec    to_syslog_bsd();
</Input>

DHCP output in Syslog format

<14>Jul 12 14:01:25 NXLog.co Microsoft-Windows-DHCP-Server[1836]: Scope: [[171.40.0.0]Test 3] for IPv4 is Deleted by NXLOG\Administrator.

Figure 1. DHCP as raw data in Rapid7

Converting and sending logs in Universal Event Format (UEF)

Rapid7 InsightIDR allows certain types of non-InsightIDR event sources to access the same functionality as InsightIDR event sources such as User Behavior Analytics. Use the previous steps to add a new Data Source in InsightIDR keeping in mind the following:

The Event Source should be one of the supported Rapid7 Universal Event Format types.
The logs need to be converted to either JSON or KVP.
Relevant fields should be rewrited, added and whitelisted to support the UEF specification for the log source type.
Confirm that the logs have no format violations and are correctly indexed by Rapid7 InsightIDR. See the Verifying Data Collection section.

The following configuration examples are based on collecting Rapid7 Ingress Authentication events. The steps, fields and input options will vary depending on the UEF source types. For more information, see the Universal Event Sources section in the Rapid7 documentation.

Use the xm_rewrite module to rename raw data fields to match SIEM and dashboard field names.

Use the xm_kvp module to delete, add and rename raw data fields. For more information, see the Universal Event Formats in InsightIDR: A Step-by-Step NXLog Guide in the Rapid7 documentation.

Example 4. Configuring xm_rewrite for Windows and Linux

NXLog Enterprise Edition exclusive feature

Use the xm_rewrite module to specify which fields to keep and rename. The fields to rewrite will depend on the operating system as shown below. For both, the fields $version and $event_type are added.

nxlog.conf

<Extension rewrite>
    Module    xm_rewrite
    # Fields associated with UEF are whitelisted
    Keep      EventTime, version, event_type, authentication_result \
              IpAddress, WorkstationName, Hostname

    # Rename the following fields to the UEF specification
    Rename    EventTime, time
    Rename    Hostname, account
    Rename    IpAddress, source_ip
    Rename    WorkstationName, authentication_target
    Rename    Version, version
</Extension>

nxlog.conf

<Extension rewrite>
    Module    xm_rewrite   
    # The syslog raw data needs to be parsed first
    Exec      parse_syslog();
    Keep      Hostname, account, version, user, custom_message, Message, \
              event_type, EventReceivedTime, authentication_result, raw_event, \
              authentication_target, source_ip
    Rename    HostName, authentication_target
    Rename    EventReceivedTime, time
    Rename    Message, custom_message
</Extension>

Example 5. Configuring Rapid7 Universal Ingress Authentication log collection

In Windows, $EventTime is converted to the required ISO 8601 date format. The SUCCESS and FAILURE results are mapped to $authentication_result based on the event ID.

nxlog.conf

<Input in_auth_windows>
    Module  im_msvistalog
    <QueryXML>
        <QueryList>
            <Query Id="0">
            <Select Path="Security">*[System[(Level&lt;=4)]]</Select>
            </Query>
        </QueryList>
    </QueryXML>
        <Exec>
            # Convert the $EventTime string to ISO 8601 extended format.
            $EventTime = strftime($EventTime, '%Y-%m-%dT%H:%M:%SZ');

            # Add the required input for $version
            $version = "v1";

            # Add the required input for $event_type
            $event_type = "INGRESS_AUTHENTICATION";

            # Add the required authentication results for Windows Event Log IDs
            if ($EventID IN (4625)) { $authentication_result = "FAILURE"; } \
            else if ($EventID IN (4624)) { $authentication_result = "SUCCESS"; } \
            # Drop all other event IDs
            else drop();

            # Add the process to rewrite the fields and convert to JSON
            rewrite->process();
            to_json();
        </Exec>
</Input>

In Linux, $EventReceivedTime is used and converted to the ISO 8601 format. The SUCCESS and FAILURE results are mapped to $authentication_result based on string results in the $raw_event field. Additional parsing of the $raw_event field is made to obtain the string data for the $account and $source_ip values.

nxlog.conf

<Input in_auth_linux>
    Module    im_file
    File      "/var/log/auth.log"
    <Exec>
        # Convert the $EventReceivedTime string to ISO 8601 extended format
        $EventReceivedTime = strftime($EventReceivedTime, '%Y-%m-%dT%H:%M:%SZ');

        # Add the required input for $version
        $version = "v1";

        # Add the required input for $event_type
        $event_type = "INGRESS_AUTHENTICATION";

        # Use xm_rewrite module for the $source_ip and $account fields
        rewrite->process();

        # Obtain the $source_ip and $account string data from $raw_event
        if ($raw_event =~ /Accepted publickey for\ (\S+)\ from\ (\S+)/)
        {
            $account = $1;
            $source_ip = $2;
        }

        # Success and Authentication messages based on the $raw_event field
        if ($raw_event =~ /authentication failure/) \
        { $authentication_result = "FAILURE"; } \
        else if ($raw_event =~ /successfully authenticated/) \
        { $authentication_result = "SUCCESS"; } \

        # Ensure that only failure and success messages are logged
        else drop();

        # Convert to JSON
        to_json();
    </Exec>
</Input>

Example 6. Ingress authentication UEF event samples in JSON

The following examples display the JSON output based on the NXLog configuration files above. It is recommended to first test the input to determine that the fields are renamed and added. There is an option to provide a $custom_message, as displayed in the Linux example.

UEF event sample for Ingress authentication on Windows

{
  "time": "2019-07-26T19:34:03Z",
  "version": "v1",
  "account": "ADMINISTRATOR",
  "authentication_target": "HR Workstation",
  "source_ip": "121.137.167.158",
  "event_type": "INGRESS_AUTHENTICATION",
  "authentication_result": "FAILURE"
}

UEF event sample for Ingress authentication on Linux

{
  "time": "2019-08-17T18:15:27Z",
  "version": "v1",
  "event_type": "INGRESS_AUTHENTICATION",
  "account": "ubuntu",
  "source_ip": "172.5.160.165",
  "authentication_result": "SUCCESS",
  "authentication_target": "ip-172-31-17-116",
  "custom_message": "Accepted publickey for ubuntu from 172.5.160.165 port 58788 ssh2: RSA SHA256:5kZ3eXnEIFf4orffpf924pbJCgPj57EQRHWBj7E"
}

Example 7. Full Ingress authentication event sample indexed in Rapid7

The following is an event sample in JSON format as indexed by Rapid7 InsightIDR.

Windows UEF event sample on Rapid7 InsightIDR log view

{
  "timestamp": "2019-07-26T19:34:03.000Z",
  "user": "administrator",
  "account": "administrator",
  "result": "FAILED_OTHER",
  "source_ip": "121.137.167.158",
  "service": "CUSTOM UNIVERSAL EVENT",
  "geoip_organization": "Korea Telecom",
  "geoip_country_code": "KR",
  "geoip_country_name": "South Korea",
  "geoip_city": "Pyeongtaek-si",
  "geoip_region": "41",
  "authentication_target": "HR Workstation",
  "source_json": {
    "time": "2019-07-26T19:34:03Z",
    "version": "v1",
    "account": "ADMINISTRATOR",
    "authentication_target": "WorkstationName",
    "source_ip": "121.137.167.158",
    "event_type": "INGRESS_AUTHENTICATION",
    "authentication_result": "FAILURE"
  }
}

Verifying data collection

To verify data collection, check that the event source is collecting the raw logs, and that Rapid7 InsightIDR is indexing them.

In the Data Collection Management pane, go to the Event Sources tab.
Select View raw log to see recent raw logs for the event source.
To verify log indexing, go to the Log Search pane and find the logs via the Logs or Log sets options.

Once indexed, logs collected using NXLog can be further processed in Rapid7 InsightIDR.

Disclaimer

While we endeavor to keep the information in this topic up to date and correct, NXLog makes no representations or warranties of any kind, express or implied about the completeness, accuracy, reliability, suitability, or availability of the content represented here. We update our screenshots and instructions on a best-effort basis.

Last revision: 23 November 2020