Rapid7 InsightIDR SIEM
Rapid7 InsightIDR is an intruder analytics suite that helps detect and investigate security incidents. It works with data collected from network logs, authentication logs, and other log sources from endpoint devices.
NXLog Agent can be configured to collect and forward event logs to Rapid7 SIEM. It can also be used to rewrite event fields to meet the log field name requirements of InsightIDR’s Universal Event Format (UEF).
Configuring InsightIDR for log collection
This topic provides information about setting up log sources for InsightIDR. This will need to be done once for each log source, making sure that the correct details are provided for each log type collected from that source. In addition to this guide, please see the Rapid7 InsightIDR documentation.
-
Create, deploy and activate an InsightIDR Collector. A Collector is required before adding any data sources to InsightIDR.
Read more about the requirements in Rapid7’s InsightIDR Collector Requirements documentation before you install and deploy the InsightIDR Collector.
-
To confirm that the Collector is running, select Data Collection in the left side panel, then under the Data Collection Management pane, select the Collectors tab.
Here you can check the state of the Collectors. If the Collector is not running, review the Collector Troubleshooting page in the Rapid7’s Collector Troubleshooting documentation.
-
To add a new Data Source, in the Data Collection Management pane, select the Event Sources tab, then in the Product Types list, adjacent to Rapid7, click Add.
The Add Event source wizard opens.
-
To configure the Event Source, select the name of the Collector and the Event Source from the corresponding dropdown lists, optionally enter the Display Name, and then select the Timezone from the dropdown list.
For the Event Source, select either Rapid 7 Raw Data (if using JSON) or Rapid7 Generic Syslog (if using Syslog-formatted logs).
-
For the Collection Method, select the Listen For Syslog button, in the Port field enter the port number, then from the dropdown list select a Protocol.
-
If TCP was selected for the Protocol, optionally select Encrypted, then click Save.
The newly created Event Source is visible under the Event Sources tab of the Data Collection Management Pane.
Configuring NXLog Agent for log processing
This section shows example configurations to send event logs to InsightIDR.
Sending generic structured logs
NXLog Agent can be used to send structured logs (as JSON or other KVP), and generic log formats like Snare or Syslog to InsightIDR. To illustrate this, the following examples show configurations collecting and sending event logs from common Windows log sources.
This configuration uses the im_etw module to collect Windows DNS Server logs and sends it to InsightIDR in JSON format.
<Input etw_in>
Module im_etw
Provider Microsoft-Windows-DNSServer
Exec to_json();
</Input>
{
"SourceName": "Microsoft-Windows-DNSServer",
"ProviderGuid": "{EB79061A-A566-4698-9119-3ED2807060E7}",
"EventID": 256,
"EventTime": "2019-02-07T11:03:18.320983+00:00",
"Domain": "NT AUTHORITY",
"AccountName": "SYSTEM",
"UserID": "S-1-5-18",
"AccountType": "User",
"Source": "172.31.33.197",
"QNAME": "1.0.0.127.in-addr.arpa.",
"QTYPE": "12",
"XID": "2767",
"EventReceivedTime": "2019-02-07T11:03:19.330496+00:00",
"SourceModuleName": "etw_in",
"SourceModuleType": "im_etw"
}
This example sends Windows Event Log collected from the Security Channel using the im_msvistalog module. The events are sent to InsightIDR in Snare format. When sending Windows Event Log security logs, create a data source with the type Rapid7 Generic Windows Event Log.
<Input eventlog_in>
Module im_msvistalog
<QueryXML>
<QueryList>
<Query Id='0'>
<Select Path='Security'>*</Select>
</Query>
</QueryList>
</QueryXML>
<Exec>
$Message = replace($Message, "\t", " "); $Message = replace \
($Message, "\n", " "); $Message = replace($Message, "\r", " ");
$raw_event = $Message;
to_syslog_snare();
</Exec>
</Input>
<14>Dec 13 18:12:48 DC71.nxlog.internal MSWinEventLog 1 Security 1 Fri Dec 13 18:12:48 2019 4634 Microsoft-Windows-Security-Auditing N/A N/A Success Audit DC71.nxlog.internal Logoff An account was logged off. Subject: Security ID: S-1-5-18 Account Name: DC01$ Account Domain: NXLOG Logon ID: 0x4DD51 Logon Type: 3 25885
In this configuration, the im_msvistalog module is configured to collect Windows DHCP logs and send them as JSON, but other types of Windows events can be collected too. In this case, the Rapid7 log source is set as Rapid7 Generic Syslog, so the logs are indexed and parsed.
<Input dhcp_server_eventlog>
Module im_msvistalog
<QueryXML>
<QueryList>
<Query Id="0">
<Select Path="DhcpAdminEvents">*</Select>
<Select Path="Microsoft-Windows-Dhcp-Server/FilterNotifications"> \
*</Select>
<Select Path="Microsoft-Windows-Dhcp-Server/Operational">*</Select>
</Query>
</QueryList>
</QueryXML>
Exec to_syslog_bsd();
</Input>
<14>Jul 12 14:01:25 NXLog.co Microsoft-Windows-DHCP-Server[1836]: Scope: [[171.40.0.0]Test 3] for IPv4 is Deleted by NXLOG\Administrator.
Converting and sending logs in Universal Event Format (UEF)
Rapid7 InsightIDR allows certain types of non-InsightIDR event sources to access the same functionality as InsightIDR event sources such as User Behavior Analytics. Use the previous steps to add a new Data Source in InsightIDR keeping in mind the following:
-
The Event Source should be one of the supported Rapid7 Universal Event Format types.
-
The logs need to be converted to either JSON or KVP.
-
Relevant fields should be rewritten, added and whitelisted to support the UEF specification for the log source type.
-
Confirm that the logs have no format violations and are correctly indexed by Rapid7 InsightIDR. See the Verifying Data Collection section.
The following configuration examples are based on collecting Rapid7 Ingress Authentication events. The steps, fields and input options will vary depending on the UEF source types. For more information, see the Universal Event Sources section in the Rapid7 documentation.
Use the xm_rewrite module to rename raw data fields to match SIEM and dashboard field names. |
Use the xm_kvp module to delete, add and rename raw data fields. For more information, see the Universal Event Formats in InsightIDR: A Step-by-Step NXLog Guide in the Rapid7 documentation. |
Use the xm_rewrite module to specify which fields to keep and
rename. The fields to rewrite will depend on the operating system as shown
below. For both, the fields $version
and $event_type
are added.
<Extension rewrite>
Module xm_rewrite
# Fields associated with UEF are whitelisted
Keep EventTime, version, event_type, authentication_result \
IpAddress, WorkstationName, Hostname
# Rename the following fields to the UEF specification
Rename EventTime, time
Rename Hostname, account
Rename IpAddress, source_ip
Rename WorkstationName, authentication_target
Rename Version, version
</Extension>
<Extension rewrite>
Module xm_rewrite
# The syslog raw data needs to be parsed first
Exec parse_syslog();
Keep Hostname, account, version, user, custom_message, Message, \
event_type, EventReceivedTime, authentication_result, raw_event, \
authentication_target, source_ip
Rename HostName, authentication_target
Rename EventReceivedTime, time
Rename Message, custom_message
</Extension>
In Windows, $EventTime
is converted to the required ISO 8601 date format. The
SUCCESS
and FAILURE
results are mapped to $authentication_result
based on
the event ID.
<Input in_auth_windows>
Module im_msvistalog
<QueryXML>
<QueryList>
<Query Id="0">
<Select Path="Security">*[System[(Level<=4)]]</Select>
</Query>
</QueryList>
</QueryXML>
<Exec>
# Convert the $EventTime string to ISO 8601 extended format.
$EventTime = strftime($EventTime, '%Y-%m-%dT%H:%M:%SZ');
# Add the required input for $version
$version = "v1";
# Add the required input for $event_type
$event_type = "INGRESS_AUTHENTICATION";
# Add the required authentication results for Windows Event Log IDs
if ($EventID IN (4625)) { $authentication_result = "FAILURE"; } \
else if ($EventID IN (4624)) { $authentication_result = "SUCCESS"; } \
# Drop all other event IDs
else drop();
# Add the process to rewrite the fields and convert to JSON
rewrite->process();
to_json();
</Exec>
</Input>
In Linux, $EventReceivedTime
is used and converted to the ISO 8601 format.
The SUCCESS
and FAILURE
results are mapped to $authentication_result
based on string results in the $raw_event
field. Additional parsing of the
$raw_event
field is made to obtain the string data for the $account
and
$source_ip
values.
<Input in_auth_linux>
Module im_file
File "/var/log/auth.log"
<Exec>
# Convert the $EventReceivedTime string to ISO 8601 extended format
$EventReceivedTime = strftime($EventReceivedTime, '%Y-%m-%dT%H:%M:%SZ');
# Add the required input for $version
$version = "v1";
# Add the required input for $event_type
$event_type = "INGRESS_AUTHENTICATION";
# Use xm_rewrite module for the $source_ip and $account fields
rewrite->process();
# Obtain the $source_ip and $account string data from $raw_event
if ($raw_event =~ /Accepted publickey for\ (\S+)\ from\ (\S+)/)
{
$account = $1;
$source_ip = $2;
}
# Success and Authentication messages based on the $raw_event field
if ($raw_event =~ /authentication failure/) \
{ $authentication_result = "FAILURE"; } \
else if ($raw_event =~ /successfully authenticated/) \
{ $authentication_result = "SUCCESS"; } \
# Ensure that only failure and success messages are logged
else drop();
# Convert to JSON
to_json();
</Exec>
</Input>
The following examples display the JSON output based on the NXLog Agent
configuration files above. It is recommended to first test the input to
determine that the fields are renamed and added. There is an option to provide
a $custom_message
, as displayed in the Linux example.
{
"time": "2019-07-26T19:34:03Z",
"version": "v1",
"account": "ADMINISTRATOR",
"authentication_target": "HR Workstation",
"source_ip": "121.137.167.158",
"event_type": "INGRESS_AUTHENTICATION",
"authentication_result": "FAILURE"
}
{
"time": "2019-08-17T18:15:27Z",
"version": "v1",
"event_type": "INGRESS_AUTHENTICATION",
"account": "ubuntu",
"source_ip": "172.5.160.165",
"authentication_result": "SUCCESS",
"authentication_target": "ip-172-31-17-116",
"custom_message": "Accepted publickey for ubuntu from 172.5.160.165 port 58788 ssh2: RSA SHA256:5kZ3eXnEIFf4orffpf924pbJCgPj57EQRHWBj7E"
}
The following is an event sample in JSON format as indexed by Rapid7 InsightIDR.
{
"timestamp": "2019-07-26T19:34:03.000Z",
"user": "administrator",
"account": "administrator",
"result": "FAILED_OTHER",
"source_ip": "121.137.167.158",
"service": "CUSTOM UNIVERSAL EVENT",
"geoip_organization": "Korea Telecom",
"geoip_country_code": "KR",
"geoip_country_name": "South Korea",
"geoip_city": "Pyeongtaek-si",
"geoip_region": "41",
"authentication_target": "HR Workstation",
"source_json": {
"time": "2019-07-26T19:34:03Z",
"version": "v1",
"account": "ADMINISTRATOR",
"authentication_target": "WorkstationName",
"source_ip": "121.137.167.158",
"event_type": "INGRESS_AUTHENTICATION",
"authentication_result": "FAILURE"
}
}
Verifying data collection
To verify data collection, check that the event source is collecting the raw logs, and that Rapid7 InsightIDR is indexing them.
-
In the Data Collection Management pane, go to the Event Sources tab.
-
Select View raw log to see recent raw logs for the event source.
-
To verify log indexing, go to the Log Search pane and find the logs via the Logs or Log sets options.
Once indexed, logs collected using NXLog Agent can be further processed in Rapid7 InsightIDR.