Send logs to Microsoft Sentinel
Microsoft Sentinel is Microsoft’s security information event management (SIEM), offered as a service within Azure. Because of its presence within Azure and close integration with other Azure services, Microsoft refers to Sentinel as "a scalable, cloud-native, and security orchestration automated response (SOAR) solution." For more information about Microsoft Sentinel, see Microsoft Sentinel documentation.
NXLog can be configured as a log collector agent for Microsoft Sentinel, collecting and forwarding logs to its Azure Log Analytics workspaces. The logs that NXLog can forward to Microsoft Sentinel include Windows DNS Server logs, Linux audit logs, and AIX audit logs. NXLog can also send security logs directly to Microsoft Sentinel using the Microsoft Sentinel (om_azure) module. NXLog’s advanced log collection, processing, and forwarding capabilities make it a perfect all-in-one solution for sending logs to Microsoft Sentinel.
Prerequisites
To send any type of log to Microsoft Sentinel from NXLog, a few prerequisites need to be met.
-
A subscription to Microsoft Azure
-
An Azure Log Analytics workspace in the Azure portal
-
Contributor permissions to the subscription in which the Microsoft Sentinel workspace resides
-
Enabling Microsoft Sentinel if it is not already activated
Microsoft Sentinel authentication
The first step in preparing to configure om_azure is to retrieve the Workspace ID and either the Primary key or the Secondary key. These keys can be found by navigating the Azure portal to Log Analytics workspace > Settings > Agents and clicking the collapsed Log Analytics agent instructions section. The same set of keys can be viewed under either the Windows servers or Linux servers tab.
The following om_azure directives are required to authenticate with Microsoft Sentinel:
| Directive | Description | Sample Value |
|---|---|---|
|
Your organization’s Workspace ID |
|
|
Either your Primary key or your Secondary key |
|
|
The Azure endpoint |
|
When assigning values to directives that are subject to change by a third-party vendor — or that might change from one organizational unit to another — using define directives to establish the constants needed for a module instance is a best practice.
In the following partial configuration file, the values for the om_azure directives that will rarely change are defined.
The contents of this file will be read by the main nxlog.conf per file inclusion:
define WORKSPACE DUMMY_WORKSPACE
define SHAREDKEY DUMMY_SHAREDKEY
define SUBDOMAIN ods.opinsights.azure.com
define RESOURCE api/logs
define APIVER api-version=2016-04-01Likewise, below is another partial configuration containing a complete, generic om_azure instance that is platform-agnostic and thus well-suited for automated deployments.
It will also be read by the main nxlog.conf configuration file per file inclusion:
<Output azure>
Module om_azure
URL https://%WORKSPACE%.%SUBDOMAIN%/%RESOURCE%?%APIVER%
WorkspaceID %WORKSPACE%
SharedKey %SHAREDKEY%
TableName "%TABLE%"
HTTPSCAFile %CAFILE%
</Output>These two partial configuration files comprise an almost complete configuration for sending logs to Microsoft Sentinel and will be used for the following configuration examples.
Only two om_azure directives are missing values: %TABLE% because it will need to change frequently depending on the log source being monitored, and %CAFile% because it can change depending on the platform where NXLog Enterprise Edition is running.
They will be defined within the main nxlog.conf configuration file immediately before the inclusion of om-azure.conf.
Gathering and forwarding security logs to Microsoft Sentinel
The following three examples collect security logs from three different platforms and forward them to Microsoft Sentinel:
Since the output instances for all four examples are almost identical, but their log sources differ greatly, the primary focus will be on configuring their input instances. Once the security logs have been captured and prepared for sending to Microsoft Sentinel, the steps for verifying the ingested data are the same, aside from their respective table names and queries:
-
After logging in to Microsoft Azure, navigate to Microsoft Sentinel and select the Workspace that will receive the forwarded security events.
-
Under the General heading immediately below Overview click Logs, which will open up the following Azure Queries modal. Close this modal.
-
Under the Tables tab, expand the Custom Logs heading to reveal the list of table names that should correspond to the value assigned
define TABLEin each of the configurations. This value is eventually assigned to theTableNamedirective in the generic om-azure.conf file.
The om_azure module uses the Azure Monitor HTTP Data Collector API to forward events to Microsoft Sentinel. Azure classifies all log sources using this API as Custom Logs. When Log Analytics ingests JSON data from Custom Logs, it appends
_CLto the value of%TABLE%thus table names will appear with this modification under the Custom Logs heading of table types as seen below.Additionally, Log Analytics renames each event record’s fields: “To identify a property’s data type, Azure Monitor adds a suffix to the property name.” For example, datetime fields like
$EventTimewill have_tappended to their names:EventTime_t. Fields used for numeric data like$EventIDwill have_dappended:EventID_d. Fields representing string values like$Hostnamewill have_sappended:Hostname_s.For details about this process, see the Azure Monitor Documentation - Custom logs: Record type and properties.
-
In the input area beneath the Run button, compose the Kusto queries specific to each table for verifying the successful ingestion of the security events.
The Kusto Query Language (KQL) provides a rich set of tools that allow one to manipulate existing tables to create new table views. Some of the following examples provide instructions with KQL sample functions for achieving this. Using this technique, fields can be normalized according to Microsoft Sentinel specifications so that the ingested logs can be better integrated with Microsoft Sentinel analytics. Another use case is to undo the effects of Log Analytics' automated renaming of all fields on ingestion. This could prove useful in cases where custom log analytics have been developed that rely on the original field names.
Forwarding Windows DNS Server logs to Microsoft Sentinel
In this example, NXLog collects DNS Server logs via ETW from the Microsoft-Windows-DNSServer provider using the Event Tracing for Windows (im_etw) module.
This Microsoft Sentinel data connector will collect both audit and analytical DNS Server events, however in the Windows Event Viewer, the analytical events are not shown by default.
This example focuses on analytical events.
To view analytical events in the Event Viewer, you have to enable analytical event logging: open Event Viewer, navigate to Applications and Services Logs\Microsoft\Windows\DNS-Server, and use View > Show Analytic and Debug Logs to change the properties of the Analytical log.
However, the im_etw module will collect both audit and analytical events regardless of the settings in Event Viewer.
For additional details, see DNS logging via ETW Providers.
In the following configuration, the Exec block contains a call to the core function host_ip() since Microsoft Sentinel prefers having a non-loopback IP address. Each record is converted to JSON using the to_json() procedure of the xm_json module to enrich the event log with the NXLog core fields.
Right after the external azure-defines.conf is included, TABLE and CAFILE are defined since they have been intentionally excluded from the generic, external om-azure.conf output instance.
They will be referenced by the azure output instance once loaded via the next include.
<Extension json>
Module xm_json
</Extension>
<Input dnsserver>
Module im_etw
Provider Microsoft-Windows-DNSServer
<Exec>
$HostIP = host_ip();
to_json();
</Exec>
</Input>
include %INSTALLDIR%\conf\azure-defines.conf
define TABLE NXLog_DNS_Server
define CAFILE %CERTDIR%\ca-certificates.crt
include %INSTALLDIR%\conf\om-azure.conf
<Route r1>
Path dnsserver => azure
</Route>The following Windows DNS Server analytical event was parsed by the im_etw module and converted to JSON before it was sent to Microsoft Sentinel.
{
"SourceName": "Microsoft-Windows-DNSServer",
"ProviderGuid": "{EB79061A-A566-4698-9119-3ED2807060E7}",
"EventID": 261,
"Version": 0,
"ChannelID": 16,
"OpcodeValue": 0,
"TaskValue": 2,
"Keywords": "9223372036854775840",
"EventTime": "2021-10-05T22:19:59.457092-07:00",
"ExecutionProcessID": 2080,
"ExecutionThreadID": 2940,
"EventType": "INFO",
"SeverityValue": 2,
"Severity": "INFO",
"Hostname": "WIN-93NOI1UVL29",
"Domain": "NT AUTHORITY",
"AccountName": "SYSTEM",
"UserID": "S-1-5-18",
"AccountType": "User",
"Flags": "32768",
"TCP": "0",
"Source": "2603:1061::cd",
"InterfaceIP": "::",
"AA": "0",
"AD": "0",
"QNAME": "8c70dfcb-83e1-4959-934d-fcb4860bf1de.ods.opinsights.azure.com.",
"QTYPE": "28",
"XID": "39719",
"RecursionDepth": "1",
"Port": "0",
"RecursionScope": ".",
"CacheScope": "Default",
"BufferSize": "224",
"PacketData": "0x9B27800000010000000400012438633730646663622D383365312D343935392D393334642D666362343836306266316465036F64730A6F70696E73696768747305617A75726503636F6D00001C0001C031000200010000012C0013066E73312D303109617A7572652D646E73C046C04F000200010000012C0016066E73322D303109617A7572652D646E73036E657400C04F000200010000012C0016066E73332D303109617A7572652D646E73036F726700C04F000200010000012C0017066E73342D303109617A7572652D646E7304696E666F0000002904D0000080000000",
"AdditionalInfo": ".",
"GUID": "{18F714C3-99C7-41EA-A194-15C7EA1EF5EB}",
"EventReceivedTime": "2021-10-05T22:20:00.460347-07:00",
"SourceModuleName": "dnsserver",
"SourceModuleType": "im_etw",
"HostIP": "192.168.1.36"
}The most basic query in the Microsoft Sentinel query editor is simply the table name itself, in this case, NXLog_DNS_Server_CL.
Here is the same sample event after Log Analytics ingested it and renamed the fields.
The following KQL ASimDnsMicrosoftNXLog() function serves as an additional parser for defining a table view that adheres to the Microsoft Sentinel DNS normalization schema.
This is one of the various schemas Microsoft Sentinel has defined "to enable source-agnostic analytics."
With this normalized DNS parser, access to an ever-growing number of built-in analytics rules is now available.
This KQL function uses the same NXLog_DNS_Server_CL as its data source.
// Usage Instructions:
// Paste the query below into the Log Analytics query editor.
// Click the "Save" button and select "Save as function".
// Enter "ASimDnsMicrosoftNXLog" in the "Function name" field.
// For "Legacy category:" enter "DNS Server logs".
// "Parameters" are not needed.
// Function usually takes 10-15 minutes to activate.
// You can then use this function from any other queries (e.g. ASimDnsMicrosoftNXLog | take 10).
// Reference: Using functions in Azure monitor log queries: https://docs.microsoft.com/azure/azure-monitor/log-query/functions
let ASimDnsMicrosoftNXLog = (disabled:bool=false) {
let EventTypeTable=datatable(EventOriginalType:real,EventType:string)[
256, 'Query'
, 257, 'Query'
, 258, 'Query'
, 259, 'Query'
, 260, 'Query'
, 261, 'Query'
, 262, 'Query'
, 263, 'Dynamic update'
, 264, 'Dynamic update'
, 265, 'Zone XFR'
, 266, 'Zone XFR'
, 267, 'Zone XFR'
, 268, 'Zone XFR'
, 269, 'Zone XFR'
, 270, 'Zone XFR'
, 271, 'Zone XFR'
, 272, 'Zone XFR'
, 273, 'Zone XFR'
, 274, 'Zone XFR'
, 275, 'Zone XFR'
, 276, 'Zone XFR'
, 277, 'Dynamic update'
, 278, 'Dynamic update'
, 279, 'Query'
, 280, 'Query'
];
let EventSubTypeTable=datatable(EventOriginalType:real,EventSubType:string)[
256, 'request'
, 257, 'response'
, 258, 'response'
, 259, 'response'
, 260, 'request'
, 261, 'response'
, 262, 'response'
, 263, 'request'
, 264, 'response'
, 265, 'request'
, 266, 'request'
, 267, 'response'
, 268, 'response'
, 269, 'request'
, 270, 'request'
, 271, 'response'
, 272, 'response'
, 273, 'request'
, 274, 'request'
, 275, 'response'
, 276, 'response'
, 277, 'request'
, 278, 'response'
, 279, 'response'
, 280, 'response'
];
let EventResultTable=datatable(EventOriginalType:real,EventResult:string)[
256, 'NA'
, 257, 'Success'
, 258, 'Failure'
, 259, 'Failure'
, 260, 'NA'
, 261, 'NA'
, 262, 'Failure'
, 263, 'NA'
, 264, 'Based on RCODE'
, 265, 'NA'
, 266, 'NA'
, 267, 'Based on RCODE'
, 268, 'Based on RCODE'
, 269, 'NA'
, 270, 'NA'
, 271, 'Based on RCODE'
, 272, 'Based on RCODE'
, 273, 'NA'
, 274, 'NA'
, 275, 'Success'
, 276, 'Success'
, 277, 'NA'
, 278, 'Based on RCODE'
, 279, 'NA'
, 280, 'NA'
];
let RCodeTable=datatable(DnsResponseCode:int,ResponseCodeName:string)[
0,'NOERROR'
, 1,'FORMERR'
, 2,'SERVFAIL'
, 3,'NXDOMAIN'
, 4,'NOTIMP'
, 5,'REFUSED'
, 6,'YXDOMAIN'
, 7,'YXRRSET'
, 8,'NXRRSET'
, 9,'NOTAUTH'
, 10,'NOTZONE'
, 11,'DSOTYPENI'
, 16,'BADVERS'
, 16,'BADSIG'
, 17,'BADKEY'
, 18,'BADTIME'
, 19,'BADMODE'
, 20,'BADNAME'
, 21,'BADALG'
, 22,'BADTRUNC'
, 23,'BADCOOKIE'
];
let QTypeTable=datatable(DnsQueryType:int,QTypeName:string)[
0, 'Reserved'
, 1, 'A'
, 2, 'NS'
, 3, 'MD'
, 4, 'MF'
, 5, 'CNAME'
, 6, 'SOA'
, 7, 'MB'
, 8 ,'MG'
, 9 ,'MR'
, 10,'NULL'
, 11,'WKS'
, 12,'PTR'
, 13,'HINFO'
, 14,'MINFO'
, 15,'MX'
, 16,'TXT'
, 17,'RP'
, 18,'AFSDB'
, 19,'X25'
, 20,'ISDN'
, 21,'RT'
, 22,'NSAP'
, 23,'NSAP-PTR'
, 24,'SIG'
, 25,'KEY'
, 26,'PX'
, 27,'GPOS'
, 28,'AAAA'
, 29,'LOC'
, 30,'NXT'
, 31,'EID'
, 32,'NIMLOC'
, 33,'SRV'
, 34,'ATMA'
, 35,'NAPTR'
, 36,'KX'
, 37,'CERT'
, 38,'A6'
, 39,'DNAME'
, 40,'SINK'
, 41,'OPT'
, 42,'APL'
, 43,'DS'
, 44,'SSHFP'
, 45,'IPSECKEY'
, 46,'RRSIG'
, 47,'NSEC'
, 48,'DNSKEY'
, 49,'DHCID'
, 50,'NSEC3'
, 51,'NSEC3PARAM'
, 52,'TLSA'
, 53,'SMIMEA'
, 55,'HIP'
, 56,'NINFO'
, 57,'RKEY'
, 58,'TALINK'
, 59,'CDS'
, 60,'CDNSKEY'
, 61,'OPENPGPKEY'
, 62,'CSYNC'
, 63,'ZONEMD'
, 64,'SVCB'
, 65,'HTTPS'
, 99,'SPF'
, 100,'UINFO'
, 101,'UID'
, 102,'GID'
, 103,'UNSPEC'
, 104,'NID'
, 105,'L32'
, 106,'L64'
, 107,'LP'
, 108,'EUI48'
, 109,'EUI64'
, 249,'TKEY'
, 250,'TSIG'
, 251,'IXFR'
, 252,'AXFR'
, 253,'MAILB'
, 254,'MAILA'
, 255,'*'
, 256,'URI'
, 257,'CAA'
, 258,'AVC'
, 259,'DOA'
, 32768,'TA'
, 32769,'DLV'
];
NXLog_DNS_Server_CL | where not(disabled)
| where EventID_d < 281
| project-rename
DnsFlags=Flags_s,
DnsQuery=QNAME_s,
DnsQueryType=QTYPE_s,
DnsResponseCode=RCODE_s,
DnsResponseName=PacketData_s,
Dvc=Hostname_s,
EventOriginalType=EventID_d,
EventOriginalUid=GUID_g,
EventStartTime=EventTime_t,
SrcIpAddr=Source_s,
EventUid=_ItemId
| extend
DnsQuery=trim_end(".",DnsQuery),
DnsQueryType=toint(DnsQueryType),
DnsResponseCode=toint(DnsResponseCode),
SrcPortNumber=toint(Port_s),
DvcHostname=Dvc,
DvcIpAddr=HostIP_s,
EventEndTime=EventStartTime,
EventProduct = "DNS Server",
EventSchemaVersion = "0.1.7",
EventVendor = "Microsoft",
EventSchema = "Dns",
EventCount = int(1),
NetworkProtocol=iff(TCP_s == "0","UDP","TCP"),
TransactionIdHex=tohex(toint(XID_s)),
DnsFlagsAuthenticated = tobool(AD_s),
DnsFlagsAuthoritative = tobool(AA_s),
DnsFlagsRecursionDesired = tobool(RD_s)
| lookup EventTypeTable on EventOriginalType
| lookup EventSubTypeTable on EventOriginalType
| lookup EventResultTable on EventOriginalType
| lookup RCodeTable on DnsResponseCode
| lookup QTypeTable on DnsQueryType
| extend
EventResultDetails = case (isnotempty(ResponseCodeName), ResponseCodeName
, DnsResponseCode between (3841 .. 4095), 'Reserved for Private Use'
, 'Unassigned'),
EventOriginalType = tostring(EventOriginalType)
| extend
Domain=DnsQuery,
DnsResponseCodeName=EventResultDetails,
DnsQueryTypeName = case (isnotempty(QTypeName), QTypeName
, DnsQueryType between (66 .. 98), 'Unassigned'
, DnsQueryType between (110 .. 248), 'Unassigned'
, DnsQueryType between (261 .. 32767), 'Unassigned'
, 'Unassigned'),
EventResult=iff (EventResult == "Based on RCODE", iff(DnsResponseCode == 0, "Success", "Failure"),EventResult)
| extend
// Aliases
IpAddr = SrcIpAddr,
Src = SrcIpAddr
| project-away
*_s, *_d, QTypeName, TenantId, SourceSystem, MG, ManagementGroupName, Computer, RawData, ResponseCodeName, EventReceivedTime_t, ProviderGuid_g, _ResourceId
};
ASimDnsMicrosoftNXLog(disabled=disabled)Here are some simple aggregation queries that use this schema.
Forwarding Linux audit logs to Microsoft Sentinel
In this example, the NXLog Linux Audit System im_linuxaudit module is used as the event source.
The xm_resolver module is needed for the ResolveValues directive in the LinuxAudit input instance, where it is used for resolving some of the numeric values to human-readable string values.
The LinuxAudit input instance gets its ruleset from an external rules file that includes rules for monitoring changes to DNS zone files.
The Exec block contains a call to the core function hostname() since the Linux Audit system does not include a hostname field by default.
This creates a new $Hostname field.
The to_json() procedure of the xm_json module is also invoked to format the events as JSON records and enrich them with the NXLog core fields.
Right after the external azure-defines.conf is included, TABLE and CAFILE are defined since they have been intentionally excluded from the generic, external om-azure.conf output instance.
They will be referenced by the azure output instance once loaded via the next include.
<Extension json>
Module xm_json
</Extension>
<Extension resolver>
Module xm_resolver
</Extension>
<Input LinuxAudit>
Module im_linuxaudit
FlowControl FALSE
LoadRule /opt/nxlog/etc/im_linuxaudit.rules
ResolveValues TRUE
<Exec>
$Hostname = hostname();
to_json();
</Exec>
</Input>
define WORKSPACE DUMMY_WORKSPACE
define SHAREDKEY DUMMY_SHAREDKEY
define SUBDOMAIN ods.opinsights.azure.com
define RESOURCE api/logs
define APIVER api-version=2016-04-01
define TABLE LinuxAudit
define CAFILE /etc/ssl/certs/ca-certificates.crt
<Output azure>
Module om_azure
URL https://%WORKSPACE%.%SUBDOMAIN%/%RESOURCE%?%APIVER%
WorkspaceID %WORKSPACE%
SharedKey %SHAREDKEY%
TableName "%TABLE%"
HTTPSCAFile %CAFILE%
</Output>
<Route r1>
Path LinuxAudit => azure
</Route>{
"type": "PATH",
"time": "2021-04-28T04:04:04.030000+00:00",
"seq": 2582,
"item": 0,
"name": "/etc/bind/zones/db.example.com",
"inode": 524363,
"dev": "fc:02",
"mode": "file,644",
"ouid": "root",
"ogid": "bind",
"rdev": 0,
"nametype": "NORMAL",
"cap_fp": 0,
"cap_fi": 0,
"cap_fe": 0,
"cap_fver": 0,
"cap_frootid": "0",
"Hostname": "u20server-1",
"EventReceivedTime": "2021-04-28T04:04:04.048076+00:00",
"SourceModuleName": "LinuxAudit",
"SourceModuleType": "im_linuxaudit"
}
Forwarding AIX audit logs to Microsoft Sentinel
NXLog supports capturing AIX audit logs directly from the AIX Audit Subsystem using the AIX auditing (im_aixaudit) input module.
This configuration uses the default settings for the DeviceFile directive to collect events streaming from /dev/audit.
Additionally, a few manual configuration steps need to be taken to ensure that the im_aixaudit input module can read the audit events from the AIX operating system.
These steps are described in the prerequisites section of the documentation of the input module.
Since Microsoft Sentinel’s analytics needs each event’s origin, i.e., its hostname and IP address, the core function hostname_fqdn() is called to determine the AIX host’s fully qualified domain name (FQDN), while the host_ip() returns the IP address.
Each record is converted to JSON using the to_json() procedure of the xm_json module to enrich the event log with the NXLog core fields.
Right after the external azure-defines.conf is included, TABLE and CAFILE are defined since they have been intentionally excluded from the generic, external om-azure.conf output instance.
They will be referenced by the azure output instance once loaded via the next include.
<Extension json>
Module xm_json
</Extension>
<Input aixaudit>
Module im_aixaudit
DeviceFile /dev/audit
<Exec>
$Hostname = hostname_fqdn();
$MessageSourceAddress = host_ip();
to_json();
</Exec>
</Input>
include /opt/nxlog/etc/azure-defines.conf
define TABLE AIX_Audit
define CAFILE %INSTALLDIR%/etc/cert.pem
include /opt/nxlog/etc/om-azure.conf{
"Verbose": "filename /audit/tempfile.04718774",
"Status": 0,
"EventType": "FILE_Unlink",
"Command": "compress",
"PID": 9961666,
"ParentPID": 4718774,
"Thread": 38862985,
"EventTime": "2021-09-09T11:33:01.379327-04:00",
"Login": "root",
"Real": "builder",
"LoginUID": 0,
"RealUID": 206,
"WPARkey": 0,
"WPARname": "Global",
"EventReceivedTime": "2021-09-09T11:33:01.605055-04:00",
"SourceModuleName": "aixaudit",
"SourceModuleType": "im_aixaudit",
"Hostname": "p1220-pvm1.p1220.cecc.ihost.com",
"MessageSourceAddress": "10.10.0.3"
}
The following KQL NXLog_parsed_AIX_Audit_view() function was created to wrap the default AIX_Audit_CL query and use the project-rename operator to restore the fields to their original names before ingestion.
Any fields having a Microsoft Sentinel Information Model (ASIM) equivalent are renamed accordingly so that the ingested logs can be better integrated with Microsoft Sentinel analytics.
let NXLog_parsed_AIX_Audit_view = view () {
AIX_Audit_CL
| project-rename
CommandLine=Command_s,
EventReceivedTime=EventReceivedTime_t,
EventEndTime=EventTime_t,
EventType=EventType_s,
DvcHostname=Hostname_s,
Username=Login_s,
UserId=LoginUID_d,
MessageSourceAddress=MessageSourceAddress_s,
ParentProcessId=ParentPID_d,
ProcessId=PID_d,
RealUsername=Real_s,
RealUserId=RealUID_d,
SourceModuleName=SourceModuleName_s,
SourceModuleType=SourceModuleType_s,
EventResultDetails=Status_d,
Thread=Thread_d,
Verbose=Verbose_s,
WPARkey=WPARkey_d,
WPARname=WPARname_s
};
NXLog_parsed_AIX_Audit_view();After saving the NXLog_Parsed_AIX_Audit_view() function (shown above) as a user-defined function, it can be used like a regular SQL view:
When these field names are compared with the original ingested field names, it is clear that the KQL function effectively renamed them back to the original parsed field names.
The following examples illustrate some of the KQL aggregation queries that can be created for analyzing AIX audit logs.
