Event Tracing for Windows (im_etw)

This module can be used to collect events through Event Tracing for Windows (ETW).

ETW is a mechanism in Windows designed for efficient logging of both kernel and user-mode applications. Debug and Analytical channels are based on ETW and cannot be collected as regular Windows Event Log channels via the im_msvistalog module. Various Windows services such as the Windows Firewall and DNS Server can be configured to log events through Windows Event Tracing.

The im_etw module reads event tracing data directly for maximum efficiency. Unlike other solutions, im_etw does not save ETW event data into intermediary trace files that need to be parsed again.

The im_etw module is only available on the Windows platform.

Configuration

The im_etw module accepts the following directives in addition to the common module directives. It is required to specify one of File, KernelFlags, Provider, or Session directives.

Required directives

One of the following four directives must be specified for the module to start.

File

This optional directive can be used to specify a full path to a log file. Supported file types have the .etl extension. The File directive can only be specified once.

Due to a Windows ETW API limitation, the im_etw module cannot seek inside .etl files. It can only read the configured .etl file from start to finish. Note that you can convert .etl files to the .evtx file format and process them with im_msvistalog without this limitation.

KernelFlags

This directive specifies that kernel trace logs should be collected, and accepts a comma-separated list of flags to use for filtering the logs. The following values are allowed: ALPC, CSWITCH, DBGPRINT, DISK_FILE_IO, DISK_IO, DISK_IO_INIT, DISPATCHER, DPC, DRIVER, FILE_IO, FILE_IO_INIT, IMAGE_LOAD, INTERRUPT, MEMORY_HARD_FAULTS, MEMORY_PAGE_FAULTS, NETWORK_TCPIP, NO_SYSCONFIG, PROCESS, PROCESS_COUNTERS, PROFILE, REGISTRY, SPLIT_IO, SYSTEMCALL, THREAD, VAMAP, and VIRTUAL_ALLOC.

Only users with administrative privileges and services running as LocalSystem can subscribe to kernel logger sessions. When the NXLog Agent service is configured to run under a custom or virtual service account, the account must be a member of the Administrators group for NXLog Agent to be able to collect kernel trace logs. If not, the following error will be logged in the NXLog Agent log file:

ERROR [im_etw in] StartTrace() failed for session 'nxlog-in'; Access is denied

Note that services running with such privileges may expose the system to higher security risks.

Provider

This directive specifies the name (not GUID) of the ETW provider from which to collect trace logs. Providers available for tracing can be listed by running the command logman query providers in a command prompt. The Windows Kernel Trace provider is not supported; instead, the KernelFlags directive should be used to open a kernel logger session.

When the NXLog Agent service is configured to run under a custom or virtual service account, the account must be a member of the Performance Log Users group for NXLog Agent to be able to collect ETW provider logs. If not, the following error will be logged in the NXLog Agent log file:

ERROR [im_etw in] StartTrace() failed for session 'nxlog-in'; Access is denied

In addition to specifying a provider name, the Provider directive can also be used as a block directive, allowing multiple providers to be enabled within the same session. In this case, the following sub-directives can be used:

Name: This mandatory directive specifies the provider name. See Provider for more details.

Level: This optional directive specifies the provider level. See Level for details.

MatchAllKeyword: Optional directive, specifies the "all keyword" filter. See MatchAllKeyword for details.

MatchAnyKeyword: Optional directive, specifies the "any keyword" filter. See MatchAnyKeyword for details.

Session: This directive specifies the name of an existing ETW session. The available sessions can be obtained by running the command logman query -ets.

When the NXLog Agent service is configured to run under a custom or virtual service account, the account must be a member of the Performance Log Users group for NXLog Agent to be able to collect ETW session logs. If not, the following error will be logged in the NXLog Agent log file:

ERROR [im_etw in] ControlTrace(EVENT_TRACE_CONTROL_QUERY) failed for session 'Session name'; Access is denied

Like Provider, the Session can only decode messages from MOF or Manifest-based providers. Refer to About Event Tracing in Microsoft Docs fo more information about provider types.

Optional directives

EnableWppSupport

Optional directive. Defaults to TRUE. If set to FALSE, disables support for WPP providers.

Level

This optional directive specifies the log level for collecting trace events. Because kernel log sessions do not provide log levels, this directive is only available in combination with the Provider directive. Valid values include Critical, Error, Warning, Information, and Verbose. If this directive is not specified, the verbose log level is used.

MatchAllKeyword

This optional directive is used for filtering ETW events based on keywords. Defaults to 0x00000000. For more information, see System ETW Provider Event Keyword-Level Settings in the Microsoft documentation.

MatchAnyKeyword

This optional directive is used for filtering Windows ETW events based on keywords. Defaults to 0x00000000. For more information, see System ETW Provider Event Keyword-Level Settings in the Microsoft documentation.

ParsePacketData

This optional directive is used to specify whether DNS, UDP and TCP payloads from the $PacketData field should be parsed. Defaults to TRUE.

TMFSearchPath

This optional directive specifies the path to be set to the TRACE_FORMAT_SEARCH_PATH environment variable. This path is searched for a file with a matching GUID filename when parsing WPP events.

ShowExtendedInfo

This optional directive enables the output of additional fields. Default value is FALSE. When this directive is set to TRUE, the properties that are referred by another element in the event property array like "countPropertyIndex" and "lengthPropertyIndex" are also shown. For example, to print the "BufferSize" property when tracing the "Microsoft-Windows-DNSServer" provider, this directive has to be set to TRUE.

Fields

The following fields are used by im_etw.

Depending on the ETW provider from which NXLog collects trace logs, the set of fields generated by the im_etw module may slightly vary. In addition to the fields listed below, the module can generate special provider-specific fields. If the module is configured to collect trace logs from a custom provider (for example, from a custom user-mode application), the module will also generate fields derived from the custom provider trace logs.

$raw_event (type: string): A list of event fields in key-value pairs.

$AccountName (type: string): The username associated with the event.

$AccountType (type: string): The type of the account. Possible values are: User, Group, Domain, Alias, Well Known Group, Deleted Account, Invalid, Unknown, and Computer.

$ActivityID (type: string): The ID of the activity corresponding to the event.

$Category (type: string): The name of a particular component of the provider.

$Channel (type: string): The name of the channel where the event log was written to.

$ChannelID (type: integer): The ID of the channel where the event log was written to.

$dns.*.class (type: string): Field common to all Resource Records. Contains the class of the RR as string. The class has one of the following values:

IN (1) // Internet class IN_QU (32769) // Internet class with QU flag set to True CH (3) // Chaos class HS (4) // Hesiod class ANY (255) // ANY class

$dns.*.type (type: string): Field common to all Resource Records. Contains the type of the RR as string. The type has one of the following values:

A (1) // IPv4 address record NS (2) // Name Server record MD (3) // Obsolete, replaced by MX MF (4) // Obsolete, replaced by MX CNAME (5) // Canonical name record SOA (6) // Start of Authority record MB (7) // mailbox domain name record MG (8) // mail group member record MR (9) // mail rename domain name record NULL_R (10) // NULL record WKS (11) // well known service description record PTR (12) // Pointer record HINFO (13) // Host information record MINFO (14) // mailbox or mail list information record MX (15) // Mail exchanger record TXT (16) // Text record RP (17) // Responsible person record AFSDB (18) // AFS database record X25 (19) // DNS X25 resource record ISDN (20) // Integrated Services Digital Network record RT (21) // Route Through record NSAP (22) // network service access point address record NSAP_PTR (23) // network service access point address pointer record SIG (24) // Signature record (RFC 2535) KEY (25) // Key record (RFC 2535) PX (26) // Mail Mapping Information record (RFC 1664) GPOS (27) // DNS Geographical Position record AAAA (28) // IPv6 address record LOC (29) // Location record NXT (30) // Obsolete record EID (31) // DNS Endpoint Identifier record NIMLOC (32) // DNS Nimrod Locator record SRV (33) // Service locator record (RFC 2052) ATMA (34) // Asynchronous Transfer Mode address record NAPTR (35) // Naming Authority Pointer record (RFC 2168) KX (36) // Key eXchanger record (RFC 2230) CERT (37) // Certificate record (RFC 2538) A6 (38) // Obsolete, replaced by AAAA type DNAM (39) // Delegation Name record (RFC 2672) SINK (40) // Kitchen sink record OPT (41) // Option record (RFC 2671) APL (42) // Address Prefix List record DS (43) // Delegation signer record SSHFP (44) // SSH Public Key Fingerprint record IPSECKEY (45) // IPsec Key record RRSIG (46) // DNSSEC signature record NSEC (47) // Next-Secure record DNSKEY (48) // DNS Key record DHCID (49) // DHCP identifier record NSEC3 (50) // NSEC record version 3 NSEC3PARAM (51) // NSEC3 parameters PTR_NAME (192) // 0xC0 - pointer to a name TKEY (249) // Transaction Key (RFC 2930) TSIG (250) // Transaction Signature (RFC 2845) IXFR (251) // IXFR (Incremental zone transfer) AXFR (252) // AXFR (Transfer of an entire zone) ALL (255) // All cached records CAA (257) // Certification Authority Authorization (RFC 6844)

$dns.additional (type: string)

A string dump of a JSON object, containing an array with the details of each DNS additional RR.

{
"dns.additional.name": <string>,
"dns.additional.type": <string>,
"dns.additional.class": <string>,
"dns.additional.ttl": <integer>, // Time to live
"dns.additional.data": <string>, // Data in a format specific to the DNS type
}

$dns.answer (type: string)

A string dump of a JSON object, containing an array with the details of each DNS answer RR.

{
"dns.answer.name": <string>,
"dns.answer.type": <string>,
"dns.answer.class": <string>,
"dns.answer.ttl": <integer>, // Time to live
"dns.answer.data": <string>, // Data in a format specific to the DNS type
}

$dns.authority (type: string)

A string dump of a JSON object, containing an array with the details of each DNS authority RR.

{
"dns.authority.name": <string>,
"dns.authority.type": <string>,
"dns.authority.class": <string>,
"dns.authority.ttl": <integer>, // Time to live
"dns.authority.data": <string>, // Data in a format specific to the DNS type
}

$dns.flags.authentic_data (type: boolean): DNS header flag. Authentic Data [RFC4035][RFC6840][RFC Errata 4924]

$dns.flags.authoritative (type: boolean): DNS header flag. Authoritative Answer [RFC1035]

$dns.flags.checking_disabled (type: boolean): DNS header flag. Checking Disabled [RFC4035][RFC6840][RFC Errata 4927]

$dns.flags.query_or_response (type: boolean): DNS header flag. Indicates whether the DNS packet is a query ro a response

$dns.flags.recursion_desired (type: boolean): DNS header flag. Recursion Available [RFC1035]
$dns.flags.recursion_desired (type: boolean): DNS header flag. Recursion Desired [RFC1035]

$dns.flags.truncated_response (type: boolean): DNS header flag. Truncated Response [RFC1035]

$dns.id (type: integer): An integer indicating the ID of the DNS packet.

$dns.mx.data.preference (type: integer): An integer containing the MX Preference

$dns.opcode (type: string): A string indicating the operation code of the DNS packet. The value is one of the following:

QUERY (0) Query [RFC1035] IQUERY (1) IQuery (Inverse Query, OBSOLETE) [RFC3425] STATUS (2) Status [RFC1035] UNASSIGNED (3) Status [RFC1035] NOTIFY (4) Notify [RFC1996] UPDATE (5) Update [RFC2136] DSO (6) DNS Stateful Operations (DSO) [RFC8490]

$dns.query (type: string)

A string dump of a JSON object, containing an array with the details of each DNS query RR.

{
"dns.query.name":  <string>, // DNS Query Name (QNAME)
"dns.query.type": <string>,
"dns.query.class": <string>,
}

$dns.response.code (type: string): A string indicating the response code (RCODE) of the DNS packet. The value is one of the following:

NOERROR (0) No Error [RFC1035] FORMERR (1) Format Error [RFC1035] SERVFAIL (2) Server Failure [RFC1035] NXDOMAIN (3) Non-Existent Domain [RFC1035] NOTIMP (4) Not Implemented [RFC1035] REFUSED (5) Query Refused [RFC1035] YXDOMAIN (6) Name Exists when it should not [RFC2136][RFC6672] YXRRSET (7) RR Set Exists when it should not [RFC2136] NXRRSET (8) RR Set that should exist does not [RFC2136] NOTAUTH (9) Server Not Authoritative for zone [RFC2136] // Not Authorized [RFC8945] NOTZONE (10) Name not contained in zone [RFC2136] DSOTYPENI (11) DSO-TYPE Not Implemented [RFC8490] BADVERS_BADSIG (16) BADVERS - Bad OPT Version [RFC6891] // BADSIG - TSIG Signature Failure [RFC8945] BADKEY (17) Key not recognized [RFC8945] BADTIME (18) Signature out of time window [RFC8945] BADMODE (19) Bad TKEY Mode [RFC2930] BADNAME (20) Duplicate key name [RFC2930] BADALG (21) Algorithm not supported [RFC2930] BADTRUNC (22) Bad Truncation [RFC8945] BADCOOKIE (23) Bad/missing Server Cookie [RFC7873]

$dns.soa.data.auth_mailbox (type: string): A string containing the DNS SOA type Responsible authority’s mailbox

$dns.soa.data.expire_limit (type: string): A string containing the DNS SOA type Expire Limit

$dns.soa.data.min_ttl (type: string): A string containing the DNS SOA type Minimum Time To Live

$dns.soa.data.name_server (type: string): A string containing the DNS SOA type Primary name server

$dns.soa.data.refresh_interval (type: string): A string containing the DNS SOA type Refresh Interval

$dns.soa.data.retry_interval (type: string): A string containing the DNS SOA type Retry Interval

$dns.soa.data.serial_number (type: string): A string containing the DNS SOA type Serial Number

$Domain (type: string): The domain name of the user.

$EventID (type: integer): The Event ID, corresponding to the provider, that indicates the type of event.

$EventTime (type: datetime): The time when the event was generated.

$EventType (type: string): One of CRITICAL, ERROR, WARNING, DEBUG, AUDIT_FAILURE, AUDIT_SUCCESS, or INFO.

$ExecutionProcessID (type: integer): The ID of the process that generated the event.

$ExecutionThreadID (type: integer): The ID of the thread that generated the event.

$Hostname (type: string): The name of the system where the event was generated.

$Keywords (type: string): A keyword bit mask corresponding to the current event.

$Level (type: string): The level of the event that was generated.

$LevelValue (type: string): An integer indicating the level of the event that was generated.

$Opcode (type: string): The name of the operation corresponding to the event.

$OpcodeValue (type: integer): An integer indicating the operation corresponding to the event.

$PacketData (type: string): The UDP or TCP payload of DNS event data as HEX. ($ParsedPacketData) contains the human readable representation of this data.

$ParsedPacketData (type: string): Human readable representation in JSON format of the data extracted by parsing the PacketData field.

$ProviderGuid (type: string): The GUID of the trace provider, corresponding to the $SourceName.

$Severity (type: string): The normalized severity name of the event. See $SeverityValue.

$SeverityValue (type: integer)

The normalized severity number of the event, mapped as follows.

Event Log Severity	Normalized Severity
0/Audit Success	2/INFO
0/Audit Failure	4/ERROR
1/Critical	5/CRITICAL
2/Error	4/ERROR
3/Warning	3/WARNING
4/Information	2/INFO
5/Verbose	1/DEBUG

Event Log Severity

Normalized Severity

0/Audit Success

2/INFO

0/Audit Failure

4/ERROR

1/Critical

5/CRITICAL

2/Error

4/ERROR

3/Warning

3/WARNING

4/Information

2/INFO

5/Verbose

1/DEBUG

$SourceName (type: string): The name of the trace provider.

$TaskValue (type: integer): An integer indicating a particular component of the provider.

$Version (type: integer): The version of the event type.

Examples

Example 1. Collecting events from the Windows kernel trace

With this configuration, NXLog Agent will collect trace events from the Windows kernel. Only events matching the PROCESS and THREAD flags will be collected.

nxlog.conf

<Input etw>
    Module      im_etw
    KernelFlags PROCESS, THREAD
</Input>

Example 2. Collecting events from an ETW provider

With this configuration, NXLog Agent will collect events from the Microsoft-Windows-Firewall trace provider.

nxlog.conf

<Input etw>
    Module      im_etw
    Provider    Microsoft-Windows-Firewall
</Input>

Example 3. Collecting events from multiple ETW providers

With this configuration, NXLog Agent will collect events from the specified providers.

nxlog.conf

<Input etw>
    Module      im_etw
    <Provider>
        Name    Microsoft-Windows-Firewall
        Level   Verbose
    </Provider>
    <Provider>
        Name    Microsoft-Windows-DNS-Client
        Level   Critical
    </Provider>
    <Provider>
        Name    Microsoft-Windows-Audit
        Level   Information
    </Provider>
</Input>

Example 4. Collecting events from a Session

With this configuration, NXLog Agent will collect events from the specified session.

nxlog.conf

<Input etw>
    Module      im_etw
    Session     DiagLog
</Input>

Example 5. Setting level directive

With this configuration, NXLog Agent will assign an event log level for a specified provider.

nxlog.conf

<Input etw>
    Module          im_etw
    Provider        Microsoft-Windows-DNS-Client
    Level           verbose
    MatchAnyKeyword 0xFFFFFFFF
    MatchAllKeyword 0x0
</Input>