NXLog Legacy Documentation

Event Tracing for Windows (im_etw)

This module can be used to collect events through Event Tracing for Windows (ETW).

ETW is a mechanism in Windows designed for efficient logging of both kernel and user-mode applications. Debug and Analytical channels are based on ETW and cannot be collected as regular Windows Event Log channels via the im_msvistalog module. Various Windows services such as the Windows Firewall and DNS Server can be configured to log events through Windows Event Tracing.

The im_etw module reads event tracing data directly for maximum efficiency. Unlike other solutions, im_etw does not save ETW event data into intermediary trace files that need to be parsed again.

The im_etw module is only available on the Windows platform.

Configuration

The im_etw module accepts the following directives in addition to the common module directives. It is required to specify one of File, KernelFlags, Provider, or Session directives.

File

This optional directive can be used to specify a full path to a log file. Supported file types have the .etl extension. The File directive can only be specified once.

Due to a Windows ETW API limitation, the im_etw module cannot seek inside .etl files. It can only read the configured .etl file from start to finish. Note that you can convert .etl files to the .evtx file format and process them with im_msvistalog without this limitation.

KernelFlags

This directive specifies that kernel trace logs should be collected, and accepts a comma-separated list of flags to use for filtering the logs. The following values are allowed: ALPC, CSWITCH, DBGPRINT, DISK_FILE_IO, DISK_IO, DISK_IO_INIT, DISPATCHER, DPC, DRIVER, FILE_IO, FILE_IO_INIT, IMAGE_LOAD, INTERRUPT, MEMORY_HARD_FAULTS, MEMORY_PAGE_FAULTS, NETWORK_TCPIP, NO_SYSCONFIG, PROCESS, PROCESS_COUNTERS, PROFILE, REGISTRY, SPLIT_IO, SYSTEMCALL, THREAD, VAMAP, and VIRTUAL_ALLOC.

Only users with administrative privileges and services running as LocalSystem can subscribe to kernel logger sessions. When the NXLog service is configured to run under a custom or virtual service account, the account must be a member of the Administrators group for NXLog to be able to collect kernel trace logs. If not, the following error will be logged in the NXLog log file:

ERROR [im_etw|in] StartTrace() failed for session 'nxlog-in'; Access is denied

Note that services running with such privileges may expose the system to higher security risks.

Provider

This directive specifies the name (not GUID) of the ETW provider from which to collect trace logs. Providers available for tracing can be listed by running the command logman query providers in a command prompt. The Windows Kernel Trace provider is not supported; instead, the KernelFlags directive should be used to open a kernel logger session.

When the NXLog service is configured to run under a custom or virtual service account, the account must be a member of the Performance Log Users group for NXLog to be able to collect ETW provider logs. If not, the following error will be logged in the NXLog log file:

ERROR [im_etw|in] StartTrace() failed for session 'nxlog-in'; Access is denied

In addition to specifying a provider name, the Provider directive can also be used as a block directive, allowing multiple providers to be enabled within the same session. In this case, the following sub-directives can be used:

Name: This mandatory directive specifies the provider name. See Provider for more details.

Level: This optional directive specifies the provider level. See Level for details.

MatchAllKeyword: Optional directive, specifies the "all keyword" filter. See MatchAllKeyword for details.

MatchAnyKeyword: Optional directive, specifies the "any keyword" filter. See MatchAnyKeyword for details.

Session

This directive specifies the name of an existing ETW session. The available sessions can be obtained by running the command logman query -ets.

When the NXLog service is configured to run under a custom or virtual service account, the account must be a member of the Performance Log Users group for NXLog to be able to collect ETW session logs. If not, the following error will be logged in the NXLog log file:

ERROR [im_etw|in] ControlTrace(EVENT_TRACE_CONTROL_QUERY) failed for session 'Session name'; Access is denied

Like Provider, the Session can only decode messages from MOF or Manifest-based providers. Refer to About Event Tracing in Microsoft Docs fo more information about provider types.

Level: This optional directive specifies the log level for collecting trace events. Because kernel log sessions do not provide log levels, this directive is only available in combination with the Provider directive. Valid values include Critical, Error, Warning, Information, and Verbose. If this directive is not specified, the verbose log level is used.

MatchAllKeyword: This optional directive is used for filtering Windows ETW events based on keywords. It enables you to specify an optional 8-byte, hexadecimal bitmask value that restricts the events category that an ETW event provider writes. Suppose the provider keyword for an event meets the condition of this directive. In that case, the provider writes the event only if the provider’s event keyword bits also match all of the bits set in the MatchAllKeyword bitmask. Note that this mask is not used if MatchAnyKeyword is set to zero. The default value is 0x00000000. For more information, see the System ETW Provider Event Keyword-Level Settings in the Microsoft documentation.

MatchAnyKeyword: This optional directive is used for filtering Windows ETW events based on keywords. It enables you to specify an optional 8-byte, hexadecimal bitmask value representing one or more keywords to set the category of events that an ETW event provider writes. The provider will write a particular event if the provider’s event keyword bits match any of the bits set in the MatchAnyKeyword bitmask. The default value is 0x00000000. For more information, see the System ETW Provider Event Keyword-Level Settings in the Microsoft documentation.

To make the concept more explicit, look at this examples:

If you use both MatchAllKeyword and MatchAnyKeyword:
MatchAllKeyword: 0x10010001
MatchAnyKeyword: 0x10010001
If an event’s keyword bits are 0x10010001, it matches both, MatchAllKeyword and MatchAnyKeyword. The event is recorded.

If you use only MatchAnyKeyword:
MatchAnyKeyword: 0x10010001
If an event’s keyword bits are 0x00000001, it matches one of the bits in MatchAnyKeyword (the last bit we set in the filter). The event is recorded.

ParsePacketData: This optional directive is used to specify whether DNS, UDP and TCP payloads from the $PacketData field should be parsed. Defaults to FALSE.

ShowExtendedInfo: This optional directive enables the output of additional fields. Default value is FALSE. When this directive is set to TRUE, the properties that are referred by another element in the event property array like "countPropertyIndex" and "lengthPropertyIndex" are also shown. For example, to print the "BufferSize" property when tracing the "Microsoft-Windows-DNSServer" provider, this directive has to be set to TRUE.

Fields

The following fields are used by im_etw.

Depending on the ETW provider from which NXLog collects trace logs, the set of fields generated by the im_etw module may slightly vary. In addition to the fields listed below, the module can generate special provider-specific fields. If the module is configured to collect trace logs from a custom provider (for example, from a custom user-mode application), the module will also generate fields derived from the custom provider trace logs.

$raw_event (type: string): A list of event fields in key-value pairs.

$AccountName (type: string)

The username associated with the event.

$AccountType (type: string)

The type of the account. Possible values are: `User`, `Group`,
`Domain`, `Alias`, `Well Known Group`, `Deleted Account`,
`Invalid`, `Unknown`, and `Computer`.

$ActivityID (type: string)

The ID of the activity corresponding to the event.

$Category (type: string)

The name of a particular component of the provider.

$Channel (type: string)

The name of the channel where the event log was written to.

$ChannelID (type: integer)

The ID of the channel where the event log was written to.

$dns.*.class (type: string)

Field common to all Resource Records. Contains the class of the RR as string.
The class has one of the following values:

IN (1) // Internet class
IN_QU (32769) // Internet class with QU flag set to True
CH (3) // Chaos class
HS (4) // Hesiod class
ANY (255) // ANY class

$dns.*.type (type: string)

Field common to all Resource Records. Contains the type of the RR as string.
The type has one of the following values:

A (1) // IPv4 address record
NS (2) // Name Server record
MD (3) // Obsolete, replaced by MX
MF (4) // Obsolete, replaced by MX
CNAME (5) // Canonical name record
SOA (6) // Start of Authority record
MB (7) // mailbox domain name record
MG (8) // mail group member record
MR (9) // mail rename domain name record
NULL_R (10) // NULL record
WKS (11) // well known service description record
PTR (12) // Pointer record
HINFO (13) // Host information record
MINFO (14) // mailbox or mail list information record
MX (15) // Mail exchanger record
TXT (16) // Text record
RP (17) // Responsible person record
AFSDB (18) // AFS database record
X25 (19) // DNS X25 resource record
ISDN (20) // Integrated Services Digital Network record
RT (21) // Route Through record
NSAP (22) // network service access point address record
NSAP_PTR (23) // network service access point address pointer record
SIG (24) // Signature record (RFC 2535)
KEY (25) // Key record (RFC 2535)
PX (26) // Mail Mapping Information record (RFC 1664)
GPOS (27) // DNS Geographical Position record
AAAA (28) // IPv6 address record
LOC (29) // Location record
NXT (30) // Obsolete record
EID (31) // DNS Endpoint Identifier record
NIMLOC (32) // DNS Nimrod Locator record
SRV (33) // Service locator record (RFC 2052)
ATMA (34) // Asynchronous Transfer Mode address record
NAPTR (35) // Naming Authority Pointer record (RFC 2168)
KX (36) // Key eXchanger record (RFC 2230)
CERT (37) // Certificate record (RFC 2538)
A6 (38) // Obsolete, replaced by AAAA type
DNAM (39) // Delegation Name record (RFC 2672)
SINK (40) // Kitchen sink record
OPT (41) // Option record (RFC 2671)
APL (42) // Address Prefix List record
DS (43) // Delegation signer record
SSHFP (44) // SSH Public Key Fingerprint record
IPSECKEY (45) // IPsec Key record
RRSIG (46) // DNSSEC signature record
NSEC (47) // Next-Secure record
DNSKEY (48) // DNS Key record
DHCID (49) // DHCP identifier record
NSEC3 (50) // NSEC record version 3
NSEC3PARAM (51) // NSEC3 parameters
PTR_NAME (192) // 0xC0 - pointer to a name
TKEY (249) // Transaction Key (RFC 2930)
TSIG (250) // Transaction Signature (RFC 2845)
IXFR (251) // IXFR (Incremental zone transfer)
AXFR (252) // AXFR (Transfer of an entire zone)
ALL (255) // All cached records
CAA (257) // Certification Authority Authorization (RFC 6844)

$dns.additional (type: string)

A string dump of a JSON object, containing an array with the details of each DNS additional RR.

{
    "dns.additional.name": <string>,
    "dns.additional.type": <string>,
    "dns.additional.class": <string>,
    "dns.additional.ttl": <integer>, // Time to live
    "dns.additional.data": <string>, // Data in a format specific to the DNS type
}

$dns.answer (type: string)

A string dump of a JSON object, containing an array with the details of each DNS answer RR.

{
    "dns.answer.name": <string>,
    "dns.answer.type": <string>,
    "dns.answer.class": <string>,
    "dns.answer.ttl": <integer>, // Time to live
    "dns.answer.data": <string>, // Data in a format specific to the DNS type
}

$dns.authority (type: string)

A string dump of a JSON object, containing an array with the details of each DNS authority RR.

{
    "dns.authority.name": <string>,
    "dns.authority.type": <string>,
    "dns.authority.class": <string>,
    "dns.authority.ttl": <integer>, // Time to live
    "dns.authority.data": <string>, // Data in a format specific to the DNS type
}

$dns.flags.authentic_data (type: boolean)

DNS header flag. Authentic Data [RFC4035][RFC6840][RFC Errata 4924]

$dns.flags.authoritative (type: boolean)

DNS header flag. Authoritative Answer [RFC1035]

$dns.flags.checking_disabled (type: boolean)

DNS header flag. Checking Disabled [RFC4035][RFC6840][RFC Errata 4927]

$dns.flags.query_or_response (type: boolean)

DNS header flag. Indicates whether the DNS packet is a query ro a response

$dns.flags.recursion_desired (type: boolean)

DNS header flag. Recursion Available [RFC1035]

$dns.flags.recursion_desired (type: boolean)

DNS header flag. Recursion Desired [RFC1035]

$dns.flags.truncated_response (type: boolean)

DNS header flag. Truncated Response [RFC1035]

$dns.id (type: integer)

An integer indicating the ID of the DNS packet.

$dns.mx.data.preference (type: integer)

An integer containing the MX Preference

$dns.opcode (type: string)

A string indicating the operation code of the DNS packet.
The value is one of the following:

QUERY (0) Query [RFC1035]
IQUERY (1) IQuery (Inverse Query, OBSOLETE) [RFC3425]
STATUS (2) Status [RFC1035]
UNASSIGNED (3) Status [RFC1035]
NOTIFY (4) Notify [RFC1996]
UPDATE (5) Update [RFC2136]
DSO (6) DNS Stateful Operations (DSO) [RFC8490]

$dns.query (type: string)

A string dump of a JSON object, containing an array with the details of each DNS query RR.

{
    "dns.query.name":  <string>, // DNS Query Name (QNAME)
    "dns.query.type": <string>,
    "dns.query.class": <string>,
}

$dns.response.code (type: string)

A string indicating the response code (RCODE) of the DNS packet.
The value is one of the following:

NOERROR (0) No Error [RFC1035]
FORMERR (1) Format Error [RFC1035]
SERVFAIL (2) Server Failure [RFC1035]
NXDOMAIN (3) Non-Existent Domain [RFC1035]
NOTIMP (4) Not Implemented [RFC1035]
REFUSED (5) Query Refused [RFC1035]
YXDOMAIN (6) Name Exists when it should not [RFC2136][RFC6672]
YXRRSET (7) RR Set Exists when it should not [RFC2136]
NXRRSET (8) RR Set that should exist does not [RFC2136]
NOTAUTH (9) Server Not Authoritative for zone [RFC2136]
		   // Not Authorized [RFC8945]
NOTZONE (10) Name not contained in zone [RFC2136]
DSOTYPENI (11) DSO-TYPE Not Implemented [RFC8490]
BADVERS_BADSIG (16) BADVERS - Bad OPT Version [RFC6891]
			   // BADSIG - TSIG Signature Failure [RFC8945]
BADKEY (17) Key not recognized [RFC8945]
BADTIME (18) Signature out of time window [RFC8945]
BADMODE (19) Bad TKEY Mode [RFC2930]
BADNAME (20) Duplicate key name [RFC2930]
BADALG (21) Algorithm not supported [RFC2930]
BADTRUNC (22) Bad Truncation [RFC8945]
BADCOOKIE (23) Bad/missing Server Cookie [RFC7873]

$dns.soa.data.auth_mailbox (type: string)

A string containing the DNS SOA type
Responsible authority's mailbox

$dns.soa.data.expire_limit (type: string)

A string containing the DNS SOA type
Expire Limit

$dns.soa.data.min_ttl (type: string)

A string containing the DNS SOA type
Minimum Time To Live

$dns.soa.data.name_server (type: string)

A string containing the DNS SOA type
Primary name server

$dns.soa.data.refresh_interval (type: string)

A string containing the DNS SOA type
Refresh Interval

$dns.soa.data.retry_interval (type: string)

A string containing the DNS SOA type
Retry Interval

$dns.soa.data.serial_number (type: string)

A string containing the DNS SOA type
Serial Number

$Domain (type: string)

The domain name of the user.

$EventID (type: integer)

The Event ID, corresponding to the provider, that indicates the
type of event.

$EventTime (type: datetime)

The time when the event was generated.

$EventType (type: string)

One of `CRITICAL`, `ERROR`, `WARNING`, `DEBUG`, `AUDIT_FAILURE`,
`AUDIT_SUCCESS`, or `INFO`.

$ExecutionProcessID (type: integer)

The ID of the process that generated the event.

$ExecutionThreadID (type: integer)

The ID of the thread that generated the event.

$Hostname (type: string)

The name of the system where the event was generated.

$Keywords (type: string)

A keyword bit mask corresponding to the current event.

$Level (type: string)

The level of the event that was generated.

$LevelValue (type: string)

An integer indicating the level of the event that was generated.

$Opcode (type: string)

The name of the operation corresponding to the event.

$OpcodeValue (type: integer)

An integer indicating the operation corresponding to the event.

$PacketData (type: string)

The UDP or TCP payload of DNS event data as HEX.
(<<parsedpacketdata,$ParsedPacketData>>)
contains the human readable representation of this data.

$ParsedPacketData (type: string)

Human readable representation in JSON format of the data extracted by
parsing the PacketData field.

$ProviderGuid (type: string)

The GUID of the trace provider, corresponding to the
<<sourcename,$SourceName>>.

$Severity (type: string)

The normalized severity name of the event. See
<<severityvalue,$SeverityValue>>.

$SeverityValue (type: integer)

The normalized severity number of the event, mapped as follows.

Event Log Severity	Normalized Severity
0/Audit Success	2/INFO
0/Audit Failure	4/ERROR
1/Critical	5/CRITICAL
2/Error	4/ERROR
3/Warning	3/WARNING
4/Information	2/INFO
5/Verbose	1/DEBUG

Event Log Severity

Normalized Severity

0/Audit Success

2/INFO

0/Audit Failure

4/ERROR

1/Critical

5/CRITICAL

2/Error

4/ERROR

3/Warning

3/WARNING

4/Information

2/INFO

5/Verbose

1/DEBUG

$SourceName (type: string)

The name of the trace provider.

$TaskValue (type: integer)

An integer indicating a particular component of the provider.

$Version (type: integer)

The version of the event type.

Examples

Example 1. Collecting events from the Windows kernel trace

With this configuration, NXLog will collect trace events from the Windows kernel. Only events matching the PROCESS and THREAD flags will be collected.

nxlog.conf

<Input etw>
    Module      im_etw
    KernelFlags PROCESS, THREAD
</Input>

Example 2. Collecting events from an ETW provider

With this configuration, NXLog will collect events from the Microsoft-Windows-Firewall trace provider.

nxlog.conf

<Input etw>
    Module      im_etw
    Provider    Microsoft-Windows-Firewall
</Input>

Example 3. Collecting events from multiple ETW providers

With this configuration, NXLog will collect events from the specified providers.

nxlog.conf

<Input etw>
    Module      im_etw
    <Provider>
        Name    Microsoft-Windows-Firewall
        Level   Verbose
    </Provider>
    <Provider>
        Name    Microsoft-Windows-DNS-Client
        Level   Critical
    </Provider>
    <Provider>
        Name    Microsoft-Windows-Audit
        Level   Information
    </Provider>
</Input>

Example 4. Collecting events from a Session

With this configuration, NXLog will collect events from the specified session.

nxlog.conf

<Input etw>
    Module      im_etw
    Session     DiagLog
</Input>

Example 5. Setting level directive

With this configuration, NXLog will assign event log level for a specified provider.

nxlog.conf

<Input etw>
    Module          im_etw
    Provider        Microsoft-Windows-DNS-Client
    Level           verbose
    MatchAnyKeyword 0xFFFFFFFF
    MatchAllKeyword 0x0
</Input>