Event Log for Windows 2008/Vista/later (im_msvistalog)

This module can be used to collect Windows Event Log messages on Microsoft Windows platforms that support the newer Event Log API (also known as the Crimson Event Log subsystem), namely Windows 2008/Vista and later. See the official Microsoft documentation about Event Logs. The module supports reading all System, Application, and Custom events. It looks up the available channels and monitors events in each unless the Query and Channel directives are explicitly defined. Event logs can be collected from remote servers over MSRPC.

To examine the supported platforms, see the list of installation packages.

For Windows 2003 and earlier, use the im_mseventlog module because the new Windows Event Log API is only available in Windows Vista, Windows 2008, and later.

Use the im_etw module to collect Analytic and Debug logs as the Windows Event Log subsystem, which im_msvistalog uses, does not support subscriptions to Debug or Analytic channels.

NXLog Agent can alter a field’s line ending according to W3C recommendations. Any CR LF and any CR that is not followed by an LF will be translated to a single LF.

In addition to the standard set of fields that are listed under the System section, event providers can define additional schema which enables logging additional data under the EventData section. The Security log makes use of this new feature and such additional fields can be seen in the following XML snippet:

<EventData>
  <Data Name="SubjectUserSid">S-1-5-18</Data>
  <Data Name="SubjectUserName">WIN-OUNNPISDHIG$</Data>
  <Data Name="SubjectDomainName">WORKGROUP</Data>
  <Data Name="SubjectLogonId">0x3e7</Data>
  <Data Name="TargetUserSid">S-1-5-18</Data>
  <Data Name="TargetUserName">SYSTEM</Data>
  <Data Name="TargetDomainName">NT AUTHORITY</Data>
  <Data Name="TargetLogonId">0x3e7</Data>
  <Data Name="LogonType">5</Data>
  <Data Name="LogonProcessName">Advapi</Data>
  <Data Name="AuthenticationPackageName">Negotiate</Data>
  <Data Name="WorkstationName" />
  <Data Name="LogonGuid">{00000000-0000-0000-0000-000000000000}</Data>
  <Data Name="TransmittedServices">-</Data>
  <Data Name="LmPackageName">-</Data>
  <Data Name="KeyLength">0</Data>
  <Data Name="ProcessId">0x1dc</Data>
  <Data Name="ProcessName">C:\Windows\System32\services.exe</Data>
  <Data Name="IpAddress">-</Data>
  <Data Name="IpPort">-</Data>
</EventData>

NXLog Agent can extract this data when fields are logged using this schema. The values will be available in the fields of the internal NXLog Agent log structure. This is especially useful because there is no need to write pattern-matching rules to extract this data from the message. These fields can be used in filtering rules, be written into SQL tables, or be used to trigger actions. The Exec directive can be used for filtering:

<Input in>
    Module  im_msvistalog
    Exec    if ($TargetUserName == 'SYSTEM') OR \
               ($EventType == 'VERBOSE') drop();
</Input>

Configuration

The im_msvistalog module accepts the following directives in addition to the common module directives.

Optional directives

AddPrefix

If this boolean directive is set to TRUE, names of fields parsed from the <EventData> portion of the event XML will be prefixed with EventData.. For example, $EventData.SubjectUserName will be added to the event record instead of $SubjectUserName. The same applies to <UserData>. This directive defaults to FALSE: field names will not be prefixed.

CaptureEventXML

This boolean directive defines whether the module should store raw XML-formatted event data. If set to TRUE, the module stores raw XML data in the $EventXML field. By default, the value is set to FALSE, and the $EventXML field is not added to the record.

CaptureMessage

This boolean directive defines whether the module should create the $Message field. The default value is TRUE, and the value of the $Message field is created from the event’s UserData and EventData fields. If $Message is not essential, it is recommended to set it to FALSE to improve event processing speed.

Channel

The name of the Channel to query. If not specified, the module will read from all sources defined in the registry. See the MSDN documentation about Event Selection.

File

This optional directive can be used to specify a full path to a log file. Log file types that can be used have the .evt and .evtx extensions. If the File directive is specified, the SavePos directive will be overridden to TRUE.

The File directive can also be used for reading .etl files with the limitation that the _im_msvistalog module can not seek in .etl files. It can only read these files from start to end.

The File directive can be specified multiple times to read from multiple files. This module finds files only when the module instance is started; any files added later will not be read until it is restarted. If the log file specified by this directive is updated with new event records while NXLog Agent is running (the file size or modification date attribute changes), the module detects the newly appended records on the fly without requiring the module instance to be restarted. Reading a Windows Event Log file directly is mostly useful for forensics purposes. The System log would be read directly with the following:

File "C:\Windows\System32\winevt\Logs\System.evtx"

You can use wildcards to specify file names and directories. Wildcards are not regular expressions but are patterns commonly used by Unix shells to expand filenames (also known as "globbing").

?: Matches any single character.
*: Matches any string, including the empty string.
\*: Matches the asterisk (*) character.
\?: Matches the question mark (?) character.
[…]: Matches one character specified within the brackets. The brackets should contain a single character (for example, [a]) or a range of characters ([a-z]). If the first character in the brackets is ^ or !, it reverses the wildcard matching logic (the wildcard matches any character not in the brackets). The backslash (\) characters are ignored and should be used to escape ] and - characters as well as ^ and ! at the beginning of the pathname.

Language

This optional directive specifies a language to use for rendering the events. The language should be given as a hyphen-separated language/region code (for example, fr-FR for French). Note that the required language support must be installed on the system. If this directive is not given, the system’s default locale is used.

PollInterval

This directive specifies how frequently the module will check for new events, in seconds. If this directive is not specified, the default is 1 second. Fractional seconds may be specified (PollInterval 0.5 will check twice every second).

Query

This directive specifies the query for returning only specific Windows Event Log sources. See the MSDN documentation about Event Selection. Note that this directive requires a single-line parameter, so multi-line query XML should be specified using line continuation:

Query <QueryList>
        <Query Id='1'>
         <Select Path='Security'>*[System/Level=4]</Select>
        </Query>
      </QueryList>

When the Query contains an XPath style expression, the Channel must also be specified. Otherwise, if an XML Query is specified, the Channel should not be used.

QueryXML

This directive is the same as the Query directive above, except it can be used as a block. Multi-line XML queries can be used without line continuation, and the XML Query can be copied directly from Event Viewer.

<QueryXML>
  <QueryList>
    <!-- XML-style comments can
         span multiple lines in
         QueryXML blocks like this.
    -->
    <Query Id='1'>
      <Select Path='Security'>*[System/Level=4]</Select>
    </Query>
  </QueryList>
</QueryXML>

Commenting with the # mark does not work within multi-line Query directives or QueryXML blocks. In this case, use XML-style comments  as shown in the example above. Failure to follow this syntax for comments within queries will render the module instance useless. Since NXLog Agent does not parse the content of QueryXML blocks, this behavior is expected.

ReadBatchSize

This optional directive can be used to specify the number of events the Windows Event Log API will pass to the module for processing. Larger sizes may increase throughput. Note that there is a known issue in the Windows Event Log subsystem: when this value is higher than 31 it may fail to retrieve some events on busy systems, returning the error "EvtNext failed with error 1734: The array bounds are invalid." For this reason, increasing this value is not recommended. The default is 31.

ReadFromLast

This optional boolean directive instructs the module to only read logs that arrive after NXLog Agent is started. This directive comes into effect if a saved position is not found, for example on the first start, or when the SavePos directive is FALSE. When the SavePos directive is TRUE and a previously saved position is found, the module will always resume reading from the saved position. If ReadFromLast is FALSE, the module will read all logs from Windows Event Log. This can result in a lot of messages and is usually not the expected behavior. If this directive is not specified, it defaults to TRUE.

The following matrix shows the outcome of this directive in conjunction with the SavePos directive:

ReadFromLast SavePos SavedPosition Outcome

TRUE

Reads events that are logged after NXLog Agent is started.

TRUE

Yes

Reads events from the saved position.

TRUE

FALSE

Reads events that are logged after NXLog Agent is started.

TRUE

FALSE

Yes

Reads events that are logged after NXLog Agent is started.

FALSE

TRUE

Reads all events.

FALSE

TRUE

Yes

Reads events from the saved position.

FALSE

Reads all events.

FALSE

Yes

Reads all events.

RemoteAuthMethod

This optional directive specifies the authentication method to use. Available values are Default, Negotiate, Kerberos, and NTLM. When the directive is not specified, Default is used, which is actually Negotiate.

RemoteDomain

Domain of the user used for authentication when logging on the remote server to collect event logs.

RemotePassword

Password of the user used for authentication when logging on to the remote server to collect event logs.

RemoteServer

This optional directive specifies the name of the remote server to collect event logs. If not specified, the module will collect locally.

RemoteUser

Name of the user used for authentication when logging on to the remote server to collect event logs.

ResolveGUID

This optional boolean directive specifies that GUID values should be resolved to their object names in the $Message field. If ResolveGUID is set to TRUE, it produces two output fields. One that retains the non-resolved form of the GUID, and another which resolves to the above-mentioned object name. To differentiate the two output fields, the resolved field name will have the DN suffix added to it. If the field already exists with the same name the resolved field will not be added and the original is preserved. The default setting is FALSE; the module will not resolve GUID values. Windows Event Viewer shows the Message with the GUID values resolved, and this must be enabled to get the same output with NXLog Agent.

ResolveSID

This optional boolean directive specifies that SID values should be resolved to user names in the $Message field. If ResolveSID is set to TRUE, it produces two output fields. One that retains the non-resolved form of the SID, and another which resolves to the above-mentioned user name. To differentiate the two output fields, the resolved field name will have the Name suffix added to it. If the field already exists with the same name the resolved field will not be added and the original is preserved. The default setting is FALSE; the module will not resolve SID values. Windows Event Viewer shows the Message with the SID values resolved, and this must be enabled to get the same output with NXLog Agent.

SavePos

If this boolean directive is set to TRUE, the file position will be saved when NXLog Agent exits. The file position will be read from the cache file upon startup. The default is TRUE, the file position will be saved if this directive is not specified. This directive affects the outcome of the ReadFromLast directive. The SavePos directive can be overridden by the global NoCache directive.

TolerateQueryErrors

This boolean directive specifies that im_msvistalog should ignore any invalid sources in the query. The default is FALSE: im_msvistalog will fail to start if any source is invalid.

Fields

The following fields are used by im_msvistalog.

$raw_event (type: string): A list of event fields in key-value pairs.

$AccountName (type: string): The username associated with the event.

$AccountType (type: string): The type of the account. Possible values are: User, Group, Domain, Alias, Well Known Group, Deleted Account, Invalid, Unknown, and Computer.

$ActivityID (type: string): A globally unique identifier for the current activity, as stored in EvtSystemActivityID.

$Category (type: string): The category name resolved from Task.

$Channel (type: string): The Channel of the event source (for example, Security or Application).

$Domain (type: string): The domain name of the user.

$ERROR_EVT_UNRESOLVED (type: boolean): This field is set to TRUE if the event message cannot be resolved and the insertion strings are not present.

$EventID (type: integer): The event ID (specific to the event source) from the EvtSystemEventID field.

$EventTime (type: datetime): The EvtSystemTimeCreated field.

$EventType (type: string): The type of the event, which is a string describing the severity. This is translated to its string representation from EvtSystemLevel. Possible values are: CRITICAL, ERROR, AUDIT_FAILURE, AUDIT_SUCCESS, INFO, WARNING, and VERBOSE.

$EventXML (type: string): The raw event data in XML format. This field is available if the module’s CaptureEventXML directive is set to TRUE.

$ExecutionProcessID (type: integer): The process identifier of the event producer as in EvtSystemProcessID.

$Hostname (type: string): The EvtSystemComputer field.

$Keywords (type: string): The value of the Keywords field from EvtSystemKeywords.

$Level (type: string): The level of the event that was generated.

$LevelValue (type: string): An integer indicating the level of the event that was generated.

$Message (type: string): The message from the event.

$Opcode (type: string): The Opcode string resolved from OpcodeValue.

$OpcodeValue (type: integer): The Opcode number of the event as in EvtSystemOpcode.

$ProviderGuid (type: string): The globally unique identifier of the event’s provider as stored in EvtSystemProviderGuid. This corresponds to the name of the provider in the $SourceName field.

$RecordNumber (type: integer): The number of the event record.

$RelatedActivityID (type: string): The RelatedActivityID as stored in EvtSystemRelatedActivityID.

$Severity (type: string): The normalized severity name of the event. See $SeverityValue.

$SeverityValue (type: integer)

The normalized severity number of the event, mapped as follows.

Event Log Severity	Normalized Severity
0/Audit Success	2/INFO
0/Audit Failure	4/ERROR
1/Critical	5/CRITICAL
2/Error	4/ERROR
3/Warning	3/WARNING
4/Information	2/INFO
5/Verbose	1/DEBUG

Event Log Severity

Normalized Severity

0/Audit Success

2/INFO

0/Audit Failure

4/ERROR

1/Critical

5/CRITICAL

2/Error

4/ERROR

3/Warning

3/WARNING

4/Information

2/INFO

5/Verbose

1/DEBUG

$SourceName (type: string): The event source which produced the event, from the EvtSystemProviderName field.

$TaskValue (type: integer): The task number from the EvtSystemTask field.

$ThreadID (type: integer): The thread identifier of the event producer as in EvtSystemThreadID.

$UserID (type: string): The Security Identifier (SID) which resolves to $AccountName, stored in EvtSystemUserID.

$Version (type: integer): The Version number of the event as in EvtSystemVersion.

Fields by providers

For the fields used by the providers, see the fields by providers list.

Examples

Due to a limitation in the Windows Event Log API, queries with more than 22 clauses will fail, producing the following error message:

ERROR failed to subscribe to events with this module, the Query is invalid: This operator is unsupported by this implementation of the filter.; [error code: 15001]

The workaround for this limitation is grouping clauses and/or splitting the filter across multiple queries. In the example below, the filter consists of 6 EventIDs; however, these count as 2 in terms of the aforementioned limitation.

Combining clauses in groups

<QueryXML>
    <QueryList>
        <Query Id="0">
            <Select Path="Security">*[System[(EventID=6416 or EventID=6424 or EventID=4732) or (EventID=1102 or EventID=4719 or EventID=4704)]]
            </Select>
        </Query>
    </QueryList>
</QueryXML>

Example 1. Forwarding logs from Windows Event Log to a remote host in syslog format

This configuration collects events from Windows Event Log with the specified query. BSD Syslog headers are added and the messages are forwarded to a remote host via TCP.

nxlog.conf

<Extension syslog>
    Module          xm_syslog
</Extension>

<Input eventlog>
    Module          im_msvistalog
    <QueryXML>
        <QueryList>
            <Query Id='0'>
                <Select Path='Application'>*</Select>
                <Select Path='Security'>*[System/Level&lt;4]</Select>
                <Select Path='System'>*</Select>
            </Query>
        </QueryList>
    </QueryXML>
</Input>

<Output tcp>
    Module          om_tcp
    Host            192.168.1.1:514
    Exec            to_syslog_bsd();
</Output>

<Route eventlog_to_tcp>
    Path            eventlog => tcp
</Route>

Example 2. Retaining the original EventData and UserData XML

This configuration collects events from Windows Event Log and uses the extract_xml() function of the xm_xml module to retain the original EventData and UserData XML.

nxlog.conf

<Extension xml>
    Module           xm_xml
</Extension>

<Extension json>
    Module           xm_json
</Extension>

<Input eventlog>
    Module           im_msvistalog
    CaptureEventXML  TRUE (1)
    <QueryXML>
        <QueryList>
            <Query Id='0'>
                <Select Path='Application'>*</Select>
                <Select Path='Security'>*[System/Level&lt;4]</Select>
                <Select Path='System'>*</Select>
            </Query>
        </QueryList>
    </QueryXML>
    <Exec>
        $EventData = extract_xml("/Event/EventData", $EventXML);
        if $EventData == ""
        {
            delete($EventData);
        }

        $UserData = extract_xml("/Event/UserData", $EventXML);
        if $UserData == ""
        {
            delete($UserData);
        }

        delete($EventXML); (2)
        to_json(); (3)
    </Exec>
</Input>

1	The CaptureEventXML directive is set to `TRUE` to store the raw XML-formatted event data in the `$EventXML` field.
2	Use the delete() procedure to remove unnecessary fields after processing.
3	The to_json() procedure of the xm_json module is called to convert the record to JSON.

Output sample

{
  "EventTime": "2022-07-05T08:24:39.522394+02:00",
  "Hostname": "WINAB-2JR3FR9RD",
  "Keywords": "9259400833873739776",
  "LevelValue": 4,
  "EventType": "INFO",
  "SeverityValue": 2,
  "Severity": "INFO",
  "EventID": 7036,
  "SourceName": "Service Control Manager",
  "ProviderGuid": "{555908D1-A6D7-4695-8E1E-26931D2012F4}",
  "Version": 0,
  "TaskValue": 0,
  "OpcodeValue": 0,
  "RecordNumber": 3356,
  "ExecutionProcessID": 528,
  "ExecutionThreadID": 1640,
  "Channel": "System",
  "Message": "The nxlog service entered the running state.",
  "Level": "Information",
  "param1": "nxlog",
  "param2": "running",
  "EventData.Binary": "6E0078006C006F0067002F0034000000",
  "EventReceivedTime": "2022-07-05T08:24:40.725436+02:00",
  "SourceModuleName": "eventlog",
  "SourceModuleType": "im_msvistalog",
  "EventData": "<EventData><Data Name=\"param1\">nxlog</Data><Data Name=\"param2\">running</Data><Binary>6E0078006C006F0067002F0034000000</Binary></EventData>"
}

If the NXLog Agent service is running as a custom user, errors may be logged in the log file such as:

WARNING [im_msvistalog|windows] failed to subscribe to msvistalog events,access denied [error code: 5]: Access is denied.

WARNING [im_msvistalog|windows] ignoring source as it cannot be subscribed to (error code: 5)

This happens when the user account does not have permission to access the specified Windows Event Log channel. Refer to Access denied to a Windows Event Log channel for instructions on how to resolve these errors.