Parse XML logs
Extensible Markup Language (XML) is an interoperable data and file format. It supports storing data in a structured format that is both human and machine-readable. NXLog Agent includes an XML log parser that significantly simplifies XML log processing.
Below, we provide examples of collecting and processing logs in the XML format.
Parse basic XML
NXLog Agent’s xm_xml module uses <Event> as the RootTag, does not parse attributes, and does not prefix fields with the root tag by default.
This configuration collects XML logs from a file with the im_file input module and parses records with the xm_xml module. It expects log records that comply with xm_xml's default settings.
<Extension xml_parser>
Module xm_xml
</Extension>
<Input xml_log>
Module im_file
File '/path/to/file'
Exec parse_xml(); (1)
</Input>| 1 | Calls the parse_xml() procedure to parse the record into structured data. |
The following log sample adheres to the expected XML format. Although it is pretty printed for legibility, the configuration expects one record per line.
<Event>
<Timestamp>2023-11-09T16:13:02.775963+01:00</Timestamp>
<Hostname>SERVER-1</Hostname>
<Severity>NOTICE</Severity>
<Message>The server has started.</Message>
</Event>When the NXLog Agent configuration above processes this log event, it adds the following fields to the log record in addition to the core fields.
| Field | Value |
|---|---|
$Timestamp |
2023-11-09T16:13:02.775963+01:00 |
$Hostname |
SERVER-1 |
$Severity |
NOTICE |
$Message |
The server has started. |
Parse complex XML
You can customize the XML log parser settings to parse any XML structure. In addition, you can combine it with the xm_multiline module to parse multiline log records.
This configuration collects multiline logs from a file with the im_file input module and uses the xm_multiline and xm_xml modules to parse the records.
<Extension multiline> (1)
Module xm_multiline
HeaderLine /^<LogRecord>/
EndLine /^<\/LogRecord>/
</Extension>
<Extension xml_parser>
Module xm_xml
RootTag LogRecord
ParseAttributes TRUE (2)
</Extension>
<Input xml_log>
Module im_file
File '/path/to/file'
InputType multiline (3)
Exec parse_xml(); (4)
</Input>| 1 | Defines regular expressions to detect the first and last line of a record. |
| 2 | Enables the ParseAttributes directive to create fields from element attributes. |
| 3 | Sets the InputType to the xm_multiline module instance. |
| 4 | Calls the parse_xml() procedure to parse the record into structured data. |
<LogRecord>
<Timestamp>2023-11-09T16:13:02.775963+01:00</Timestamp>
<Hostname>SERVER-1</Hostname>
<Event ID="4605" Category="System">
<Severity>NOTICE</Severity>
<Message>The server has started.</Message>
</Event>
</LogRecord>When the NXLog Agent configuration above processes this log event, it adds the following fields to the log record in addition to the core fields.
| Field | Value |
|---|---|
$Timestamp |
2023-11-09T16:13:02.775963+01:00 |
$Hostname |
SERVER-1 |
$Event.ID |
4605 |
$Event.Category |
System |
$Event.Severity |
NOTICE |
$Event.Message |
The server has started. |
Parse Windows Event Log XML
The XML log parser provides specialized procedures for parsing Windows events in XML format without requiring custom settings.
This configuration collects Windows event logs from a file with the im_file input module and parses records with the xm_xml module. It expects log records in the Windows Event Log XML format.
<Extension xml>
Module xm_xml
PrefixWinEvent TRUE (1)
</Extension>
<Input windows_event>
Module im_file
File 'C:\logs\security_evt.xml'
Exec parse_windows_eventlog_xml(); (2)
</Input>| 1 | Enables the PrefixWinEvent directive to prefix EventData and UserData child elements with the parent field name. |
| 2 | Calls the parse_windows_eventlog_xml() procedure to parse the record into structured data. |
The following is a Windows security event sample. Although it is pretty printed for legibility, the configuration expects one record per line.
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Microsoft-Windows-Security-Auditing" Guid="{54849625-5478-4994-a5ba-3e3b0328c30d}" />
<EventID>4672</EventID>
<Version>0</Version>
<Level>0</Level>
<Task>12548</Task>
<Opcode>0</Opcode>
<Keywords>0x8020000000000000</Keywords>
<TimeCreated SystemTime="2023-11-09T16:55:28.1200980Z" />
<EventRecordID>1637767</EventRecordID>
<Correlation ActivityID="{485b6933-09f7-0002-ba69-5b48f709da01}" />
<Execution ProcessID="1004" ThreadID="3152" />
<Channel>Security</Channel>
<Computer>CLIENT-1</Computer>
<Security />
</System>
<EventData>
<Data Name="SubjectUserSid">S-1-5-18</Data>
<Data Name="SubjectUserName">SYSTEM</Data>
<Data Name="SubjectDomainName">NT AUTHORITY</Data>
<Data Name="SubjectLogonId">0x3e7</Data>
<Data Name="PrivilegeList">SeAssignPrimaryTokenPrivilege SeTcbPrivilege SeSecurityPrivilege SeTakeOwnershipPrivilege SeLoadDriverPrivilege SeBackupPrivilege SeRestorePrivilege SeDebugPrivilege SeAuditPrivilege SeSystemEnvironmentPrivilege SeImpersonatePrivilege SeDelegateSessionUserImpersonatePrivilege</Data>
</EventData>
</Event>When the NXLog Agent configuration above processes this log event, it adds the following fields to the log record in addition to the core fields.
Notice the fields prefixed with EventData.
If the PrefixWinEvent directive is not enabled, the configuration will still parse the fields but omit the prefix.
| Field | Value |
|---|---|
$SourceName |
Microsoft-Windows-Security-Auditing |
$ProviderGuid |
{54849625-5478-4994-a5ba-3e3b0328c30d} |
$EventID |
4672 |
$Version |
0 |
$LevelValue |
0 |
$TaskValue |
12548 |
$OpcodeValue |
0 |
$Keywords |
0x8020000000000000 |
$EventTime |
2023-11-09T17:55:28.120098+01:00 |
$RecordNumber |
1637767 |
$ActivityID |
{485b6933-09f7-0002-ba69-5b48f709da01} |
$ExecutionProcessID |
1004 |
$ExecutionThreadID |
3152 |
$Channel |
Security |
$EventData.SubjectUserSid |
S-1-5-18 |
$EventData.SubjectUserName |
SYSTEM |
$EventData.SubjectDomainName |
NT AUTHORITY |
$EventData.SubjectLogonId" |
0x3e7 |
$EventData.PrivilegeList |
SeAssignPrimaryTokenPrivilege SeTcbPrivilege SeSecurityPrivilege SeTakeOwnershipPrivilege SeLoadDriverPrivilege SeBackupPrivilege SeRestorePrivilege SeDebugPrivilege SeAuditPrivilege SeSystemEnvironmentPrivilege SeImpersonatePrivilege SeDelegateSessionUserImpersonatePrivilege |
$EventType |
AUDIT_SUCCESS |
$SeverityValue |
2 |
$Severity |
INFO |