Trim log events
NXLog Agent can reduce the size of log data by trimming (removing) unnecessary content or fields from log events. Trimming logs at the source means you transfer and store less data, reducing the data size during all subsequent log processing stages. This helps to reduce bandwidth usage, storage requirements, and licensing costs for commercial SIEMs that bill by data volume.
When trimming logs, it is essential to understand how NXLog Agent parses log records and which fields the modules you use transfer or write. See Log records and fields for more details, especially the Outgoing log records section.
| For other strategies to reduce the size of log data, see Filter logs and Bandwidth usage. |
Discard unnecessary fields
Not all fields in your logs may be of interest to you. NXLog Agent provides several ways to remove such fields, including defining a list of allowed or unnecessary fields and deleting fields.
This configuration collects Windows events with the im_msvistalog input module, automatically parsing log records into fields. It also uses an xm_rewrite module instance to define a list of allowed fields and discard the rest.
You can use the xm_rewrite instance in this configuration with other log sources. For example, the same fields also apply to syslog messages processed with xm_syslog.
The xm_rewrite module does not remove the $raw_event field.
See Discarding the raw event below for more details.
|
<Extension allowlist>
Module xm_rewrite
Keep AccountName, Channel, EventID, EventReceivedTime, EventTime, Hostname, \
Severity, SeverityValue, SourceName
</Extension>
<Input eventlog>
Module im_msvistalog
<QueryXML>
<QueryList>
<Query Id='0'>
<Select Path='Security'>*[System/Level<=4]</Select>
</Query>
</QueryList>
</QueryXML>
Exec allowlist->process();
</Input>If all the information you require is available in the event fields and you do not need the original event, you can reduce the event size by setting the $raw_event core field to an empty string.
This configuration collects Windows events with the im_msvistalog input module, automatically parsing log records into fields.
If the following modules in your route do not need the $raw_event field, such as when sending logs to another NXLog Agent instance with om_batchcompress, you can discard the field content.
Stream-oriented output modules, such as om_file and om_tcp, use the $raw_event field to write or forward the log event.
For this reason, this example is not suitable when using these modules.
|
<Input eventlog>
Module im_msvistalog
<QueryXML>
<QueryList>
<Query Id='1'>
<Select Path='Security'>*[System/Level<=4]</Select>
</Query>
</QueryList>
</QueryXML>
</Input>
<Output agent_relay>
Module om_batchcompress
Host 192.168.0.123:2514
Exec $raw_event = '';
</Output>Remove the Windows events description
In some cases, event messages contain extra data duplicated across events of the same type. An example is the descriptive event data in some Windows events. Removing this verbose text from events can significantly reduce data size while preserving all the event details.
This configuration collects Windows events from the Application, Security, and System logs.
It then uses regular expressions to truncate the $Message field of Security event IDs 4688 and 4769.
<Input eventlog>
Module im_msvistalog
<QueryXML>
<QueryList>
<Query Id="0">
<Select Path="Application">
*[System[(Level<=4)]]</Select>
<Select Path="Security">
*[System[(Level<=4)]]</Select>
<Select Path="System">
*[System[(Level<=4)]]</Select>
</Query>
</QueryList>
</QueryXML>
<Exec>
if ($Channel == 'Security') and ($EventID == 4688)
{
$Message =~ s/\s*Token Elevation Type indicates the type of.*$//s;
}
else if ($Channel == 'Security') and ($EventID == 4769)
{
$Message =~ s/\s*This event is generated every time access is.*$//s;
}
# Optionally, update the $raw_event field.
$raw_event = $EventTime + ' ' + $Message; (1)
</Exec>
</Input>| 1 | You can optionally update the $raw_event field with the new event message.
If you don’t, the $raw_event field retains the original value of the $Message field. |
The following is an event ID 4769 (A Kerberos ticket was requested) message.
A Kerberos service ticket was requested.
Account Information:
Account Name: WINAD$@EXAMPLE.COM
Account Domain: EXAMPLE.COM
Logon GUID: {55a7f67c-a32c-150a-29f1-7e173ff130a7}
Service Information:
Service Name: WINAD$
Service ID: EXAMPLE\WINAD$
Network Information:
Client Address: ::1
Client Port: 0
Additional Information:
Ticket Options: 0x40810000
Ticket Encryption Type: 0x12
Failure Code: 0x0
Transited Services: -
This event is generated every time access is requested to a resource such as a computer or a Windows service. The service name indicates the resource to which access was requested.
This event can be correlated with Windows logon events by comparing the Logon GUID fields in each event. The logon event occurs on the machine that was accessed, which is often a different machine than the domain controller which issued the service ticket.
Ticket options, encryption types, and failure codes are defined in RFC 4120.
The following shows the same message after NXLog Agent processed it.
A Kerberos service ticket was requested.
Account Information:
Account Name: WINAD$@EXAMPLE.COM
Account Domain: EXAMPLE.COM
Logon GUID: {55a7f67c-a32c-150a-29f1-7e173ff130a7}
Service Information:
Service Name: WINAD$
Service ID: EXAMPLE\WINAD$
Network Information:
Client Address: ::1
Client Port: 0
Additional Information:
Ticket Options: 0x40810000
Ticket Encryption Type: 0x12
Failure Code: 0x0
Transited Services: -
Truncate events to a specific size
There are cases when large events may cause an issue during transport, such as potential packet fragmentation when using UDP. To prevent this, you can truncate events to ensure they don’t exceed a specific size.
The following configuration collects Windows events with im_msvistalog and uses the substr() function to truncate them to 1000 bytes.
| This method discards any data after the specified position. Only use it if losing some data is acceptable. |
<Input eventlog>
Module im_msvistalog
<QueryXML>
<QueryList>
<Query Id='0'>
<Select Path="Security">*[System/Level<4]</Select>
</Query>
</QueryList>
</QueryXML>
Exec $raw_event = substr($raw_event, 0, 1000);
</Input>