Prioritize log forwarding with NXLog Agent
Log prioritization involves defining when and how you forward your logs to their destination. Not all logs are of equal importance. While some log events may indicate a problem, others are informational, and you collect them for a potential investigation or compliance. A well-defined log-forwarding strategy helps you focus your efforts and resources on critical logs, avoiding overlooking important information.
Below, we provide examples of how you can process log events conditionally with NXLog Agent and forward them according to their criticality.
| While our examples use syslog messages and filter on the severity field, you can use any other field according to the log type. It is also possible to filter the contents of the data using a regular expression; however, this can increase resource consumption and may not be feasible for a production environment. |
Set route priority
NXLog Agent supports defining multiple routes and can transmit logs from the same input instance to different output instances. For example, you can route logs from the same source to be processed by different output instances based on event severity. You can then prioritize multiple routes by using the Priority route directive.
This example utilizes two routes: one for high-priority events and errors and another for low-priority events.
The configuration receives logs on TCP port 514, and parses log records with the parse_syslog() procedure of the xm_syslog module. This procedure populates the $Severity field, which the output instances use to drop() log records that do not match their severity.
The critical_route forwards high-priority events and errors immediately to the destination.
On the other hand, the info_route stores logs in memory using a pm_buffer instance.
The Priority route directive specifies that critical_route has higher priority; thus, NXLog Agent will process logs from this route first.
<Extension syslog>
Module xm_syslog
</Extension>
<Processor buffer>
Module pm_buffer
MaxSize 102400
Type Mem
WarnLimit 51200
BatchSize 10000
BatchFlushInterval 30000
</Processor>
<Input tcp_in>
Module im_tcp
ListenAddr 0.0.0.0:514
Exec parse_syslog();
</Input>
<Output high_priority>
Module om_tcp
Host 10.0.0.123:514
Exec if $Severity != "CRITICAL" and $Severity != "ERROR" drop();
</Output>
<Output low_priority>
Module om_tcp
Host 10.0.0.123:514
Exec if $Severity != "INFO" drop();
</Output>
<Route critical_route>
Path tcp_in => high_priority
Priority 1
</Route>
<Route info_route>
Path tcp_in => buffer => low_priority
Priority 10
</Route>Define a log forwarding period
In some cases, you might want to send lower-priority logs during a specific time. For example, limiting log forwarding to a particular time frame is especially useful when sending logs over the network to ensure the load does not disrupt business operations during peak hours. NXLog Agent can achieve this by using the pm_blocker module.
This example uses the pm_blocker module to forward logs outside working hours.
The configuration receives logs on TCP port 514 and forwards them to a remote destination via TCP. It uses a pm_buffer instance to store logs on disk until log forwarding is enabled between 7 PM and 6 AM.
<Processor buffer>
Module pm_buffer
MaxSize 102400
Type disk
WarnLimit 51200
</Processor>
<Processor blocker>
Module pm_blocker
<Schedule>
When @startup
<Exec>
if hour(now()) >= 6 and hour(now()) < 19
{
blocker->block(TRUE);
}
else
{
blocker->block(FALSE);
}
</Exec>
</Schedule>
<Schedule>
When 0 6 * * *
<Exec>
blocker->block(TRUE);
log_info("Route is blocked");
</Exec>
</Schedule>
<Schedule>
When 0 19 * * *
<Exec>
blocker->block(FALSE);
log_info("Route is unblocked");
</Exec>
</Schedule>
</Processor>
<Input tcp_in>
Module im_tcp
ListenAddr 0.0.0.0:514
</Input>
<Output low_priority>
Module om_tcp
Host 10.0.0.123:514
</Output>
<Route info_route>
Path tcp_in => buffer => blocker => low_priority
</Route>The following diagram illustrates log processing with the configuration above.
-
During the day, the pm_blocker instance is blocked. However, the
tcp_ininstance continues collecting logs and saving them in a buffer on disk. -
The pm_blocker instance is unblocked at night. The logs are released from the buffer and forwarded to the destination.
This example uses the pm_blocker module to collect logs outside working hours only.
The configuration reads events from Windows Event Log and forwards them to a remote NXLog Agent instance. Because log collection is blocked during the day, it is not necessary to configure extra buffering.
<Input eventlog>
Module im_msvistalog
</Input>
<Processor schedule>
Module pm_blocker
<Schedule>
# Start blocking at 7:00
When 0 7 * * *
Exec schedule->block(TRUE);
</Schedule>
<Schedule>
# Stop blocking at 19:00
When 0 19 * * *
Exec schedule->block(FALSE);
</Schedule>
</Processor>
<Output batch>
Module om_batchcompress
Host 10.0.0.123:2514
</Output>
<Route scheduled_batches>
Path eventlog => schedule => batch
</Route>The following diagram illustrates log processing with the configuration above.
-
During the day, the pm_blocker instance is blocked. Once its log queue is full, the
eventloginstance stops collecting logs. -
The pm_blocker instance is unblocked at night. It processes the events in its log queue and once enough space is available, the
eventloginstance resumes collecting events.
