Prioritize data forwarding with NXLog Agent
Prioritization involves defining when and how you forward your telemetry data to its destination. Not all telemetry data is of equal importance. For example, while some events may indicate a problem, others are informational, and you collect them for a potential investigation or compliance. A well-defined data-forwarding strategy helps you focus your efforts and resources on critical data, avoiding overlooking important information.
Below, we provide examples of how you can process telemetry data conditionally with NXLog Agent and forward it according to its criticality.
| While our examples use syslog messages and filter on the severity field, you can use any other field according to the data type. It is also possible to filter the data using a regular expression; however, this can increase resource consumption and may not be feasible for a production environment. |
Set route priority
NXLog Agent supports defining multiple routes and can transmit data from the same input instance to different output instances. For example, you can route data from the same source to be processed by different output instances based on severity. You can then prioritize multiple routes by using the Priority route directive.
This example utilizes two routes: one for high-priority events and errors and another for low-priority events.
The configuration receives events on TCP port 514 and parses them with the parse_syslog() procedure of the xm_syslog module. This procedure populates the $Severity field, which the output instances use to drop() events that do not match their severity.
The critical_route forwards high-priority events and errors immediately to the destination.
On the other hand, the info_route stores events in memory using a pm_buffer instance.
The Priority route directive specifies that critical_route has higher priority; thus, NXLog Agent will process events from this route first.
<Extension syslog>
Module xm_syslog
</Extension>
<Processor buffer>
Module pm_buffer
MaxSize 102400
Type Mem
WarnLimit 51200
BatchSize 10000
BatchFlushInterval 30000
</Processor>
<Input tcp_in>
Module im_tcp
ListenAddr 0.0.0.0:514
Exec parse_syslog();
</Input>
<Output high_priority>
Module om_tcp
Host 10.0.0.123:514
Exec if $Severity != "CRITICAL" and $Severity != "ERROR" drop();
</Output>
<Output low_priority>
Module om_tcp
Host 10.0.0.123:514
Exec if $Severity != "INFO" drop();
</Output>
<Route critical_route>
Path tcp_in => high_priority
Priority 1
</Route>
<Route info_route>
Path tcp_in => buffer => low_priority
Priority 10
</Route>Define a data forwarding period
In some cases, you might want to send lower-priority telemetry data during a specific time. For example, limiting forwarding to a particular time frame is especially useful when sending data over the network to ensure the load does not disrupt business operations during peak hours. NXLog Agent can achieve this by using the pm_blocker module.
This example uses the pm_blocker module to forward telemetry data outside working hours.
The configuration receives data on TCP port 514 and forwards them to a remote destination via TCP. It uses a pm_buffer instance to store data on disk until forwarding is enabled between 7 PM and 6 AM.
<Processor buffer>
Module pm_buffer
MaxSize 102400
Type disk
WarnLimit 51200
</Processor>
<Processor blocker>
Module pm_blocker
<Schedule>
When @startup
<Exec>
if hour(now()) >= 6 and hour(now()) < 19
{
blocker->block(TRUE);
}
else
{
blocker->block(FALSE);
}
</Exec>
</Schedule>
<Schedule>
When 0 6 * * *
<Exec>
blocker->block(TRUE);
log_info("Route is blocked");
</Exec>
</Schedule>
<Schedule>
When 0 19 * * *
<Exec>
blocker->block(FALSE);
log_info("Route is unblocked");
</Exec>
</Schedule>
</Processor>
<Input tcp_in>
Module im_tcp
ListenAddr 0.0.0.0:514
</Input>
<Output low_priority>
Module om_tcp
Host 10.0.0.123:514
</Output>
<Route info_route>
Path tcp_in => buffer => blocker => low_priority
</Route>The following diagram illustrates data processing with the configuration above.
-
During the day, the pm_blocker instance is blocked. However, the
tcp_ininstance continues collecting data and saving it in a buffer on disk. -
The pm_blocker instance is unblocked at night. The data is released from the buffer and forwarded to the destination.
This example uses the pm_blocker module to collect data outside working hours only.
The configuration reads events from Windows Event Log and forwards them to a remote NXLog Agent instance. Because telemetry data collection is blocked during the day, it is not necessary to configure extra buffering.
<Input eventlog>
Module im_msvistalog
</Input>
<Processor schedule>
Module pm_blocker
<Schedule>
# Start blocking at 7:00
When 0 7 * * *
Exec schedule->block(TRUE);
</Schedule>
<Schedule>
# Stop blocking at 19:00
When 0 19 * * *
Exec schedule->block(FALSE);
</Schedule>
</Processor>
<Output batch>
Module om_batchcompress
Host 10.0.0.123:2514
</Output>
<Route scheduled_batches>
Path eventlog => schedule => batch
</Route>The following diagram illustrates data processing with the configuration above.
-
During the day, the pm_blocker instance is blocked. Once its log queue is full, the
eventloginstance stops collecting data. -
The pm_blocker instance is unblocked at night. It processes the data in its log queue and once enough space is available, the
eventloginstance resumes collecting events.
