Log prioritization

You may need to implement log prioritization to ensure that critical events are processed and sent to the destination with priority over informational events. NXLog can process events conditionally and forward them according to their criticality.

While the examples in this document use syslog messages and filter on the severity field, you can use any other field according to the log type. It is also possible to filter the contents of the data using a regular expression; however, this can increase resource consumption and may not be feasible for a production environment.

Setting route priority

NXLog supports defining multiple routes and can transmit logs from the same input instance to different output instances. For example, you can route logs from the same source to be processed by different output instances based on event severity. You can then prioritize multiple routes by using the Priority route directive.

Example 1. Routing according to log priority

This example utilizes two routes, one for high priority events and errors and another for low priority events.

The tcp_in input instance of the im_tcp module listens for syslog messages and parses log records using the parse_syslog() procedure of the xm_syslog module. As a result of parsing records with this procedure, events can be filtered based on the $Severity field in the output instances. Events that do not match the respective severity for each output instance are discarded with the drop() procedure.

The critical_route forwards high priority events and errors immediately to the destination. On the other hand, the info_route stores logs in a memory buffer using the buffer instance of the pm_buffer module. The Priority route directive specifies that critical_route has higher priority, and thus, logs from this route will be processed first.

nxlog.conf

<Extension syslog>
    Module                xm_syslog
</Extension>

<Processor buffer>
    Module                pm_buffer
    MaxSize               102400
    Type                  Mem
    WarnLimit             51200
    BatchSize             10000
    BatchFlushInterval    30000
</Processor>

<Input tcp_in>
    Module                im_tcp
    ListenAddr            0.0.0.0:514
    Exec                  parse_syslog();
</Input>

<Output high_priority>
    Module                om_tcp
    Host                  10.0.0.1:514
    Exec                  if $Severity != "CRITICAL" and $Severity != "ERROR" drop();
</Output>

<Output low_priority>
    Module                om_tcp
    Host                  10.0.0.1:1514
    Exec                  if $Severity != "INFO" drop();
</Output>

<Route critical_route>
    Path                  tcp_in => high_priority
    Priority              1
</Route>

<Route info_route>
    Path                  tcp_in => buffer => low_priority
    Priority              10
</Route>

Limiting log forwarding to a specific period

In some cases, you might want to send lower priority logs during a specific time. For example, limiting log forwarding to a particular time frame is especially useful when sending logs over the network to ensure that the load does not disrupt business operations during peak hours. NXLog can achieve this by using the pm_blocker module.

Example 2. Blocking log forwarding during peak hours

This example uses the pm_blocker module to only forward events outside working hours.

The tcp_in input instance of the im_tcp module listens for syslog messages and parses log records using the parse_syslog() procedure of the xm_syslog module. As a result of parsing records with this procedure, events can be filtered based on the $Severity field in the output instance. Events that are not of INFO severity are discarded with the drop() procedure.

The info_route stores logs in a disk-based buffer using the buffer instance of the pm_buffer module. The buffer stores the logs on disk until the blocker instance of the pm_blocker module unblocks the route between 7 PM and 6 AM.

nxlog.conf

<Extension syslog>
    Module        xm_syslog
</Extension>

<Processor buffer>
    Module        pm_buffer
    MaxSize       102400
    Type          disk
    WarnLimit     51200
</Processor>

<Processor blocker>
    Module        pm_blocker
    <Schedule>
        When      @startup
        <Exec>
            if hour(now()) >= 6 and hour(now()) < 19
            {
                blocker->block(TRUE);
            }
            else
            {
                blocker->block(FALSE);
            }
        </Exec>
    </Schedule>
    <Schedule>
        When      0 6 * * *
        <Exec>
            blocker->block(TRUE);
            log_info("Route is locked");
        </Exec>
    </Schedule>
    <Schedule>
        When      0 19 * * *
        <Exec>
            blocker->block(FALSE);
            log_info("Route is unlocked");
        </Exec>
    </Schedule>
</Processor>

<Input tcp_in>
    Module        im_tcp
    ListenAddr    0.0.0.0:514
    Exec          parse_syslog();
</Input>

<Output low_priority>
    Module        om_tcp
    Host          10.0.0.1:1514
    Exec          if $Severity != "INFO" drop();
</Output>

<Route info_route>
    Path          tcp_in => buffer => blocker => low_priority
</Route>