Correlation

Event correlation in NXLog Manager is architected similarly to the Patterns system. It is performed in real-time by the NXLog agents. This way it is possible to do local event correlation on the client side (at the agent). This will not only reduce the load on the central server but the system can send alerts over another channel (e.g. SMS) even if the network is down and the log messages would not reach the central log server.

A trivial example for this is to detect 3 failed login attempts into a system. When this happens, the user will be likely locked out. If the log analysis system is capable to detect such situation, a lot of things can be automated. You will not need to wait for the user to come asking for a new password.

For more information about the correlation capabilities of NXLog Manager, please consult the Event Correlator (pm_evcorr) module documentation in the NXLog Reference Manual.

Correlation rulesets

Correlation rules are grouped into rulesets. Because there can be different correlation rules depending on the agent or log source, rulesets ease the use of correlation rules. To view the list of correlation rulesets, click the Correlation menu item. The list of existing correlation rulesets will appear such as in the following screenshot:

Correlation ruleset list

To create a new ruleset, click the Add button.

Adding a correlation ruleset

Correlation rules

Correlation rules check whether the conditions specified in the rule are satisfied and execute an action. Correlation rules are evaluated linearly.

Clicking the name of a correlation ruleset will show a list of correlation rules within the ruleset:

Correlation rules in a ruleset

The order of the rules within the rulset matter because they are evaluated by NXLog’s pm_evcorr module in the order they appear. To change the order of the rules, use the Up and Down buttons.

Creating correlation rules

To create a new correlation rule, click the Add button on the bottom of the list. The following dialog window will appear:

Adding a correlation rule to a ruleset

Each correlation rule has a mandatory Name, Type and Action parameter and one or more type-specific parameters where the conditions can be specified. The following correlation rule types are available:

Simple
Suppressed
Pair
Absence
Thresholded

Please consult the Event Correlator (pm_evcorr) module documentation in the NXLog Reference Manual to view these rules provided by this module. There are two modes available to specify a condition:

Matched pattern is

This will generate a simple test to check whether the specified pattern matches. The generated NXLog config will contain a similar snippet:

if $PatternID == 42 {\
   ACTION            \
}

For this to work, the pm_pattern module must be configured and must be in front of the pm_evcorr module in the route. The pm_pattern module is responsible for setting the PatternID field.

Expert

The Expert field expects a statement (a boolean condition) which evaluates to TRUE or FALSE. The above expressed in Expert form would look like the following:

$PatternID == 42

Using the language constructs provided by NXLog, it is possible to
specify more complex conditions here, e.g.

($EventTime > now() - 10000) and ($PatternID == 42 or $PatternID == 142)

Example 1. Correlation rule for ssh brute-force attack detection

In this example, we will create a correlation rule which will detect ssh brute-force attempts. We define a brute-force attempt as 5 login failures within a 20-second interval. In this example, only an internal warning message is generated, but it is possible to trigger any other action, such as executing an external script to block the IP or sending an alert by email.

This correlation rule depends on a pattern that matches an ssh authentication failure event. See the Creating patterns section on how to do this. Once the pattern is available in our database, the correlation rule should be configured as shown in the following screenshot:

Correlation rule parameters

Exporting and importing correlation rules

NXLog Manager can export and import correlation rules and rulesets in an XML format. To export a correlation rule or a ruleset, check its checkbox in the list and click Export. You can import a correlation rule XML file by clicking on the Import button under the ruleset list.

Unlike patterns, correlation rules used by NXLog are not in XML. Correlation rules exported from NXLog Manager cannot be used by NXLog because the NXLog agent uses an Apache-style configuration file for the rules and this is part of (or included in) the pm_evcorr module configuration block in nxlog.conf.

Using correlation rules

Similarly to patterns, correlation rules are also used and executed by the NXLog engine. Unlike other log analysis solutions which utilize a single pattern correlation engine in the central log server, the architecture of NXLog Manager allows correlation rules to be evaluated on the agents as well.

To use the correlation rules in an NXLog agent, add a pm_evcorr processor module and select the appropriate correlation ruleset:

Configuring the pm_evcorr module

It is recommended to use only one ruleset per agent. The correlation rules will be pushed to the NXLog agent after clicking Update config and they will take effect after a restart. See the Agents chapter for more information about agent configuration details.

In many cases, correlation rules depend on patterns (and the PatternID field). For this reason, a pm_pattern module should be in the processor chain before the pm_evcorr module.