NXLog Legacy Documentation

NXLog language

The NXLog core has a built-in interpreted language. This language can be used to make complex decisions or build expressions in the NXLog configuration file. Code written in the NXLog language is similar to Perl, which is commonly used by developers and administrators for log processing tasks. When NXLog starts and reads its configuration file, directives containing NXLog language code are parsed and compiled into pseudo-code. If a syntax error is found, NXLog will print the error. This pseudo-code is then evaluated at run-time, as with other interpreted languages.

The features of the NXLog language are not limited to those in the NXLog core: modules can register functions and procedures to supplement built-in functions and procedures (see the xm_syslog functions, for example).

Due to the simplicity of the language there is no error handling available to the user, except for function return values. If an error occurs during the execution of the NXLog pseudo-code, usually the error is printed in the NXLog logs. If an error occurs during log message processing it is also possible that the message will be dropped. If sophisticated error handling or more complex processing is required, additional message processing can be implemented in an external script or program via the xm_exec module, in a dedicated NXLog module, or in Perl via the xm_perl module.

The NXLog language is described in five sections.

Types: All fields and other expressions in the NXLog language are typed.
Expressions: An expression is evaluated to a value at run-time and the value is used in place of the expression. All expressions have types. Expressions can be used as arguments for some module directives.
Statements: The evaluation of a statement will cause a change in the state of the NXLog engine, the state of a module instance, or the current event. Statements often contain expressions. Statements are used as arguments for the Exec module directive, where they are then executed for each event (unless scheduled).
Variables: Variables store data persistently in a module instance, across multiple event records.
Statistical counters: NXLog provides statistical counters with various algorithms that can be used for realtime analysis.

Example 1. Statements vs. configurations

While this Guide provides many configuration examples, in some cases only statement examples are given. Statements must be used with the Exec directive (or Exec block). The following statement example shows one way to use the parsedate() function.

if $raw_event =~ /^(\w{3} \d{2} \d{2}:\d{2}:\d{2})/
    $EventTime = parsedate($1);

The following configuration example uses the above statement in an Exec block.

nxlog.conf

<Input in>
    Module  im_file
    File    '/var/log/app.log'
    <Exec>
        if $raw_event =~ /^(\w{3} \d{2} \d{2}:\d{2}:\d{2})/
            $EventTime = parsedate($1);
    </Exec>
</Input>

Types

The NXLog language is a typed language. Fields, literals, and other expressions evaluate to values with specific types. This allows for stricter type-safety syntax checking when parsing the configuration. Note that fields and some functions can return values with types that can only be determined at run-time.

The language provides only simple types. Complex types such as arrays and hashes (associative arrays) are not supported. The language does support the undefined value similar to that in Perl. See the xm_perl module if you require more complex types.

A log’s format must be parsed before its individual parts can be used for processing (see Fields). But even after the message has been parsed into its parts, additional processing may still be required, for example, to prepare a timestamp for comparison with another timestamp. This is a situation where typing is helpful: by converting all timestamps to the datetime type they can be easily compared—and converted back to strings later if required—using the functions and procedures provided. The same applies to other types.

Example 2. Typed fields in a syslog event record

The following illustrates the four steps NXLog performs with this configuration as it manually processes a syslog event record using only regular expressions on the core field $raw_event and the core function parsedate().

nxlog.conf

<Input in>
    # 1. New event record created
    Module  im_udp
    Host    0.0.0.0
    Port    514
    <Exec>
        # 2. Timestamp parsed from Syslog header
        if $raw_event =~ /^(\w{3} \d{2} \d{2}:\d{2}:\d{2})/
        {
            # 3. parsedate() function converts from string to datetime
            $EventTime = parsedate($1);
            # 4. Datetime fields compared
            if ( $EventReceivedTime - $EventTime ) > 60000000
                log_warning('Message delayed more than 1 minute');
        }
    </Exec>
</Input>

NXLog creates a new event record for the incoming log message. The new event record contains the $raw_event string type field, with the contents of the entire syslog string.
A regular expression is used to parse the timestamp from the event. The captured sub-string is a string type, not a datetime type.
The parsedate() function converts the captured string to a datetime type.
Two datetime fields are compared to determine if the message was delayed during delivery. The datetime type $EventReceivedTime field is added by NXLog to each event when it is received.

Normally the parse_syslog() procedure (provided by the xm_syslog extension module) would be used to parse a syslog event. It will create fields with the appropriate types during parsing, eliminating the need to directly call the parsedate() function. See Collecting and Parsing Syslog.

For a full list of types, see the Reference Manual Types section. For NXLog language core functions that can be used to work with types, see Functions. For functions and procedures that can work with types related to a particular format, see the module corresponding to the required format.

Expressions

An expression is a language element that is dynamically evaluated to a value at run-time. The value is then used in place of the expression. Each expression evaluates to a type, but not always to the same type.

The following language elements are expressions: literals, regular expressions, fields, operations, and functions.

Expressions can be bracketed by parentheses ( ) to help improve code readability.

Example 3. Using parentheses (round brackets) around expressions

There are three statements below, one per line. Each statement contains multiple expressions, with parentheses added in various ways.

if 1 + 1 == (1 + 1) log_info("2");
if (1 + 1) == (1 + 1) log_info("2");
if ((1 + 1) == (1 + 1)) log_info("2");

Expressions are often used in statements.

Example 4. Using an expression in a statement

This simple statement uses the log_info() procedure with an expression as its argument. In this case the expression is a literal.

In NXLog Enterprise Edition version 5, the log_info() procedure will truncate messages larger than 1024 bytes. In subsequent versions, messages longer than specified at LogSizeLimit will be truncated.

log_info('This message will be logged.');

Here is a function (also an expression) that is used in the same procedure. It generates an internal event with the current time when each event is processed.

log_info(now());

Expressions can be used with module directives that support them.

Example 5. Expressions for directives

The File directive of the om_file module supports expressions. This allows the output filename to be set dynamically for each individual event.

nxlog.conf

<Output out>
    Module  om_file
    File    "/var/log/nxlog/out_" + strftime($EventTime, "%Y%m%d")
</Output>

See Using dynamic filenames for more information.

Literals

A literal is a simple expression that represents a fixed value. Common literals include booleans, integers, and strings. The type of literal is detected by the syntax used to declare it.

This section demonstrates the use of literals by using examples with assignment statements.

Boolean literals can be declared using the constants TRUE or FALSE. Both are case-insensitive.

Setting Boolean literals

$Important = FALSE;
$Local = true;

Integer literals are declared with an unquoted integer. Negative integers, hexademical notation, and base-2 modifiers (Kilo, Mega, and Giga) are supported.

Setting integer literals

$Count = 42;
$NegativeCount = -42;
$BigCount = 42M;
$HexCount = 0x2A;

String literals are declared by quoting characters with single or double quotes. Escape sequences are available when using double quotes.

Setting string literals

$Server = 'Alpha';
$Message = 'This is a test message.';
$NumberAsString = '12';
$StringWithNewline = "This is line 1.\nThis is line 2.";

For a list of all available literals, see the Reference Manual Literals section.

Regular expressions

NXLog supports regular expressions for matching, parsing, and modifying event records. In the context of the NXLog language, a regular expression is an expression that is evaluated to a boolean value at run-time. Regular expressions can be used to define complex search and replacement patterns for text matching and substitution.

Examples in this section use only simple patterns. See Extracting data and other topic-specific sections for more extensive examples.

Matching can be used with an if statement to conditionally execute a statement.

Example 6. Matching a field with a regular expression

The event record will be discarded if the $raw_event field matches the regular expression.

if $raw_event =~ /TEST: / drop();

Regular expression matching can also be used for extensive parsing, by capturing sub-strings for field assignment.

Example 7. Parsing fields with a regular expression

If the $raw_event field contains the regular expression, the two fields will be set to the corresponding captured sub-strings.

if $raw_event =~ /TEST(\d): (.+)/
{
    $TestNumber = $1;
    $TestName = $2;
}

Regular expression matching also supports named capturing groups. This can be useful when writing long regular expressions. Each captured group is automatically added to the event record as a field with the same name.

Example 8. Named capturing groups

This regular expression uses the named groups TestNumber and TestName to add corresponding $TestNumber and $TestName fields to the event record.

if $raw_event =~ /TEST(?<TestNumber>\d): (?<TestName>.+)/
{
    $Message = $TestNumber + ' ' + $TestName;
}

Regular expression substitution can be used to modify a string. In this case, the regular expression follows the form s/pattern/replace/. The result of the expression will be assigned to the field to the left of the operator.

Example 9. Performing substitution using a regular expression

The first regular expression match will be removed from the $raw_event field.

$raw_event =~ s/TEST: //;

Global substitution is supported with the /g modifier. Without the /g modifier, only the first match in the string will be replaced.

Example 10. Global regular expression substitution

Every whitespace character in the $AlertType field will be replaced with an underscore (_).

$AlertType =~ s/\s/_/g;

A statement can be conditionally executed according to the success of a regular expression substitution.

Example 11. Regular expression substitution with conditional execution

If the substitution succeeds, an internal log message will also be generated.

if $Hostname =~ s/myhost/yourhost/ log_info('Updated hostname');

For more information, see the following sections in the Reference Manual: Regular Expressions, =~, and !~.

Fields

When NXLog receives a log message, it creates an event record for it. An event record is a set of fields (see Fields for more information). A field is an expression which evaluates to a value with a specific type. Each field has a name, and in the NXLog language it is represented with the dollar sign ($) prefixed to the name of the field, like Perl’s scalar variables.

Fields are only available in an evaluation context which is triggered by a log message. For example, using a value of a field in the Exec directive of a Schedule block will result in a run-time error because the scheduled execution is not triggered by a log message.

Because it is through fields that the NXLog language accesses the contents of an event record, they are frequently referenced. The following examples show some common ways that fields are used in NXLog configurations.

Example 12. Assigning a value to a field

This statement uses assignment to set the $Department field on log messages.

$Department = 'customer-service';

Example 13. Testing a field value

If the $Hostname field does not match, the message will be discarded with the drop() procedure.

if $Hostname != 'webserver' drop();

Example 14. Using a field in a procedure

This statement will generate an internal event if $SeverityValue integer field is greater than 2 (NXLog INFO severity). The generated event will include the contents of the $Message field.

if $SeverityValue > 2 log_warning("ALERT: " + $Message);

Operations

Like other programming languages and especially Perl, the NXLog language has unary operations, binary operations, and the conditional ternary operation. These operations are expressions and evaluate to values.

Unary operations

Unary operations work with a single operand and evaluate to a boolean value.

Example 15. Using a unary operation

This statement uses the defined operator to log a message only if the $Hostname field is defined in the event record.

if defined $Hostname log_info('Event received');

Binary operations

Binary operations work with two operands and evaluate to a value. The type of the evaluated value depends on the type of the operands. Execution might result in a run-time error if the type of the operands are unknown at compile time and then evaluate to types which are incompatible with the binary operation when executed.

Example 16. Using binary operations

This statement uses the == operator to drop the event if the $Hostname field matches.

if $Hostname == 'testbox' drop();

Here, the + operator is used to concatenate two strings.

log_info('Event received from ' + $Hostname);

Ternary operation

The conditional or ternary operation requires three operands. The first is an expression that evaluates to a boolean. The second is an expression that is evaluated if the first expression is TRUE. The third is an expression that is evaluated if the first expression is FALSE.

Example 17. Using the ternary operation

This statement sets the $Important field to TRUE if $SeverityValue is greater than 2, or FALSE otherwise. The parentheses are optional and have been added here for clarity.

$Important = ( $SeverityValue > 2 ? TRUE : FALSE );

For a full list of supported operations, see the Reference Manual Operations section.

Functions

A function is an expression which returns a value. The returned value can be used to set fields, output log data, or make logic decisions. Functions can be polymorphic, meaning that the same function can take different argument types.

Many NXLog language features are provided through functions. As with other types of expressions, and unlike procedures, functions do not modify the state of the NXLog engine, the state of the module, or the current event.

See the list of available core functions. Modules can provide additional functions for use with the NXLog language.

Example 18. Function calls

These expressions call the now() function to return the current time and the hostname() function to return the hostname of the system where NXLog is installed. The returned values are used to set the $EventTime and $Relay fields.

$EventTime = now();
$Relay = hostname();

In the example below, the size() function is called to calculate the size of the $Message field. If the field is over 4096 bytes, an internal log is generated.

if size($Message) > 4096 log_info('Large message received.');

Example 19. Calling a function of a specific module instance

Functions for a specific module instance can be called using the -> operator. This expression calls the file_name() and file_size() functions of an om_file instance named out. The returned values are used to log the name and size of its current output file.

x('Size of output file ' + out->file_name() + ' is ' + out->file_size());

Calling functions of a specific instance is especially useful when the configuration contains more than one instance of the same module. These expressions call the to_xml() function of two xm_xml instances, one named xml_a and the other named xml_b.

xml_a->to_xml();
xml_b->to_xml();

Statements

The evaluation of a statement will usually result in a change in the state of the NXLog engine, the state of a module, or the log message.

Statements are used with the Exec module directive. A statement is terminated by a semicolon (;).

Example 20. Using a statement with Exec

With this input configuration, an internal NXLog log message will be generated for each message received.

nxlog.conf

<Input in>
    Module  im_udp
    Host    0.0.0.0
    Port    514
    Exec    log_info("Message received on UDP port 514");
</Input>

Multiple statements can be specified, these will be evaluated and executed in order. Statements can also be given on multiple lines by using line continuation or by enclosing the statements in an Exec block.

Example 21. Using multiple statements with Exec

This configuration generates an internal log message and sets the $File field.

nxlog.conf

<Input in1>
    Module  im_file
    File    '/var/log/app.log'
    Exec    log_info("App message read from log"); $File = file_name();
</Input>

This is the same, but the backslash (\) is used to continue the Exec directive to the next line.

nxlog.conf

<Input in2>
    Module  im_file
    File    '/var/log/app.log'
    Exec    log_info("App message read from log"); \
            $File = file_name();
</Input>

The following configuration is functionally equivalent to the previous configuration above. However, by creating an Exec block, multiple statements can be specified without the need for a backslash (\) line continuation at the end of each line.

nxlog.conf

<Input in3>
    Module  im_file
    File    '/var/log/app.log'
    <Exec>
        log_info("App message read from log");
        $File = file_name();
    </Exec>
</Input>

Statements can also be executed based on a schedule by using the Exec directive of a Schedule block. The Exec directive is slightly different in this example. Because its execution depends solely on a schedule instead of any incoming log events, there is no event record that can be associated with it. The $File field assignment in the example above would be impossible.

Example 22. Using a statement in a schedule

This input instance will generate an hourly internal log event.

nxlog.conf

<Input syslog_udp>
    Module  im_udp
    Host    0.0.0.0
    Port    514
    <Schedule>
        When    @hourly
        Exec    log_info("The syslog_udp input module instance is active.");
    </Schedule>
</Input>

Similar functionality is implemented by the im_mark module.

Assignment

Each event record is made up of fields, and assignment is the primary way that a value is written to a field in the NXLog language. The assignment operation is declared with an equal sign (=). This operation loads the value from the expression evaluated on the right into an event record field on the left.

Example 23. Using field assignment

This input instance uses assignment operations to add two fields to each event record.

nxlog.conf

<Input in>
    Module  im_file
    File    '/var/log/messages'
    <Exec>
        $Department = 'processing';
        $Tier = 1;
    </Exec>
</Input>

Block

Statements can be declared inside a block by surrounding them with curly braces ({}). A statement block in the configuration is parsed as if it were a single statement. Blocks are typically used with conditional statements.

Example 24. Using statement blocks

This statement uses a block to execute two statements if the $Message field matches.

nxlog.conf

<Input in>
    Module  im_file
    File    '/var/log/messages'
    <Exec>
        if $Message =~ /^br0:/
        {
            log_warning('br0 interface state changed');
            $Tag = 'network';
        }
    </Exec>
</Input>

Procedures

A procedure is a statement that performs a set of actions. Procedures can accept arguments. Unlike a function, a procedure modifies its arguments, the state of the NXLog engine, the state of a module, or the current event. Procedures can be polymorphic, meaning that the same procedure can take different argument types.

Many NXLog language features are provided through procedures. See the list of available core procedures. Modules can provide additional procedures for use with the NXLog language.

Example 25. Calling a procedure

This example uses the parse_syslog() procedure, provided by the xm_syslog module, to parse syslog records received via UDP.

nxlog.conf

<Input in>
    Module  im_udp
    Host    0.0.0.0
    Port    514
    Exec    parse_syslog();
</Input>

Example 26. Calling a procedure of a specific module instance

Procedures for a specific module instance can be called using the -> operator. This statement calls the rotate_to() procedure of an om_file instance named out to rotate the current output file.

out->rotate_to("output_logs");

Calling procedures of a specific instance is especially useful when the configuration contains more than one instance of the same module. These statements call the parse_xml() procedure of two xm_xml instances, one named xml_a and the other named xml_b.

xml_a->parse_xml();
xml_b->parse_xml();

If-else

The if or conditional statement allows a statement to be executed based on the boolean value of an expression. When the boolean is TRUE, the statement is executed. An optional else keyword can be followed by another statement to be executed if the boolean is FALSE.

Example 27. Using If statements

This example uses an if statement and the drop() procedure to discard any event that matches the regular expression.

nxlog.conf

<Input in1>
    Module  im_file
    File    '/var/log/messages'
    Exec    if $raw_event =~ /junk/ drop();
</Input>

Here, any event not matching the regular expression will be dropped.

nxlog.conf

<Input in2>
    Module  im_file
    File    '/var/log/messages'
    Exec    if not ($raw_event =~ /important/) drop();
</Input>

Finally, this statement shows more extensive use of the if statement, with an else clause and blocks defined by curly braces ({}).

nxlog.conf

<Input in3>
    Module  im_file
    File    '/var/log/messages'
    <Exec>
        if $raw_event =~ /alert/
        {
            log_warning('Detected alert message');
        }
        else
        {
            log_info('Discarding non-alert message');
            drop();
        }
    </Exec>
</Input>

Variables

While NXLog provides fields for storing data during the processing of an event, they are only available for the duration of that event record and can not be used to store a value across multiple events. For this purpose, module variables can be used. A variable stores a value for the module instance where it is set. It can only be accessed from the same module where it was created: a variable with the same name is a different variable when referenced from another module.

Each module variable can be created with an expiry value or an infinite lifetime. If an expiry is used, the variable will be destroyed automatically when the lifetime expires. This can be used as a garbage collection method or to reset variable values automatically.

A module variable is referenced by a string value and can store a value of any type. Module variables are supported by all modules. See the create_var(), delete_var(), set_var(), and get_var() procedures.

Example 28. Using module variables

If the number of login failures exceeds 3 within 45 seconds, then an internal log message is generated.

nxlog.conf

<Input in>
    Module  im_file
    File    '/var/log/messages'
    <Exec>
    if $Message =~ /login failure/
    {
       if not defined get_var('login_failures')
       { # create the variable if it doesn't exist
           create_var('login_failures', 45);
           set_var('login_failures', 1);
       }
       else
       { # increase the variable and check if it is over the limit
           set_var('login_failures', get_var('login_failures') + 1);
           if get_var('login_failures') >= 3 
              log_warning(">= 3 login failures within 45 seconds");
       }
    }
    </Exec>
</Input>

The pm_evcorr module is recommended instead for this case. This algorithm does not reliably detect failures because the lifetime of the variable is not affected by set_var(). For example, consider login failures at 0, 44, 46, and 47 seconds. The lifetime of the variable will be set when the first failure occurs, causing the variable to be cleared at 45 seconds. The variable is created with a new expiry at 46 seconds, but then only two failures are noticed. Also, this method can only work in real-time because the timing is not based on values available in the log message (although the event time could be stored in another variable).

Statistical counters

Like variables, statistical counters provide data storage for a module instance. Counters only support integers, but a counter can use an algorithm to recalculate its value every time it is updated or read. With NXLog Enterprise Edition v4.x and earlier, a statistical counter will only return a value if the time specified in the interval argument has elapsed since it was created. Statistical counters can also be created with a lifetime. When a counter expires, it is destroyed, like module variables.

A statistical counter can be created with the create_stat() procedure call. After it is created, it can be updated with the add_stat() procedure call. The value of the counter can be read with the get_stat() function call. Note that the value of the statistical counter is only recalculated during these calls, rather than happening automatically. This can result in some slight distortion of the calculated value if the add and read operations are infrequent.

A time value can also be specified during creation, updating, and reading. This makes it possible for statistical counters to be used with offline log processing.

Example 29. Using statistical counters

This input configuration uses a Schedule block and a statistical counter with the RATEMAX algorithm to calculate the maximum rate of events over a 1 hour period. An internal log message is generated if the rate exceeds 500 events/second at any point during the 1 hour period.

nxlog.conf

<Input in>
    Module  im_tcp
    Host    0.0.0.0
    Port    1514
    <Exec>
        parse_syslog();
        if defined get_stat('eps') add_stat('eps', 1, $EventReceivedTime);
    </Exec>
    <Schedule>
        Every   1 hour
        <Exec>
            create_stat('eps', 'RATEMAX', 1, now(), 3600);
            if get_stat('eps') > 500
                log_info('Inbound TCP rate peaked at ' + get_stat('eps')
                         + ' events/second during the last hour');
        </Exec>
    </Schedule>
</Input>