NXLog Legacy Documentation

Language

Types

The following types are provided by the NXLog language.

Unknown: This is a special type for values where the type cannot be determined at compile time and for uninitialized values. The undef literal and fields without a value also have an unknown type. The unknown type can also be thought of as "any" in case of function and procedure API declarations.

Boolean: A boolean value is TRUE, FALSE or undefined. Note that an undefined value is not the same as a FALSE value.

Integer: An integer can hold a signed 64 bit value in addition to the undefined value. Floating point values are not supported.

String: A string is an array of characters in any character set. The binary type should be used for values where the NULL byte can also occur. An undefined string is not the same as an empty string. Strings have a limited length to prevent resource exhaustion problems, this is a compile-time value currently set to 1M.

Datetime: A datetime holds a microsecond, millisecond, and second value of time elapsed since the Epoch, and NXLog can handle the corresponding value. It is always stored in UTC/GMT.

IP Address: The ipaddr type can store IP addresses in an internal format. This type is used to store both dotted-quad IPv4 addresses (for example, 192.168.1.1) and IPv6 addresses (for example, 2001:0db8:85a3:0000:0000:8a2e:0370:7334).

Regular expression: A regular expression type can only be used with the =~ or !~ operators.

Binary: This type can hold an array of bytes.

Array: This type is a container and it can hold an arbitrary number values of any kind of types supported by the NXLog language. The inividual members are accessed by a non-negative integer type of index value.

Hash: This type is a container and it can hold an arbitrary number of key-value pairs. The key is a string while the value is one of the value types supported by the NXLog language. The inividual values can be accessed by the key strings.

Variadic arguments: This is a special type only used in function and procedure API declarations to indicate variadic arguments.

JSON string

A specific format of json-string.

JSON string is not yet a separate data type and is just a simple string. It was described separately to specify the string format.

JSON map: A specific format for json-string. Presents as an object description for specific json-array.

Expressions

Literals

Undef

The undef literal has an unknown type. It can be also used in an assignment to unset the value of a field.

Example 1. Un-Setting the Value of a Field

This statement unsets the $ProcessID field.

$ProcessID = undef;

Boolean: A boolean literal is either TRUE or FALSE. It is case-insensitive, so True, False, true, and false are also valid.

Integer

An integer starts with a minus (-) sign if it is negative. A "0X" or "0x" prepended modifier indicates a hexadecimal notation. The "K", "M" and "G" modifiers are also supported; these mean Kilo (1024), Mega (1024^2), or Giga (1024^3) respectively when appended.

Example 2. Setting an Integer Value

This statement uses a modifier to set the $Limit field to 44040192 (42×1024^2).

$Limit = 42M;

String

String literals are quoted characters using either single or double quotes.

Enclosing in single quotes means that the string contains no escape sequences. For example, the '\n' character combination will be interpreted as \n. Similarly, the asterisk '*' character inside single quotes will be processed by NXLog as the asterisk.

String literals specified within double quotes can contain escape sequences. For instance, the "\n" character combination will be interpreted as a newline (LF) character as opposed to '\n' in single quotes, which is treated by NXLog as \n.

Below is the full list of escape sequences supported by NXLog.

\\: The backslash (\) character.
\": The double quote (") character.
\n: Line feed (LF).
\r: Carriage return (CR).
\t: Horizontal tab.
\b: Audible bell.
\xXX: A single byte in the form of a two digit hexadecimal number. For example the line-feed character can also be expressed as \x0A.

The asterisk ("*") character in double quotes is interpreted as a wildcard character. In such case, the "file*.log" literal can be used to specify both file1.log and filea.log. For more details, see the Configuration section of the im_file module documentation.

Extra care should be taken when specifying file paths within double quotes since the backslash is used as the directory separator on Windows. For more information about the possible complications, see this note of the im_file module.

The examples below explain how to use single and double quotes with strings.

Example 3. Setting a string without escaping

The statement below assigns the Test message string to the $Message field.

$Message = 'Test message';

Using single quotes means that character escaping is not allowed.

Output

Test message

Example 4. Escaping double quotes within a string

Using the \" character combination, the statement below wraps the message substring with double quotes.

$QuotedMessage = "Test \"message\"";

Below is the value which was assigned to the $QuotedMessage field.

Field value

Test "message"

Wildcard characters are useful for reading data from multiple files based on their naming pattern.

Example 5. Escaping wildcard characters

The NXLog configuration below uses the * asterisk character to read data from files with names starting with the log string and having the .txt extension.

nxlog.conf

<Input from_file>
    Module    im_file
    File      "/tmp/log*.txt"
</Input>

Based on the naming pattern, both log1.txt and log_new.txt will be read.

Datetime

A datetime literal is an unquoted representation of a time value expressing local time in the format of YYYY-MM-DD hh:mm:ss.

Example 6. Setting a Datetime Value

This statement sets the $EventTime field to the specified datetime value.

$EventTime = 2000-01-02 03:04:05;

IP Address: An IP address literal can be expressed in the form of a dotted quad notation for IPv4 (192.168.1.1) or by using 8 colon-separated (:) groups of 16-bit hexadecimal values for IPv6 (2001:0db8:85a3:0000:0000:8a2e:0370:7334). When specifying an address and port separated by colon (host:port), IPv6 addresses must be enclosed in square brackets ([host]:port).
Array: An array literal is a comma-separated list of expressions, enclosed in square brackets, where the list is optional. E.g. [1, 2, hostname(), now()] or [ ] for empty array.
Hash: A hash literal is a comma-separated list of key-value pairs, enclosed in parentheses. A key-value pair consists a key-expression and a value-expression separated by =>. E.g. ( 'key1' => 123, 'key2' => now() ) or ( ) for empty hash. The key-expression can evaluate to any type but the actual key will always be to_string(key-expression).

Regular Expressions

The PCRE engine is used to execute regular expressions in NXLog. For more information about the PCRE syntax, see the pcre2syntax(3) and pcre2pattern(3) man pages.

Regular expressions must be used with one of the =~ and !~ operators, and must be quoted with slashes (/) as in Perl. Captured sub-strings are accessible through numeric reference, and the full subject string is placed into $0.

Example 7. A Regular Expression Match Operation

If the regular expression matches the $Message field, the log_info() procedure is executed. The captured sub-string is used in the logged string ($1).

if $Message =~ /^Test (\S+)/ log_info("captured: " + $1);

It is also possible to use named capturing such that the resulting field name is defined in the regular expression.

Example 8. Regular Expression Match Using Named Capturing

This statement causes the same behavior as the previous example, but it uses named capturing instead.

if $Message =~ /^Test: (?<test>\S+)/ log_info("captured: " + $test);

Substitution is supported with the s/// operator. Fields or captured sub-string references cannot be used inside the regular expression but these can be used inside the substitution operator using the usual $fieldname, ${relaxed field name}, $0..n and $captured_name notations, e.g., if $subject =~ s/(\w+)/($1→${replace})/g log_info($subject);.

Only the \\, \/, and \$ escape sequences are supported for the replacement string of the substitution operator. Other sequences, like \n, will be replaced literally.

Regular Expression Modifiers

The following regular expression modifiers are supported:

g

The /g modifier can be used for global replacement.

Example 9. Replace Whitespace Occurrences

If any whitespace is found in the $SourceName field, it is replaced with underscores (_) and a log message is generated.

if $SourceName =~ s/\s/_/g log_info("removed all whitespace in SourceName");

s

The dot (.) normally matches any character except newline. The /s modifier causes the dot to match all characters including line terminator characters (LF and CRLF).

Example 10. Dot Matches All Characters

The regular expression in this statement will match a message that begins and ends with the given keywords, even if the message spans multiple lines.

if $Message =~ /^Backtrace.*END$/s drop();

m

The /m modifier can be used to treat the string as multiple lines (^ and $ match newlines within data).

i

The /i modifier does case insensitive matching.

Fields

See Fields for a list of fields provided by the NXLog core. Additional fields are available through input modules.

Fields are referenced in the NXLog language by prepending a dollar sign ($) to the field name.

Field names are case-insensitive. For example, the $SourceModuleName and $sourcemodulename field names are equivalent.

Normally, a field name may contain letters, digits, the period (.), and the underscore (_). Additionally, field names must begin with a letter or an underscore. The corresponding regular expression is:

[a-zA-Z_][a-zA-Z0-9._]*

However, those restrictions are relaxed if the field name is specified with curly braces ({}). In this case, the field name may also contain hyphens (-), parentheses (()), and spaces. The field name may also begin with any one of the allowed characters. The regular expression in this case is:

[a-zA-Z0-9._() -]+

Example 11. Referencing a Field

This statement generates an internal log message indicating the time when the message was received by NXLog.

log_debug('Message received at ' + $EventReceivedTime);

This statement uses curly braces ({}) to refer to a field with a hyphenated name.

log_info('The file size is ' + ${file-size});

A field which does not exist has an unknown type.

Arrays

An array of values can be created and initialized with the following syntax:

[ expression, expression, ...., expression ]

Where an expression is a valid NXLog expression which evaluates to a value including the container type of values and the undef value.

The array then can be assigned to a field:

$field = [1, 'aaa', TRUE, 2022-12-22 10:25:01, 2001:db8:85a3::8a2e:370:7334, hostname() ];

Or to a module variable:

$$modvar = [TRUE, 2022-12-22 10:25:01, [], undef, [1, 2, 3] ];

The individual array elements can be accessed by the array operator [] and an index expression:

$field[idx_expression];
$$modvar[idx_expression];

Where an idx_expression is a valid NXLog expression which evaluates to an integer value. If idx_expression evaluates to a string type then NXLog automatically tries to convert it to an integer value. The first (the smallest) index value of an array is 0.

An array element can be at the left part of an assignment. This way one can assign new values to the individual elements of the array.

$array[idx_expression] = expression;
# OR
$$array[idx_expression] = expression;

One can insert new elements into an array by referring to a not existing element. E.g.:

$$array = [1, 2, 3];
$$array[5] = "test str";

which yields this array:

[1, 2, 3, undef, undef, 'test str']

There are empty arrays which can be created too.

# empty array:
$array = [ ];

Hashes

Hashes (aka associative arrays) can be created and initialized with the following syntax:

( key_expression => value_expression, key_expression => value_expression, .... , key_expression => value_expression )

Where a key_expression is a valid NXLog expression which evaluates to a value which is then converted to a string. The value_expression is a valid NXLog expression which evaluates to any type of value, including the container type of values and the undef value.

The hash then can be assigned to a field:

$field = ('key1' => 1 + 1, 'key2' => 'aa' + 'bb', 'key3' => 2022-12-22 10:25:01, 'key4' => hostname() );

or to a module variable:

$$modvar = ('key1' => 1 + 1, 'key2' => 'aa' + 'bb', 'key3' => 2022-12-22 10:25:01, 'key4' => hostname() );

The individual hash values can be accessed by the hash operator () and a key expression:

$field(key_expression);
$$modvar(key_expression);

A hash value can be at the left part of an assignment. This way one can assign new values to the individual values of the hash container.

$hash(key_expression] = expression;
# OR
$$hash[key_expression] = expression;

One can insert new values into a hash container with the following syntax:

$$hash = (); # empty container
$$hash=('key1') = "value1";
$$hash=('key2') = "value2";

which yields this hash container:

['key1' => 'value1', 'key2' => 'value2']

There are empty hash containers which can be created too.

# empty hash:
$hash = ( );

Operations

Unary Operations

The following unary operations are available. It is possible to use brackets around the operand to make it look like a function call as in the "defined" example below.

not

The not operator expects a boolean value. It will evaluate to undef if the value is undefined. If it receives an unknown value which evaluates to a non-boolean, it will result in a run-time execution error.

Example 12. Using the "not" Operand

If the $Success field has a value of false, an error is logged.

if not $Success log_error("Job failed");

defined

The defined operator will evaluate to TRUE if the operand is defined, otherwise FALSE.

Example 13. Using the Unary "defined" Operation

This statement is a no-op, it does nothing.

if defined undef log_info("never printed");

If the $EventTime field has not been set (due perhaps to failed parsing), it will be set to the current time.

if not defined($EventTime) $EventTime = now();

Binary Operations

The following binary operations are available.

The operations are described with the following syntax:

LEFT_OPERAND_TYPE OPERATION RIGHT_OPERAND_TYPE = EVALUATED_VALUE_TYPE

=~

This is the regular expression match operation as in Perl. This operation takes a string and a regular expression operand and evaluates to a boolean value which will be TRUE if the regular expression matches the subject string. Captured sub-strings are accessible through numeric reference (such as $1) and the full subject string is placed into $0. Regular expression based string substitution is supported with the s/// operator. For more details, see Regular Expressions.

string =~ regexp = boolean
regexp =~ string = boolean

Example 14. Regular Expression Based String Matching

A log message will be generated if the $Message field matches the regular expression.

if $Message =~ /^Test message/ log_info("matched");

!~

This is the opposite of =~

The expression will evaluate to TRUE if the regular expression does not match on the subject string. It can be also written as not LEFT_OPERAND =~ RIGHT_OPERAND. The s/// substitution operator is supported.

string !~ regexp = boolean
regexp !~ string = boolean

Example 15. Regular Expression Based Negative String Matching

A log message will be generated if the $Message field does not match the regular expression.

if $Message !~ /^Test message/ log_info("didn't match");

==

This operator compares two values for equality. Comparing a defined value with an undefined results in undef.

undef == undef = TRUE
string == string = boolean
integer == integer = boolean
boolean == boolean = boolean
datetime == datetime = boolean
ipaddr == ipaddr = boolean
ipaddr == string = boolean
string == ipaddr = boolean

Example 16. Equality

A log message will be generated if $SeverityValue is 1.

if $SeverityValue == 1 log_info("severity is one");

!=

This operator compares two values for inequality. Comparing a defined value with an undefined results in undef.

undef != undef = FALSE
string != string = boolean
integer != integer = boolean
boolean != boolean = boolean
datetime != datetime = boolean
ipaddr != ipaddr = boolean
ipaddr != string = boolean
string != ipaddr = boolean

Example 17. Inequality

A log message will be generated if $SeverityValue is not 1.

if $SeverityValue != 1 log_info("severity is not one");

<

This operation will evaluate to TRUE if the left operand is less than the right operand, and FALSE otherwise. Comparing a defined value with an undefined results in undef.

integer < integer = boolean
datetime < datetime = boolean

Example 18. Less

A log message will be generated if $SeverityValue is less than 1.

if $SeverityValue < 1 log_info("severity is less than one");

<=

This operation will evaluate to TRUE if the left operand is less than or equal to the right operand, and FALSE otherwise. Comparing a defined value with an undefined results in undef.

integer <= integer = boolean
datetime <= datetime = boolean

Example 19. Less or Equal

A log message will be generated if $SeverityValue is less than or equal to 1.

if $SeverityValue < 1 log_info("severity is less than or equal to one");

>

This operation will evaluate to TRUE if the left operand is greater than the right operand, and FALSE otherwise. Comparing a defined value with an undefined results in undef.

integer > integer = boolean
datetime > datetime = boolean

Example 20. Greater

A log message will be generated if $SeverityValue is greater than 1.

if $SeverityValue > 1 log_info("severity is greater than one");

>=

This operation will evaluate to TRUE if the left operand is greater than or equal to the right operand, and FALSE otherwise. Comparing a defined value with an undefined results in undef.

integer >= integer = boolean
datetime >= datetime = boolean

Example 21. Greater or Equal

A log message will be generated if $SeverityValue is greater than or equal to 1.

if $SeverityValue >= 1 log_info("severity is greater than or equal to one");

and

This operation evaluates to TRUE if and only if both operands are TRUE. The operation will evaluate to undef if either operand is undefined.

boolean and boolean = boolean

Example 22. And Operation

A log message will be generated only if both $SeverityValue equals 1 and $FacilityValue equals 2.

if $SeverityValue == 1 and $FacilityValue == 2 log_info("1 and 2");

or

This operation evaluates to TRUE if either operand is TRUE. The operation will evaluate to undef if both operands are undefined.

boolean or boolean = boolean

Example 23. Or Operation

A log message will be generated if $SeverityValue is equal to either 1 or 2.

if $SeverityValue == 1 or $SeverityValue == 2 log_info("1 or 2");

+

This operation will result in an integer if both operands are integers. If either operand is a string, the result will be a string where non-string typed values are converted to strings. In this case it acts as a concatenation operator, like the dot (.) operator in Perl. Adding an undefined value to a non-string will result in undef.

integer + integer = integer
string + undef = string
undef + string = string
undef + undef = undef
string + string = string (Concatenate two strings.)
datetime + integer = datetime (Add the number of seconds in the right value to the datetime stored in the left value.)
integer + datetime = datetime (Add the number of seconds in the left value to the datetime stored in the right value.)

Example 24. Concatenation