Expressions

Expressions in the NXLog language are fundamental building blocks used to define and manipulate data. They encompass a range of types, including literals, booleans, integers, strings, and more complex structures like arrays and hashes. Understanding these expressions is key to configuring NXLog Agent with customized log processing.

Literals

Undef

The undef literal has an unknown type. It can be also used in an assignment to unset the value of a field.

Example 1. Unsetting the value of a field

This statement unsets the $ProcessID field.

$ProcessID = undef;

Boolean: A boolean literal is either TRUE or FALSE. It is case-insensitive, so True, False, true, and false are also valid.

Integer

An integer starts with a minus (-) sign if it is negative. A "0X" or "0x" prepended modifier indicates a hexadecimal notation. The "K", "M" and "G" modifiers are also supported; these mean Kilo (1024), Mega (1024^2), or Giga (1024^3) respectively when appended.

Example 2. Setting an integer value

This statement uses a modifier to set the $Limit field to 44040192 (42×1024^2).

$Limit = 42M;

String

String literals are quoted characters using either single or double quotes.

Enclosing in single quotes means that the string contains no escape sequences. For example, the '\n' character combination will be interpreted as \n. Similarly, the asterisk '*' character inside single quotes will be processed by NXLog Agent as the asterisk.

String literals specified within double quotes can contain escape sequences. For instance, the "\n" character combination will be interpreted as a newline (LF) character as opposed to '\n' in single quotes, which is treated by NXLog Agent as \n.

Below is the full list of escape sequences supported by NXLog Agent.

\\: The backslash (\) character.
\": The double quote (") character.
\n: Line feed (LF).
\r: Carriage return (CR).
\t: Horizontal tab.
\b: Audible bell.
\xXX: A single byte in the form of a two-digit hexadecimal number. For example, the line-feed character can also be expressed as \x0A.

The asterisk ("*") character in double quotes is interpreted as a wildcard character. In such case, the "file*.log" literal can be used to specify both file1.log and filea.log. For more details, see the Configuration section of the im_file module documentation.

Extra care should be taken when specifying file paths within double quotes since the backslash is used as the directory separator on Windows. For more information about the possible complications, see this note of the im_file module.

The examples below explain how to use single and double quotes with strings.

Example 3. Setting a string without escaping

The statement below assigns the Test message string to the $Message field.

$Message = 'Test message';

Using single quotes means that character-escaping is not allowed.

Output

Test message

Example 4. Escaping double quotes within a string

Using the \" character combination, the statement below wraps the message substring with double quotes.

$QuotedMessage = "Test \"message\"";

Below is the value that was assigned to the $QuotedMessage field.

Field value

Test "message"

Wildcard characters are useful for reading data from multiple files based on their naming pattern.

Example 5. Escaping wildcard characters

The NXLog Agent configuration below uses the * asterisk character to read data from files with names starting with the log string and having the .txt extension.

nxlog.conf

<Input from_file>
    Module    im_file
    File      "/tmp/log*.txt"
</Input>

Based on the naming pattern, both log1.txt and log_new.txt will be read.

Datetime

A datetime literal is an unquoted representation of a time value expressing local time in the format of YYYY-MM-DD hh:mm:ss.

Example 6. Setting a datetime value

This statement sets the $EventTime field to the specified datetime value.

$EventTime = 2024-01-02 03:04:05;

IP Address: An IP address literal can be expressed in the form of a dotted quad notation for IPv4 (192.168.1.1) or by using 8 colon-separated (:) groups of 16-bit hexadecimal values for IPv6 (2001:0db8:85a3:0000:0000:8a2e:0370:7334). When specifying an address and port separated by a colon (host:port), IPv6 addresses must be enclosed in square brackets ([host]:port).

Array: An array literal is a comma-separated list of expressions, enclosed in square brackets, where the list is optional. E.g. [1, 2, hostname(), now()] or [ ] for empty array.

Hash: A hash literal is a comma-separated list of key-value pairs, enclosed in parentheses. A key-value pair consists of a key-expression and a value-expression separated by =>. E.g. ( 'key1' => 123, 'key2' => now() ) or ( ) for empty hash. The key-expression can evaluate to any type but the actual key will always be to_string(key-expression).

Regular Expressions

The PCRE engine is used to execute regular expressions in NXLog Agent. For more information about the PCRE syntax, see the pcre2syntax(3) and pcre2pattern(3) man pages.

Regular expressions must be used with one of the =~ and !~ operators, and must be quoted with slashes (/) as in Perl. Captured sub-strings are accessible through numeric reference, and the full subject string is placed into $0.

Example 7. A regular expression match operation

If the regular expression matches the $Message field, the log_info() procedure is executed. The captured sub-string is used in the logged string ($1).

if $Message =~ /^Test (\S+)/ log_info("captured: " + $1);

It is also possible to use named capturing such that the resulting field name is defined in the regular expression.

Example 8. Regular expression match using named capturing

This statement causes the same behavior as the previous example, but it uses named capturing instead.

if $Message =~ /^Test: (?<test>\S+)/ log_info("captured: " + $test);

Substitution is supported with the s/// operator. Fields or captured sub-string references cannot be used inside the regular expression but these can be used inside the substitution operator using the usual $fieldname, ${relaxed field name}, $0..n and $captured_name notations, e.g., if $subject =~ s/(\w+)/($1→${replace})/g log_info($subject);.

Only the \\, \/, and \$ escape sequences are supported for the replacement string of the substitution operator. Other sequences, like \n, will be replaced literally.

Regular Expression Modifiers

The following regular expression modifiers are supported:

g

The /g modifier can be used for global replacement.

Example 9. Replace whitespaces

If any whitespace is found in the $SourceName field, it is replaced with underscores (_) and a log message is generated.

if $SourceName =~ s/\s/_/g log_info("removed all whitespace in SourceName");

s

The dot (.) normally matches any character except newline. The /s modifier causes the dot to match all characters including line terminator characters (LF and CRLF).

Example 10. Dot matches all characters

The regular expression in this statement will match a message that begins and ends with the given keywords, even if the message spans multiple lines.

if $Message =~ /^Backtrace.*END$/s drop();

m

The /m modifier can be used to treat the string as multiple lines (^ and $ match newlines within data).

i

The /i modifier does case insensitive matching.

Fields

See Core Fields for a list of fields provided by the NXLog Agent core. Additional fields are available through input modules.

Fields are referenced in the NXLog language by prefixing the field name with a dollar sign ($).

Field names are case-insensitive. For example, $SourceModuleName and $sourcemodulename are equivalent.

Normally, a field name may contain letters, digits, the period (.), and the underscore (_). Additionally, field names must begin with a letter or an underscore. The corresponding regular expression is:

[a-zA-Z_][a-zA-Z0-9._]*

However, those restrictions are relaxed if the field name is specified with curly braces ({}). In this case, the field name may also contain hyphens (-), parentheses (()), and spaces. The field name may also begin with any one of the allowed characters. The regular expression in this case is:

[a-zA-Z0-9._() -]+

Example 11. Referencing a Field

This statement generates an internal log message indicating the time when the message was received by NXLog Agent.

log_debug('Message received at ' + $EventReceivedTime);

This statement uses curly braces ({}) to refer to a field with a hyphenated name.

log_info('The file size is ' + ${file-size});

A field that does not exist has an unknown type.

Arrays

You can create and initialize an array of values with the following syntax:

[ expression, expression, ...., expression ]

In the above example, an expression is a valid NXLog language expression that evaluates to a value, including the container type of values and the undef value.

You can assign an array to a field, as displayed in the example below:

$field = [1, 'aaa', TRUE, 2022-12-22 10:25:01, 2001:db8:85a3::8a2e:370:7334, hostname() ];

You can also assign an array to a module variable:

$$modvar = [TRUE, 2022-12-22 10:25:01, [], undef, [1, 2, 3] ];

You can access the individual array elements by the array operator [] and an index expression:

$field[idx_expression];
$$modvar[idx_expression];

In the above example, an idx_expression is a valid NXLog language expression that evaluates to an integer value. If the idx_expression evaluates to a string type, then NXLog language automatically tries to convert it to an integer value. The first (smallest) index value of an array is 0.

You can place an array element at the left part of an assignment. This way, you can assign new values to the individual elements of the array:

$array[idx_expression] = expression;
# OR
$$array[idx_expression] = expression;

You can insert new elements into an array by referring to a nonexistent element:

$$array = [1, 2, 3];
$$array[5] = "test str";

The above example results in the following array:

[1, 2, 3, undef, undef, 'test str']

You can also create empty arrays:

# empty array:
$array = [ ];

Hashes

You can create and initialize hashes (associative arrays) with the following syntax:

( key_expression => value_expression, key_expression => value_expression, .... , key_expression => value_expression )

In the above example, a key_expression is a valid NXLog language expression that evaluates to a value, which is then converted to a string. The value_expression is a valid NXLog language expression that evaluates to any type of value, including the container type of values and the undef value.

You can assign a hash to a field using the following syntax:

$field = ('key1' => 1 + 1, 'key2' => 'aa' + 'bb', 'key3' => 2022-12-22 10:25:01, 'key4' => hostname() );

You can also assign a hash to a module variable:

$$modvar = ('key1' => 1 + 1, 'key2' => 'aa' + 'bb', 'key3' => 2022-12-22 10:25:01, 'key4' => hostname() );

You can access the individual hash values by the hash operator () and a key expression:

$field(key_expression);
$$modvar(key_expression);

You can place the hash value at the left part of an assignment. This way, you can assign new values to the individual values of the hash container.

$hash(key_expression) = expression;
# OR
$$hash(key_expression) = expression;

You can insert new values into a hash container with the following syntax:

$$hash = (); # empty container
$$hash=('key1') = "value1";
$$hash=('key2') = "value2";

The above example results in the following container:

['key1' => 'value1', 'key2' => 'value2']

You can also create empty hash containers:

# empty hash:
$hash = ( );