Expressions
Expressions in the NXLog language are fundamental building blocks used to define and manipulate data. They encompass a range of types, including literals, booleans, integers, strings, and more complex structures like arrays and hashes. Understanding these expressions is key to configuring NXLog Agent with customized log processing.
Literals
- Undef
-
The undef literal has an unknown type. It can be also used in an assignment to unset the value of a field.
Example 1. Un-Setting the Value of a FieldThis statement unsets the
$ProcessID
field.$ProcessID = undef;
- Boolean
-
A boolean literal is either TRUE or FALSE. It is case-insensitive, so
True
,False
,true
, andfalse
are also valid.
- Integer
-
An integer starts with a minus (
-
) sign if it is negative. A "0X" or "0x" prepended modifier indicates a hexadecimal notation. The "K", "M" and "G" modifiers are also supported; these mean Kilo (1024), Mega (1024^2), or Giga (1024^3) respectively when appended.Example 2. Setting an Integer ValueThis statement uses a modifier to set the
$Limit
field to 44040192 (42×1024^2).$Limit = 42M;
- String
-
String literals are quoted characters using either single or double quotes.
Enclosing in single quotes means that the string contains no escape sequences. For example, the
'\n'
character combination will be interpreted as\n
. Similarly, the asterisk'*'
character inside single quotes will be processed by NXLog Agent as the asterisk.String literals specified within double quotes can contain escape sequences. For instance, the
"\n"
character combination will be interpreted as a newline (LF) character as opposed to'\n'
in single quotes, which is treated by NXLog Agent as\n
.Below is the full list of escape sequences supported by NXLog Agent.
- \\
-
The backslash (
\
) character. - \"
-
The double quote (
"
) character. - \n
-
Line feed (LF).
- \r
-
Carriage return (CR).
- \t
-
Horizontal tab.
- \b
-
Audible bell.
- \xXX
-
A single byte in the form of a two-digit hexadecimal number. For example, the line-feed character can also be expressed as
\x0A
.
The asterisk (
"*"
) character in double quotes is interpreted as a wildcard character. In such case, the"file*.log"
literal can be used to specify bothfile1.log
andfilea.log
. For more details, see the Configuration section of the im_file module documentation.Extra care should be taken when specifying file paths within double quotes since the backslash is used as the directory separator on Windows. For more information about the possible complications, see this note of the im_file module. The examples below explain how to use single and double quotes with strings.
Example 3. Setting a string without escapingThe statement below assigns the
Test message
string to the$Message
field.$Message = 'Test message';
Using single quotes means that character-escaping is not allowed.
OutputTest message
Example 4. Escaping double quotes within a stringUsing the
\"
character combination, the statement below wraps themessage
substring with double quotes.$QuotedMessage = "Test \"message\"";
Below is the value that was assigned to the
$QuotedMessage
field.Field valueTest "message"
Wildcard characters are useful for reading data from multiple files based on their naming pattern.
Example 5. Escaping wildcard charactersThe NXLog Agent configuration below uses the
*
asterisk character to read data from files with names starting with thelog
string and having the.txt
extension.nxlog.conf<Input from_file> Module im_file File "/tmp/log*.txt" </Input>
Based on the naming pattern, both
log1.txt
andlog_new.txt
will be read.
- Datetime
-
A datetime literal is an unquoted representation of a time value expressing local time in the format of
YYYY-MM-DD hh:mm:ss
.Example 6. Setting a Datetime ValueThis statement sets the
$EventTime
field to the specified datetime value.$EventTime = 2024-01-02 03:04:05;
- IP Address
-
An IP address literal can be expressed in the form of a dotted quad notation for IPv4 (
192.168.1.1
) or by using 8 colon-separated (:
) groups of 16-bit hexadecimal values for IPv6 (2001:0db8:85a3:0000:0000:8a2e:0370:7334
). When specifying an address and port separated by colon (host:port
), IPv6 addresses must be enclosed in square brackets ([host]:port
). - Array
-
An array literal is a comma-separated list of expressions, enclosed in square brackets, where the list is optional. E.g.
[1, 2, hostname(), now()]
or[ ]
for empty array. - Hash
-
A hash literal is a comma-separated list of key-value pairs, enclosed in parentheses. A key-value pair consists a
key-expression
and avalue-expression
separated by=>
. E.g.( 'key1' => 123, 'key2' => now() )
or( )
for empty hash. Thekey-expression
can evaluate to any type but the actual key will always beto_string(key-expression)
.
Regular Expressions
The PCRE engine is used to execute regular expressions in NXLog Agent. For more information about the PCRE syntax, see the pcre2syntax(3) and pcre2pattern(3) man pages.
Regular expressions must be used with one of the
=~ and !~
operators, and must be quoted with slashes (/
) as in Perl. Captured
sub-strings are accessible through numeric reference, and the full
subject string is placed into $0
.
If the regular expression matches the $Message
field, the
log_info() procedure is executed. The captured
sub-string is used in the logged string ($1
).
if $Message =~ /^Test (\S+)/ log_info("captured: " + $1);
It is also possible to use named capturing such that the resulting field name is defined in the regular expression.
This statement causes the same behavior as the previous example, but it uses named capturing instead.
if $Message =~ /^Test: (?<test>\S+)/ log_info("captured: " + $test);
Substitution is supported with the s///
operator.
Fields or captured sub-string references cannot be used inside the regular expression but these can be used inside the substitution operator using the usual $fieldname
, ${relaxed field name}
, $0..n
and $captured_name
notations, e.g., if $subject =~ s/(\w+)/($1→${replace})/g log_info($subject);
.
Only the \\
, \/
, and \$
escape sequences are supported for the replacement string of the substitution operator.
Other sequences, like \n
, will be replaced literally.
Regular Expression Modifiers
The following regular expression modifiers are supported:
- g
-
The
/g
modifier can be used for global replacement.Example 9. Replace Whitespace OccurrencesIf any whitespace is found in the
$SourceName
field, it is replaced with underscores (_
) and a log message is generated.if $SourceName =~ s/\s/_/g log_info("removed all whitespace in SourceName");
- s
-
The dot (
.
) normally matches any character except newline. The/s
modifier causes the dot to match all characters including line terminator characters (LF and CRLF).Example 10. Dot Matches All CharactersThe regular expression in this statement will match a message that begins and ends with the given keywords, even if the message spans multiple lines.
if $Message =~ /^Backtrace.*END$/s drop();
- m
-
The
/m
modifier can be used to treat the string as multiple lines (^
and$
match newlines within data). - i
-
The
/i
modifier does case insensitive matching.
Fields
See Core fields for a list of fields provided by the NXLog Agent core. Additional fields are available through input modules.
Fields are referenced in the NXLog language by prepending a
dollar sign ($
) to the field name.
Field names are case-insensitive. For example, the $SourceModuleName
and $sourcemodulename
field names are equivalent.
Normally, a field name may contain letters, digits, the period (.
), and the
underscore (_
). Additionally, field names must begin with a letter or
an underscore. The corresponding regular expression is:
[a-zA-Z_][a-zA-Z0-9._]*
However, those restrictions are relaxed if the field name is specified with
curly braces ({}
). In this case, the field name may also contain hyphens
(-
), parentheses (()
), and spaces. The field name may also begin with any
one of the allowed characters. The regular expression in this case is:
[a-zA-Z0-9._() -]+
This statement generates an internal log message indicating the time when the message was received by NXLog Agent.
log_debug('Message received at ' + $EventReceivedTime);
This statement uses curly braces ({}
) to refer to a field with a hyphenated
name.
log_info('The file size is ' + ${file-size});
A field that does not exist has an unknown type.
Arrays
You can create and initialize an array of values with the following syntax:
[ expression, expression, ...., expression ]
Where an expression
is a valid NXLog language expression which evaluates to a value including the container
type of values and the undef value.
You can assign an array to a field, as displayed in the example below:
$field = [1, 'aaa', TRUE, 2022-12-22 10:25:01, 2001:db8:85a3::8a2e:370:7334, hostname() ];
You can also assign an array to a module variable:
$$modvar = [TRUE, 2022-12-22 10:25:01, [], undef, [1, 2, 3] ];
You can access the individual array elements by the array operator []
and an index expression:
$field[idx_expression]; $$modvar[idx_expression];
In the above example, an idx_expression
is a valid NXLog language expression that evaluates to an integer value.
If the idx_expression
evaluates to a string type, then NXLog Agent automatically tries to convert it to an integer value.
The first (smallest) index value of an array is 0
.
You can place an array element on the left part of an assignment. This way, you can assign new values to individual elements of the array:
$array[idx_expression] = expression; # OR $$array[idx_expression] = expression;
You can add elements to an array by referring to a nonexistent element:
$$array = [1, 2, 3]; $$array[5] = "test str";
The above example results in the following array:
[1, 2, 3, undef, undef, 'test str']
You can also create empty arrays:
# empty array: $array = [ ];
Hashes
You can create and initialize hashes (associative arrays) with the following syntax:
( key_expression => value_expression, key_expression => value_expression, .... , key_expression => value_expression )
In the above example, a key_expression
is a valid expression that evaluates to a value, which is then converted to a string.
The value_expression
is an expression that evaluates to any type of value, including the container type of values and the undef value.
You can assign a hash to a field using the following syntax:
$field = ('key1' => 1 + 1, 'key2' => 'aa' + 'bb', 'key3' => 2022-12-22 10:25:01, 'key4' => hostname() );
You can also assign a hash to a module variable:
$$modvar = ('key1' => 1 + 1, 'key2' => 'aa' + 'bb', 'key3' => 2022-12-22 10:25:01, 'key4' => hostname() );
You can access the individual hash values with the hash operator ()
and a key expression:
$field(key_expression); $$modvar(key_expression);
You can place the hash value at the left part of an assignment. This way, you can assign new values to the individual values of the hash container.
$hash(key_expression) = expression; # OR $$hash(key_expression) = expression;
You can insert new values into a hash container with the following syntax:
$$hash = (); # empty container $$hash=('key1') = "value1"; $$hash=('key2') = "value2";
The above example results in the following container:
['key1' => 'value1', 'key2' => 'value2']
You can also create empty hash containers:
# empty hash: $hash = ( );