NXLog search
Let's talk Start free
Let's talk Start free

NXLog Legacy Documentation

6.01

Language

Types

The following types are provided by the NXLog language.

Unknown

This is a special type for values where the type cannot be determined at compile time and for uninitialized values. The undef literal and fields without a value also have an unknown type. The unknown type can also be thought of as "any" in case of function and procedure API declarations.

Boolean

A boolean value is TRUE, FALSE or undefined. Note that an undefined value is not the same as a FALSE value.

Integer

An integer can hold a signed 64 bit value in addition to the undefined value. Floating point values are not supported.

String

A string is an array of characters in any character set. The binary type should be used for values where the NULL byte can also occur. An undefined string is not the same as an empty string. Strings have a limited length to prevent resource exhaustion problems, this is a compile-time value currently set to 1M.

Datetime

A datetime holds a microsecond, millisecond, and second value of time elapsed since the Epoch, and NXLog can handle the corresponding value. It is always stored in UTC/GMT.

IP Address

The ipaddr type can store IP addresses in an internal format. This type is used to store both dotted-quad IPv4 addresses (for example, 192.168.1.1) and IPv6 addresses (for example, 2001:0db8:85a3:0000:0000:8a2e:0370:7334).

Regular expression

A regular expression type can only be used with the =~ or !~ operators.

Binary

This type can hold an array of bytes.

Array

This type is a container and it can hold an arbitrary number values of any kind of types supported by the NXLog language. The inividual members are accessed by a non-negative integer type of index value.

Hash

This type is a container and it can hold an arbitrary number of key-value pairs. The key is a string while the value is one of the value types supported by the NXLog language. The inividual values can be accessed by the key strings.

Variadic arguments

This is a special type only used in function and procedure API declarations to indicate variadic arguments.

JSON string

A specific format of json-string.

JSON string is not yet a separate data type and is just a simple string. It was described separately to specify the string format.
JSON map

A specific format for json-string. Presents as an object description for specific json-array.

Expressions

Literals

Undef

The undef literal has an unknown type. It can be also used in an assignment to unset the value of a field.

Example 1. Un-Setting the Value of a Field

This statement unsets the $ProcessID field.

$ProcessID = undef;
Boolean

A boolean literal is either TRUE or FALSE. It is case-insensitive, so True, False, true, and false are also valid.

Integer

An integer starts with a minus (-) sign if it is negative. A "0X" or "0x" prepended modifier indicates a hexadecimal notation. The "K", "M" and "G" modifiers are also supported; these mean Kilo (1024), Mega (1024^2), or Giga (1024^3) respectively when appended.

Example 2. Setting an Integer Value

This statement uses a modifier to set the $Limit field to 44040192 (42×1024^2).

$Limit = 42M;
String

String literals are quoted characters using either single or double quotes.

Enclosing in single quotes means that the string contains no escape sequences. For example, the '\n' character combination will be interpreted as \n. Similarly, the asterisk '*' character inside single quotes will be processed by NXLog as the asterisk.

String literals specified within double quotes can contain escape sequences. For instance, the "\n" character combination will be interpreted as a newline (LF) character as opposed to '\n' in single quotes, which is treated by NXLog as \n.

Below is the full list of escape sequences supported by NXLog.

\\

The backslash (\) character.

\"

The double quote (") character.

\n

Line feed (LF).

\r

Carriage return (CR).

\t

Horizontal tab.

\b

Audible bell.

\xXX

A single byte in the form of a two digit hexadecimal number. For example the line-feed character can also be expressed as \x0A.

The asterisk ("*") character in double quotes is interpreted as a wildcard character. In such case, the "file*.log" literal can be used to specify both file1.log and filea.log. For more details, see the Configuration section of the im_file module documentation.

Extra care should be taken when specifying file paths within double quotes since the backslash is used as the directory separator on Windows. For more information about the possible complications, see this note of the im_file module.

The examples below explain how to use single and double quotes with strings.

Example 3. Setting a string without escaping

The statement below assigns the Test message string to the $Message field.

$Message = 'Test message';

Using single quotes means that character escaping is not allowed.

Output
Test message
Example 4. Escaping double quotes within a string

Using the \" character combination, the statement below wraps the message substring with double quotes.

$QuotedMessage = "Test \"message\"";

Below is the value which was assigned to the $QuotedMessage field.

Field value
Test "message"

Wildcard characters are useful for reading data from multiple files based on their naming pattern.

Example 5. Escaping wildcard characters

The NXLog configuration below uses the * asterisk character to read data from files with names starting with the log string and having the .txt extension.

nxlog.conf
<Input from_file>
    Module    im_file
    File      "/tmp/log*.txt"
</Input>

Based on the naming pattern, both log1.txt and log_new.txt will be read.

Datetime

A datetime literal is an unquoted representation of a time value expressing local time in the format of YYYY-MM-DD hh:mm:ss.

Example 6. Setting a Datetime Value

This statement sets the $EventTime field to the specified datetime value.

$EventTime = 2000-01-02 03:04:05;
IP Address

An IP address literal can be expressed in the form of a dotted quad notation for IPv4 (192.168.1.1) or by using 8 colon-separated (:) groups of 16-bit hexadecimal values for IPv6 (2001:0db8:85a3:0000:0000:8a2e:0370:7334). When specifying an address and port separated by colon (host:port), IPv6 addresses must be enclosed in square brackets ([host]:port).

Array

An array literal is a comma-separated list of expressions, enclosed in square brackets, where the list is optional. E.g. [1, 2, hostname(), now()] or [ ] for empty array.

Hash

A hash literal is a comma-separated list of key-value pairs, enclosed in parentheses. A key-value pair consists a key-expression and a value-expression separated by =>. E.g. ( 'key1' => 123, 'key2' => now() ) or ( ) for empty hash. The key-expression can evaluate to any type but the actual key will always be to_string(key-expression).

Regular Expressions

The PCRE engine is used to execute regular expressions in NXLog. For more information about the PCRE syntax, see the pcre2syntax(3) and pcre2pattern(3) man pages.

Regular expressions must be used with one of the =~ and !~ operators, and must be quoted with slashes (/) as in Perl. Captured sub-strings are accessible through numeric reference, and the full subject string is placed into $0.

Example 7. A Regular Expression Match Operation

If the regular expression matches the $Message field, the log_info() procedure is executed. The captured sub-string is used in the logged string ($1).

if $Message =~ /^Test (\S+)/ log_info("captured: " + $1);

It is also possible to use named capturing such that the resulting field name is defined in the regular expression.

Example 8. Regular Expression Match Using Named Capturing

This statement causes the same behavior as the previous example, but it uses named capturing instead.

if $Message =~ /^Test: (?<test>\S+)/ log_info("captured: " + $test);

Substitution is supported with the s/// operator. Fields or captured sub-string references cannot be used inside the regular expression but these can be used inside the substitution operator using the usual $fieldname, ${relaxed field name}, $0..n and $captured_name notations, e.g., if $subject =~ s/(\w+)/($1→${replace})/g log_info($subject);.

Only the \\, \/, and \$ escape sequences are supported for the replacement string of the substitution operator. Other sequences, like \n, will be replaced literally.

Regular Expression Modifiers

The following regular expression modifiers are supported:

g

The /g modifier can be used for global replacement.

Example 9. Replace Whitespace Occurrences

If any whitespace is found in the $SourceName field, it is replaced with underscores (_) and a log message is generated.

if $SourceName =~ s/\s/_/g log_info("removed all whitespace in SourceName");
s

The dot (.) normally matches any character except newline. The /s modifier causes the dot to match all characters including line terminator characters (LF and CRLF).

Example 10. Dot Matches All Characters

The regular expression in this statement will match a message that begins and ends with the given keywords, even if the message spans multiple lines.

if $Message =~ /^Backtrace.*END$/s drop();
m

The /m modifier can be used to treat the string as multiple lines (^ and $ match newlines within data).

i

The /i modifier does case insensitive matching.

Fields

See Fields for a list of fields provided by the NXLog core. Additional fields are available through input modules.

Fields are referenced in the NXLog language by prepending a dollar sign ($) to the field name.

Field names are case-insensitive. For example, the $SourceModuleName and $sourcemodulename field names are equivalent.

Normally, a field name may contain letters, digits, the period (.), and the underscore (_). Additionally, field names must begin with a letter or an underscore. The corresponding regular expression is:

[a-zA-Z_][a-zA-Z0-9._]*

However, those restrictions are relaxed if the field name is specified with curly braces ({}). In this case, the field name may also contain hyphens (-), parentheses (()), and spaces. The field name may also begin with any one of the allowed characters. The regular expression in this case is:

[a-zA-Z0-9._() -]+
Example 11. Referencing a Field

This statement generates an internal log message indicating the time when the message was received by NXLog.

log_debug('Message received at ' + $EventReceivedTime);

This statement uses curly braces ({}) to refer to a field with a hyphenated name.

log_info('The file size is ' + ${file-size});

A field which does not exist has an unknown type.

Arrays

An array of values can be created and initialized with the following syntax:

[ expression, expression, ...., expression ]

Where an expression is a valid NXLog expression which evaluates to a value including the container type of values and the undef value.

The array then can be assigned to a field:

$field = [1, 'aaa', TRUE, 2022-12-22 10:25:01, 2001:db8:85a3::8a2e:370:7334, hostname() ];

Or to a module variable:

$$modvar = [TRUE, 2022-12-22 10:25:01, [], undef, [1, 2, 3] ];

The individual array elements can be accessed by the array operator [] and an index expression:

$field[idx_expression];
$$modvar[idx_expression];

Where an idx_expression is a valid NXLog expression which evaluates to an integer value. If idx_expression evaluates to a string type then NXLog automatically tries to convert it to an integer value. The first (the smallest) index value of an array is 0.

An array element can be at the left part of an assignment. This way one can assign new values to the individual elements of the array.

$array[idx_expression] = expression;
# OR
$$array[idx_expression] = expression;

One can insert new elements into an array by referring to a not existing element. E.g.:

$$array = [1, 2, 3];
$$array[5] = "test str";

which yields this array:

[1, 2, 3, undef, undef, 'test str']

There are empty arrays which can be created too.

# empty array:
$array = [ ];

Hashes

Hashes (aka associative arrays) can be created and initialized with the following syntax:

( key_expression => value_expression, key_expression => value_expression, .... , key_expression => value_expression )

Where a key_expression is a valid NXLog expression which evaluates to a value which is then converted to a string. The value_expression is a valid NXLog expression which evaluates to any type of value, including the container type of values and the undef value.

The hash then can be assigned to a field:

$field = ('key1' => 1 + 1, 'key2' => 'aa' + 'bb', 'key3' => 2022-12-22 10:25:01, 'key4' => hostname() );

or to a module variable:

$$modvar = ('key1' => 1 + 1, 'key2' => 'aa' + 'bb', 'key3' => 2022-12-22 10:25:01, 'key4' => hostname() );

The individual hash values can be accessed by the hash operator () and a key expression:

$field(key_expression);
$$modvar(key_expression);

A hash value can be at the left part of an assignment. This way one can assign new values to the individual values of the hash container.

$hash(key_expression) = expression;
# OR
$$hash(key_expression) = expression;

One can insert new values into a hash container with the following syntax:

$$hash = (); # empty container
$$hash=('key1') = "value1";
$$hash=('key2') = "value2";

which yields this hash container:

['key1' => 'value1', 'key2' => 'value2']

There are empty hash containers which can be created too.

# empty hash:
$hash = ( );

Operations

Unary Operations

The following unary operations are available. It is possible to use brackets around the operand to make it look like a function call as in the "defined" example below.

not

The not operator expects a boolean value. It will evaluate to undef if the value is undefined. If it receives an unknown value which evaluates to a non-boolean, it will result in a run-time execution error.

Example 12. Using the "not" Operand

If the $Success field has a value of false, an error is logged.

if not $Success log_error("Job failed");
defined

The defined operator will evaluate to TRUE if the operand is defined, otherwise FALSE.

Example 13. Using the Unary "defined" Operation

This statement is a no-op, it does nothing.

if defined undef log_info("never printed");

If the $EventTime field has not been set (due perhaps to failed parsing), it will be set to the current time.

if not defined($EventTime) $EventTime = now();

Binary Operations

The following binary operations are available.

The operations are described with the following syntax:

LEFT_OPERAND_TYPE OPERATION RIGHT_OPERAND_TYPE = EVALUATED_VALUE_TYPE

=~

This is the regular expression match operation as in Perl. This operation takes a string and a regular expression operand and evaluates to a boolean value which will be TRUE if the regular expression matches the subject string. Captured sub-strings are accessible through numeric reference (such as $1) and the full subject string is placed into $0. Regular expression based string substitution is supported with the s/// operator. For more details, see Regular Expressions.

Example 14. Regular Expression Based String Matching

A log message will be generated if the $Message field matches the regular expression.

if $Message =~ /^Test message/ log_info("matched");
!~

This is the opposite of =~

The expression will evaluate to TRUE if the regular expression does not match on the subject string. It can be also written as not LEFT_OPERAND =~ RIGHT_OPERAND. The s/// substitution operator is supported.

Example 15. Regular Expression Based Negative String Matching

A log message will be generated if the $Message field does not match the regular expression.

if $Message !~ /^Test message/ log_info("didn't match");
==

This operator compares two values for equality. Comparing a defined value with an undefined results in undef.

Example 16. Equality

A log message will be generated if $SeverityValue is 1.

if $SeverityValue == 1 log_info("severity is one");
!=

This operator compares two values for inequality. Comparing a defined value with an undefined results in undef.

Example 17. Inequality

A log message will be generated if $SeverityValue is not 1.

if $SeverityValue != 1 log_info("severity is not one");
<

This operation will evaluate to TRUE if the left operand is less than the right operand, and FALSE otherwise. Comparing a defined value with an undefined results in undef.

Example 18. Less

A log message will be generated if $SeverityValue is less than 1.

if $SeverityValue < 1 log_info("severity is less than one");
<=

This operation will evaluate to TRUE if the left operand is less than or equal to the right operand, and FALSE otherwise. Comparing a defined value with an undefined results in undef.

Example 19. Less or Equal

A log message will be generated if $SeverityValue is less than or equal to 1.

if $SeverityValue < 1 log_info("severity is less than or equal to one");
>

This operation will evaluate to TRUE if the left operand is greater than the right operand, and FALSE otherwise. Comparing a defined value with an undefined results in undef.

Example 20. Greater

A log message will be generated if $SeverityValue is greater than 1.

if $SeverityValue > 1 log_info("severity is greater than one");
>=

This operation will evaluate to TRUE if the left operand is greater than or equal to the right operand, and FALSE otherwise. Comparing a defined value with an undefined results in undef.

Example 21. Greater or Equal

A log message will be generated if $SeverityValue is greater than or equal to 1.

if $SeverityValue >= 1 log_info("severity is greater than or equal to one");
and

This operation evaluates to TRUE if and only if both operands are TRUE. The operation will evaluate to undef if either operand is undefined.

boolean and boolean = boolean

Example 22. And Operation

A log message will be generated only if both $SeverityValue equals 1 and $FacilityValue equals 2.

if $SeverityValue == 1 and $FacilityValue == 2 log_info("1 and 2");
or

This operation evaluates to TRUE if either operand is TRUE. The operation will evaluate to undef if both operands are undefined.

boolean or boolean = boolean

Example 23. Or Operation

A log message will be generated if $SeverityValue is equal to either 1 or 2.

if $SeverityValue == 1 or $SeverityValue == 2 log_info("1 or 2");
+

This operation will result in an integer if both operands are integers. If either operand is a string, the result will be a string where non-string typed values are converted to strings. In this case it acts as a concatenation operator, like the dot (.) operator in Perl. Adding an undefined value to a non-string will result in undef.

Example 24. Concatenation

This statement will always cause a log message to be generated.

if 1 + "a" == "1a" log_info("this will be printed");
-

Subtraction. The result will be undef if either operand is undefined.

Example 25. Subtraction

This statement will always cause a log message to be generated.

if 4 - 1 == 3 log_info("four minus one is three");
*

Multiply an integer with another. The result will be undef if either operand is undefined.

integer * integer = integer

Example 26. Multiplication

This statement will always cause a log message to be generated.

if 4 * 2 == 8 log_info("four times two is eight");
/

Divide an integer with another. The result will be undef if either operand is undefined. Since the result is an integer, a fractional part is lost.

integer / integer = integer

Example 27. Division

This statement will always cause a log message to be generated.

if 9 / 4 == 2 log_info("9 divided by 4 is 2");
%

The modulo operation divides an integer with another and returns the remainder. The result will be undef if either operand is undefined.

integer % integer = integer

Example 28. Modulo

This statement will always cause a log message to be generated.

if 3 % 2 == 1 log_info("three mod two is one");
IN

This operation will evaluate to TRUE if the left operand is equal to any of the expressions in the list on the right, and FALSE otherwise. Comparing a undefined value results in undef.

unknown IN unknown, unknown …​ = boolean

Example 29. IN

A log message will be generated if $EventID is equal to any one of the values in the list.

if $EventID IN (1000, 1001, 1004, 4001) log_info("EventID found");
NOT IN

This operation is equivalent to NOT expr IN expr_list.

unknown NOT IN unknown, unknown …​ = boolean

Example 30. NOT IN

A log message will be generated if $EventID is not equal to any of the values in the list.

if $EventID NOT IN (1000, 1001, 1004, 4001) log_info("EventID not in list");

Ternary Operation

The ternary operator expr1 ? expr2 : expr3 evaluates to expr2 if expr1 is TRUE, otherwise to expr3. The parentheses as shown here are optional.

Example 31. Using the Ternary Operator

The $Important field is set to TRUE if $SeverityValue is greater than 2, or FALSE otherwise.

$Important = ( $SeverityValue > 2 ? TRUE : FALSE );

Functions

A function is an expression which returns a value. The returned value can be used to set fields, output log data, or make logic decisions. Functions can be polymorphic, meaning that the same function can take different argument types.

Many NXLog language features are provided through functions. As with other types of expressions, and unlike procedures, functions do not modify the state of the NXLog engine, the state of the module, or the current event.

See the list of available core functions. Modules can provide additional functions for use with the NXLog language.

Example 32. Function calls

These expressions call the now() function to return the current time and the hostname() function to return the hostname of the system where NXLog is installed. The returned values are used to set the $EventTime and $Relay fields.

$EventTime = now();
$Relay = hostname();

In the example below, the size() function is called to calculate the size of the $Message field. If the field is over 4096 bytes, an internal log is generated.

if size($Message) > 4096 log_info('Large message received.');
Example 33. Calling a function of a specific module instance

Functions for a specific module instance can be called using the -> operator. This expression calls the file_name() and file_size() functions of an om_file instance named out. The returned values are used to log the name and size of its current output file.

log_info('Size of output file ' + out->file_name() + ' is ' + out->file_size());

Calling functions of a specific instance is especially useful when the configuration contains more than one instance of the same module. These expressions call the to_xml() function of two xm_xml instances, one named xml_a and the other named xml_b.

xml_a->to_xml();
xml_b->to_xml();

Statements

The following elements can be used in statements.

Assignment

The assignment operation is declared with an equal sign (=). It loads the value from the expression evaluated on the right into a field on the left.

Example 34. Field Assignment

This statement sets the $EventReceivedTime field to the value returned by the now() function.

$EventReceivedTime = now();

Block

A block consists of one or more statements within curly braces ({}). This is typically used with conditional statements as in the example below.

Example 35. Conditional Statement Block

If the expression matches, both log messages will be generated.

if now() > 2000-01-01 00:00:00
{
   log_info("we are in the");
   log_info("21st century");
}

Procedures

A procedure is a statement that performs a set of actions. Procedures can accept arguments. Unlike a function, a procedure modifies its arguments, the state of the NXLog engine, the state of a module, or the current event. Procedures can be polymorphic, meaning that the same procedure can take different argument types.

Many NXLog language features are provided through procedures. See the list of available core procedures. Modules can provide additional procedures for use with the NXLog language.

Example 36. Calling a procedure

This example uses the parse_syslog() procedure, provided by the xm_syslog module, to parse syslog records received via UDP.

nxlog.conf
<Input in>
    Module  im_udp
    Host    0.0.0.0
    Port    514
    Exec    parse_syslog();
</Input>
Example 37. Calling a procedure of a specific module instance

Procedures for a specific module instance can be called using the -> operator. This statement calls the rotate_to() procedure of an om_file instance named out to rotate the current output file.

out->rotate_to("output_logs");

Calling procedures of a specific instance is especially useful when the configuration contains more than one instance of the same module. These statements call the parse_xml() procedure of two xm_xml instances, one named xml_a and the other named xml_b.

xml_a->parse_xml();
xml_b->parse_xml();

If-Else

A conditional statement starts with the if keyword followed by a boolean expression and a statement. The else keyword, followed by another statement, is optional. Brackets around the expression are also optional.

Example 38. Conditional Statements

A log message will be generated if the expression matches.

if now() > 2000-01-01 00:00:00 log_info("we are in the 21st century");

This statement is the same as the previous, but uses brackets.

if ( now() > 2000-01-01 00:00:00 ) log_info("we are in the 21st century");

This is a conditional statement block.

if now() > 2000-01-01 00:00:00
{
   log_info("we are in the 21st century");
}

This conditional statement block includes an else branch.

if now() > 2000-01-01 00:00:00
{
   log_info("we are in the 21st century");
}
else log_info("we are not yet in the 21st century");

Like Perl, the NXLog language does not have a switch statement. Instead, this can be accomplished by using conditional if-else statements.

Example 39. Emulating "switch" With "if-else"

The generated log message various based on the value of the $value field.

if ( $value == 1 )
   log_info("1");
else if ( $value == 2 )
   log_info("2");
else if ( $value == 3 )
   log_info("3");
else 
   log_info("default");
The Perl elsif and unless keywords are not supported.

While

A basic looping construct in NXLog language. The syntax is:

while ( cond_expression )
{
   statement;
   statement;
   ...
   statement;
}

Where cond_expression is any valid NXLog expression which evaluates to a boolean value. The statements will be executed repeatedly as long as the cond_expression evaluates to TRUE.

It is important to note that currently nothing prevents infinite loops therefore it is the responsibility of the user to not create such loops.

An example:

$$i = 0;
while ( $$i < 5 )
{
   log_info("Repeated message");
   $$i = $$i + 1;
}

while loops can be nested without limitations but the best practice is to limit the nesting level in order to avoid long running exec blocks.

Foreach

A looping construct in NXLog language designed to work on containers. The syntax is:

foreach (expression, module-variable)
{
   statement;
   statement;
   ...
   statement;
}

Where expression is any valid NXLog expression which evaluates to a container type of value like array or hash. But if one wants to change the values in that container then the expression must "address" a container or a sub-container already assigned to a field or to a module variable. Any temporary container, which can be a container literal or a function’s return value, will be disposed outside the loop.

The module-variable is a $$ notation for a module variable.

The statements will be executed as many times as many elements the container has (expression) and for each repetition the module-variable will be updated with the access key for a container element. In case of arrays this module-variable will hold the index values, while in case of a hash containers this will hold the keys for the stored values.

An example:

$$array = [1, 2, 3];
foreach ($$array, $$idx)
{
   $$array[$$idx] = $$array[$$idx] * 10;
}

$$hash = ( '1' => 100, '2' => 500);
foreach($$hash, $$key)
{
   $$hash[$$key] = $$hash[$$key] - 5;
}

foreach loops can be nested without limitations but the best practice is to limit the nesting level in order to avoid long running exec blocks.

Break, Continue

Like programming or scripting languages NXLog loops also have these statements and the effects are the same. In loop body break; causes immediate leaving of the loop while continue; causes skipping the rest of the statements after the continue;. In case of nested loops, these statements affect the loop in which the break or the continue resides.

E.g.:

$ar = [9,8,7,6,5,4,3,2,1];
foreach($ar, $$i)
{
  if ( $ar[$$i] == 4 )
  {
    continue;
  }
  if ( $ar[$$i] == 2 )
  {
    break;
  }
  if ( $ar[$$i] % 2 == 0 )  $ar[$$i] = "xxx";
}

Variables

A module variable can only be accessed from the same module instance where it was created. A variable is referenced by a string value and can store a value of any type.

See the create_var(), delete_var(), set_var(), and get_var() procedures.

In NXLog language there is another, simplified syntax for using module variables. Similar to the $ notation for fields, it is possible to access module variables with the $$ notation. In that case, the variable name is not dynamic but in return, the value can be manipulated the same way as the NXLog fields.

$$ival_1 = 10;
$$ival_2 = $$ival_1 + 200;
$$str = string($$ival_2);

The $$ notation is directly supported by the foreach(expr, $$key) NXLog language construct.

Statistical Counters

The following types are available for statistical counters:

COUNT

Added values are aggregated, and the value of the counter is increased if only positive integers are added until the counter is destroyed or indefinitely if the counter has no expiry.

COUNTMIN

This calculates the minimum value of the counter.

COUNTMAX

This calculates the maximum value of the counter.

AVG

This algorithm calculates the average over the specified interval.

AVGMIN

This algorithm calculates the average over the specified interval, and the value of the counter is always the lowest which was ever calculated during the lifetime of the counter.

AVGMAX

Like AVGMIN, but this returns the highest value calculated during the lifetime of the counter.

RATE

This calculates the value over the specified interval. It can be used to calculate events per second (EPS) values.

RATEMIN

This calculates the value over the specified interval, and returns the lowest rate calculated during the lifetime of the counter.

RATEMAX

Like RATEMIN, but this returns the highest rate calculated during the lifetime of the counter.

GRAD

This calculates the change of the rate of the counter over the specified interval, which is the gradient.

GRADMIN

This calculates the gradient and returns the lowest gradient calculated during the lifetime of the counter.

GRADMAX

Like GRADMIN, but this returns the highest gradient calculated during the lifetime of the counter.

Fields

The following fields are used by core.

$raw_event (type: string)

The data received from stream modules (im_file, im_tcp, etc.).

$EventReceivedTime (type: datetime)

The time when the event is received. The value is not modified if the field already exists.

$Hostname (type: string)

The name of the host where the nxlog is running. The value is not modified if the field already exists.

$SourceModuleName (type: string)

The name of the module instance, for input modules. The value is not modified if the field already exists.

$SourceModuleType (type: string)

The type of module instance (such as im_file), for input modules. The value is not modified if the field already exists.

Functions

The following functions are exported by core.

boolean array_pop(array array_value)

This removes the highest index element from the container array_value. It returns TRUE on succesful removal or FALSE on empty container or if the type of array_value is not an array.

binary base64decode(string base64str)

Return the decoded binary value of base64str.

string base64encode(unknown arg)

Return the BASE64 encoded string of arg, which can be either string or binary.

string bin2str(binary arg)

Return the raw string from the binary value of arg. ASCII null characters in arg will be converted to periods (.) in the returned string value. This function is intended for debugging purposes.

datetime datetime(integer arg)

Convert the integer argument, expressing the number of microseconds since epoch, to datetime.

integer day(datetime datetime)

Return the day part of the time value.

integer day(datetime datetime, boolean utc)

Return the day part of the time value. Optionally in UTC time if utc is set to TRUE, localtime if FALSE. If not set, honors the GenerateDateInUTC directive.

integer dayofweek(datetime datetime)

Return the number of days since Sunday in the range of 0-6.

integer dayofweek(datetime datetime, boolean utc)

Return the number of days since Sunday in the range of 0-6. Optionally in UTC time if utc is set to TRUE, localtime if FALSE. If not set, honors the GenerateDateInUTC directive.

integer dayofyear(datetime datetime)

Return the day number of the year in the range of 1-366.

integer dayofyear(datetime datetime, boolean utc)

Return the day number of the year in the range of 1-366. Optionally in UTC time if utc is set to TRUE, localtime if FALSE. If not set, honors the GenerateDateInUTC directive.

unknown deserialize(binary value)

It returns the de-serialized value of the value. If value is not a binary type or the format does not match the nxlog internal format then it returns an undef value.

boolean dropped()

Return TRUE if the currently processed event has already been dropped.

string escape_html(string html)

Return the HTML escaped html string.

string escape_json(string jsonstr)

Escape and return jsonstr according to the JSON specification.

string escape_url(string url)

Return the URL encoded string for url.

string escape_xml(string xmlstr)

Return the XML escaped xmlstr string.

boolean failed_over()

Returns TRUE if the current module is not connected to the first configured Host.

boolean failed_over(string modulename)

Returns TRUE if the module given by modulename is not connected to it’s first configured Host.

datetime fix_year(datetime datetime)

Return a corrected datetime value for a datetime which was parsed with a missing year, such as BSD Syslog or Cisco timestamps. The current year is used unless it would result in a timestamp that is more than 30 days in the future, in which case the previous year is used instead. If using the current year results in a timestamp that is less than or equal to 30 days in the future, it is assumed that the source device’s clock is incorrect (and the returned datetime value will be up to 30 days in the future).

string get_instance_id()

Return the NXLog agent unique identifier (UID).

integer get_rand()

Return a random integer value.

integer get_rand(integer max)

Return a random integer value between 0 and max.

unknown get_registryvalue(string mainkey, string subkeys, string valuename, boolean 64bit_view)

Return a value from the Windows Registry. mainkey must be one of the following predefined registry keys: HKCC, HKU, HKCU, HKCR, or HKLM. subkeys must be a series of backslash-separated valid Registry keys to open from mainkey. valuename must be a valid name of a value in last key of the subkeys. If 64bit_view is FALSE, then it indicates that 64-bit Windows should operate on the 32-bit Registry view; otherwise 64-bit Windows should operate on the 64-bit Registry view. Returns the value belonging to valuename. Returns undef if valuename or any of the subkeys can not be accessed in the Registry.

integer get_sequence(string name)

Return a number for the specified sequence that is incremented after each call to this function.

integer get_stat(string statname)

Return the value of the statistical counter or undef if it does not exist.

integer get_stat(string statname, datetime time)

Return the value of the statistical counter or undef if it does not exist. The time argument specifies the current time.

string get_uuid()

Return a UUID string.

unknown get_var(string varname)

Return the value of the variable or undef if it does not exist.

ipaddr host_ip()

Return the first non-loopback IP address the hostname resolves to.

ipaddr host_ip(integer nth)

Return the nth non-loopback IP address the hostname resolves to. The nth argument starts from 1.

string hostname()

Return the hostname (short form).

string hostname_fqdn()

Return the FQDN hostname. This function will return the short form if the FQDN hostname cannot be determined.

integer hour(datetime datetime)

Return the hour part of the time value.

integer hour(datetime datetime, boolean utc)

Return the hour part of the time value. Optionally in UTC time if utc is set to TRUE, localtime if FALSE. If not set, honors the GenerateDateInUTC directive.

integer integer(unknown arg)

Parse and convert the string argument to an integer. For datetime type it returns the number of microseconds since epoch.

ipaddr ipaddr(integer arg)

Convert the integer argument to an ipaddr type.

ipaddr ipaddr(integer arg, boolean ntoa)

Convert the integer argument to an ipaddr type. If ntoa is set to true, the integer is assumed to be in network byte order. Instead of 1.2.3.4 the result will be 4.3.2.1.

string lc(string arg)

Convert the string to lower case.

string md5sum(unknown arg)

Return the MD5 hash of arg as a hexadecimal string. arg can be either string or binary.

unknown md5sum(unknown arg, boolean isbinary)

Return the MD5 hash of arg as a binary value or a hexadecimal string. When isbinary is TRUE, the return type will be binary. arg can be either string or binary.

integer microsecond(datetime datetime)

Return the microsecond part of the time value.

integer microsecond(datetime datetime, boolean utc)

Return the microsecond part of the time value. Optionally in UTC time if utc is set to TRUE, localtime if FALSE. If not set, honors the GenerateDateInUTC directive.

integer minute(datetime datetime)

Return the minute part of the time value.

integer minute(datetime datetime, boolean utc)

Return the minute part of the time value. Optionally in UTC time if utc is set to TRUE, localtime if FALSE. If not set, honors the GenerateDateInUTC directive.

integer month(datetime datetime)

Return the month part of the datetime value.

integer month(datetime datetime, boolean utc)

Return the month part of the datetime value. Optionally in UTC time if utc is set to TRUE, localtime if FALSE. If not set, honors the GenerateDateInUTC directive.

datetime now()

Return the current time.

string nxlog_version()

Return the NXLog version string.

datetime parsedate(string arg)

Parse a string containing a timestamp. Dates without timezone information are treated as local time. The current year is used for formats that do not include the year. An undefined datetime type is returned if the argument cannot be parsed, so that the user can fix the error (for example, $EventTime = parsedate($somestring); if not defined($EventTime) $EventTime = now();). Supported timestamp formats are listed below.

RFC 3164 (legacy Syslog) and variations
Nov 6 08:49:37
Nov  6 08:49:37
Nov 06 08:49:37
Nov 3 14:50:30.403
Nov  3 14:50:30.403
Nov 03 14:50:30.403
Nov 3 2005 14:50:30
Nov  3 2005 14:50:30
Nov 03 2005 14:50:30
Nov 3 2005 14:50:30.403
Nov  3 2005 14:50:30.403
Nov 03 2005 14:50:30.403
Nov 3 14:50:30 2005
Nov  3 14:50:30 2005
Nov 03 14:50:30 2005
RFC 1123

RFC 1123 compliant dates are also supported, including a couple others which are similar such as those defined in RFC 822, RFC 850, and RFC 1036.

Sun, 06 Nov 1994 08:49:37 GMT  ; RFC 822, updated by RFC 1123
Sunday, 06-Nov-94 08:49:37 GMT ; RFC 850, obsoleted by RFC 1036
Sun Nov  6 08:49:37 1994       ; ANSI C's asctime() format
Sun, 6 Nov 1994 08:49:37 GMT   ; RFC 822, updated by RFC 1123
Sun, 06 Nov 94 08:49:37 GMT    ; RFC 822
Sun,  6 Nov 94 08:49:37 GMT    ; RFC 822
Sun, 6 Nov 94 08:49:37 GMT     ; RFC 822
Sun, 06 Nov 94 08:49 GMT       ; Unknown
Sun, 6 Nov 94 08:49 GMT        ; Unknown
Sun, 06 Nov 94 8:49:37 GMT     ; Unknown [Elm 70.85]
Sun, 6 Nov 94 8:49:37 GMT      ; Unknown [Elm 70.85]
Mon,  7 Jan 2002 07:21:22 GMT  ; Unknown [Postfix]
Sun, 06-Nov-1994 08:49:37 GMT  ; RFC 850 with four digit years

The above formats are also recognized when the leading day of week and/or the timezone are omitted.

Apache/NCSA date

This format can be found in Apache access logs and other sources.

24/Aug/2009:16:08:57 +0200
ISO 8601 and RFC 3339

NXLog can parse the ISO format with or without sub-second resolution, and with or without timezone information. It accepts either a comma (,) or a dot (.) in case there is sub-second resolution.

1977-09-06 01:02:03
1977-09-06 01:02:03.004
1977-09-06T01:02:03.004Z
1977-09-06T01:02:03.004+02:00
2011-5-29 0:3:21
2011-5-29 0:3:21+02:00
2011-5-29 0:3:21.004
2011-5-29 0:3:21.004+02:00
Windows timestamps
20100426151354.537875
20100426151354.537875-000
20100426151354.537875000
3/13/2017 8:42:07 AM      ; Microsoft DNS Server
Integer timestamp

This format is XXXXXXXXXX.USEC. The value is expressed as an integer showing the number of seconds elapsed since the epoch UTC. The fractional microsecond part is optional.

1258531221.650359
1258531221
BIND9 timestamps
23-Mar-2017 06:38:30.143
23-Mar-2017 06:38:30
2017-Mar-23 06:38:30.143
2017-Mar-23 06:38:30
datetime parsedate(string arg, boolean utc)

Dates without timezone information are treated as UTC when utc is TRUE. If utc is FALSE, input strings are parsed in local time—the same behavior as parsedate(arg).

string replace(string subject, string src, string dst)

Replace all occurrences of src with dst in the subject string.

string replace(string subject, string src, string dst, integer count)

Replace count number occurrences of src with dst in the subject string.

integer second(datetime datetime)

Return the second part of the time value.

integer second(datetime datetime, boolean utc)

Return the second part of the time value. Optionally in UTC time if utc is set to TRUE, localtime if FALSE. If not set, honors the GenerateDateInUTC directive.

binary serialize(unknown value)

It returns the serialized (binary) representation of the value.

string sha1sum(unknown arg)

Return the SHA1 hash of arg as a hexadecimal string. arg can be either string or binary.

unknown sha1sum(unknown arg, boolean isbinary)

Return the SHA1 hash of arg as a binary value or a hexadecimal string. When isbinary is TRUE, the return type will be binary. arg can be either string or binary.

string sha512sum(unknown arg)

Return the SHA512 hash of arg as a hexadecimal string. arg can be either string or binary.

unknown sha512sum(unknown arg, boolean isbinary)

Return the SHA512 hash of arg as a binary value or a hexadecimal string. When isbinary is TRUE, the return type will be binary. arg can be either string or binary.

unknown size(unknown value)

If value is a string or binary type, it returns the size of value in bytes. If value is a map or array type, it returns the number of elements in value. Otherwise, if value is another type or undef, the function returns undef.

string strftime(datetime datetime, string fmt)

Convert a datetime value to a string with the given format. The format must be one of:

  • YYYY-MM-DD hh:mm:ss,

  • YYYY-MM-DDThh:mm:ssTZ,

  • YYYY-MM-DDThh:mm:ss.sTZ,

  • YYYY-MM-DD hh:mm:ssTZ,

  • YYYY-MM-DD hh:mm:ss.sTZ,

  • YYYY-MM-DDThh:mm:ssUTC,

  • YYYY-MM-DDThh:mm:ss.sUTC,

  • YYYY-MM-DD hh:mm:ssUTC,

  • YYYY-MM-DD hh:mm:ss.sUTC, or

  • a format string accepted by the C strftime() function (see the strftime(3) manual or the Windows strftime reference for the format specification).

string string(unknown arg)

Convert the argument to a string.

datetime strptime(string input, string fmt)

Convert the string to a datetime with the given format. See the manual of strptime(3) for the format specification.

string substr(string src, integer from)

Return the string starting at the byte offset specified in from.

string substr(string src, integer from, integer to)

Return a sub-string specified with the starting and ending positions as byte offsets from the beginning of the string.

string type(unknown arg)

Return the type of the variable, which can be boolean, integer, string, datetime, ipaddr, regexp, or binary. For values with the unknown type, it returns undef.

string uc(string arg)

Convert the string to upper case.

string unescape_html(string html)

Return the HTML unescaped html string.

string unescape_json(string jsonstr)

Unescape and return jsonstr according to the JSON specification.

string unescape_url(string url)

Return the URL decoded string for url.

string unescape_xml(string xmlstr)

Return the XML unescaped xmlstr string.

integer year(datetime datetime)

Return the year part of the datetime value.

integer year(datetime datetime, boolean utc)

Return the year part of the datetime value. Optionally in UTC time if utc is set to TRUE, localtime if FALSE. If not set, honors the GenerateDateInUTC directive.

Procedures

The following procedures are exported by core.

add_stat(string statname, integer value);

Add value to the statistical counter using the current time.

add_stat(string statname, integer value, datetime time);

Add value to the statistical counter using the time specified in the argument named time.

add_to_route(string routename);

Copy the currently processed event data to the route specified. This procedure makes a copy of the data. The original will be processed normally. Note that flow control is explicitly disabled when moving data with add_to_route() and the data will not be added if the queue of the target module(s) is full.

array_append(array dst_array, array src_array, integer src_index, integer src_num_elements);

This procedure copies elements from src_array to the end of dst_array. The src_index specifies the first src_array element index and src_num_elements specifies the number of elements to copy. The size of dst_array will increase by src_num_elements at most. dst_array and src_array must both be array types. The operation is not executed if src_index exceeds the src_array size or src_num_elements is 0.

array_push(array array_value, varargs value);

Pushing the value(s) into the array_value after the highest index value. The size of the array_value will increase by the number of value arguments. If array_value is not an array type then no operation(s) will be carried out.

array_trim(array array_value);

Removing the undef values from the end of the array_value container. The size of the array will decrease if such element(s) was(were) found at the end of the container. If array_value is not an array type then no operation(s) will be carried out.

create_stat(string statname, string type);

Create a module statistical counter with the specified name using the current time. The statistical counter will be created with an infinite lifetime. The type argument must be one of the following to select the required algorithm for calculating the value of the statistical counter: COUNT, COUNTMIN, COUNTMAX, AVG, AVGMIN, AVGMAX, RATE, RATEMIN, RATEMAX, GRAD, GRADMIN, or GRADMAX (see Statistical Counters).

The interval parameter is optional for COUNT type statistical counters. It is mandatory for all other types.

This procedure will do nothing if a counter with the specified name already exists.

create_stat(string statname, string type, integer interval);

Create a module statistical counter with the specified name to be calculated over interval seconds and using the current time. The statistical counter will be created with an infinite lifetime.

create_stat(string statname, string type, integer interval, datetime time);

Create a module statistical counter with the specified name to be calculated over interval seconds and the time value specified in the time argument. The statistical counter will be created with an infinite lifetime.

create_stat(string statname, string type, integer interval, datetime time, integer lifetime);

Create a module statistical counter with the specified name to be calculated over interval seconds and the time value specified in the time argument. The statistical counter will expire after lifetime seconds.

create_stat(string statname, string type, integer interval, datetime time, datetime expiry);

Create a module statistical counter with the specified name to be calculated over interval seconds and the time value specified in the time argument. The statistical counter will expire at expiry.

create_var(string varname);

Create a module variable with the specified name. The variable will be created with an infinite lifetime.

create_var(string varname, integer lifetime);

Create a module variable with the specified name and the lifetime given in seconds. When the lifetime expires, the variable will be deleted automatically and get_var(name) will return undef.

create_var(string varname, datetime expiry);

Create a module variable with the specified name. The expiry specifies when the variable should be deleted automatically.

debug(unknown arg, varargs args);

Print the argument(s) at DEBUG log level. Same as log_debug().

delete(unknown arg);

Delete a field or a module variable from the event. For example, delete($field) or delete(variable)`. Note that `$field = undef` or `variable = undef is not the same, though after both operations the field or the module variable will be undefined. If the argument is an expression which is not purely a field ($field) or a module variable ($$variable) then the expression will be evaluated and that value will be treated as a name of the field to be deleted. If the expression yields a non-string type value then an error message will be issued.

delete(string arg);

Delete the field from the event. For example, delete("field").

delete_all();

Delete all of the fields from the event except raw_event field.

delete_stat(string statname);

Delete a module statistical counter with the specified name. This procedure will do nothing if a counter with the specified name does not exist (e.g. was already deleted).

delete_var(string varname);

Delete the module variable with the specified name if it exists.

drop();

Drop the event record that is currently being processed. Any further action on the event record will result in a "missing record" error.

duplicate_guard();

Guard against event duplication.

hash_remove(hash hash_value, varargs key);

Removing the keys(s) from the hash_value. The size of the hash_value will increase by the number of key arguments. To perform this operation, the key argument(s) must be of string type. If hash_value is not a hash type then no operation(s) will be carried out.

log_debug(unknown arg, varargs args);

Print the argument(s) at DEBUG log level. Same as debug(). Messages longer than specified at LogSizeLimit will be truncated.

log_error(unknown arg, varargs args);

Print the argument(s) at ERROR log level. Messages longer than specified at LogSizeLimit will be truncated.

log_info(unknown arg, varargs args);

Print the argument(s) at INFO log level. Messages longer than specified at LogSizeLimit will be truncated.

log_warning(unknown arg, varargs args);

Print the argument(s) at WARNING log level. Messages longer than specified at LogSizeLimit will be truncated.

module_restart();

Issue module_stop and then a module_start events for the calling module. Cross-module calls are supported: other_module→module_restart().

module_start();

Issue a module_start event for the calling module. Cross-module calls are supported: other_module→module_start().

module_stop();

Issue a module_stop event for the calling module. Cross-module calls are supported: other_module→module_stop().

rename_field(unknown old, unknown new);

Rename a field. For example, rename_field($old, $new).

rename_field(string old, string new);

Rename a field. For example, rename_field("old", "new").

reroute(string routename);

Move the currently processed event data to the route specified. The event data will enter the route as if it was received by an input module there. Note that flow control is explicitly disabled when moving data with reroute() and the data will be dropped if the queue of the target module(s) is full.

set_var(string varname, unknown value);

Set the value of a module variable. If the variable does not exist, it will be created with an infinite lifetime.

sleep(integer interval);

Sleep the specified number of microseconds. This procedure is provided for testing purposes primarily. It can be used as a poor man’s rate limiting tool, though this use is not recommended.