Perl (xm_perl)

The Perl programming language is widely used for log processing and comes with a broad set of modules bundled or available from CPAN. Code can be written more quickly in Perl than in C, and code execution is safer because exceptions (croak/die) are handled properly and will only result in an unfinished attempt at log processing rather than taking down the whole NXLog process.

While the NXLog language is already a powerful framework, it is not intended to be a fully featured programming language and does not provide lists, arrays, hashes, and other features available in many high-level languages. With this module, Perl can be used to process event data via a built-in Perl interpreter. See also the im_perl and om_perl modules.

The Perl interpreter is only loaded if the module is declared in the configuration. The module will parse the file specified in the PerlCode directive when NXLog starts the module. This file should contain one or more methods which can be called from the Exec directive of any module that will use Perl for log processing. See the example below.

Perl code defined via this module must not be called from the im_perl and om_perl modules as that would involve two Perl interpreters and will likely result in a crash.

To use the xm_perl module on Windows, a separate Perl environment must be installed. Currently, the only environment supported is a specific version of Strawberry Perl, 5.28.2.1. Newer versions will not work. See the release notes. To download the MSI installer for this version (100 MB), click here.

To access event data, the Log::Nxlog module must be included, which provides the following methods.

log_debug(msg): Send the message msg to the internal logger on DEBUG log level. This method does the same as the log_debug() procedure in NXLog.
log_info(msg): Send the message msg to the internal logger on INFO log level. This method does the same as the log_info() procedure in NXLog.
log_warning(msg): Send the message msg to the internal logger on WARNING log level. This method does the same as the log_warning() procedure in NXLog.
log_error(msg): Send the message msg to the internal logger on ERROR log level. This method does the same as the log_error() procedure in NXLog.
delete_field(event, key): Delete the value associated with the field named key.
field_names(event): Return a list of the field names contained in the event data. This method can be used to iterate over all of the fields.
field_type(event, key): Return a string representing the type of the value associated with the field named key.
get_field(event, key): Retrieve the value associated with the field named key. This method returns a scalar value if the key exists and the value is defined, otherwise it returns undef.
set_field_boolean(event, key, value): Set the boolean value in the field named key.
set_field_integer(event, key, value): Set the integer value in the field named key.
set_field_string(event, key, value): Set the string value in the field named key.

For the full NXLog Perl API, see the POD documentation in Nxlog.pm. The documentation can be read with perldoc Log::Nxlog.

Configuration

The xm_perl module accepts the following directives in addition to the common module directives.

PerlCode

This mandatory directive expects a file containing valid Perl code. This file is read and parsed by the Perl interpreter. Methods defined in this file can be called with the call() procedure.

On Windows, the Perl script invoked by the PerlCode directive must define the Perl library paths at the beginning of the script to provide access to the Perl modules.

nxlog-windows.pl

use lib 'c:\Program Files\nxlog\data';

Config

This optional directive allows you to pass configuration strings to the script file defined by the PerlCode directive. This is a block directive and any text enclosed within <Config></Config> is submitted as a single string literal to the Perl code.

If you pass several values using this directive (for example, separated by the \n delimiter) be sure to parse the string correspondingly inside the Perl code.

Procedures

The following procedures are exported by xm_perl.

call(string subroutine);: Call the given Perl subroutine.

perl_call(string subroutine, varargs args);: Call the given Perl subroutine.

Examples

Example 1. Using the built-in Perl interpreter

In this example, logs are parsed as Syslog and then are passed to a Perl method which does a GeoIP lookup on the source address of the incoming message.

nxlog.conf

<Extension syslog>
    Module      xm_syslog
</Extension>

<Extension perl>
    Module      xm_perl
    PerlCode    "modules/extension/perl/processlogs.pl"
</Extension>

<Output fileout>
    Module      om_file
    File        'tmp/output'

    # First we parse the input natively from nxlog
    Exec        parse_syslog_bsd();

    # Now call the 'process' subroutine defined in 'processlogs.pl'
    Exec        perl_call("process");

    # You can also invoke this public procedure 'call' in case
    # of multiple xm_perl instances like this:
    # Exec      perl->call("process");
</Output>

processlogs.pl

use strict;
use warnings;

# Without Log::Nxlog you cannot access (read or modify) the event data
use Log::Nxlog;

use Geo::IP;

my $geoip;

BEGIN
{
    # This will be called once when nxlog starts so you can use this to
    # initialize stuff here
    #$geoip = Geo::IP->new(GEOIP_MEMORY_CACHE);
    $geoip = Geo::IP->open('modules/extension/perl/GeoIP.dat', GEOIP_MEMORY_CACHE);
}

# This is the method which is invoked from 'Exec' for each event
sub process
{
    # The event data is passed here when this method is invoked by the module
    my ( $event ) = @_;

    # We look up the county of the sender of the message
    my $msgsrcaddr = Log::Nxlog::get_field($event, 'MessageSourceAddress');
    if ( defined($msgsrcaddr) )
    {
	my $country = $geoip->country_code_by_addr($msgsrcaddr);
	$country = "unknown" unless ( defined($country) );
	Log::Nxlog::set_field_string($event, 'MessageSourceCountry', $country);
    }

    # Iterate over the fields
    foreach my $fname ( @{Log::Nxlog::field_names($event)} )
    {
	# Delete all fields except these
	if ( ! (($fname eq 'raw_event') ||
		($fname eq 'AccountName') ||
		($fname eq 'MessageSourceCountry')) )
	{
	    Log::Nxlog::delete_field($event, $fname);
	}
    }

    # Check a field and rename it if it matches
    my $accountname = Log::Nxlog::get_field($event, 'AccountName');
    if ( defined($accountname) && ($accountname eq 'John') )
    {
	Log::Nxlog::set_field_string($event, 'AccountName', 'johnny');
	Log::Nxlog::log_info('renamed john');
    }
}