Python (im_python)

This module enables you to collect and process logs using a custom Python script. Only Python version 3 and its minor releases are currently supported. See the Python prerequisites for using this module on Windows.

The script specified by the PythonCode directive should contain a read_data() function, which the im_python module instance will call. In addition, see also the xm_python and om_python modules.

The Python script must import the nxlog module to access the following classes and methods.

nxlog.log_debug(msg): Send the message msg to the internal logger at DEBUG log level. This function does the same as the core log_debug() procedure.
nxlog.log_info(msg): Send the message msg to the internal logger at INFO log level. This function does the same as the core log_info()procedure.
nxlog.log_warning(msg): Send the message msg to the internal logger at WARNING log level. This function does the same as the core log_warning() procedure.
nxlog.log_error(msg): Send the message msg to the internal logger at ERROR log level. This function does the same as the core log_error() procedure.

class nxlog.Module

This class will be instantiated by NXLog Agent and passed to the read_data() method in the script.

logdata_new(): This method returns a new LogData event object.
set_read_timer(delay): This method sets a trigger for another read after a specified delay in seconds (float).

class nxlog.LogData

This class represents a Logdata event object.

delete_field(name): This method removes the field name from the event record.
field_names(): This method returns a list with the names of all the fields currently in the event record.
get_field(name): This method returns the value of the field name in the event.
post(): This method will submit the LogData event to NXLog Agent for processing by the next module in the route.
set_field(name, value): This method sets the value of field name to value.
module: This attribute is set to the Module object associated with the event.

Python prerequisites for Windows

im_python is available on Windows as of NXLog Agent version 5.5 and newer. Before using this module, you must ensure that the correct Python version is installed and that NXLog Agent can load it.

Install Python manually

These steps install Python for all users on the machine, which is required when the NXLog Agent service is running under the default Local System account. If you are using a custom service user, you may install Python for only that user.

Download the required Python version according to your NXLog Agent installation. See the NXLog Agent and Python version matrix below.
Execute the installation wizard and in the first step, choose Customize installation.
Select any optional features and click Next.
Select Install Python 3.0 for all users and Add Python to environment variables advanced options.
Take note of the install location and click Install.
When the installation is complete, open the Python installation folder.
Copy the Python DLL, e.g., python310.dll, to the NXLog Agent installation folder.
Rename the file to libpython<major_version>.<minor_version>.dll. See the table below.
Restart the NXLog Agent service.

Automated Python installation

The following PowerShell script downloads Python, installs it, and copies the necessary DLL file to the NXLog Agent installation folder. You must specify the required Python version according to your NXLog Agent installation in the $ver variable. See the NXLog Agent and Python version matrix below.

install_python.ps1

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

$ver="3.10.8"

$baseuri="https://www.python.org/ftp/python/" + $ver
$msifile="python-" + $ver +"-amd64.exe"
Invoke-WebRequest -uri $baseuri/$msifile -OutFile $msifile

$sdotver=$ver -replace '([0-9]*)\.([0-9]*)\.([0-9]*)', '$1.$2'
$sver=$sdotver -replace '\.',''

$pydll='C:\Program Files\Python' + $sver + '\python' + $sver + '.dll'
$nxpydll='C:\Program Files\nxlog\libpython' + $sdotver + '.dll'

& ./$msifile /quiet InstallAllUsers=1 PrependPath=1 CompileAll=1

DO
{
   Start-Sleep -s 2
   Write-Host -NoNewline .
}
until (Test-Path $pydll )

Write-Host
Write-Host "Copying DLL: $pydll -> $nxpydll"

cp $pydll $nxpydll

This script is provided "AS IS" without warranty of any kind, either expressed or implied. Use at your own risk.

NXLog Agent and Python version matrix

NXLog Agent version	Python version	DLL filename
5.5.x	3.9.x	libpython3.9.dll
5.6.x	3.10.x	libpython3.10.dll

NXLog Agent version

Python version

DLL filename

5.5.x

3.9.x

libpython3.9.dll

5.6.x

3.10.x

libpython3.10.dll

Configuration

The im_python module accepts the following directives in addition to the common module directives.

Required directives

The following directives are required for the module to start.

PythonCode

This mandatory directive specifies a file containing Python code. The im_python instance will call a read_data() function which must accept an nxlog.Module object as its only argument.

Optional directives

Call

This optional directive specifies the Python method to invoke. With this directive, you can call only specific methods from your Python code. If the directive is not specified, the default method read_data is invoked.

Examples

Example 1. Using im_python to generate event data

In this example, a Python script is used to read syslog events from multiple log files bundled in tar archives, which may be compressed. The parse_syslog() procedure is also used to parse the events.

To avoid re-reading archives, each one should be removed after reading (see the comments in the script) or other similar functionality implemented.

nxlog.conf

</Output>

<Extension _syslog>
    Module      xm_syslog
</Extension>

<Input in>
    Module      im_python
    PythonCode  modules/input/python/2_python.py
    Exec        parse_syslog();
</Input>

2_python.py

import os
import tarfile

import nxlog

LOG_DIR = 'modules/input/python/2_logdir'
POLL_INTERVAL = 30

def read_data(module):
    nxlog.log_debug('Checking for new archives')
    for file in os.listdir(LOG_DIR):
        path = os.path.join(LOG_DIR, file)
        nxlog.log_debug("Attempting to read from '{}'".format(path))
        try:
            for line in read_tar(path):
                event = module.logdata_new()
                event.set_field('ImportFile', path)
                event.set_field('raw_event', line.decode('utf-8'))
                event.post()
                nxlog.log_debug("Added event from '{}'".format(path))
            nxlog.log_debug("Added all events from '{}'".format(path))
            # Each archive should be removed after reading to prevent reading
            # the same file again. Requires adequate permissions.
            #nxlog.log_debug("Deleting file '{}'".format(path))
            #os.remove(path)
        except tarfile.ReadError:
            msg = "Skipping invalid tar file '{}'".format(path)
            nxlog.log_error(msg)
    # Check for files again after specified delay
    msg = 'Adding a read event with {} seconds delay'.format(POLL_INTERVAL)
    nxlog.log_debug(msg)
    module.set_read_timer(POLL_INTERVAL)

def read_tar(path):
    """Yield a string for each line in each file in tar file."""
    with tarfile.open(path) as tar:
        for file in tar:
            inner_file = tar.extractfile(file)
            for line in inner_file:
                yield line