Python (xm_python)

This module enables you to extend NXLog Agent’s log processing capabilities with a custom Python script. Only Python version 3 and its minor releases are currently supported. See the Python prerequisites for using this module on Windows.

The script specified by the PythonCode directive should contain one or more functions. You then call these functions using the Exec directive from any module instance. In addition, see also the im_python and om_python modules.

The Python script must import the nxlog module to access the following classes and methods.

nxlog.log_debug(msg)

Send the message msg to the internal logger at DEBUG log level. This function does the same as the core log_debug() procedure.

nxlog.log_info(msg)

Send the message msg to the internal logger at INFO log level. This function does the same as the core log_info()procedure.

nxlog.log_warning(msg)

Send the message msg to the internal logger at WARNING log level. This function does the same as the core log_warning() procedure.

nxlog.log_error(msg)

Send the message msg to the internal logger at ERROR log level. This function does the same as the core log_error() procedure.

class nxlog.Module

This class is instantiated by NXLog Agent and can be accessed via the LogData.module attribute. This can be used to set or access variables associated with the module (see the example below).

class nxlog.LogData

This class represents an event. It is instantiated by NXLog Agent and passed to the method specified by the python_call() procedure.

delete_field(name)

This method removes the field name from the event record.

field_names()

This method returns a list with the names of all the fields currently in the event record.

get_field(name)

This method returns the value of the field name in the event.

set_field(name, value)

This method sets the value of field name to value.

module

This attribute is set to the Module object associated with the event.

Python prerequisites for Windows

xm_python is available on Windows as of NXLog Agent version 5.5 and newer. Before using this module, you must ensure that the correct Python version is installed and that NXLog Agent can load it.

Install Python manually

These steps install Python for all users on the machine, which is required when the NXLog Agent service is running under the default Local System account. If you are using a custom service user, you may install Python for only that user.

  1. Download the required Python version according to your NXLog Agent installation. See the NXLog Agent and Python version matrix below.

  2. Execute the installation wizard and in the first step, choose Customize installation.

    Python installation wizard
  3. Select any optional features and click Next.

  4. Select Install Python 3.0 for all users and Add Python to environment variables advanced options.

    Python installation advanced options
  5. Take note of the install location and click Install.

  6. When the installation is complete, open the Python installation folder.

  7. Copy the Python DLL, e.g., python310.dll, to the NXLog Agent installation folder.

  8. Rename the file to libpython<major_version>.<minor_version>.dll. See the table below.

  9. Restart the NXLog Agent service.

Automated Python installation

The following PowerShell script downloads Python, installs it, and copies the necessary DLL file to the NXLog Agent installation folder. You must specify the required Python version according to your NXLog Agent installation in the $ver variable. See the [nxlog-and-python-version-matrix] below.

install_python.ps1
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

$ver="3.10.8"

$baseuri="https://www.python.org/ftp/python/" + $ver
$msifile="python-" + $ver +"-amd64.exe"
Invoke-WebRequest -uri $baseuri/$msifile -OutFile $msifile

$sdotver=$ver -replace '([0-9]*)\.([0-9]*)\.([0-9]*)', '$1.$2'
$sver=$sdotver -replace '\.',''

$pydll='C:\Program Files\Python' + $sver + '\python' + $sver + '.dll'
$nxpydll='C:\Program Files\nxlog\libpython' + $sdotver + '.dll'

& ./$msifile /quiet InstallAllUsers=1 PrependPath=1 CompileAll=1

DO
{
   Start-Sleep -s 2
   Write-Host -NoNewline .
}
until (Test-Path $pydll )

Write-Host
Write-Host "Copying DLL: $pydll -> $nxpydll"

cp $pydll $nxpydll
This script is provided "AS IS" without warranty of any kind, either expressed or implied. Use at your own risk.

NXLog Agent and Python version matrix

NXLog Agent version Python version DLL filename

5.5.x

3.9.x

libpython3.9.dll

5.6.x

3.10.x

libpython3.10.dll

Configuration

The xm_python module accepts the following directives in addition to the common module directives.

Required directives

The following directives are required for the module to start.

PythonCode

This mandatory directive specifies a file containing Python code. The python_call() procedure can be used to call a Python function defined in the file. The function must accept an nxlog.LogData object as its argument.

Procedures

The following procedures are exported by xm_python.

call(string subroutine);

Call the given Python subroutine.

python_call(string function);

Call the specified function, which must accept an nxlog.LogData() object as its only argument.

Examples

Example 1. Using Python for log processing

This configuration calls two Python functions to modify each event record. The add_checksum() uses Python’s hashlib module to add a $ChecksumSHA1 field to the event; the add_counter() function adds a $Counter field for non-DEBUG events.

The pm_hmac module offers a more complete implementation for checksumming. See Statistical Counters for a native way to add counters.
nxlog.conf
</Input>

<Extension _json>
    Module      xm_json
    DateFormat YYYY-MM-DD hh:mm:ss
</Extension>

<Extension _syslog>
    Module      xm_syslog
</Extension>

<Extension python>
    Module      xm_python
    PythonCode  "modules/extension/python/py/processlogs2.py"
</Extension>

<Output out>
    Module      om_file
    File        'tmp/output'
    <Exec>
        # The $SeverityValue field is added by this procedure.
        # Most other parsers also add a normalized severity value.
        parse_syslog();

        # Add a counter for each event with log level above DEBUG.
        python_call('add_counter');

        # Calculate a checksum (after the counter field is added).
        python_call('add_checksum');

        # Convert to JSON format
        to_json();
    </Exec>
</Output>
processlogs2.py
import hashlib

import nxlog

def add_checksum(event):
    # Convert field list to dictionary
    all = {}
    for field in event.field_names():
        all.update({field: event.get_field(field)})

    # Calculate checksum and add to event record
    checksum = hashlib.sha1(repr(sorted(all)).encode('utf-8')).hexdigest()
    event.set_field('ChecksumSHA1', checksum)
    nxlog.log_debug('Added checksum field')

def add_counter(event):
    # Get module object and initialize counter
    module = event.module
    if not 'counter' in module:
        module['counter'] = 0
        nxlog.log_debug('Initialized counter field')

    # Skip DEBUG messages
    severity = event.get_field('SeverityValue')
    if severity > 1:
        # Add field
        event.set_field('Counter', module['counter'])
        nxlog.log_debug('Added counter field')

        # Increment counter
        module['counter'] += 1
        nxlog.log_debug('Incremented counter')