Python (xm_python)
This module enables you to extend NXLog Agent’s log processing capabilities with a custom Python script. Only Python version 3 and its minor releases are currently supported. See the Python prerequisites for using this module on Windows.
The script specified by the PythonCode directive should contain one or more functions. You then call these functions using the Exec directive from any module instance. In addition, see also the im_python and om_python modules.
The Python script must import the nxlog
module to access the following classes and methods.
- nxlog.log_debug(msg)
-
Send the message msg to the internal logger at DEBUG log level. This function does the same as the core log_debug() procedure.
- nxlog.log_info(msg)
-
Send the message msg to the internal logger at INFO log level. This function does the same as the core log_info()procedure.
- nxlog.log_warning(msg)
-
Send the message msg to the internal logger at WARNING log level. This function does the same as the core log_warning() procedure.
- nxlog.log_error(msg)
-
Send the message msg to the internal logger at ERROR log level. This function does the same as the core log_error() procedure.
- class nxlog.Module
-
This class is instantiated by NXLog Agent and can be accessed via the LogData.module attribute. This can be used to set or access variables associated with the module (see the example below).
- class nxlog.LogData
-
This class represents an event. It is instantiated by NXLog Agent and passed to the method specified by the python_call() procedure.
- delete_field(name)
-
This method removes the field name from the event record.
- field_names()
-
This method returns a list with the names of all the fields currently in the event record.
- get_field(name)
-
This method returns the value of the field name in the event.
- set_field(name, value)
-
This method sets the value of field name to value.
- module
-
This attribute is set to the Module object associated with the event.
Python prerequisites for Windows
xm_python is available on Windows as of NXLog Agent version 5.5 and newer. Before using this module, you must ensure that the correct Python version is installed and that NXLog Agent can load it.
Install Python manually
These steps install Python for all users on the machine, which is required when the NXLog Agent service is running under the default Local System account. If you are using a custom service user, you may install Python for only that user.
-
Download the required Python version according to your NXLog Agent installation. See the NXLog Agent and Python version matrix below.
-
Execute the installation wizard and in the first step, choose Customize installation.
-
Select any optional features and click Next.
-
Select Install Python 3.0 for all users and Add Python to environment variables advanced options.
-
Take note of the install location and click Install.
-
When the installation is complete, open the Python installation folder.
-
Copy the Python DLL, e.g.,
python310.dll
, to the NXLog Agent installation folder. -
Rename the file to
libpython<major_version>.<minor_version>.dll
. See the table below. -
Restart the NXLog Agent service.
Automated Python installation
The following PowerShell script downloads Python, installs it, and copies the necessary DLL file to the NXLog Agent installation folder.
You must specify the required Python version according to your NXLog Agent installation in the $ver
variable.
See the [nxlog-and-python-version-matrix] below.
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$ver="3.10.8"
$baseuri="https://www.python.org/ftp/python/" + $ver
$msifile="python-" + $ver +"-amd64.exe"
Invoke-WebRequest -uri $baseuri/$msifile -OutFile $msifile
$sdotver=$ver -replace '([0-9]*)\.([0-9]*)\.([0-9]*)', '$1.$2'
$sver=$sdotver -replace '\.',''
$pydll='C:\Program Files\Python' + $sver + '\python' + $sver + '.dll'
$nxpydll='C:\Program Files\nxlog\libpython' + $sdotver + '.dll'
& ./$msifile /quiet InstallAllUsers=1 PrependPath=1 CompileAll=1
DO
{
Start-Sleep -s 2
Write-Host -NoNewline .
}
until (Test-Path $pydll )
Write-Host
Write-Host "Copying DLL: $pydll -> $nxpydll"
cp $pydll $nxpydll
This script is provided "AS IS" without warranty of any kind, either expressed or implied. Use at your own risk. |
Configuration
The xm_python module accepts the following directives in addition to the common module directives.
Required directives
The following directives are required for the module to start.
This mandatory directive specifies a file containing Python code. The python_call() procedure can be used to call a Python function defined in the file. The function must accept an nxlog.LogData object as its argument. |
Procedures
The following procedures are exported by xm_python.
call(string subroutine);
-
Call the given Python subroutine.
python_call(string function);
-
Call the specified function, which must accept an nxlog.LogData() object as its only argument.
Examples
This configuration calls two Python functions to modify each event record.
The add_checksum()
uses Python’s hashlib module to add a $ChecksumSHA1
field to the event; the add_counter()
function adds a $Counter
field for non-DEBUG events.
The pm_hmac module offers a more complete implementation for checksumming. See Statistical Counters for a native way to add counters. |
</Input>
<Extension _json>
Module xm_json
DateFormat YYYY-MM-DD hh:mm:ss
</Extension>
<Extension _syslog>
Module xm_syslog
</Extension>
<Extension python>
Module xm_python
PythonCode "modules/extension/python/py/processlogs2.py"
</Extension>
<Output out>
Module om_file
File 'tmp/output'
<Exec>
# The $SeverityValue field is added by this procedure.
# Most other parsers also add a normalized severity value.
parse_syslog();
# Add a counter for each event with log level above DEBUG.
python_call('add_counter');
# Calculate a checksum (after the counter field is added).
python_call('add_checksum');
# Convert to JSON format
to_json();
</Exec>
</Output>
import hashlib
import nxlog
def add_checksum(event):
# Convert field list to dictionary
all = {}
for field in event.field_names():
all.update({field: event.get_field(field)})
# Calculate checksum and add to event record
checksum = hashlib.sha1(repr(sorted(all)).encode('utf-8')).hexdigest()
event.set_field('ChecksumSHA1', checksum)
nxlog.log_debug('Added checksum field')
def add_counter(event):
# Get module object and initialize counter
module = event.module
if not 'counter' in module:
module['counter'] = 0
nxlog.log_debug('Initialized counter field')
# Skip DEBUG messages
severity = event.get_field('SeverityValue')
if severity > 1:
# Add field
event.set_field('Counter', module['counter'])
nxlog.log_debug('Added counter field')
# Increment counter
module['counter'] += 1
nxlog.log_debug('Incremented counter')