Python (xm_python)

This module provides support for processing NXLog log data with methods written in the Python language. Currently, only versions 3.x of Python are supported.

The file specified by the PythonCode directive should contain one or more methods which can be called from the Exec directive of any module. See also the im_python and om_python modules.

To examine the supported platforms, see the list of installer packages in the Available Modules chapter.

The Python script should import the nxlog module, and will have access to the following classes and functions.

nxlog.log_debug(msg): Send the message msg to the internal logger at DEBUG log level. This function does the same as the core log_debug() procedure.
nxlog.log_info(msg): Send the message msg to the internal logger at INFO log level. This function does the same as the core log_info()procedure.
nxlog.log_warning(msg): Send the message msg to the internal logger at WARNING log level. This function does the same as the core log_warning() procedure.
nxlog.log_error(msg): Send the message msg to the internal logger at ERROR log level. This function does the same as the core log_error() procedure.
class nxlog.Module: This class is instantiated by NXLog and can be accessed via the LogData.module attribute. This can be used to set or access variables associated with the module (see the example below).

class nxlog.LogData

This class represents an event. It is instantiated by NXLog and passed to the method specified by the python_call() procedure.

delete_field(name): This method removes the field name from the event record.
field_names(): This method returns a list with the names of all the fields currently in the event record.
get_field(name): This method returns the value of the field name in the event.
set_field(name, value): This method sets the value of field name to value.

module: This attribute is set to the Module object associated with the event.

Configuration

The xm_python module accepts the following directives in addition to the common module directives.

PythonCode: This mandatory directive specifies a file containing Python code. The python_call() procedure can be used to call a Python function defined in the file. The function must accept an nxlog.LogData object as its argument.

Procedures

The following procedures are exported by xm_python.

call(string subroutine);: Call the given Python subroutine.

python_call(string function);: Call the specified function, which must accept an nxlog.LogData() object as its only argument.

Examples

Example 1. Using Python for log processing

This configuration calls two Python functions to modify each event record. The add_checksum() uses Python’s hashlib module to add a $ChecksumSHA1 field to the event; the add_counter() function adds a $Counter field for non-DEBUG events.

The pm_hmac module offers a more complete implementation for checksumming. See Statistical Counters for a native way to add counters.

nxlog.conf

<Extension _json>
    Module      xm_json
    DateFormat YYYY-MM-DD hh:mm:ss
</Extension>

<Extension _syslog>
    Module      xm_syslog
</Extension>

<Extension python>
    Module      xm_python
    PythonCode  "modules/extension/python/py/processlogs2.py"
</Extension>

<Output out>
    Module      om_file
    File        'tmp/output'
    <Exec>
        # The $SeverityValue field is added by this procedure.
        # Most other parsers also add a normalized severity value.
        parse_syslog();

        # Add a counter for each event with log level above DEBUG.
        python_call('add_counter');

        # Calculate a checksum (after the counter field is added).
        python_call('add_checksum');

        # Convert to JSON format
        to_json();
    </Exec>
</Output>

processlogs2.py

import hashlib

import nxlog

def add_checksum(event):
    # Convert field list to dictionary
    all = {}
    for field in event.field_names():
        all.update({field: event.get_field(field)})

    # Calculate checksum and add to event record
    checksum = hashlib.sha1(repr(sorted(all)).encode('utf-8')).hexdigest()
    event.set_field('ChecksumSHA1', checksum)
    nxlog.log_debug('Added checksum field')

def add_counter(event):
    # Get module object and initialize counter
    module = event.module
    if not 'counter' in module:
        module['counter'] = 0
        nxlog.log_debug('Initialized counter field')

    # Skip DEBUG messages
    severity = event.get_field('SeverityValue')
    if severity > 1:
        # Add field
        event.set_field('Counter', module['counter'])
        nxlog.log_debug('Added counter field')

        # Increment counter
        module['counter'] += 1
        nxlog.log_debug('Incremented counter')