Python (im_python)

This module provides support for collecting log data with methods written in the Python language. Currently, only versions 3.x of Python are supported.

The file specified by the PythonCode directive should contain a read_data() method which is called by the im_python module instance. See also the xm_python and om_python modules.

To examine the supported platforms, see the list of installer packages in the Available Modules chapter.

The Python script should import the nxlog module, and will have access to the following classes and functions.

nxlog.log_debug(msg): Send the message msg to the internal logger at DEBUG log level. This function does the same as the core log_debug() procedure.
nxlog.log_info(msg): Send the message msg to the internal logger at INFO log level. This function does the same as the core log_info()procedure.
nxlog.log_warning(msg): Send the message msg to the internal logger at WARNING log level. This function does the same as the core log_warning() procedure.
nxlog.log_error(msg): Send the message msg to the internal logger at ERROR log level. This function does the same as the core log_error() procedure.

class nxlog.Module

This class will be instantiated by NXLog and passed to the read_data() method in the script.

logdata_new(): This method returns a new LogData event object.
set_read_timer(delay): This method sets a trigger for another read after a specified delay in seconds (float).

class nxlog.LogData

This class represents a Logdata event object.

delete_field(name): This method removes the field name from the event record.
field_names(): This method returns a list with the names of all the fields currently in the event record.
get_field(name): This method returns the value of the field name in the event.
post(): This method will submit the LogData event to NXLog for processing by the next module in the route.
set_field(name, value): This method sets the value of field name to value.
module: This attribute is set to the Module object associated with the event.

Configuration

The im_python module accepts the following directives in addition to the common module directives.

PythonCode: This mandatory directive specifies a file containing Python code. The im_python instance will call a read_data() function which must accept an nxlog.Module object as its only argument.

Call: This optional directive specifies the Python method to invoke. With this directive, you can call only specific methods from your Python code. If the directive is not specified, the default method read_data is invoked.

Examples

Example 1. Using im_python to generate event data

In this example, a Python script is used to read syslog events from multiple log files bundled in tar archives, which may be compressed. The parse_syslog() procedure is also used to parse the events.

To avoid re-reading archives, each one should be removed after reading (see the comments in the script) or other similar functionality implemented.

nxlog.conf

</Output>

<Extension _syslog>
    Module      xm_syslog
</Extension>

<Input in>
    Module      im_python
    PythonCode  modules/input/python/2_python.py
    Exec        parse_syslog();
</Input>

2_python.py

import os
import tarfile

import nxlog

LOG_DIR = 'modules/input/python/2_logdir'
POLL_INTERVAL = 30

def read_data(module):
    nxlog.log_debug('Checking for new archives')
    for file in os.listdir(LOG_DIR):
        path = os.path.join(LOG_DIR, file)
        nxlog.log_debug("Attempting to read from '{}'".format(path))
        try:
            for line in read_tar(path):
                event = module.logdata_new()
                event.set_field('ImportFile', path)
                event.set_field('raw_event', line.decode('utf-8'))
                event.post()
                nxlog.log_debug("Added event from '{}'".format(path))
            nxlog.log_debug("Added all events from '{}'".format(path))
            # Each archive should be removed after reading to prevent reading
            # the same file again. Requires adequate permissions.
            #nxlog.log_debug("Deleting file '{}'".format(path))
            #os.remove(path)
        except tarfile.ReadError:
            msg = "Skipping invalid tar file '{}'".format(path)
            nxlog.log_error(msg)
    # Check for files again after specified delay
    msg = 'Adding a read event with {} seconds delay'.format(POLL_INTERVAL)
    nxlog.log_debug(msg)
    module.set_read_timer(POLL_INTERVAL)

def read_tar(path):
    """Yield a string for each line in each file in tar file."""
    with tarfile.open(path) as tar:
        for file in tar:
            inner_file = tar.extractfile(file)
            for line in inner_file:
                yield line