External programs (im_exec)

This module will execute a program or script on startup and read its standard output. It can be used to easily integrate with exotic log sources which can be read only with the help of an external script or program.

If you are using a Perl script, consider using im_perl instead or turning on Autoflush with $| = 1;, otherwise im_exec might not receive data immediately due to Perl’s internal buffering. See the Perl language reference for more information about $|.

If you are using a Python script, we recommend disabling buffering of the stdout and stderr streams using the -u command-line option. Otherwise, NXLog Agent might not receive data immediately due to Python’s internal buffering.

Configuration

The im_exec module accepts the following directives in addition to the common module directives. The Command directive is required.

Required directives

The following directives are required for the module to start.

Command

This mandatory directive specifies the name of the program or script to be executed.

Programs, scripts, and commands are executed under the context of the user running NXLog Agent. When NXLog Agent is running as a service, the service user will be used. If the program, script, or command accesses environment variables, make sure that these are available for the NXLog Agent user.

Optional directives

Arg

This is an optional parameter. Arg can be specified multiple times, once for each argument that needs to be passed to the Command. Note that specifying multiple arguments with one Arg directive, with arguments separated by spaces, will not work (the Command would receive it as one argument).

InputType

See the InputType description in the global module configuration section.

Restart

Restart the process if it exits. There is a one-second delay before it is restarted to avoid a denial-of-service when a process is not behaving. Looping should be implemented in the script itself, this directive is only to provide some safety against malfunctioning scripts and programs. This boolean directive defaults to FALSE: the Command will not be restarted if it exits.

Creating and populating fields

im_exec populates the $raw_event core field with the log message read from the program or script’s standard output. Further processing of this field can be done to parse the message into structured data or convert it to a different output format, such as JSON or XML. See Parsing and converting log records below and Parsing standard log formats in the NXLog Platform User Guide for examples.

Examples

Example 1. Emulating im_file on Linux

This configuration uses the Linux tail command-line tool to read lines from a log file. The first Arg directive specifies the -f argument, which means that tail should monitor the file for new lines. The second Arg directive specifies the path of the log file. This is equivalent to executing the following command:

$ tail -f /var/log/messages

The im_file module should be used to read log messages from files. This example is only intended to demonstrate the use of the im_exec module.

nxlog.conf

<Input messages>
    Module     im_exec
    Command    /usr/bin/tail
    Arg        -f
    Arg        /var/log/messages
</Input>

Example 2. Executing an application

This configuration executes an application to read logs from a third-party source. The Command directive specifies the path to the application executable and the Arg directive specifies an application argument. This is equivalent to executing the following command:

$ /path/to/myapp --level=info

nxlog.conf

<Input myapp>
    Module      im_exec
    Command     /path/to/myapp

    # On Windows the path to the application executable
    # should include the file extension.
    #Command    C:\Program Files\MyApp\myapp.exe

    Arg         --level=info
</Input>

Example 3. Executing a script

This configuration executes a Python script to read logs from a third-party source. The Command directive specifies the path to the Python executable. The first Arg directive specifies the -u command-line option to disable buffering for the stdout and stderr streams. It is recommended to disable buffering because it may lead to a delay in receiving the logs. The second Arg directive specifies the path to the script. This is equivalent to executing the following command:

> python -u C:\Scripts\myscript.py

nxlog.conf

<Input python_script>
    Module     im_exec
    Command    C:\Python39\python.exe
    Arg        -u
    Arg        C:\Scripts\myscript.py
</Input>

Example 4. Executing commands under a specific shell

To execute commands under a specific shell, the Command directive should specify the path to the shell executable. The commands to execute can be passed as arguments according to the shell being used. The configuration below executes PowerShell commands from a file.

nxlog.conf

<Input powershell_script>
    Module     im_exec
    Command    C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
    Arg        C:\Scripts\myscript.ps1
</Input>

Example 5. Parsing and converting log records

This configuration executes a script and parses the $raw_event field with a regular expression. If the regular expression matches, fields are created according to the captured groups, otherwise the log record is dropped. Finally, the record is converted to JSON format using the to_json() procedure of the xm_json module.

nxlog.conf

<Extension json>
    Module    xm_json
</Extension>

<Input powershell_script>
    Module    im_exec
    Command   C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
    Arg       C:\Scripts\myscript.ps1
    <Exec>
        if $raw_event =~ /(?x)^(\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d\+\d\d:\d\d),
                          (.+),(.+)$/
        {
            $EventTime = parsedate($1);
            $Severity = $2;
            $Message = $3;
        }
        else
        {
            drop();
        }

        to_json();
    </Exec>
</Input>

Input sample

2021-11-05T14:03:40+01:00,INFO,The service started successfully

Output sample in JSON format

{
  "EventReceivedTime": "2021-11-05T14:04:24.244343+01:00",
  "SourceModuleName": "powershell_script",
  "SourceModuleType": "im_exec",
  "EventTime": "2021-11-05T14:03:40.000000+01:00",
  "Severity": "INFO",
  "Message": "The service started successfully"
}