File (om_file)

This module writes data to one or more output files. You can specify a static or a dynamic filename that NXLog Agent evaluates as an expression for each record, allowing you to route records to different files based on field values such as $Hostname. The module does not manage log rotation or limit file size on its own, so you must configure rotation to prevent output files from growing indefinitely; see Rotating files based on size below for an example.

To examine the supported platforms, see the list of installation packages.

Managing open files with dynamic filenames

When you use a dynamic File expression, NXLog Agent can route records to any number of distinct output files over time. By default, the module keeps only one file open at a time. Whenever the filename changes, it closes the current file before opening the new one. This keeps resource usage low, but repeatedly opening and closing files adds overhead if the module frequently switches between the same set of filenames.

Use CacheSize to keep multiple files open concurrently instead. Once the number of open files reaches the configured limit, the module closes the least recently used file to make room for a new one. NXLog Agent does not enforce a maximum value for the cache size.

Avoiding file descriptor exhaustion

Because NXLog Agent does not cap CacheSize, setting it too high can exceed the number of file descriptors your operating system allows a single process to hold open. When this happens, the module fails to open new files until existing ones are closed.

Keep the following in mind when choosing a value for CacheSize:

Match it to concurrency, not volume: Set CacheSize to the number of files you expect the module to write concurrently, for example the number of distinct $Hostname values active at once, not the total number of files it will ever create.
Check your system’s limit: On Unix/Linux systems, run ulimit -n to see the per-process open file descriptor limit, and stay below it, accounting for other files and sockets NXLog Agent may already have open.
Prefer the default on Unix/Linux: Caching output files generally improves performance on Windows. On Unix/Linux systems, leave it unset unless you have measured a benefit.

Configuration

The om_file module accepts the following directives in addition to the common module directives.

Required directives

The following directive is required for the module to start.

File

Set this directive to specify the output file. The value must be a string expression.

Quote the filename to form a valid string literal; File does not accept an unquoted value. For relative filenames, NXLog Agent changes its working directory to / unless you change the global SpoolDir value. Below are three variations that specify the same output file on a Windows system:

File    'C:\Logs\nxlog-output.log'
File    "C:\\Logs\\nxlog-output.log"
File    'C:/Logs/nxlog-output.log'

If the File value is not a literal string but contains functions, field names, or operators, the module evaluates the expression for each record it processes. Processing high event rates increases CPU usage and can significantly degrade performance. If you notice performance issues with a dynamic File, use the simplest possible expression in your file-naming strategy.

If you write to a mapped network drive while NXLog Agent is running as a service, it might not be able to see the drive. See Cannot write logs to a mapped network drive for more information.

File permissions directives

The following directives are for specifying permissions for the created socket, pipe, or file.

Group

Set this directive to specify the group who owns the created socket, pipe, or file.

The default is the group set by the global Group directive. This directive does not work on Windows.

Perms

Set this directive to specify the permissions for the created socket, pipe, or file. Enter the value as a four-digit octal number starting with zero.

The default is the operating system’s default permissions. This directive does not work on Windows.

If the path includes non-existent directories, NXLog Agent adds execute permission (allowing directory browsing) to each newly created directory along the path for the current user.

User

Set this directive to specify the user who owns the created socket, pipe, or file.

The default is the user set by the global User directive. This directive does not work on Windows.

Optional directives

CacheSize

Set this directive to cache open files when using dynamic filenames, reducing the overhead of repeatedly opening and closing files. Set the value to the number of files you expect the module to write concurrently, but do not exceed the number of open files allowed by your system. See Managing open files with dynamic filenames for more details on using this directive.

By default, the module does not cache open files.

CreateDir

Set this directive to TRUE to create the output directory before opening the file for writing, if it does not already exist.

The default is FALSE.

OutputType

Set this directive to specify how the module formats output records. See the common OutputType directive for the available values, including support for data converters.

The default is LineBased, terminating each record with CRLF on Windows or LF on Unix.

RenameCheck

Set this directive to TRUE to check if an external application has rotated the open file. If the module detects that the file was rotated, it reopens the original file set by the File directive.

The default is FALSE.

Sync

Set this directive to TRUE to synchronize the file to disk after writing each record, flushing it from the buffers. This can degrade performance in write-heavy environments.

The default is FALSE.

Truncate

Set this directive to TRUE to truncate the file before each write, keeping only the most recent record.

The default is FALSE; the module appends records to the output file.

Functions

The following functions are exported by om_file.

type: string file_name()

Returns the current output filename, as specified by the File directive.

type: integer file_size()

Returns the size of the current output file in bytes. The function returns undef if the file doesn’t exist, which can happen if no event records have been written to the file yet.

The File Operations extension also provides a file_size() function with different functionality and parameters. Specify the module instance name with the arrow operator to ensure you call the correct function, e.g., MyInstanceName->file_size().

Procedures

The following procedures are exported by om_file.

reopen();: Reopens the current file. Call this procedure if the file is removed or renamed, for example with the file_cycle(), file_remove(), or file_rename() procedures of the File Operations extension. You do not need to call this procedure after rotate_to() because that procedure reopens the file automatically.
rotate_to(type: string filename);: Rotates the current output file to the specified filename. The target folder must exist, or the procedure fails to rotate the file. The module then recreates the original file set by the File directive.

This procedure uses the rename(2) system call, which does not support moving files across different devices on some platforms. To address this, first rotate the file on the same device, then use the exec_async() procedure of the External Programs extension to copy it to another device. Alternatively, use the file_copy() procedure of the File Operations extension.

Examples

Example 1. Basic om_file configuration

The following is a basic configuration that writes logs to a file on Linux.

nxlog.conf

<Output file>
    Module    om_file
    File      '/var/log/nxlog-output.log'
</Output>

Example 2. Rotating files based on size

This configuration rotates the output file once it exceeds 10 MB and retains up to two log files.

If you do not configure rotation, the module keeps writing to the same file indefinitely, which can fill up your disk.

nxlog.conf

define OUTPUTFILE  '/var/log/nxlog-output.log'

<Extension fileop>
    Module       xm_fileop
</Extension>

<Output file>
    Module       om_file
    File         %OUTPUTFILE%
    <Schedule>
        Every    60 sec
        <Exec>
            if (file_size() >= 10M)
            {
                fileop->file_cycle(%OUTPUTFILE%, 2);
                reopen();
            }
        </Exec>
    </Schedule>
</Output>

Example 3. Using dynamic filenames and compression

This configuration writes records to a file, using the $Hostname field as the filename. When the output file size exceeds 15 MB, it rotates it to a new file in a separate folder and compresses the new file.

If you use unsanitized fields for filenames, you risk exposing NXLog Agent to injection attacks.

nxlog.conf

<Extension exec> (1)
    Module    xm_exec
</Extension>

<Extension fileop> (2)
    Module    xm_fileop
</Extension>

<Output file>
    Module    om_file
    File      'tmp/' + $SanitizedHostname
    <Exec>
        $SanitizedHostname = $Hostname;
        if $SanitizedHostname =~ s/[\\\/:*?"<>|]//g {}; (3)

        if (file->file_size() > 15M) {
            $targetDir = 'tmp/' + month(now());

            if not fileop->dir_exists($targetDir) { (4)
                fileop->dir_make($targetDir);
            }

            $newfile = $targetDir + '/' + $SanitizedHostname + '_' + strftime(now(), '%Y%m%d%H%M%S');
            file->rotate_to($newfile);
            exec_async("/bin/bzip2", $newfile);
        }
    </Exec>
</Output>

1	The External Programs extension provides the exec_async() procedure.
2	The File Operations extension provides the dir_exists() function and the dir_make() procedure.
3	Removes illegal characters.
4	Creates the target folder if it does not exist.