Files (om_file)

This module can be used to write log messages to a file.

To examine the supported platforms, see the list of installation packages.

Configuration

The om_file module accepts the following directives in addition to the common module directives. The File directive is required.

Required directives

The following directives are required for the module to start.

File

This mandatory directive specifies the name of the output file to open. It must be an expression of type string.

Note that the filename must be quoted to be a valid string literal, unlike in other directives which take a filename argument. For relative filenames, note that NXLog Agent changes its working directory to "/" unless the global SpoolDir is set to something else. Below are three variations for specifying the same output file on a Windows system:

File 'C:\logs\logmsg.txt'
File "C:\\logs\\logmsg.txt"
File 'C:/logs/logmsg.txt'

If the value of the File directive is an expression containing functions, field names, or operators, and not a literal string, the module evaluates the expression for each event it processes. When processing high event rates, this increases CPU usage and can significantly degrade performance. If you detect performance issues while using a dynamic File, consider changing the file naming strategy to use the simplest expression possible.

If the output location is a mapped network drive and NXLog Agent is running as a service, it might not be able to see the drive. See Cannot write logs to a mapped network drive for more information.

File permissions

The following directives are for specifying permissions for the created socket, pipe, or, file.

Perms

This directive specifies the permissions to use for the created socket or pipe or file. This must be a four-digit octal value beginning with a zero. By default, OS default permissions will be set. This directive is not currently supported on Windows.

If the specified path includes non-existent directories, execute permissions (allowing directory browsing) will be added to each newly created directory along the path for the current user.

User

Use this directive to set the user ownership for the created socket, pipe, or file. By default, this is the user NXLog Agent is running as (which may be specified by the global User directive). This directive is not currently supported on Windows.

Optional directives

CacheSize

In the case of dynamic filenames, a cache can be utilized to keep files open. This increases performance by reducing the overhead caused by many open/close operations. It is recommended to set this to the number of expected files to be written. Note that this should not be set to more than the number of open files allowed by the system. This caching provides performance benefits on Windows only. Caching is disabled by default.

CreateDir

If set to TRUE, this optional boolean directive instructs the module to create the output directory before opening the file for writing if it does not exist. The default is FALSE.

Group

Use this directive to set the group ownership for the created socket, pipe, or file. By default, this is the group NXLog Agent is running as (which may be specified by the global Group directive). This directive is not currently supported on Windows.

OutputType

See the OutputType directive in the list of common module directives. If this directive is not specified the default is LineBased (the module will use CRLF as the record terminator on Windows, or LF on Unix).

This directive also supports data converters, see the description in the OutputType section.

RenameCheck

If set to TRUE, this optional boolean directive instructs the module to check if an external application has rotated the open file. If the file has been rotated, the module will reopen the original file specified with the File directive. The default value is FALSE.

Sync

This optional boolean directive instructs the module to sync the file after each log message is written, ensuring that it is written to disk from the buffers. Because this can hurt performance, the default is FALSE.

Truncate

This optional boolean directive instructs the module to truncate the file before each write, causing only the most recent log message to be saved. The default is FALSE: messages are appended to the output file.

Functions

The following functions are exported by om_file.

type: string file_name()

Return the current output file name, as specified by the File directive.

type: integer file_size()

Return the size of the current output file in bytes. The function returns undef if the file doesn’t exist, which can happen if no event records have been written to the file yet.

The xm_fileop extension also provides a file_size() function with different functionality and parameters. Specify the module instance name with the arrow operator to ensure you call the correct function, e.g., MyInstanceName->file_size().

Procedures

The following procedures are exported by om_file.

reopen();: Reopen the current file. This procedure should be called if the file has been removed or renamed, for example with the file_cycle(), file_remove(), or file_rename(); procedures of the xm_fileop module. This does not need to be called after rotate_to() because that procedure reopens the file automatically.
rotate_to(type: string filename);: Rotate the current output file to the specified filename. The target folder must exist, or the procedure will fail to rotate the file. The module will then recreate the original file specified with the File directive.

This procedure uses the rename(2) system call, which does not support moving files across different devices on some platforms. To address this, first rotate the file on the same device, then use the xm_exec exec_async() procedure to copy it to another device, or use the xm_fileop file_copy() procedure.

Examples

Example 1. Storing raw syslog messages into a log file

This configuration reads log messages from a socket and writes the messages to a file. No additional processing is done.

nxlog.conf

<Input uds>
    Module  im_uds
    UDS     /dev/log
</Input>

<Output file>
    Module  om_file
    File    '/var/log/nxlog-output.log'
</Output>

<Route uds_to_file>
    Path    uds => file
</Route>

Example 2. File rotation based on size

The om_file module can exhaust space on a file system if left on its own without proper rotation.

This configuration rotates the file if it is larger than 10 MB and no more than two past log files are retained.

nxlog.conf

<Input tcp>
    Module        im_tcp
    ListenAddr    0.0.0.0:5125
</Input>

<Extension fileop>
    Module        xm_fileop
</Extension>

define LOCALFILE  '/opt/nxlog/var/log/nxlog/local_output'
<Output file>
    Module        om_file
    File          %LOCALFILE%
    <Schedule>
        Every 60 sec
        <Exec>
            if (file_size() >= 10M)
            {
                fileop->file_cycle(%LOCALFILE%, 2);
                reopen();
            }
        </Exec>
    </Schedule>
</Output>

<Route tcp_to_file>
    Path          tcp => file
</Route>

Example 3. Using dynamic filenames and compression

This configuration listens for data over TCP and writes records to a file, using the $Hostname field as the filename. When the output file size exceeds 15 MB, it will rotate it to a new file in a separate folder and compress it.

Using unsanitized fields for filenames is risky as it exposes the agent to injection attacks.

nxlog.conf

<Extension exec> (1)
    Module        xm_exec
</Extension>

<Extension fileop> (2)
    Module        xm_fileop
</Extension>

<Input tcp>
    Module        im_tcp
    ListenAddr    0.0.0.0:1514
</Input>

<Output file>
    Module        om_file
    File          'tmp/' + $SanitizedHostname
    <Exec>
        # Remove filename illegal characters
        $SanitizedHostname = $Hostname;
        if $SanitizedHostname =~ s/[\\\/:*?"<>|]//g {};

        if (file->file_size() > 15M) {
            $targetDir = 'tmp/' + month(now());

            # Create the target folder if it doesn't exist
            if not fileop->dir_exists($targetDir) {
                fileop->dir_make($targetDir);
            }

            $newfile = $targetDir + '/' + $SanitizedHostname + '_' + strftime(now(), '%Y%m%d%H%M%S');
            file->rotate_to($newfile);
            exec_async("/bin/bzip2", $newfile);
        }
    </Exec>
</Output>

<Route tcp_to_file>
    Path          tcp => file
</Route>

1	The External Programs extension provides the exec_async() procedure.
2	The File Operations extension provides the dir_exists() function and the dir_make() procedure.