Files (om_file)

This module can be used to write log messages to a file.

Configuration

The om_file module accepts the following directives in addition to the common module directives. The File directive is required.

Required directives

The following directives are required for the module to start.

File

This mandatory directive specifies the name of the output file to open. It must be a string type expression. If the expression in the File directive is not a constant string (it contains functions, field names, or operators), it will be evaluated before each event is written to the file (and after the Exec is evaluated). Note that the filename must be quoted to be a valid string literal, unlike in other directives which take a filename argument. For relative filenames, note that NXLog Agent changes its working directory to "/" unless the global SpoolDir is set to something else.

Below are three variations for specifying the same output file on a Windows system:

File 'C:\logs\logmsg.txt'
File "C:\\logs\\logmsg.txt"
File 'C:/logs/logmsg.txt'

If the output location is a mapped network drive and NXLog Agent is running as a service, it might not be able to see the drive.

File permissions

The following directives are for specifying permissions for the created socket, pipe, or, file.

Perms

This directive specifies the permissions to use for the created socket or pipe or file. This must be a four-digit octal value beginning with a zero. By default, OS default permissions will be set. This directive is not currently supported on Windows.

User

Use this directive to set the user ownership for the created socket, pipe, or file. By default, this is the user NXLog Agent is running as (which may be specified by the global User directive). This directive is not currently supported on Windows.

Optional directives

CacheSize

In the case of dynamic filenames, a cache can be utilized to keep files open. This increases performance by reducing the overhead caused by many open/close operations. It is recommended to set this to the number of expected files to be written. Note that this should not be set to more than the number of open files allowed by the system. This caching provides performance benefits on Windows only. Caching is disabled by default.

CreateDir

If set to TRUE, this optional boolean directive instructs the module to create the output directory before opening the file for writing if it does not exist. The default is FALSE.

Group

Use this directive to set the group ownership for the created socket, pipe, or file. By default, this is the group NXLog Agent is running as, (which may be specified by the global Group directive). This directive is not currently supported on Windows.

OutputType

See the OutputType directive in the list of common module directives. If this directive is not specified the default is LineBased (the module will use CRLF as the record terminator on Windows, or LF on Unix).

This directive also supports data converters, see the description in the OutputType section.

RenameCheck

If set to TRUE, this optional boolean directive instructs the module to check if an external application has rotated the open file. If the file has been rotated, the module will reopen the original file specified with the File directive. The default value is FALSE.

Sync

This optional boolean directive instructs the module to sync the file after each log message is written, ensuring that it is written to disk from the buffers. Because this can hurt performance, the default is FALSE.

Truncate

This optional boolean directive instructs the module to truncate the file before each write, causing only the most recent log message to be saved. The default is FALSE: messages are appended to the output file.

Functions

The following functions are exported by om_file.

string file_name(): Return the current output file name, as specified by the File directive.

integer file_size()

Return the size of the current output file in bytes. The function returns undef if the file doesn’t exist, which can happen if no event records have been written to the file yet.

The xm_fileop extension also provides a file_size() function with different functionality and parameters. Specify the module instance name with the arrow operator to ensure you call the correct function, e.g., MyInstanceName->file_size().

Procedures

The following procedures are exported by om_file.

reopen();: Reopen the current file. This procedure should be called if the file has been removed or renamed, for example with the file_cycle(), file_remove(), or file_rename(); procedures of the xm_fileop module. This does not need to be called after rotate_to() because that procedure reopens the file automatically.

rotate_to(string filename);: Rotate the current file to the filename specified. The module will then open the original file specified with the File directive. Note that the rename(2) system call is used internally which does not support moving files across different devices on some platforms. If this is a problem, first rotate the file on the same device. Then use the xm_exec exec_async() procedure to copy it to another device or file system, or use the xm_fileop file_copy() procedure.

Examples

Example 1. Storing raw syslog messages into a log file

This configuration reads log messages from a socket and writes the messages to a file. No additional processing is done.

nxlog.conf

<Input uds>
    Module  im_uds
    UDS     /dev/log
</Input>

<Output file>
    Module  om_file
    File    "/var/log/messages"
</Output>

<Route uds_to_file>
    Path    uds => file
</Route>

Example 2. File rotation based on size

The om_file module can exhaust space on a file system if left on its own without proper rotation.

This configuration rotates the file if it is larger than 10 MB and no more than two past log files are retained.

nxlog.conf

<Input tcp>
    Module        im_tcp
    ListenAddr    0.0.0.0:5125
</Input>

<Extension fileop>
    Module        xm_fileop
</Extension>

define LOCALFILE  '/opt/nxlog/var/log/nxlog/local_output'
<Output file>
    Module        om_file
    File          %LOCALFILE%
    <Schedule>
        Every 60 sec
        <Exec>
            if (file_size() >= 10M)
            {
                fileop->file_cycle(%LOCALFILE%, 2);
                reopen();
            }
        </Exec>
    </Schedule>
</Output>

<Route tcp_to_file>
    Path          tcp => file
</Route>

Example 3. Using dynamic filenames and compression

With this configuration, NXLog accepts log messages via TCP and parses them as BSD Syslog. A separate output file is used for log messages from each host. When the output file size exceeds 15 MB, it will be automatically rotated and compressed.

Using unsanitized fields for filenames is risky as it exposes the agent to injection attacks.

nxlog.conf

<Extension exec>
    Module        xm_exec
</Extension>

<Extension syslog>
    Module        xm_syslog
</Extension>

<Input tcp>
    Module        im_tcp
    ListenAddr    0.0.0.0:1514
    Exec          parse_syslog_bsd();
</Input>

<Output file>
    Module        om_file
    File          'tmp/output_' + $SanitizedHostname + '_' + month(now())
    <Exec>
        # Remove filename illegal characters
        $SanitizedHostname = $Hostname;
        if $SanitizedHostname =~ s/[\\\/:*?"<>|]//g {};
        if (file->file_size() > 15M)
        {
            $newfile = 'tmp/output_' + $SanitizedHostname + '_' +
                        strftime(now(), '%Y%m%d%H%M%S');
            file->rotate_to($newfile);
            exec_async("/bin/bzip2", $newfile);
        }
    </Exec>
</Output>

<Route tcp_to_file>
    Path          tcp => file
</Route>