Files (om_file)

This module can be used to write log messages to a file.

To examine the supported platforms, see the list of installer packages in the Available Modules chapter.

Configuration

The om_file module accepts the following directives in addition to the common module directives. The File directive is required.

File

This mandatory directive specifies the name of the output file to open. It must be a string type expression. If the expression in the File directive is not a constant string (it contains functions, field names, or operators), it will be evaluated before each event is written to the file (and after the Exec is evaluated). Note that the filename must be quoted to be a valid string literal, unlike in other directives which take a filename argument. For relative filenames, note that NXLog changes its working directory to "/" unless the global SpoolDir is set to something else.

Below are three variations for specifying the same output file on a Windows system:

File 'C:\logs\logmsg.txt'
File "C:\\logs\\logmsg.txt"
File 'C:/logs/logmsg.txt'

If the output location is a mapped network drive and NXLog is running as a service, it might not be able to see the drive. See Cannot write logs to a mapped network drive in the User Guide for more information.

CacheSize: In case of dynamic filenames, a cache can be utilized to keep files open. This increases performance by reducing the overhead caused by many open/close operations. It is recommended to set this to the number of expected files to be written. Note that this should not be set to more than the number of open files allowed by the system. This caching provides performance benefits on Windows only. Caching is disabled by default.

CreateDir: If set to TRUE, this optional boolean directive instructs the module to create the output directory before opening the file for writing if it does not exist. The default is FALSE.

Group: Use this directive to set the group ownership for the created socket or pipe or file. By default, this is the group NXLog is running as, (which may be specified by the global Group directive). This directive is not currently supported on Windows.

OutputType: See the OutputType directive in the list of common module directives. If this directive is not specified the default is LineBased (the module will use CRLF as the record terminator on Windows, or LF on Unix).
This directive also supports data converters, see the description in the OutputType section.

Perms: This directive specifies the permissions to use for the created socket or pipe or file. This must be a four-digit octal value beginning with a zero. By default, OS default permissions will be set. This directive is not currently supported on Windows.

RenameCheck: If set to TRUE, this optional boolean directive instructs the module to check if an external application has rotated the open file. If the file has been rotated, the module will reopen the original file specified with the File directive. The default value is FALSE.

Sync: This optional boolean directive instructs the module to sync the file after each log message is written, ensuring that it is really written to disk from the buffers. Because this can hurt performance, the default is FALSE.

Truncate: This optional boolean directive instructs the module to truncate the file before each write, causing only the most recent log message to be saved. The default is FALSE: messages are appended to the output file.

User: Use this directive to set the user ownership for the created socket or pipe or file. By default, this is the user NXLog is running as (which may be specified by the global User directive). This directive is not currently supported on Windows.

Functions

The following functions are exported by om_file.

string file_name(): Return the name of the currently open file which was specified using the File directive. Note that this will be the old name if the filename changes dynamically; for the new name, use the expression specified for the File directive instead of using

integer file_size(): Return the size of the currently open output file in bytes. Returns undef if the file is not open. This can happen if File is not a string literal expression and there was no log message.

Procedures

The following procedures are exported by om_file.

reopen();: Reopen the current file. This procedure should be called if the file has been removed or renamed, for example with the file_cycle(), file_remove(), or file_rename(); procedures of the xm_fileop module. This does not need to be called after rotate_to() because that procedure reopens the file automatically.

rotate_to(string filename);: Rotate the current file to the filename specified. The module will then open the original file specified with the File directive. Note that the rename(2) system call is used internally which does not support moving files across different devices on some platforms. If this is a problem, first rotate the file on the same device. Then use the xm_exec exec_async() procedure to copy it to another device or file system, or use the xm_fileop file_copy() procedure.

Examples

Example 1. Storing raw syslog messages into a log file

This configuration reads log messages from socket and writes the messages to file. No additional processing is done.

nxlog.conf

<Input uds>
    Module  im_uds
    UDS     /dev/log
</Input>

<Output file>
    Module  om_file
    File    "/var/log/messages"
</Output>

<Route uds_to_file>
    Path    uds => file
</Route>

Example 2. File rotation based on size

The om_file module can exhaust space on a file system if left on its own without proper rotation.

This configuration rotates the file if it is larger than 10 MB and no more than two past log files are retained.

nxlog.conf

<Input tcp>
    Module        im_tcp
    ListenAddr    0.0.0.0:5125
</Input>

<Extension fileop>
    Module        xm_fileop
</Extension>

define LOCALFILE  '/opt/nxlog/var/log/nxlog/local_output'
<Output file>
    Module        om_file
    File          %LOCALFILE%
    <Schedule>
        Every 60 sec
        <Exec>
            if (file_size() >= 10M)
            {
                fileop->file_cycle(%LOCALFILE%, 2);
                reopen();
            }
        </Exec>
    </Schedule>
</Output>

<Route tcp_to_file>
    Path          tcp => file
</Route>

Example 3. Using dynamic filenames and compression

With this configuration, NXLog accepts log messages via TCP and parses them as BSD Syslog. A separate output file is used for log messages from each host. When the output file size exceeds 15 MB, it will be automatically rotated and compressed.

Using unsanitized fields for filenames is risky as it exposes the agent to injection attacks.

nxlog.conf

<Extension exec>
    Module        xm_exec
</Extension>

<Extension syslog>
    Module        xm_syslog
</Extension>

<Input tcp>
    Module        im_tcp
    ListenAddr    0.0.0.0:1514
    Exec          parse_syslog_bsd();
</Input>

<Output file>
    Module        om_file
    File          'tmp/output_' + $SanitizedHostname + '_' + month(now())
    <Exec>
        # Remove filename illegal characters
        $SanitizedHostname = $Hostname;
        if $SanitizedHostname =~ s/[\\\/:*?"<>|]//g {};
        if (file->file_size() > 15M)
        {
            $newfile = 'tmp/output_' + $SanitizedHostname + '_' +
                        strftime(now(), '%Y%m%d%H%M%S');
            file->rotate_to($newfile);
            exec_async("/bin/bzip2", $newfile);
        }
    </Exec>
</Output>

<Route tcp_to_file>
    Path          tcp => file
</Route>