File Operations (xm_fileop)

This module provides functions and procedures to manipulate files. Coupled with a Schedule block, this module allows various log rotation and retention policies to be implemented, including:

log file retention based on file size,
log file retention based on file age, and
cyclic log file rotation and retention.

To examine the supported platforms, see the list of installation packages.

Rotating, renaming, or removing the file written by om_file is also supported with the help of the om_file reopen() procedure.

Configuration

The xm_fileop module accepts only the common module directives.

Functions

The following functions are exported by xm_fileop.

type: boolean dir_exists(type: string path)

Return TRUE if path exists and is a directory. On error undef is returned and an error is logged.

type: string dir_temp_get()

Return the name of a directory suitable as a temporary storage location.

type: string file_basename(type: string file)

Strip the directory name from the full file path. For example, file_basename('/var/log/app.log') will return app.log.

type: datetime file_ctime(type: string file)

Return the creation or inode-changed time of file. On error undef is returned and an error is logged.

type: string file_dirname(type: string file)

Return the directory name of the full file path. For example, file_dirname('/var/log/app.log') will return /var/log. Returns an empty string if file does not contain any directory separators.

type: boolean file_exists(type: string file)

Return TRUE if file exists and is a regular file.

type: binary file_hash(type: string file, type: string digest)

Return the calculated hash of file using digest algorithm. Available digest values are blake2b512, blake2s256, gost, md4, md5, rmd160, sha1, sha224, sha256, sha384, sha512 (see openssl dgst command in openssl’s manual). On error undef is returned and an error is logged.

type: integer file_inode(type: string file)

Return the inode number of file. On error undef is returned and an error is logged.

type: datetime file_mtime(type: string file)

Return the last modification time of file. On error undef is returned and an error is logged.

type: string file_read(type: string file)

Return the contents of file as a string value. On error undef is returned and an error is logged.

type: integer file_size(type: string file)

Return the file size in bytes. If there’s an error, the function returns undef and logs an error.

The om_file module also provides a file_size() function with different functionality and parameters. Specify the module instance name with the arrow operator to ensure you call the correct function, e.g., MyInstanceName->file_size().

type: string file_type(type: string file)

Return the type of file. The following string values can be returned: FILE, DIR, CHAR, BLOCK, PIPE, LINK, SOCKET, and UNKNOWN. On error undef is returned and an error is logged.

Procedures

The following procedures are exported by xm_fileop.

dir_make(type: string path);

Create a directory recursively (like mkdir -p). It succeeds if the directory already exists. An error is logged if the operation fails.

dir_remove(type: string file);

Remove the directory from the filesystem.

file_append(type: string src, type: string dst);

Append the contents of the file src to dst. The dst file will be created if it does not exist. An error is logged if the operation fails.

file_chmod(type: string file, type: integer mode);

Change the permissions of file. This function is only implemented on POSIX systems where chmod() is available in the underlying operating system. An error is logged if the operation fails.

file_chown(type: string file, type: integer uid, type: integer gid);

Change the ownership of file. This function is only implemented on POSIX systems where chown() is available in the underlying operating system. An error is logged if the operation fails.

file_chown(type: string file, type: string user, type: string group);

Change the ownership of file. This function is only implemented on POSIX systems where chown() is available in the underlying operating system. An error is logged if the operation fails.

file_copy(type: string src, type: string dst);

Copy the file src to dst. If file dst already exists, its contents will be overwritten. An error is logged if the operation fails.

file_cycle(type: string file);

Performs a cyclic rotation on the specified file. NXLog Agent renames the file to "file.1". If "file.1" already exists, NXLog Agent renames it to "file.2", continuing sequentially. Wildcards are supported in both the file path and filename. On Windows, if backslashes (\) are used as directory separators with wildcards, they must be escaped (for example, C:\\test\\*.log).

The file_cycle() procedure automatically reopens only the internal LogFile if it is cycled. For all other files, reopening must be handled explicitly after using this procedure. An error is logged if the operation fails.

file_cycle(type: string file, type: integer max);

Do a cyclic rotation on file as described above. The max argument specifies the maximum number of files to keep. For example, if max is 5, "file.6" will be deleted.

file_cycle(type: string file, type: boolean copy_truncate);

Perform a cyclic rotation on file. If copy_truncate is set to FALSE or not specified, the file will be moved to "file.1". If copy_truncate is set to TRUE, the original file will be copied to "file.1", and then truncated to zero. If "file.1" already exists it will be moved to "file.2", and so on. Wildcards are supported in the file path and filename. The backslash (\) must be escaped if used as the directory separator with wildcards (for example, C:\\test\\*.log). If copy_truncate is FALSE, this procedure will reopen the LogFile if it is cycled. An error is logged if the operation fails.

Be aware, that if copy_truncate is TRUE, there is a small time delay between copying the file and truncating it, so some logging data may be lost.

file_cycle(type: string file, type: integer max, type: boolean copy_truncate);

Perform a cyclic rotation on file as described above. The max argument specifies the maximum number of files to keep. For example, if max is 5, "file.6" will be deleted.

file_link(type: string src, type: string dst);

Create a hardlink from src to dst. An error is logged if the operation fails.

file_remove(type: string file);

Remove file. It is possible to specify a wildcard in the filename (but not in the path). The backslash (\) must be escaped if used as the directory separator with wildcards (for example, C:\\test\\*.log). This procedure will reopen the LogFile if it is removed. An error is logged if the operation fails. Calling this procedure while the file is still open by nxlog or other processes, may produce undefined behavior, including silently failing to remove the file (known issue on Windows Server).

file_remove(type: string file, type: datetime older);

Remove file if its creation time is older than the value specified in older. It is possible to specify a wildcard in the filename (but not in the path). The backslash (\) must be escaped if used as the directory separator with wildcards (for example, C:\\test\\*.log). This procedure will reopen the LogFile if it is removed. An error is logged if the operation fails. Calling this procedure while the file is still open by nxlog or other processes, may produce undefined behavior, including silently failing to remove the file (known issue on Windows Server).

file_rename(type: string old, type: string new);

Rename the file old to new. If the file new exists, it will be overwritten. Moving files or directories across devices may not be possible. This procedure will reopen the LogFile if it is renamed. An error is logged if the operation fails.

file_touch(type: string file);

Update the last modification time of file or create the file if it does not exist. An error is logged if the operation fails.

file_truncate(type: string file);

Truncate file to zero length. If the file does not exist, it will be created. An error is logged if the operation fails.

file_truncate(type: string file, type: integer offset);

Truncate file to the size specified in offset. If the file does not exist, it will be created. An error is logged if the operation fails.

file_write(type: string file, type: string value);

Write value into file. The file will be created if it does not exist. An error is logged if the operation fails.

Examples

Example 1. Rotation of the NXLog Agent internal log file

In this example, the internal log file is rotated based on time and size.

nxlog.conf

#define LOGFILE C:\Program Files\nxlog\data\nxlog.log
define LOGFILE /var/log/nxlog/nxlog.log

<Extension fileop>
    Module      xm_fileop

    # Check the log file size every hour and rotate if larger than 1 MB
    <Schedule>
        Every   1 hour
        Exec    if (file_size('%LOGFILE%') >= 1M) \
                    file_cycle('%LOGFILE%', 2);
    </Schedule>

    # Rotate log file every week on Sunday at midnight
    <Schedule>
        When    @weekly
        Exec    file_cycle('%LOGFILE%', 2);
    </Schedule>
</Extension>

Example 2. Rotation of an external log file

In this example, an external log file is rotated based on its size.

nxlog.conf

<Extension fileop>
    Module    xm_fileop
</Extension>

<Output rotated_file>
    Module    om_file
    File      "C:\\Program Files\\nxlog\\data\\static.json" (1)
    <Schedule>
        Every 5 sec (2)
        <Exec>
            log_info(file_name(), " is ", file_size(), " bytes long"); (3)
            if file_size() >= 20K (4)
            {
                file_cycle(file_name(), 2); (5)
            }
            reopen(); (6)
        </Exec>
    </Schedule>
</Output>

1	Specifies `static.json` as the log file where data is written.
2	Ensures the `<Exec>` block runs every 5 seconds.
3	Logs the current file name and size.
4	Checks if the file size exceeds or equals 20 KB.
5	If the size condition is met, the `file_cycle()` function renames the file `static.json` to `static.json.1`, and so on, sequentially. The second parameter of `file_cycle(file_name(), 2)` limits the number of rotated files retained to two (`static.json.1` and `static.json.2`).
6	The `reopen()` function resumes logging to a new `static.json` file. This step is crucial because `file_cycle()` closes the current file as part of the rotation process.