Rotate and delete log files

You can implement log rotation and retention policies with NXLog Agent to organize archived logs and free up disk space. Below, we provide examples of rotating and deleting log files based on file size, number of files, retention period, and attributes such as event severity.

Rotate input log files

You can use the im_file OnEOF block directive to back up a file once NXLog Agent finishes reading it. When using this directive to rotate log files, ensure the new filename does not match the File setting. Otherwise, NXLog Agent will detect it as a new file and reread the logs.

Example 1. Renaming input log files on end-of-file

This configuration collects logs from a file with the im_file input module. If there are no new logs after 10 seconds, it uses the file_rename() procedure of the xm_fileop module to rename the file, adding the current timestamp to the filename. For example, a file /var/log/app.log would be renamed to /var/log/app.log_20231122T130100.

nxlog.conf

<Extension fileop>
    Module    xm_fileop
</Extension>

<Input app_log>
    Module    im_file
    File      '/var/log/app.log'
    <OnEOF>
        <Exec>
            file_rename(file_name(), file_name() + strftime(now(), '_%Y%m%dT%H%M%S'));
        </Exec>
        GraceTimeout    10
    </OnEOF>
</Input>

Ensure that the application logging to the file can handle removing its log file. Additionally, if the application keeps the log file open for writing, NXLog Agent will fail to rename it. You can use the xm_exec exec() procedure to issue a command to restart the application to handle such scenarios.

Rotate output log files

If you write logs to a file with the om_file output module, you have several options to implement log rotation and retention, such as rotating files by size or the number of files.

Rotate by the number of files

Example 2. Retaining a set number of log files

This configuration writes logs to a file with the om_file output module. It uses a Schedule block to rotate log files daily, keeping a maximum of 7 files.

nxlog.conf

<Extension fileop>
    Module    xm_fileop
</Extension>

<Output output_file>
    Module    om_file
    File      '/var/log/nxlog-out.log'
    <Schedule>
        When    @daily (1)
        <Exec>
            file_cycle(file_name(), 7); (2) (3)
            reopen(); (4)
        </Exec>
    </Schedule>
</Output>

1	Executes every day at midnight.
2	The file_cycle() procedure of xm_fileop renames the log file, adding the file number to the filename—for example, `nxlog-out.log.1`.
3	The file_name() function of om_file returns the current output log file.
4	Calls the reopen() procedure to recreate the output log file after renaming it.

Example 3. Retaining files by log severity

This configuration reads syslog messages from a file, and parses records into structured data using the parse_syslog() procedure of the xm_syslog module. This procedure adds the $Severity field to the event record.

It then writes the logs to a file with the om_file output module, grouping them into files according to severity. The Schedule block rotates log files daily, keeping debug logs for one week, informational logs for two weeks, and higher severity logs for four weeks.

nxlog.conf

define OUTPUT_DIR /opt/nxlog/var/log

<Extension syslog>
    Module    xm_syslog
</Extension>

<Extension fileop>
    Module    xm_fileop
</Extension>

<Input system_messages>
    Module    im_file
    File      '/var/log/messages'
    Exec      parse_syslog();
</Input>

<Output output_file>
    Module    om_file
    File      '%OUTPUT_DIR%/' + $Severity + '.log'
    <Schedule>
        When  @daily (1)
        <Exec>
            file_cycle('%OUTPUT_DIR%/DEBUG.log', 7); (2)
            file_cycle('%OUTPUT_DIR%/INFO.log', 14);
            file_cycle('%OUTPUT_DIR%/WARNING.log', 28);
            file_cycle('%OUTPUT_DIR%/ERROR.log', 28);
            file_cycle('%OUTPUT_DIR%/CRITICAL.log', 28);
            reopen(); (3)
        </Exec>
    </Schedule>
</Output>

1	Executes every day at midnight.
2	The file_cycle() procedure of xm_fileop renames the log file, adding the file number to the filename—for example, `ERROR.log.1`.
3	Calls the reopen() procedure to recreate the output log file after renaming it.

Rotate files by size

Example 4. Rotating files when they reach a size limit

This configuration writes logs to a file with the om_file output module. It uses a Schedule block to rotate log files once they reach 1 MB.

nxlog.conf

<Extension fileop>
    Module    xm_fileop
</Extension>

<Output output_file>
    Module    om_file
    File      '/var/log/nxlog-out.log'
    <Schedule>
        When  @hourly (1)
        <Exec>
            if (file_size(file_name()) >= 1M) (2) (3)
            {
                file_cycle(file_name()); (4)
                reopen(); (5)
            }
        </Exec>
    </Schedule>
</Output>

1	Executes every hour.
2	Uses the file_size() function of xm_fileop and will only rotate the log file if it’s 1 MB or bigger.
3	The file_name() function of om_file returns the current output log file.
4	The file_cycle() procedure of xm_fileop renames the log file, adding the file number to the filename—for example, `nxlog-out.log.1`.
5	Calls the reopen() procedure to recreate the output log file after renaming it.

Rotate files on a schedule

You can implement an interval-based log rotation policy with a Schedule block and om_file's rotate_to() procedure. A Schedule supports two types of interval directives:

Use the Every directive to rotate log files at an interval in seconds, minutes, days, or weeks.
Use the When directive to specify a crontab-style schedule expression, including extensions like @hourly, @daily, and @weekly.

Example 5. Rotating log files daily

This configuration demonstrates how to use the Every and When directives in a Schedule block.

nxlog.conf

<Output output_file>
    Module    om_file
    File      '/var/log/nxlog-out.log'
    <Schedule>
        # Can also be `@weekly` or `@monthly`.
        When  @daily

        # The following crontab expression is the same as `@daily` above.
        # When "0 0 * * *"

        # You can also use the `Every` directive as follows to achieve the same schedule.
        # Every 24 hour

        Exec  rotate_to(file_name() + strftime(now(), '_%Y-%m-%d')); (1)
    </Schedule>
</Output>

1	The rotate_to() procedure of om_file renames the current output log file, adding the date to the filename—for example, `nxlog-out.log_2023-11-22`. The procedure also recreates the original file.

Rotate the NXLog Agent log file

NXLog Agent writes logs to the file specified by the LogFile global directive. We recommend rotating the log file to prevent it from becoming too large.

Example 6. Rotating the log file on a schedule

This configuration uses the xm_fileop extension and a Schedule block to keep a maximum of 8 NXLog Agent log files of ~5 MB each.

nxlog.conf

define   LOGFILE /opt/nxlog/var/log/nxlog/nxlog.log
LogFile  %LOGFILE%

<Extension fileop>
    Module    xm_fileop

    # Checks the log file size every hour
    <Schedule>
        Every 1 hour
        <Exec>
            if (file_exists('%LOGFILE%') and file_size('%LOGFILE%') >= 5M) (1)
            {
                file_cycle('%LOGFILE%', 8); (2)
            }
        </Exec>
    </Schedule>

    # Rotates the log file every Sunday at midnight
    <Schedule> (3)
        When  @weekly
        <Exec>
            if (file_exists('%LOGFILE%'))
            {
                file_cycle('%LOGFILE%', 8);
            }
        </Exec>
    </Schedule>
</Extension>

1	Checks that the log file exists and will only rotate it if the file size is 5 MB or bigger.
2	The file_cycle() procedure renames the log file, adding the file number to the filename—for example, `nxlog.log.1`.
3	An additional schedule that rotates the log file weekly. You can use this as a complementary or alternative measure.

Compress old log files

NXLog Agent provides the xm_exec module to execute external scripts and applications. You can use this module to run your preferred compression tool and reduce the size of old log files.

Example 7. Compressing log files with bzip2

This configuration uses a Schedule block in the output instance to compress log files when they become too large.

nxlog.conf

<Extension exec>
    Module    xm_exec
</Extension>

<Extension fileop>
    Module    xm_fileop
</Extension>

<Output output_file>
    Module    om_file
    File      '/var/log/nxlog-out.log'
    <Schedule>
        When  @hourly (1)
        <Exec>
            if (output_file->file_size() > 15M) (2)
            {
                set_var('newfile', file_name() + strftime(now(), '_%Y%m%d%H%M%S')); (3)
                rotate_to(get_var('newfile')); (4)
                exec_async('/bin/bzip2', get_var('newfile')); (5)
            }
        </Exec>
    </Schedule>
</Output>

1	Executes every hour.
2	Uses the file_size() function of om_file to retrieve the output file size and only executes the code if it is larger than 15 MB.
3	Sets the `newfile` variable to the filename of the current output file concatenated with the current timestamp.
4	Uses the rotate_to() procedure of om_file to rename the current output file to the value of `newfile`. After that, the module recreates the output file specified by the File directive and continues writing to it.
5	The xm_exec exec_async() procedure calls `bzip2` to compress the old output file without waiting for the command to complete.

Delete log files

If you simply want to delete old log files, use the file_remove() procedure of xm_fileop. This procedure accepts an optional datetime parameter to specify that it should only delete the file if its creation time is older than this date.

Example 8. Deleting old log files

This configuration uses a Schedule block in the output instance to delete log files older than 30 days.

nxlog.conf

<Extension fileop>
    Module    xm_fileop
</Extension>

<Output output_file>
    Module    om_file
    File      '/var/log/nxlog-' + strftime(now(),'%Y%m%d') + '.log'
    <Schedule>
        When  @daily (1)
        Exec  file_remove('/var/log/nxlog-*.log', now() - 2592000); (2)
    </Schedule>
</Output>

1	Executes every day at midnight.
2	Deletes log files older than 30 days (24x60x60x30)