NXLog Legacy Documentation

Log rotation and retention

NXLog can implement many kinds of log rotation and retention policies in order to prevent overuse of disk space and to organize older logs. These policies can be applied based on file size, time intervals, or even event attributes (such as severity). Log files can be rotated out to custom filenames and then compressed and/or deleted after a specified time period. The configuration is very flexible and custom policies can be easily implemented.

NXLog supports three main approaches to log rotation. In each case, policies should usually be implemented using a Schedule block.

Most policies are implemented within the scope of an om_file module instance, where output files are being written.
The im_file module can be configured to rotate log files after they have been fully read.
Any log file on the system can be rotated under the scope of an xm_fileop module or any other module. This includes the internal log file (specified by the LogFile directive).

Example 1. Rotating om_file log files

Log files written by an om_file module often need to be rotated regularly. This example uses the om_file file_name() function and xm_fileop file_cycle() procedure to rotate the output file daily, keeping a total of 7 old log files.

nxlog.conf

<Extension _fileop>
    Module  xm_fileop
</Extension>

<Output out>
    Module  om_file
    File    '/var/log/out.log'
    <Schedule>
        When    @daily
        <Exec>
            file_cycle(file_name(), 7);
            reopen();
        </Exec>
    </Schedule>
</Output>

Example 2. Rotating the internal log file

NXLog will write its own logs to a file specified by the LogFile directive. It is good practice to set up rotation of this file. This configuration uses the xm_fileop file_size() function. The file_cycle() procedure rotates the file if it is larger than 5 MB. The file is also rotated weekly. No more than 8 past log files are retained.

nxlog.conf

define LOGFILE /opt/nxlog/var/log/nxlog/nxlog.log
LogFile %LOGFILE%

<Extension _fileop>
    Module xm_fileop

    # Check the log file size every hour and rotate if larger than 5 MB
    <Schedule>
        Every 1 hour
        <Exec>
            if (file_exists('%LOGFILE%') and file_size('%LOGFILE%') >= 5M)
                file_cycle('%LOGFILE%', 8);
        </Exec>
    </Schedule>

    # Rotate log file every week on Sunday at midnight
    <Schedule>
        When    @weekly
        Exec    if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8);
    </Schedule>
</Extension>

There are many other ways that rotation and retention can be implemented. See the following sections for more details and examples.

Rotation policies and intervals

The om_file reopen() procedure will cause NXLog to reopen the output file specified by the File directive.
The rotate_to() procedure can be used to choose a name to rotate the current file to. This procedure will reopen the output file automatically, so there is no need to use the reopen() procedure.

The file_cycle() procedure will move the selected file to "file.1". If "file.1" already exists, it will be moved to "file.2", and so on. If an integer is used as a second argument, it specifies the maximum number of previous files to keep.

If file_cycle() is used on a file that NXLog currently has open under the scope of an om_file module instance, the reopen() procedure must be used to continue logging to the file specified by the File directive. Otherwise, events will continue to be logged to the rotated file ("file.1", for example). This is not necessary if the rotated file is the LogFile.

Rotating by file size

A log file can be rotated according to a pre-defined file size. This policy can be configured with the om_file file_size() function or the xm_fileop file_size() function.

Example 3. Using the file_size() function

This example uses the file_size() function to detect if a file has grown beyond a specified size. If it has, the file_cycle() procedure is used to rotate it. The file size is checked hourly with the When directive.

nxlog.conf

<Extension _fileop>
    Module  xm_fileop
</Extension>

<Output out>
    Module  om_file
    File    '/var/log/out.log'
    <Schedule>
        When @hourly
        <Exec>
            if file_size(file_name()) >= 1M
            {
                file_cycle(file_name());
                reopen();
            }
        </Exec>
    </Schedule>
</Output>

Using time-based intervals

For time interval based rotation policies NXLog provides two directives for use in Schedule blocks.

The Every directive rotates log files according to a specific interval specified in seconds, minutes, days, or weeks.
The When directive provides crontab-style scheduling, including extensions like @hourly, @daily, and @weekly.

Example 4. Using every and when for time-based rotation

This example shows the use of the Every and When directives. The output file is rotated daily using the rotate_to() function. The name is generated in the YYYY-MM-DD format according to the current server time.

nxlog.conf

<Output out>
    Module  om_file
    File    '/var/log/out.log'
    <Schedule>
        # This can likewise be used for `@weekly` or `@monthly` time periods.
        When @daily

        # The following crontab-style is the same as `@daily` above.
        # When "0 0 * * *"

        # The `Every` directive could also be used in this case.
        # Every 24 hour

        Exec    rotate_to(file_name() + strftime(now(), '_%Y-%m-%d'));
    </Schedule>
</Output>

Example 5. Rotating into a nested directory structure

In this example, logs for each year and month are stored in separated sub-directories as shown below. The log file is rotated daily.

.../logs/YEAR/MONTH/YYYY-MM-DD.log

This is accomplished with the xm_fileop dir_make() procedure, the core strftime() function, and the om_file rotate_to() procedure.

nxlog.conf

<Extension _fileop>
    Module  xm_fileop
</Extension>

<Output out>
    define OUT_DIR /srv/logs

    Module  om_file
    File    '%OUT_DIR%/out.log'
    <Schedule>
        When @daily
        <Exec>
            # Create year/month directories if necessary
            dir_make('%OUT_DIR%/' + strftime(now(), '%Y/%m'));

            # Rotate current file into the correct directory
            rotate_to('%OUT_DIR%/' + strftime(now(), '%Y/%m/%Y-%m-%d.log'));
        </Exec>
    </Schedule>
</Output>

Using dynamic filenames

As an alternative to traditional file rotation, output filenames can be set dynamically, based on each log event individually. This is possible because the om_file File directive supports expressions.

Because dynamic filenames result in events being written to multiple files with semi-arbitrary names, they are not suitable for scenarios where a server or application expects events to be written to a particular foo.log. In this case normal rotation should be used instead.

Often one of now(), $EventReceivedTime, and $EventTime are used for dynamic filenames. Consider the following points.

The now() function uses the current server time, not when the event was created or when it was received by NXLog. If logs are delayed, they will be stored according to the time at which the NXLog output module instance processes them. This will not work with nxlog-processor(8) (see Offline log processing).
The $EventReceivedTime field timestamp is set by the input module instance when an event is received by NXLog. This will usually be practically the same as using now(), except in cases where there are processing delays in the NXLog route (such as when using buffering). This can be used with nxlog-processor(8) if the $EventReceivedTime field was previously set in the logs.
The $EventTime field is set from a timestamp in the event, so will result in correct value even if the event was delayed before reaching NXLog. Note that some parsing may be required before this field is available (for example, the parse_syslog() procedure sets the xm_syslog EventTime field). Note also that an incorrect timestamp in an event record can cause the field to be unset or filled incorrectly, resulting in data written into the wrong file.

Example 6. Timestamp-based dynamic filenames with om_file

This example accepts Syslog formatted messages via UDP. Each message is parsed by the parse_syslog() procedure. The EventTime field is set from the timestamp in the syslog header. This field is then used by the expression in the File directive to generate an output filename for the event.

Even if messages received from clients over the network are out of order or delayed, they will still be placed in the appropriate output files according to the timestamps.

nxlog.conf

<Extension _syslog>
    Module  xm_syslog
</Extension>

<Input in>
    Module  im_udp
    Port    514
    Host    0.0.0.0
    Exec    parse_syslog();
</Input>

<Output out>
    Module  om_file
    File    '/var/log/nxlog/out_' + strftime($EventTime, '%Y-%m-%d')
    Exec    to_syslog_ietf();
</Output>

Dynamic filenames can be based on other fields also.

Example 7. Attribute-based dynamic filenames with om_file

In this example, events are grouped by their source hostname.

nxlog.conf

<Extension _syslog>
    Module  xm_syslog
</Extension>

<Input in>
    Module  im_udp
    Host    0.0.0.0
    Port    514
    Exec    parse_syslog();
</Input>

<Output out>
    Module  om_file
    File    '/tmp/logs_by_host/' + $Hostname
</Output>

Rotating input files

An im_file module instance can be configured to manipulate files after they are fully processed. The im_file OnEOF block can be used for this purpose.

When using OnEOF for rotation, the rotated files must be named (or placed in a directory) such that they will not be detected as new files and re-read by the module instance.

If a logging service keeps a log file open for writing, the xm_exec exec() procedure should be used to restart the service or otherwise instruct it to re-open the log file.

Example 8. Using im_file OnEOF for input files

In this example, files matching /var/log/app/*.log are read with an im_file module instance. When each file has been fully read, it is rotated. The GraceTimeout directive will prevent NXLog from rotating the file until after there have been no events for 10 seconds.

The input files are rotated by adding a timestamp suffix to the filename. For example, an input file named /var/log/app/errors.log would be rotated to /var/log/app/errors.log_20180101T130100. The new name does not match the wildcard specified by the File directive, so the file is not re-read.

nxlog.conf

<Extension _fileop>
    Module  xm_fileop
</Extension>

<Input app_logs_rotated>
    Module  im_file
    File    '/var/log/app/*.log'
    <OnEOF>
        <Exec>
            file_rename(file_name(),
                        file_name() + strftime(now(), '_%Y%m%dT%H%M%S'));
        </Exec>
        GraceTimeout  10
    </OnEOF>
</Input>

Retention policies

NXLog can be configured to keep old log files according to a particular retention policy. Functions and procedures for retention are provided by the xm_fileop module. Additional actions, such as compressing old log files, can be implemented with the xm_exec extension module.

Using simple file cycling

The file_cycle() procedure provides simple numbered rotation and, optionally, retention.

Example 9. Cycling one year of logs with file_cycle()

This example demonstrates the use of the xm_fileop file_cycle() procedure for keeping a total of 12 log files, one for each month. Log files older than 1 year will be automatically deleted.

This policy creates following log file structure: /var/log/foo.log for the current month,/var/log/foo.log.1 for the previous month, and so on up to the maximum of 12 files.

nxlog.conf

<Extension _fileop>
    Module  xm_fileop
</Extension>

<Output out>
    Module  om_file
    File    '/var/logs/foo.log'
    <Schedule>
        When @monthly
        <Exec>
            file_cycle(file_name(), 12);
            reopen();
        </Exec>
    </Schedule>
</Output>

Different policies for different events can be implemented in combination with dynamic filenames.

Example 10. Retaining files according to severity

This example uses the $Severity field (such as $Severity set by parse_syslog()) to filter events to separate files. Then different retention policies are applied according to severity. Here, one week of debug logs, 2 weeks of informational logs, and 4 weeks of higher severity logs are retained.

nxlog.conf

<Extension _syslog>
    Module  xm_syslog
</Extension>

<Extension _fileop>
    Module  xm_fileop
</Extension>

<Input logs_in>
   Module   im_file
   File     "/var/log/messages"
   Exec     parse_syslog();
</Input>

<Output logs_out>
    define OUT_DIR /opt/nxlog/var/log

    Module  om_file
    File    '%OUT_DIR%/' + $Severity + '.log'
    <Schedule>
        When @daily
        <Exec>
            file_cycle('%OUT_DIR%/DEBUG.log', 7);
            file_cycle('%OUT_DIR%/INFO.log', 14);
            file_cycle('%OUT_DIR%/WARNING.log', 28);
            file_cycle('%OUT_DIR%/ERROR.log', 28);
            file_cycle('%OUT_DIR%/CRITICAL.log', 28);
            reopen();
        </Exec>
    </Schedule>
</Output>

Compressing old log files

The xm_exec module can be used to compress old log files to reduce disk usage.

Example 11. Using bzip2 with exec_async()

In this example, the file size of the output file is checked hourly with the om_file file_size() function. If the size is over the limit, then:

a newfile module variable is set to the name the current file will be rotated to,
the om_file rotate_to() procedure renames the current output file to the name set in newfile,
the module re-opens the original file specified by the File directive and continue logging, and
the xm_exec exec_async() procedure call bzip2 on the rotated-out file (without waiting for the command to complete).

nxlog.conf

<Extension _exec>
    Module  xm_exec
</Extension>

<Extension _fileop>
    Module  xm_fileop
</Extension>

<Output out>
    Module  om_file
    File    '/opt/nxlog/var/log/app.log'
    <Schedule>
        When @hourly
        <Exec>
            if out->file_size() > 15M
            {
                set_var('newfile', file_name() + strftime(now(), '_%Y%m%d%H%M%S'));
                rotate_to(get_var('newfile'));
                exec_async('/bin/bzip2', get_var('newfile'));
            }
        </Exec>
    </Schedule>
</Output>

Deleting old log files

For retention policies where file deletion is not handled automatically by the xm_fileop file_cycle() procedure, the xm_fileop file_remove() can be used to delete old files. This procedure can also delete files based on their creation time.

Example 12. Using file_remove() to delete old files

This example uses file_remove() to remove any files older than 30 days.

nxlog.conf

<Input in>
   Module   im_null
</Input>

<Output logs_out>
    Module  om_file
    File    '/var/log/'+ strftime(now(),'%Y%m%d') + '.log'
    <Schedule>
        When @daily

        # Delete logs older than 30 days (24x60x60x30)
        Exec    file_remove('/var/log/*.log', now() - 2592000);
    </Schedule>
</Output>