Compress and encrypt logs with NXLog Agent

Two main concerns when storing log files on disk are reducing the size to save storage space and preventing unauthorized access. NXLog Agent provides compression and encryption functionality to help you address these.

Below, we provide examples of compressing and encrypting log files and their inverse operations—decompressing and decrypting log files to process the original logs. See NXLog Agent log compression and encryption for a further explanation of how compression and encryption data converters work.

Compress and decompress logs

Text compression is very efficient and reduces your log files' size significantly. The bigger the file size, the higher the data compression ratio you will achieve. NXLog Agent’s xm_zlib module provides file compression and decompression functionality that you can use with the im_file and om_file modules. It supports the gzip and zlib file formats.

Example 1. Compressing log files

This configuration writes logs to a file with the om_file output module. It uses the default LineBased output format to write one record per line and compresses the log file in gzip format.

<Extension zlib> (1)
    Module        xm_zlib
</Extension>

<Output output_file>
    Module        om_file
    File          '/tmp/nxlog-output.txt.gz'
    OutputType    zlib.compress (2)
</Output>

1	This xm_zlib module instance is named `zlib`. You must use this name when specifying compression and decompression data converters in the rest of the configuration.
2	Sets the OutputType directive to use the compress data converter of the `zlib` instance. Since the configuration does not specify an output writer function, the om_file instance will use the default one, equivalent to setting `OutputType = LineBased, zlib.compress`.

You can read back compressed log files with an im_file instance, specifying the decompression data converter in the input type. For example, the following configuration collects the log files created with the above configuration.

Example 2. Collecting compressed log files

This configuration reads gzip-compressed files with the im_file input module. It expects text-based logs written one record per line.

<Extension zlib> (1)
    Module       xm_zlib
</Extension>

<Input input_file>
    Module       im_file
    File         '/tmp/nxlog-output.txt.gz'
    InputType    zlib.decompress  (2)
</Input>

1	This xm_zlib module instance is named `zlib`. You must use this name when specifying compression and decompression data converters in the rest of the configuration.
2	Sets the InputType directive to use the decompress data converter of the `zlib` instance. Since the configuration does not specify an input reader function, the im_file instance will use the default one, equivalent to setting `InputType = zlib.decompress, LineBased`.

Encrypt and decrypt logs

Encrypting logs adds a layer of security, preventing unauthorized access to log data. Compliance regulations often require logs to be encrypted. NXLog Agent’s xm_crypto module provides file encryption and decryption functionality that you can use with the im_file and om_file modules. It uses the AES symmetric encryption algorithm with a 256-bit key.

Example 3. Encrypting log files

This configuration writes logs to a file with the om_file output module. It uses the default LineBased output format to write one record per line. It also encrypts and password-protects the output log file.

<Extension crypto>  (1)
    Module        xm_crypto  
    Password      MySecretPassword (2)
</Extension>

<Output output_file>
    Module        om_file
    File          '/tmp/nxlog-output.txt'
    OutputType    crypto.aes_encrypt  (3)
</Output>

1	This xm_crypto module instance is named `crypto`. You must use this name when specifying compression and decompression data converters in the rest of the configuration.
2	Uses the Password directive to password-protect the log file. Alternatively, use a PasswordFile.
3	Sets the OutputType directive to use the aes_encrypt data converter of the `crypto` instance. Since the configuration does not specify an output writer function, the om_file instance will use the default one, equivalent to setting `OutputType = LineBased, crypto.aes_encrypt`.

You can read back encrypted log files with an im_file instance, specifying the decryption data converter in the input type. For example, the following configuration collects the log files created with the above configuration.

Example 4. Collecting encrypted log files

This configuration reads AES-encrypted files with the im_file input module. It expects text-based logs written one record per line.

<Extension crypto> (1)
    Module       xm_crypto
    Password     MySecretPassword (2)
</Extension>

<Input input_file>
    Module       im_file
    File         '/tmp/nxlog-output.txt'
    InputType    crypto.aes_decrypt (3)
</Input>

1	This xm_crypto module instance is named `crypto`. You must use this name when specifying compression and decompression data converters in the rest of the configuration.
2	The Password must match the password you used to encrypt the file.
3	Sets the InputType directive to use the aes_decrypt data converter of the `crypto` instance. Since the configuration does not specify an input reader function, the im_file instance will use the default one, equivalent to setting `InputType = crypto.aes_decrypt, LineBased`.

Combine compression and encryption

The previous examples demonstrated how you can compress and encrypt log files separately. However, you can combine the two to compress and encrypt your log files simultaneously.

Example 5. Compressing and encrypting logs

This configuration writes logs to a file with the om_file output module. It uses the default LineBased output format to write one record per line. In addition, it compresses the output log file in gzip format and encrypts it.

<Extension zlib>  (1)
    Module        xm_zlib
</Extension>

<Extension crypto>  (2)
    Module        xm_crypto
    Password      MySecretPassword (3)
</Extension>

<Output output_file>
    Module        om_file
    File          '/tmp/nxlog-output.txt.gz'
    OutputType    zlib.compress, crypto.aes_encrypt  (4)
</Output>

1	This xm_zlib module instance is named `zlib`. You must use this name when specifying compression and decompression data converters in the rest of the configuration.
2	This xm_crypto module instance is named `crypto`. You must use this name when specifying encryption and decryption data converters in the rest of the configuration.
3	Uses the Password directive to password-protect the log file. Alternatively, use a PasswordFile.
4	Sets the OutputType directive to use the compress data converter of the `zlib` instance and the aes_encrypt data converter of the `crypto` instance in that sequence. Since the configuration does not specify an output writer function, the om_file instance will use the default one, equivalent to setting `OutputType = LineBased, zlib.compress, crypto.aes_encrypt`.

You can read back compressed and encrypted log files with an im_file instance, specifying the decompression and decryption data converters in the input type. For example, the following configuration collects the log files created with the above configuration.

Example 6. Collecting compressed and encrypted log files

This configuration reads gzip-compressed and AES-encrypted files with the im_file input module. It expects text-based logs written one record per line.

<Extension zlib> (1)
    Module        xm_zlib
</Extension>

<Extension crypto> (2)
    Module        xm_crypto
    Password      MySecretPassword (3)
</Extension>

<Input input_file>
    Module        im_file
    File          '/tmp/nxlog-output.txt.gz'
    InputType     crypto.aes_decrypt, zlib.decompress (4)
</Input>

1	This xm_zlib module instance is named `zlib`. You must use this name when specifying compression and decompression data converters in the rest of the configuration.
2	This xm_crypto module instance is named `crypto`. You must use this name when specifying encryption and decryption data converters in the rest of the configuration.
3	The Password must match the password you used to encrypt the file.
4	Sets the InputType directive to use the decompress data converter of the `zlib` instance and the aes_decrypt data converter of the `crypto` instance in that sequence. Since the configuration does not specify an input reader function, the im_file instance will use the default one, equivalent to setting `InputType = crypto.aes_decrypt, zlib.decompress, LineBased`.