Compress and encrypt data with NXLog Agent

Two main concerns when storing telemetry data on disk are reducing the size to save storage space and preventing unauthorized access. NXLog Agent provides compression and encryption functionality to help you address these.

Below, we provide examples of compressing and encrypting files and their inverse operations—decompressing and decrypting files to process the original data. See NXLog Agent log compression and encryption for a further explanation of how compression and encryption data converters work.

Compress and decompress data

Text compression is very efficient and significantly reduces file size. The bigger the file size, the higher the data compression ratio you will achieve. NXLog Agent’s xm_zlib module provides file compression and decompression functionality that you can use with the im_file and om_file modules. It supports the gzip and zlib file formats.

Example 1. Compressing telemetry data

This configuration writes events to a file with the om_file output module. It uses the default LineBased output format to write one record per line and compresses the file in gzip format.

<Extension zlib> (1)
    Module        xm_zlib
</Extension>

<Output output_file>
    Module        om_file
    File          '/tmp/nxlog-output.txt.gz'
    OutputType    zlib.compress (2)
</Output>

1	This xm_zlib module instance is named `zlib`. You must use this name when specifying compression and decompression data converters in the rest of the configuration.
2	Sets the OutputType directive to use the compress data converter of the `zlib` instance. Since the configuration does not specify an output writer function, the om_file instance will use the default one, equivalent to setting `OutputType = LineBased, zlib.compress`.

You can read back compressed files with an im_file instance, specifying the decompression data converter in the input type. For example, the following configuration collects events from the file created with the above configuration.

Example 2. Collecting compressed telemetry data

This configuration reads gzip-compressed files with the im_file input module. It expects text-based events written one record per line.

<Extension zlib> (1)
    Module       xm_zlib
</Extension>

<Input input_file>
    Module       im_file
    File         '/tmp/nxlog-output.txt.gz'
    InputType    zlib.decompress  (2)
</Input>

1	This xm_zlib module instance is named `zlib`. You must use this name when specifying compression and decompression data converters in the rest of the configuration.
2	Sets the InputType directive to use the decompress data converter of the `zlib` instance. Since the configuration does not specify an input reader function, the im_file instance will use the default one, equivalent to setting `InputType = zlib.decompress, LineBased`.

Encrypt and decrypt data

Encrypting logs adds a layer of security, preventing unauthorized access to telemetry data. Compliance regulations often require data to be encrypted. NXLog Agent’s xm_crypto module provides file encryption and decryption functionality that you can use with the im_file and om_file modules. It uses the AES symmetric encryption algorithm with a 256-bit key.

Example 3. Encrypting telemetry data

This configuration writes events to a file with the om_file output module. It uses the default LineBased output format to write one record per line. It also encrypts and password-protects the output file.

<Extension crypto>  (1)
    Module        xm_crypto  
    Password      MySecretPassword (2)
</Extension>

<Output output_file>
    Module        om_file
    File          '/tmp/nxlog-output.txt'
    OutputType    crypto.aes_encrypt  (3)
</Output>

1	This xm_crypto module instance is named `crypto`. You must use this name when specifying compression and decompression data converters in the rest of the configuration.
2	Uses the Password directive to password-protect the file. Alternatively, use a PasswordFile.
3	Sets the OutputType directive to use the aes_encrypt data converter of the `crypto` instance. Since the configuration does not specify an output writer function, the om_file instance will use the default one, equivalent to setting `OutputType = LineBased, crypto.aes_encrypt`.

You can read back encrypted files with an im_file instance, specifying the decryption data converter in the input type. For example, the following configuration collects events from the file created with the above configuration.

Example 4. Collecting encrypted telemetry data

This configuration reads AES-encrypted files with the im_file input module. It expects text-based events written one record per line.

<Extension crypto> (1)
    Module       xm_crypto
    Password     MySecretPassword (2)
</Extension>

<Input input_file>
    Module       im_file
    File         '/tmp/nxlog-output.txt'
    InputType    crypto.aes_decrypt (3)
</Input>

1	This xm_crypto module instance is named `crypto`. You must use this name when specifying compression and decompression data converters in the rest of the configuration.
2	The Password must match the password you used to encrypt the file.
3	Sets the InputType directive to use the aes_decrypt data converter of the `crypto` instance. Since the configuration does not specify an input reader function, the im_file instance will use the default one, equivalent to setting `InputType = crypto.aes_decrypt, LineBased`.

Combine compression and encryption

The previous examples demonstrated how you can compress and encrypt files separately. However, you can combine the two to compress and encrypt files simultaneously.

Example 5. Compressing and encrypting telemetry data

This configuration writes events to a file with the om_file output module. It uses the default LineBased output format to write one record per line. In addition, it compresses the output file in gzip format and encrypts it.

<Extension zlib>  (1)
    Module        xm_zlib
</Extension>

<Extension crypto>  (2)
    Module        xm_crypto
    Password      MySecretPassword (3)
</Extension>

<Output output_file>
    Module        om_file
    File          '/tmp/nxlog-output.txt.gz'
    OutputType    zlib.compress, crypto.aes_encrypt  (4)
</Output>

1	This xm_zlib module instance is named `zlib`. You must use this name when specifying compression and decompression data converters in the rest of the configuration.
2	This xm_crypto module instance is named `crypto`. You must use this name when specifying encryption and decryption data converters in the rest of the configuration.
3	Uses the Password directive to password-protect the file. Alternatively, use a PasswordFile.
4	Sets the OutputType directive to use the compress data converter of the `zlib` instance and the aes_encrypt data converter of the `crypto` instance in that sequence. Since the configuration does not specify an output writer function, the om_file instance will use the default one, equivalent to setting `OutputType = LineBased, zlib.compress, crypto.aes_encrypt`.

You can read back compressed and encrypted files with an im_file instance, specifying the decompression and decryption data converters in the input type. For example, the following configuration collects events from the file created with the above configuration.

Example 6. Collecting compressed and encrypted telemetry data

This configuration reads gzip-compressed and AES-encrypted files with the im_file input module. It expects text-based events written one record per line.

<Extension zlib> (1)
    Module        xm_zlib
</Extension>

<Extension crypto> (2)
    Module        xm_crypto
    Password      MySecretPassword (3)
</Extension>

<Input input_file>
    Module        im_file
    File          '/tmp/nxlog-output.txt.gz'
    InputType     crypto.aes_decrypt, zlib.decompress (4)
</Input>

1	This xm_zlib module instance is named `zlib`. You must use this name when specifying compression and decompression data converters in the rest of the configuration.
2	This xm_crypto module instance is named `crypto`. You must use this name when specifying encryption and decryption data converters in the rest of the configuration.
3	The Password must match the password you used to encrypt the file.
4	Sets the InputType directive to use the decompress data converter of the `zlib` instance and the aes_decrypt data converter of the `crypto` instance in that sequence. Since the configuration does not specify an input reader function, the im_file instance will use the default one, equivalent to setting `InputType = crypto.aes_decrypt, zlib.decompress, LineBased`.