Compress and encrypt data with NXLog Agent
Two main concerns when storing telemetry data on disk are reducing the size to save storage space and preventing unauthorized access. NXLog Agent provides compression and encryption functionality to help you address these.
Below, we provide examples of compressing and encrypting files and their inverse operations—decompressing and decrypting files to process the original data. See NXLog Agent log compression and encryption for a further explanation of how compression and encryption data converters work.
Compress and decompress data
Text compression is very efficient and significantly reduces file size.
The bigger the file size, the higher the data compression ratio you will achieve.
NXLog Agent’s xm_zlib module provides file compression and decompression functionality that you can use with the im_file and om_file modules.
It supports the gzip
and zlib
file formats.
This configuration writes events to a file with the om_file output module.
It uses the default LineBased output format to write one record per line and compresses the file in gzip
format.
<Extension zlib> (1)
Module xm_zlib
</Extension>
<Output output_file>
Module om_file
File '/tmp/nxlog-output.txt.gz'
OutputType zlib.compress (2)
</Output>
1 | This xm_zlib module instance is named zlib .
You must use this name when specifying compression and decompression data converters in the rest of the configuration. |
2 | Sets the OutputType directive to use the compress data converter of the zlib instance.
Since the configuration does not specify an output writer function, the om_file instance will use the default one, equivalent to setting OutputType = LineBased, zlib.compress . |
You can read back compressed files with an im_file instance, specifying the decompression data converter in the input type. For example, the following configuration collects events from the file created with the above configuration.
This configuration reads gzip-compressed files with the im_file input module. It expects text-based events written one record per line.
<Extension zlib> (1)
Module xm_zlib
</Extension>
<Input input_file>
Module im_file
File '/tmp/nxlog-output.txt.gz'
InputType zlib.decompress (2)
</Input>
1 | This xm_zlib module instance is named zlib .
You must use this name when specifying compression and decompression data converters in the rest of the configuration. |
2 | Sets the InputType directive to use the decompress data converter of the zlib instance.
Since the configuration does not specify an input reader function, the im_file instance will use the default one, equivalent to setting InputType = zlib.decompress, LineBased . |
Encrypt and decrypt data
Encrypting logs adds a layer of security, preventing unauthorized access to telemetry data. Compliance regulations often require data to be encrypted. NXLog Agent’s xm_crypto module provides file encryption and decryption functionality that you can use with the im_file and om_file modules. It uses the AES symmetric encryption algorithm with a 256-bit key.
This configuration writes events to a file with the om_file output module. It uses the default LineBased output format to write one record per line. It also encrypts and password-protects the output file.
<Extension crypto> (1)
Module xm_crypto
Password MySecretPassword (2)
</Extension>
<Output output_file>
Module om_file
File '/tmp/nxlog-output.txt'
OutputType crypto.aes_encrypt (3)
</Output>
1 | This xm_crypto module instance is named crypto .
You must use this name when specifying compression and decompression data converters in the rest of the configuration. |
2 | Uses the Password directive to password-protect the file. Alternatively, use a PasswordFile. |
3 | Sets the OutputType directive to use the aes_encrypt data converter of the crypto instance.
Since the configuration does not specify an output writer function, the om_file instance will use the default one, equivalent to setting OutputType = LineBased, crypto.aes_encrypt . |
You can read back encrypted files with an im_file instance, specifying the decryption data converter in the input type. For example, the following configuration collects events from the file created with the above configuration.
This configuration reads AES-encrypted files with the im_file input module. It expects text-based events written one record per line.
<Extension crypto> (1)
Module xm_crypto
Password MySecretPassword (2)
</Extension>
<Input input_file>
Module im_file
File '/tmp/nxlog-output.txt'
InputType crypto.aes_decrypt (3)
</Input>
1 | This xm_crypto module instance is named crypto .
You must use this name when specifying compression and decompression data converters in the rest of the configuration. |
2 | The Password must match the password you used to encrypt the file. |
3 | Sets the InputType directive to use the aes_decrypt data converter of the crypto instance.
Since the configuration does not specify an input reader function, the im_file instance will use the default one, equivalent to setting InputType = crypto.aes_decrypt, LineBased . |
Combine compression and encryption
The previous examples demonstrated how you can compress and encrypt files separately. However, you can combine the two to compress and encrypt files simultaneously.
This configuration writes events to a file with the om_file output module.
It uses the default LineBased output format to write one record per line.
In addition, it compresses the output file in gzip
format and encrypts it.
<Extension zlib> (1)
Module xm_zlib
</Extension>
<Extension crypto> (2)
Module xm_crypto
Password MySecretPassword (3)
</Extension>
<Output output_file>
Module om_file
File '/tmp/nxlog-output.txt.gz'
OutputType zlib.compress, crypto.aes_encrypt (4)
</Output>
1 | This xm_zlib module instance is named zlib .
You must use this name when specifying compression and decompression data converters in the rest of the configuration. |
2 | This xm_crypto module instance is named crypto .
You must use this name when specifying encryption and decryption data converters in the rest of the configuration. |
3 | Uses the Password directive to password-protect the file. Alternatively, use a PasswordFile. |
4 | Sets the OutputType directive to use the compress data converter of the zlib instance and the aes_encrypt data converter of the crypto instance in that sequence.
Since the configuration does not specify an output writer function, the om_file instance will use the default one, equivalent to setting OutputType = LineBased, zlib.compress, crypto.aes_encrypt . |
You can read back compressed and encrypted files with an im_file instance, specifying the decompression and decryption data converters in the input type. For example, the following configuration collects events from the file created with the above configuration.
This configuration reads gzip-compressed and AES-encrypted files with the im_file input module. It expects text-based events written one record per line.
<Extension zlib> (1)
Module xm_zlib
</Extension>
<Extension crypto> (2)
Module xm_crypto
Password MySecretPassword (3)
</Extension>
<Input input_file>
Module im_file
File '/tmp/nxlog-output.txt.gz'
InputType crypto.aes_decrypt, zlib.decompress (4)
</Input>
1 | This xm_zlib module instance is named zlib .
You must use this name when specifying compression and decompression data converters in the rest of the configuration. |
2 | This xm_crypto module instance is named crypto .
You must use this name when specifying encryption and decryption data converters in the rest of the configuration. |
3 | The Password must match the password you used to encrypt the file. |
4 | Sets the InputType directive to use the decompress data converter of the zlib instance and the aes_decrypt data converter of the crypto instance in that sequence.
Since the configuration does not specify an input reader function, the im_file instance will use the default one, equivalent to setting InputType = crypto.aes_decrypt, zlib.decompress, LineBased . |