NXLog Legacy Documentation

Compression and Encryption

NXLog Enterprise Edition exclusive feature

Organizations may require log data to be stored on-site. In such a case, the following questions may arise:

How can we reduce the size of our log data?
How can we prevent unauthorized access to sensitive information?

To address the first question, compressing logs can help reduce the size of the data that needs to be stored, thus allowing these resources to be reallocated for other purposes. This is true for both log entries that are large in size, such as Windows Event Logs, as well as for log sources that generate many events within a short period of time, even those that are small in size.

Log encryption offers security by concealing sensitive information that may be required by compliance mandates.

Both operations, compression and encryption of data, imply that their inverse operations, decompression and decryption, can be performed to facilitate the reading and processing of the original data.

The data converters used with the InputType and OutputType directives of their respective input/output stream-based modules are invoked using dot notation:

<InstanceName>.<DataConverterName>

For the sake of consistency in these examples, all xm_zlib instances will be named zlib and all xm_crypto instances will be named crypto. For more details about using data converters with these directives, see the documentation on InputType data converters and OutputType data converters.

To learn more about compression and encryption, see the documentation for the Compression (xm_zlib) and Encryption (xm_crypto) extension modules.

Compression and Decompression of Logs

The data compression capabilities of NXLog are provided by the xm_zlib module. This module supports the gzip data format defined in RFC 1952 and the zlib data format defined in RFC 1950.

The following table shows the order of operations for compressing and decompressing log data comprised of single-line events.

Table 1. Sequential order of operations for compression and decompression
	Directive	First Operation	Second Operation
Compression	OutputType	LineBased	compress
Decompression	InputType	decompress	LineBased

Log data can be compressed and saved to file using the om_file module. To successfully read the compressed logs, decompression needs to be performed on input and can be specified using the im_file module.

The example below shows how to compress Collecting logs from Windows Event Log data using the NXLog xm_zlib extension module.

Example 1. Compressing Windows Event Logs

This configuration reads Windows Event Log messages with Event ID 4688 using the im_msvistalog input module. The collected events are formatted as JSON using the to_json() procedure of the xm_json module.

Following the sequence of operations listed in the table above for compression, the OutputType directive in the to_file output instance outputs the events using the LineBased output writer function, the default for om_file. It then invokes the compress data converter using the zlib instance of the xm_zlib module to compress the log data.

Finally, the compressed output is saved as a file specified by the mandatory File directive.

nxlog.conf

<Extension json>
    Module        xm_json
</Extension>

<Extension zlib>
    Module        xm_zlib
</Extension>

<Input from_eventlog>
    Module        im_msvistalog
    <QueryXML>
        <QueryList>
            <Query Id="0">
                <Select Path="Security">
                    *[System[Level=0 and (EventID=4688)]]
                </Select>
            </Query>
        </QueryList>
    </QueryXML>
    Exec          to_json();
</Input>

<Output to_file>
    Module        om_file
    File          'C:\outputfile.txt.gz'
    OutputType    zlib.compress
</Output>

The decompression procedure can be performed using the xm_zlib module as well, as shown below.

Example 2. Decompressing Logs

In this configuration, decompression is performed by the from_file input instance of the im_file module when files are read.

Following the sequence of operations listed in the table above for decompression, the InputType directive in the from_file input instance invokes the decompress data converter using the zlib instance of the xm_zlib module to decompress the log data. It then reads the events using the LineBased input reader function, the default for im_file.

After successful decompression and reading, events can be processed as usual. In this case, the log data is forwarded over the network with UDP using the om_udp module.

nxlog.conf

<Extension zlib>
    Module        xm_zlib
</Extension>

<Input from_file>
    Module        im_file
    File          'C:\inputtfile.txt.gz'
    InputType     zlib.decompress
</Input>

<Output to_udp>
    Module        om_udp
    Host          192.168.31.11:10500
</Output>

Encrypting and Decrypting Logs

To provide better security for logs, they can be encrypted using the xm_crypto module. This module utilizes the AES symmetric-key algorithm.

Log data can be encrypted and saved to file using the om_file module. Decryption can be performed using the im_file module.

The examples below demonstrate how to configure NXLog to successfully encrypt and decrypt log data.

The following table shows the order of operations for encrypting and decrypting log data comprised of single-line events.

Table 2. Sequential order of operations for encryption and decryption
	Directive	First Operation	Second Operation
Encryption	OutputType	LineBased	aes_encrypt
Decryption	InputType	aes_decrypt	LineBased

Example 3. Encrypting Logs

In this example NXLog receives data via the network using the im_tcp module. The received data is formatted as JSON using the to_json() procedure of the xm_json module.

Following the sequence of operations listed in the table above for encryption, the OutputType directive in the to_file output instance outputs the events using the LineBased output writer function, the deault for om_file. It then invokes the aes_encrypt data converter using the crypto instance of the xm_crypto module to encrypt the log data. Since the optional Password directive is specified in the crypto instance along with a password, it will be applied during the encryption process.

Finally, the encrypted output is saved as a file specified by the mandatory File directive.

nxlog.conf

<Extension json>
    Module        xm_json
</Extension>

<Extension crypto>
    Module        xm_crypto
    Password      <PASSWORD_TO_ENCRYPT>
</Extension>

<Input from_tcp>
    Module        im_tcp
    ListenAddr    192.168.31.25:10500
    Exec          to_json();
</Input>

<Output to_file>
    Module        om_file
    File          '/tmp/output'
    OutputType    crypto.aes_encrypt
</Output>

Decryption of logs is merely the inverse procedure of encrypting logs, which NXLog can perform as well. This example shows how to decrypt log data.

Example 4. Decrypting Logs

Following the sequence of operations listed in the table above for decryption, the InputType directive in the from_file input instance invokes the aes_decrypt data converter using the crypto instance of the xm_crypto module to decrypt the log data. Since this data was encrypted with a password, the optional Password directive is specified in the crypto instance along with the required password. Once the data has been decrypted, it then reads the events using the LineBased input reader function, the default for im_file.

Once the operations specified by the InputType directive have been completed, the events can be processed as usual. In this case, the log data is sent over UDP using the om_udp module.

nxlog.conf

<Extension json>
    Module        xm_json
</Extension>

<Extension crypto>
    Module        xm_crypto
    Password      <PASSWORD_TO_DECRYPT>
</Extension>

<Input from_file>
    Module        im_file
    File          '/tmp/input'
    InputType     crypto.aes_decrypt
</Input>

<Output to_udp>
    Module        om_udp
    Host          192.168.11.31:10500
</Output>

Compression and Encryption of Logs

The previous examples show how compression and encryption can be performed separately. In some cases, to improve workflow these might need to be combined together. Both operations can be coupled and applied sequentially to the same log data.

The following table shows the order of operations for compressing and encrypting log data comprised of single-line events.

Table 3. Sequential order of operations for compression and encryption
	Directive	First Operation	Second Operation	Third Operation
Compression + Encryption	OutputType	LineBased	compress	aes_encrypt

Below is an example of the NXLog configuration to compress and encrypt the log data.

Example 5. Compressing and Encrypting Logs

This example contains configuration to accept incoming UDP datagrams using the im_udp module. The logs formatted as JSON using the to_json() procedure of the xm_json module.

Following the sequence of operations listed in the table above for compression and encryption, the OutputType directive in the to_file output instance outputs the events using the LineBased output writer function, the default for om_file. It then invokes the compress data converter using the zlib instance of the xm_zlib module to compress the log data.

Once the log data is compressed, it invokes the aes_encrypt data converter using the crypto instance of the xm_crypto module to encrypt the log data. Since the optional Password directive is specified in the crypto instance along with a password, it will be applied during the encryption process.

Finally, the encrypted output is saved as a file specified by the mandatory File directive.

nxlog.conf

<Extension json>
    Module        xm_json
</Extension>

<Extension crypto>
    Module        xm_crypto
    Password      <PASSWORD_TO_ENCRYPT>
</Extension>

<Extension zlib>
    Module        xm_zlib
</Extension>

<Input from_udp>
    Module        im_udp
    ListenAddr    192.168.31.25:10500
    Exec          to_json();
</Input>

<Output to_file>
    Module        om_file
    File          '/tmp/output'
    OutputType    zlib.compress, crypto.aes_encrypt
</Output>

Decompression and Decryption of Logs

Compressed and encrypted data may later need to be processed. In this case, it needs to be decompressed and decrypted. The example below explains how to configure NXLog to perform these procedures.

The following table shows the order of operations for decompressing and decrypting log data comprised of single-line events.

Table 4. Sequential order of operations for decompression and decryption
	Directive	First Operation	Second Operation	Third Operation
Decompression + Decryption	InputType	aes_decrypt	decompress	LineBased

Example 6. Decompressing and Decrypting Logs

After the data has been decrypted, it invokes the decompress data converter using the zlib instance of the xm_zlib module to decompress the log data. It then reads the events using the LineBased input reader function, the default for im_file.

Finally, after restoring the events to their original state, they are transmitted over TCP using the om_tcp module to a remote host.

nxlog.conf

<Extension json>
    Module        xm_json
</Extension>

<Extension crypto>
    Module        xm_crypto
    Password      <PASSWORD_TO_DECRYPT>
</Extension>

<Extension zlib>
    Module        xm_zlib
</Extension>

<Input from_file>
    Module        im_file
    File          '/tmp/input'
    InputType     crypto.aes_decrypt, zlib.decompress
</Input>

<Output to_tcp>
    Module        om_tcp
    Host          192.168.31.11:10500
</Output>

Multiple Encryption and Compression Procedures Combined

In some cases, log data may need to be decrypted and decompressed to be processed, and then encrypted and compressed again once processing is complete. See the example below on how NXLog can be configured to achieve this process.

The following table shows the order of operations for compressing/encrypting, and decompressing/decrypting log data comprised of single-line events.

Table 5. Sequential order of operations for compression/encryption and decompression/decryption
	Directive	First Operation	Second Operation	Third Operation
Compression + Encryption	OutputType	LineBased	compress	aes_encrypt
Decompression + Decryption	InputType	aes_decrypt	decompress	LineBased

Example 7. Performing Multiple Encryption and Compression Procedures

In this example, it is assumed that the input data is log data comprised of single-line events that have been compressed and encrypted with a password.

Following the sequence of operations listed in the table above for decompression and decryption, the InputType directive in the from_file input instance invokes the aes_decrypt data converter using the crypto instance of the xm_crypto module to decrypt the log data. Since this data was encrypted with a password, the optional Password directive is specified in the crypto instance along with the required password.

After the data has been decrypted, it invokes the decompress data converter using the zlib instance of the xm_zlib module to decompress the log data. It then reads events using the LineBased input reader function, the default for im_file.

At this stage, the events are now ready for further processing or filtering as needed. In this case, the Exec block of the from_file input instance instructs NXLog to discard messages which do not contain the error string by using the drop() procedure. Events that are not dropped are then ready for output where they will be compressed and encrypted.

Following the sequence of operations listed in the table above for compression and encryption, the OutputType directive in the to_file output instance outputs events using the LineBased output write function, the default for om_file. It then invokes the compress data converter using the zlib instance of the xm_zlib module to compress the log data.

Finally, the filtered, compressed, and encrypted output is saved as a file specified by the mandatory File directive.

nxlog.conf

<Extension json>
    Module        xm_json
</Extension>

<Extension crypto>
    Module        xm_crypto
    Password      <PASSWORD_TO_DECRYPT>
</Extension>

<Extension zlib>
    Module        xm_zlib
</Extension>

<Input from_file>
    Module        im_file
    File          '/tmp/input_all'
    InputType     crypto.aes_decrypt, zlib.decompress
    Exec          if not ($raw_event =~ /error/) drop();
</Input>

<Output to_file>
    Module        om_file
    File          '/tmp/output_error'
    OutputType    zlib.compress, crypto.aes_encrypt
</Output>