Redis (om_redis)

This module can store data in a Redis server. It issues RPUSH commands using the Redis Protocol to send data.

The input counterpart, im_redis, can be used to retrieve data from a Redis server.

Configuration

The om_redis module accepts the following directives in addition to the common module directives. The Host directive is required.

Required directives

The following directive is required for the module to start.

Host

This mandatory directive specifies the IP address or DNS hostname of the Redis server to connect to.

Optional directives

Channel

This directive is interpreted the same way as the Key directive, i.e., it can be an expression that evaluates to a string. The module will publish records to a Redis channel with this name. The default is nxlog. The Command directive must be set to PUBLISH when using this directive.

Command

This optional directive specifies the command to be used. The possible commands are LPUSH, RPUSH (the default), LPUSHX, RPUSHX, and PUBLISH.

Key

This specifies the Key used by the RPUSH command. It must be a string type expression. If the expression in the Key directive is not a constant string (it contains functions field names, or operators), it will be evaluated for each event to be inserted. The default is nxlog. The usage of this directive is mutually exclusive with the usage of the PUBLISH command in the Command directive.

LocalPort

Set this directive to the local port number to use for the connection, for example to satisfy a firewall rule that expects traffic from a known port. If you do not set this directive, the module uses a random high port number, which a firewall may block.

Attempts to bind to LocalPort may fail because of the required TIME-WAIT delay in closing connections. When this happens, the module writes Address already in use to the NXLog Agent log file. If it persists, it can degrade network performance.

OutputType

See the OutputType directive in the list of common module directives. If this directive is not specified, the default Dgram formatter function is used, which writes the value of $raw_event without a line terminator. You can use Binary to preserve structured data if the logs are intended to be read from Redis by another NXLog Agent instance.

Port

This specifies the port number of the Redis server. The default is port 6379.

Reconnect

Set this directive to the number of seconds to wait between reconnection attempts. If you do not set this directive, the module starts with a 1 second interval and doubles it after each failed attempt, resetting to 1 second once it reconnects successfully.

Setting this directive on multiple systems can cause them to send reconnection requests to the same destination at the same time, potentially overloading it. This can also cause NXLog Agent to consume unusually high system resources or become unresponsive.

ReconnectOnData

Set this directive to TRUE to have the module reconnect to the remote host only when it has data to send, instead of keeping the connection open at all times. The default is FALSE; the module always keeps a connection open with the remote host.

TcpSendStallDetect

Set this directive to TRUE to monitor for pipeline backpressure by logging a warning whenever the module cannot send data to the destination. The module writes the following message to the NXLog Agent log file:

TCP send buffer full for <instance_name> - write blocked

The default is FALSE; the module does not check for send stalls.

Procedures

The following procedures are exported by om_redis.

reconnect();

Force a reconnection. This can be used from a Schedule block to periodically reconnect to the server.

The reconnect() procedure must be used with caution. If configured, it can attempt to reconnect after every event sent, potentially overloading the destination system.

Examples

Example 1. Sending logs to Redis with the default configuration

This configuration only sets the mandatory Host directive to the Redis server IP address. The module default will use the RPUSH command to write the logs to a key named nxlog.

nxlog.conf

<Input test_input>
    Module      im_testgen
    MaxCount    2
</Input>

<Output redis>
    Module      om_redis
    Host        192.168.1.213
</Output>

Output sample

127.0.0.1:6379> lrange nxlog 0 -1
1) "0@Wed Jul 13 16:47:59 2022"
2) "1@Wed Jul 13 16:47:59 2022"

Example 2. Sending logs to Redis with a custom command and key

The configuration below sets the mandatory Host directive to the Redis server IP address. It also specifies the Command and Key directives to use the LPUSH command for writing logs to a key named event_logs.

nxlog.conf

<Input test_input>
    Module      im_testgen
    MaxCount    2
</Input>

<Output redis>
    Module      om_redis
    Host        192.168.1.213
    Command     LPUSH
    Key         'event_logs'
</Output>

Output sample

127.0.0.1:6379> lrange event_logs 0 -1
1) "1@Wed Jul 13 16:52:39 2022"
2) "0@Wed Jul 13 16:52:39 2022"

Example 3. Publishing logs to a custom channel

The configuration below sets the mandatory Host directive to the Redis server IP address. It also specifies the Command and Channel directives for publishing logs to the event_logs Redis channel.

nxlog.conf

<Input test_input>
    Module      im_testgen
    MaxCount    2
</Input>

<Output redis>
    Module      om_redis
    Host        192.168.1.213
    Command     PUBLISH
    Channel     'event_logs'
</Output>

Output sample

127.0.0.1:6379> subscribe event_logs
Reading messages... (press Ctrl-C to quit)
1) "subscribe"
2) "event_logs"
3) (integer) 1
1) "message"
2) "event_logs"
3) "0@Wed Jul 13 17:00:56 2022"
1) "message"
2) "event_logs"
3) "1@Wed Jul 13 17:00:56 2022"