Reading and receiving logs

This chapter discusses log sources that you may need to use with NXLog, including:

log data received over the network,
events stored in databases,
messages read from files, and
data retrieved using executables.

Receiving over the network

This section provides information and examples about receiving log messages from the network over various protocols.

UDP

The im_udp module handles incoming messages over UDP.

Example 1. Using the im_udp module

This input module instance shows the im_udp module configured with the default options: localhost only and port 514.

nxlog.conf

<Input udp>
    Module  im_udp
    Host    localhost
    Port    514
</Input>

The UDP protocol does not guarantee reliable message delivery. It is recommended to use the TCP or SSL transport modules instead if message loss is a concern.

Though NXLog was designed to minimize message loss even in the case of UDP, adjusting the kernel buffers may reduce the likelihood of UDP message loss on a system under heavy load. The Priority directive in the Route block can also help.

TCP

The im_tcp module handles incoming messages over TCP. For TLS/SSL, use the im_ssl module.

Example 2. Using the im_tcp module

This input module instance accepts TCP connections from any host on port 1514.

nxlog.conf

<Input tcp>
    Module  im_tcp
    Host    0.0.0.0
    Port    1514
</Input>

SSL/TLS

The im_ssl module handles incoming messages over TCP with SSL/TLS security.

Example 3. Using the im_ssl module

The following input module instance listens for SSL/TLS encrypted incoming logs on port 6514. The certificate file paths are specified relative to a previously defined CERTDIR.

nxlog.conf

<Input in>
    Module      im_ssl
    Host        0.0.0.0
    Port        6514
    CAFile      %CERTDIR%/ca.pem
    CertFile    %CERTDIR%/client-cert.pem
    CertKeyFile %CERTDIR%/client-key.pem
</Input>

Syslog

To receive Syslog over the network, use one of the network modules above, coupled with xm_syslog. Syslog parsing is not required if you only need to forward or store the messages as they are. See also Accepting Syslog via UDP, TCP, or TLS.

Example 4. Receiving syslog over TCP with octet-framing

With this example configuration, NXLog listens for messages on TCP port 1514. The xm_syslog extension module provides the Syslog_TLS InputType (for octet-framing) and the parse_syslog() procedure for parsing Syslog messages.

nxlog.conf

<Extension _syslog>
    Module  xm_syslog
</Extension>

<Output out>
    Module  om_tcp
    Host    192.168.1.1
    Port    1514
    Exec    to_syslog_ietf();
</Output>

Reading from a database

With the im_dbi and im_odbc modules it is possible to read logs directly from database servers. The im_dbi module can be used on POSIX systems where libdbi is available. The im_odbc module, available in NXLog Enterprise Edition, can be used with ODBC compatible databases on Windows, Linux, and Unix.

Example 5. Using the im_dbi module

This example uses libdbi and the MySQL driver to read records from the logdb database.

nxlog.conf

<Input in>
    Module  im_dbi
    Driver  mysql
    Option  host 127.0.0.1
    Option  username mysql
    Option  password mysql
    Option  dbname logdb
    SQL     SELECT id, facility, severity, hostname, timestamp, application, \
                   message FROM log
</Input>

Example 6. Using the im_odbc module

NXLog Enterprise Edition exclusive feature

Here, the mydb database is accessed via ODBC.

nxlog.conf

<Input in>
    Module              im_odbc
    ConnectionString    DSN=mssql;database=mydb;
    SQL                 SELECT RecordNumber as id, DateOccured as EventTime, \
                               data as Message from logtable WHERE RecordNumber > ?
</Input>

Reading from files and sockets

Files

The im_file module can be used to read logs from files. See also Reading Syslog Log Files.

Example 7. Using the im_file module

This example reads from the specified file without performing any additional processing.

nxlog.conf

<Input in>
    Module  im_file
    File    "/var/log/messages"
</Input>

Unix domain socket

Use the im_uds module to read from a Unix domain socket. See also Accepting Syslog via /dev/log.

Example 8. Using the im_uds module

With this configuration, NXLog will read messages from the /dev/log socket. NXLog’s flow control feature must be disabled in this case (see the FlowControl directive in the Reference Manual).

nxlog.conf

<Input in>
    Module      im_uds
    UDS         /dev/log
    FlowControl FALSE
</Input>

Receiving from an executable

The im_exec module can be used to read logs from external programs and scripts over a pipe.

Example 9. Using the im_exec module

This example uses the tail command to read messages from a file.

The im_file module should be used to read log messages from files. This example only demonstrates the use of the im_exec module.

nxlog.conf

<Input in>
    Module  im_exec
    Command /usr/bin/tail
    Arg     -f
    Arg     /var/log/messages
</Input>