Reading and receiving logs
This chapter discusses log sources that you may need to use with NXLog, including:
log data received over the network,
events stored in databases,
messages read from files, and
data retrieved using executables.
Receiving over the network
This section provides information and examples about receiving log messages from the network over various protocols.
The im_udp module handles incoming messages over UDP.Example 1. Using the im_udp module
This input module instance shows the im_udp module configured with the default options: localhost only and port 514.nxlog.conf
<Input udp> Module im_udp Host localhost Port 514 </Input>
The UDP protocol does not guarantee reliable message delivery. It is recommended to use the TCP or SSL transport modules instead if message loss is a concern.
Though NXLog was designed to minimize message loss even in the case of UDP, adjusting the kernel buffers may reduce the likelihood of UDP message loss on a system under heavy load. The Priority directive in the Route block can also help.
The im_ssl module handles incoming messages over TCP with SSL/TLS security.Example 3. Using the im_ssl module
The following input module instance listens for SSL/TLS encrypted incoming logs on port 6514. The certificate file paths are specified relative to a previously defined
<Input in> Module im_ssl Host 0.0.0.0 Port 6514 CAFile %CERTDIR%/ca.pem CertFile %CERTDIR%/client-cert.pem CertKeyFile %CERTDIR%/client-key.pem </Input>
To receive Syslog over the network, use one of the network modules above, coupled with xm_syslog. Syslog parsing is not required if you only need to forward or store the messages as they are. See also Accepting Syslog via UDP, TCP, or TLS.Example 4. Receiving syslog over TCP with octet-framing
With this example configuration, NXLog listens for messages on TCP port 1514. The xm_syslog extension module provides the Syslog_TLS InputType (for octet-framing) and the parse_syslog() procedure for parsing Syslog messages.nxlog.conf
<Extension _syslog> Module xm_syslog </Extension> <Output out> Module om_tcp Host 192.168.1.1 Port 1514 Exec to_syslog_ietf(); </Output>
Reading from a database
With the im_dbi and im_odbc modules it is possible to read logs directly from database servers. The im_dbi module can be used on POSIX systems where libdbi is available. The im_odbc module, available in NXLog Enterprise Edition, can be used with ODBC compatible databases on Windows, Linux, and Unix.
This example uses libdbi and the MySQL driver to read records from
<Input in> Module im_dbi Driver mysql Option host 127.0.0.1 Option username mysql Option password mysql Option dbname logdb SQL SELECT id, facility, severity, hostname, timestamp, application, \ message FROM log </Input>
mydb database is accessed via ODBC.
<Input in> Module im_odbc ConnectionString DSN=mssql;database=mydb; SQL SELECT RecordNumber as id, DateOccured as EventTime, \ data as Message from logtable WHERE RecordNumber > ? </Input>
Reading from files and sockets
The im_file module can be used to read logs from files. See also Reading Syslog Log Files.Example 7. Using the im_file module
This example reads from the specified file without performing any additional processing.nxlog.conf
<Input in> Module im_file File "/var/log/messages" </Input>
- Unix domain socket
Use the im_uds module to read from a Unix domain socket. See also Accepting Syslog via /dev/log.Example 8. Using the im_uds module
With this configuration, NXLog will read messages from the
/dev/logsocket. NXLog’s flow control feature must be disabled in this case (see the FlowControl directive in the Reference Manual).nxlog.conf
<Input in> Module im_uds UDS /dev/log FlowControl FALSE </Input>
Receiving from an executable
The im_exec module can be used to read logs from external programs and scripts over a pipe.