DBI (im_dbi)
The im_dbi module allows NXLog Agent to pull log data from external databases. This module utilizes the libdbi database abstraction library, which supports various database engines such as MySQL, PostgreSQL, Oracle, SQLite, and Firebird. A SELECT statement can be specified, which will be executed periodically to check for new records.
The im_dbi and om_dbi modules support GNU/Linux only because of the libdbi library. The im_odbc and om_odbc modules provide native database access on Windows. |
libdbi needs drivers to access the database engines. These are in the libdbd-* packages on Debian and Ubuntu. CentOS 5.6 has a libdbi-drivers RPM package, but this package does not contain any driver binaries under /usr/lib64/dbd. The drivers for both MySQL and PostgreSQL are in libdbi-dbd-mysql. If these are not installed, NXLog Agent will return a libdbi driver initialization error. |
Configuration
The im_dbi module accepts the following directives in addition to the common module directives.
Required directives
The following directives are required for the module to start.
This mandatory directive specifies the name of the libdbi driver that will be used to connect to the database.
A DRIVER name must be provided here for which a loadable driver module exists under the name |
|
This directive should specify the SELECT statement to be executed every PollInterval seconds.
The module automatically appends a |
Optional directives
This directive can be used to specify additional connection options.
Refer to the libdbi driver documentation of your database engine for supported connection options.
The module works in a failover configuration if you specify the |
|
This directive specifies how frequently the module will check for new records, in seconds.
If this directive is not specified, the default is 1 second.
Fractional seconds may be specified ( |
|
If this boolean directive is set to |
Creating and populating fields
When the im_dbi module reads a record from a database, it creates and populates the fields returned by the SQL statement.
The fields can be used for further processing or to convert the log record to a different output format, such as JSON or XML.
Additionally, the $raw_event
core field is populated as follows:
EventTime Hostname Severity field1=value1 field2=value2
If the EventTime
, Hostname
, and Severity
fields are not returned as part
of the database record, the following default values are used:
- EventTime
-
The value of the
$EventReceivedTime
core field. - Hostname
-
The value returned by the hostname() function.
- Severity
-
INFO
The configuration Examples below demonstrate how database log records can be processed with NXLog Agent.
Fields
The following fields are used by im_dbi.
$raw_event
(type: string)-
A list of event fields in key-value pairs.
Examples
This example uses im_dbi and the MySQL driver to connect to the logdb
database and retrieve logs from table log
. Log records are written to file
without any further processing.
<Input dbi>
Module im_dbi
Driver mysql
Option host 127.0.0.1
Option username mysql
Option password mysql
Option dbname logdb
SQL SELECT id, timestamp AS EventTime, message \
FROM log
</Input>
<Output file>
Module om_file
File '/path/to/output/file'
</Output>
id | timestamp | message |
---|---|---|
1234 |
2021-11-08T08:42:20 |
The service started successfully |
2021-11-08 08:42:20 SERVER-01 INFO message="The service started successfully"
This example uses im_dbi to connect to a database logdb
and retrieve logs from table log
.
If the log record has a severity equal to DEBUG
, it is dropped.
Otherwise, the log record is converted to JSON using the to_json() procedure of the xm_json module.
<Extension json>
Module xm_json
</Extension>
<Input dbi>
Module im_dbi
Driver mysql
Option host 127.0.0.1
Option username mysql
Option password mysql
Option dbname logdb
SQL SELECT id, Timestamp AS EventTime, \
LogLevel AS Severity, Message \
FROM log
Exec to_json();
</Input>
id | Timestamp | LogLevel | Message |
---|---|---|---|
1234 |
2021-11-08T09:12:05 |
INFO |
The service started successfully |
{
"id": 1234,
"EventTime": "2021-11-08T09:12:05.000000+01:00",
"Severity": "INFO",
"Message": "The service started successfully",
"EventReceivedTime": "2021-11-08T09:13:16.332925+01:00",
"SourceModuleName": "odbc",
"SourceModuleType": "im_odbc"
}