Raijin (om_raijin)
This module enables logs to be forwarded to Raijin Database servers for ingestion. It connects to the URL specified in the configuration in either HTTP or HTTPS mode. Raijin accepts HTTP POST requests with multiple JSON records in the request body, assuming that there is a target database table already created on the Raijin side. Although Raijin accepts structured data in flat JSON (i.e. a list of key-value pairs) format, it does not support JSON values comprised of nested data structures or other non-scalar data such as arrays and maps. Raijin currently does not support authorization/SSL but the om_raijin module supports TLS since TLS can be enabled with an HTTP proxy.
|
This module requires the xm_json extension module to be loaded to convert the payload to JSON. See the Output log format section for information on the payload format. |
Output log format
om_raijin forwards log records over HTTP(S) as JSON payload.
The JSON format depends on the value of the $raw_event field.
The module checks if the value of $raw_event is valid JSON and applies the following rules:
-
If it is valid JSON, the value is forwarded as is.
-
If it is not valid JSON, the log record is converted to JSON in the following format:
{ "raw_event": "<json_escaped_raw_event>" }
Additional metadata, including the NXLog Agent-specific fields EventReceivedTime, SourceModuleName, and SourceModuleType, will not be included in the output unless these values have been written to the $raw_event field.
The processing required to achieve this depends on the format of the input data, but generally, it means you need to:
-
Parse the log record according to the data format.
-
If the input data is already in JSON format, use parse_json() to parse
$raw_eventinto fields. -
If the input is unstructured plain text data, copy the value of
$raw_eventto a custom field.
-
-
Create and populate any additional custom fields.
-
Use to_json() to convert the fields to JSON format and update the value of
$raw_event.
See the Examples section for NXLog Agent configuration examples of the above.
Configuration
The om_raijin module accepts the following directives in addition to the common module directives.
Required directives
The following directives are required for the module to start.
This mandatory directive specifies the name of the Raijin database containing the table where logs will be stored. |
|||
This mandatory directive specifies the name of the database table where log events will be inserted as JSON records. |
|||
This mandatory directive is comprised of the REST API URI that Raijin Server uses for receiving bulk, JSON-formatted events via POST requests: If an om_raijin module instance contains multiple URL directives, the hosts will function as a cluster in failover mode.
This directive also supports HTTPS URLs.
If the optional port number part of the URL (
|
HTTPS directives
The following directives configure secure data transfer via HTTPS.
Set this directive to |
||||||||||||||||
Set this directive to |
||||||||||||||||
Set this directive to |
||||||||||||||||
Set this directive to the path of a directory containing Certificate Authority (CA) certificates.
The module uses these certificates to verify the certificate presented by the remote host.
Name the certificate files using the OpenSSL hashed format: the hash of the certificate followed by For example, if the certificate hash is To trust a remote host’s self-signed certificate, include a copy of it in this directory. If this directive isn’t specified, NXLog Agent falls back to the default operating system root certificate store.
Unix-like operating systems commonly store root certificates in In addition, Microsoft’s PKI repository contains root certificates for Microsoft services. This directive is mutually exclusive with the HTTPSCAThumbprint, HTTPSCAPattern, and HTTPSSearchAllCertStores directives. |
||||||||||||||||
Set this directive to the path of the Certificate Authority (CA) certificate used to verify the certificate presented by the remote host. To trust a remote host’s self-signed certificate, specify the remote host’s certificate itself. For certificates signed by an intermediate CA, the specified certificate must contain the complete certificate chain (certificate bundle). This directive is mutually exclusive with the HTTPSCAThumbprint, HTTPSCAPattern, and HTTPSSearchAllCertStores directives. |
||||||||||||||||
Set this directive to a PCRE2-compliant regular expression to locate a suitable Certificate Authority (CA) certificate. This directive is supported on Windows and macOS. On Windows, the certificate must be in the Windows Certificate Store. On macOS, the certificate must be in the system keychain. NXLog Agent matches the pattern against a string built from the certificate’s fields in this order: The pattern doesn’t need to match every field.
Include only the fields you want to match, and use For example, the following pattern matches only the This produces the following output in the NXLog Agent log file:
If the pattern matches multiple certificates, NXLog Agent uses the first match. If the certificate changes, restart NXLog Agent to apply the new certificate. This directive is mutually exclusive with the HTTPSCAThumbprint, HTTPSCADir, HTTPSCAFile, and HTTPSSearchAllCertStores directives.
|
||||||||||||||||
Set this directive to the thumbprint of the Certificate Authority (CA) certificate used to verify the certificate presented by the remote host. This directive is supported on Windows and macOS. On Windows, copy the hexadecimal fingerprint string from Windows Certificate Manager (certmgr.msc). The certificate must be in a Windows certificate store accessible by NXLog Agent. On macOS, obtain the hexadecimal fingerprint string from the Keychain Access application. The certificate must be accessible in the system keychain. Whitespaces and colons are automatically removed. This directive is mutually exclusive with the HTTPSCADir, HTTPSCAFile, HTTPSCAPattern, and HTTPSSearchAllCertStores directives.
|
||||||||||||||||
Set this directive to the path of the certificate file to present to the remote host during the HTTPS handshake. |
||||||||||||||||
Set this directive to the path of the private key file corresponding to the certificate specified by HTTPSCertFile. |
||||||||||||||||
Set this directive to a PCRE2-compliant pattern to identify the certificate and its corresponding private key. This directive is supported on Windows and macOS. NXLog Agent matches the pattern against a string built from the certificate’s fields in this order: The pattern doesn’t need to match every field.
Include only the fields you want to match, and use On Windows, import the certificate in PFX format into the On macOS, ensure the certificate and its private key are present in the System Keychain ( During configuration, NXLog Agent resolves this directive into the corresponding HTTPSCertThumbprint value. If the pattern matches multiple certificates, NXLog Agent uses the first match. This directive is not dynamic; restart NXLog Agent to apply a changed certificate. We recommend keeping the certificate storage referenced by this directive well-maintained for optimal performance. This directive is mutually exclusive with the HTTPSCertThumbprint, HTTPSCertFile and HTTPSCertKeyFile directives. Configuration examples: or This produces the following output in the NXLog Agent log file:
|
||||||||||||||||
Set this directive to the thumbprint of the certificate to present to the remote host during the HTTPS handshake. Whitespaces and colons are automatically removed. This directive is supported on Windows and macOS. On Windows, copy the hexadecimal fingerprint string from Windows Certificate Manager (certmgr.msc).
Import the certificate to the When the global directive UseCNGCertificates is set to
When UseCNGCertificates is set to On macOS, obtain the hexadecimal fingerprint string from the Keychain Access application.
The certificate and its associated private key must be present in the System Keychain ( For example:
|
||||||||||||||||
Set this directive to the path of a directory containing certificate revocation list (CRL) files.
The module uses these files to reject connections from hosts presenting a revoked certificate.
Name the files using the OpenSSL hashed format: the hash of the issuer followed by For example, if the hash is |
||||||||||||||||
Set this directive to the path of the certificate revocation list (CRL) file used to reject connections from hosts presenting a revoked certificate. To generate a CRL file using OpenSSL: |
||||||||||||||||
Set this directive to the path of a file containing Diffie-Hellman parameters for key exchange. Generate the parameters using dhparam(1ssl). If this directive is not set, the module uses default parameters. See the OpenSSL Wiki for further details. |
||||||||||||||||
Set this directive to the passphrase of the private key specified by HTTPSCertKeyFile. This directive applies only to encrypted private keys. To generate a private key with Triple DES encryption using OpenSSL: |
||||||||||||||||
Set this directive to This directive is only supported on Windows. |
||||||||||||||||
Set this directive to |
||||||||||||||||
Set this directive to |
||||||||||||||||
Set this directive to override the permitted cipher list for TLSv1.2 and below.
Use the format described in the ciphers(1ssl) man page.
For example, specify
|
||||||||||||||||
Set this directive to override the permitted cipher list for TLSv1.3. Use the same format as HTTPSSSLCipher. Refer to the OpenSSL documentation for a list of valid TLS v1.3 cipher suites. The default is:
|
||||||||||||||||
Set this directive to
|
||||||||||||||||
Set this directive to restrict the TLS/SSL protocols the module accepts.
Specify a comma-separated list of the following values: |
||||||||||||||||
Set this directive to This directive depends on the values of the HTTPSAllowUntrusted and HTTPSRequireCert directives:
|
||||||||||||||||
Set this directive to the signature algorithm parameter to pass to the Windows SSL library. Accepted values depend on the available encryption providers. This directive is only supported on Windows. |
||||||||||||||||
Set this directive to the hostname to use for Server Name Indication (SNI) in HTTPS mode. If not set, the module uses the hostname specified in the URL directive. |
||||||||||||||||
Set this directive to |
Optional directives
This directive specifies the username that the incoming data are associated with. If authentication is enabled, then it will be also used in the authentication process. |
|||
This optional directive specifies the password for the user defined by |
|||
Set this directive to the local port number to use for the connection, for example to satisfy a firewall rule that expects traffic from a known port. If you do not set this directive, the module uses a random high port number, which a firewall may block.
|
|||
Specifies the IP address of the proxy server in case the module needs to connect to a Raijin Database server through a proxy.
|
|||
Specifies the port number required to connect to the proxy server. |
|||
Set this directive to the number of seconds to wait between reconnection attempts. If you do not set this directive, the module starts with a 1 second interval and doubles it after each failed attempt, resetting to 1 second once it reconnects successfully.
|
|||
Set this directive to |
|||
Set this directive to
The default is |
Examples
This configuration reads log messages from a file and forwards them to a Raijin Database server deployed on localhost.
No further processing is done on the log records.
<Extension json>
Module xm_json
</Extension>
<Input file>
Module im_file
File '/var/log/myapp*.log'
</Input>
<Output raijin_local>
Module om_raijin
URL http://localhost:2500
DBName logdb
DBTable logs
</Output>The following is a log record sample read by NXLog Agent.
Mar 24 15:58:53 pc1 systemd[1452]: tracker-store.service: Succeeded.The following is the JSON-formatted log record that will be sent to the Raijin Database server.
{
"raw_event": "Mar 24 15:58:53 pc1 systemd[1452]: tracker-store.service: Succeeded."
}This configuration reads log records from a file and adds a $Hostname metadata field.
Log records are converted to JSON using the to_json() procedure of the xm_json module before they are forwarded to the Raijin Database server.
<Extension json>
Module xm_json
</Extension>
<Input file>
Module im_file
File '/var/log/myapp*.log'
Exec $Hostname = hostname();
Exec $Message = $raw_event;
</Input>
<Output raijin_local>
Module om_raijin
URL http://localhost:2500
DBName logdb
DBTable logs
Exec to_json();
</Output>The following is a log record sample read by NXLog Agent.
Mar 24 15:58:53 pc1 systemd[1452]: tracker-store.service: Succeeded.The following is the JSON-formatted log record that will be sent to the Raijin Database server.
{
"EventReceivedTime": "2021-03-24T16:52:20.457348+01:00",
"SourceModuleName": "file",
"SourceModuleType": "im_file",
"Hostname": "pc1",
"Message": "Mar 24 15:58:53 pc1 systemd[1452]: tracker-store.service: Succeeded."
}This configuration reads syslog records from a file. It uses the parse_syslog() procedure of the xm_syslog module to parse logs into structured data. Log records are then converted to JSON using the to_json() procedure of the xm_json module before they are forwarded to the Raijin Database server.
<Extension syslog>
Module xm_syslog
</Extension>
<Extension json>
Module xm_json
</Extension>
<Input file>
Module im_file
File '/var/log/myapp*.log'
Exec parse_syslog();
</Input>
<Output raijin_local>
Module om_raijin
URL http://localhost:2500
DBName logdb
DBTable logs
Exec to_json();
</Output>The following is a log record sample read by NXLog Agent.
Mar 24 15:58:53 pc1 systemd[1452]: tracker-store.service: Succeeded.The following is the JSON-formatted log record that will be sent to the Raijin Database server.
{
"EventReceivedTime": "2021-03-24T16:30:18.920342+01:00",
"SourceModuleName": "file",
"SourceModuleType": "im_file",
"SyslogFacilityValue": 1,
"SyslogFacility": "USER",
"SyslogSeverityValue": 5,
"SyslogSeverity": "NOTICE",
"SeverityValue": 2,
"Severity": "INFO",
"Hostname": "pc1",
"EventTime": "2021-03-24T15:58:53.000000+01:00",
"SourceName": "systemd",
"ProcessID": 1452,
"Message": "tracker-store.service: Succeeded."
}This configuration reads JSON-formatted log records from a file.
It uses the parse_json() procedure of the xm_json module to parse logs into structured data and adds an $EventType metadata field.
Log records are then converted back to JSON using the to_json() procedure before they are forwarded to the Raijin Database server.
<Extension json>
Module xm_json
</Extension>
<Input file>
Module im_file
File '/var/log/myapp*.log'
Exec parse_json();
Exec $EventType = "browser-history";
</Input>
<Output raijin_local>
Module om_raijin
URL http://localhost:2500
DBName logdb
DBTable logs
Exec to_json();
</Output>The following is a log record sample read by NXLog Agent.
{
"AccessTime": "2021-03-24T16:30:43.000000+01:00",
"URL": "https://nxlog.co",
"Title": "High Performance Log Collection Solutions",
"Username": "user1"
}The following is the JSON-formatted log record that will be sent to the Raijin Database server.
{
"EventReceivedTime": "2021-03-24T17:14:23.908155+01:00",
"SourceModuleName": "file",
"SourceModuleType": "im_file",
"AccessTime": "2021-03-24T16:30:43.000000+01:00",
"URL": "https://nxlog.co",
"Title": "High Performance Log Collection Solutions",
"Username": "user1",
"EventType": "browser-history"
}This configuration sends logs to a cluster of Raijin Database servers in failover mode.
The cluster is comprised of three hosts: http://raijin-1.example.com:2500, http://localhost:2500, and http://192.168.1.123:2500.
<Extension json>
Module xm_json
</Extension>
<Output raijin_cluster>
Module om_raijin
URL http://raijin-1.example.com:2500
URL http://localhost:2500
URL http://192.168.1.123:2500
DBName logdb
DBTable logs
</Output>