NXLog Legacy Documentation

Raijin Database Engine

NXLog Enterprise Edition exclusive feature

The Raijin Database engine was designed to work with high-volume log data. It supports semi-structured data and has been optimized for aggregation queries. Its server component accepts SQL queries over a secure REST API and can operate in cluster mode. Raijin databases can be used to aggregate security logs from diverse sources as well as providing a foundation for developing customized data analytics frameworks.

NXLog Enterprise Edition can send logs directly to Raijin Server using the Raijin (om_raijin) output module.

Installing Raijin Server

Currently, Raijin Server supports only 64-bit, Debian-based or RedHat-based Linux distributions. The Raijin Quickstart page has step-by-step instructions for deploying and querying Raijin Server.

The following section covers only the basics for administering a single Raijin Server deployment. For more comprehensive documentation, see the Raijin Database User Manual which is bundled with the Raijin installation package and is located in /opt/raijin/doc if you use the default installation path.

Administering Raijin Server

To start Raijin Server:

# systemctl start raijin-server

To check the status of Raijin Server:

# systemctl status raijin-server

To stop Raijin Server

# systemctl stop raijin-server

Managing Raijin databases and tables

All supported SQL commands — including those used for creating, listing, and dropping databases and tables — can be issued using the web interface at http://<raijin-host>:2500, in the input area labeled Enter query: which supports multiple SQL statements terminated with semicolons (;).

Example 1. Create a new database named auditDB that contains a table with an undefined schema called LinuxAudit

Enter query:

create database auditDB;
use auditDB;
create table LinuxAudit();

Then click the Execute Query button.

Sending events to Raijin Server using NXLog

Example 2. Forwarding Linux Audit events to Raijin

In this example, NXLog has been deployed on a primary DNS server. The Linux Audit System im_linuxaudit module is used with a rules file containing over 100 rules to capture security-related events. Two of these rules are for monitoring changes to DNS configuration files:

-w /etc/bind/named.conf -p wa -k dns_conf_change
-w /etc/bind/zones/db.example.com -p wa -k dns_zone_change

The xm_resolver module is needed for the ResolveValues directive in the audit input instance, where it is used for resolving some of the numeric values to human readable string values.

Log records are then converted to JSON using the to_json() procedure of the xm_json module which enriches each record with the NXLog core fields before they are forwarded to Raijin Server.

This configuration is just one example of how mission-critical servers can send their audit logs to Raijin for centralized logging and for aggregating common log sources from multiple hosts.

nxlog.conf

<Extension _json>
    Module          xm_json
</Extension>

<Extension _resolver>
    Module          xm_resolver
</Extension>

<Input LinuxAudit>
    Module          im_linuxaudit
    FlowControl     FALSE
    LoadRule        '/opt/nxlog/etc/im_linuxaudit.rules'
    ResolveValues   TRUE
    Exec            to_json();
</Input>

<Output raijin>
    Module          om_raijin
    URL             http://raijin-1.example.com:2500
    DBName          auditDB
    DBTable         LinuxAudit
</Output>

These sample Linux Audit events sent to Raijin represent a change to a DNS zone file. Raijin returns query results as newline-delimited JSON (NDJSON) records:

Raijin query results of a DNS zone change logged by Linux Audit

{"_id":4016,"type":"CONFIG_CHANGE","time":"2021-04-05T15:13:22.981000+00:00","seq":5480,"auid":"root","ses":"3op=updated_rules","res":"1","Hostname":"u20server-1","EventReceivedTime":"2021-04-05T15:13:22.987380+00:00","SourceModuleName":"LinuxAudit","SourceModuleType":"im_linuxaudit","key":"dns_zone_change","list":"4","path":"/etc/bind/zones/db.example.com"}
{"_id":4017,"type":"SYSCALL","time":"2021-04-05T15:13:22.981000+00:00","seq":5480,"auid":"root","ses":"3","Hostname":"u20server-1","EventReceivedTime":"2021-04-05T15:13:22.988169+00:00","SourceModuleName":"LinuxAudit","SourceModuleType":"im_linuxaudit","arch":"x86_64","syscall":"rename","success":"yes","exit":0,"a0":"56304f629410","a1":"56304f87b770","a2":"fffffffffffffe98","a3":"81a4","items":4,"ppid":1306,"pid":32973,"uid":"root","gid":"root","euid":"root","suid":"root","fsuid":"root","egid":"root","sgid":"root","fsgid":"root","tty":"pts0","comm":"vi","exe":"/usr/bin/vim.basic","key":"dns_zone_change"}

The same Raijin query results displayed as pretty-printed JSON

{
  "_id": 4016,
  "type": "CONFIG_CHANGE",
  "time": "2021-04-05T15:13:22.981000+00:00",
  "seq": 5480,
  "auid": "root",
  "ses": "3op=updated_rules",
  "res": "1",
  "Hostname": "u20server-1",
  "EventReceivedTime": "2021-04-05T15:13:22.987380+00:00",
  "SourceModuleName": "LinuxAudit",
  "SourceModuleType": "im_linuxaudit",
  "key": "dns_zone_change",
  "list": "4",
  "path": "/etc/bind/zones/db.example.com"
}
{
  "_id": 4017,
  "type": "SYSCALL",
  "time": "2021-04-05T15:13:22.981000+00:00",
  "seq": 5480,
  "auid": "root",
  "ses": "3",
  "Hostname": "u20server-1",
  "EventReceivedTime": "2021-04-05T15:13:22.988169+00:00",
  "SourceModuleName": "LinuxAudit",
  "SourceModuleType": "im_linuxaudit",
  "arch": "x86_64",
  "syscall": "rename",
  "success": "yes",
  "exit": 0,
  "a0": "56304f629410",
  "a1": "56304f87b770",
  "a2": "fffffffffffffe98",
  "a3": "81a4",
  "items": 4,
  "ppid": 1306,
  "pid": 32973,
  "uid": "root",
  "gid": "root",
  "euid": "root",
  "suid": "root",
  "fsuid": "root",
  "egid": "root",
  "sgid": "root",
  "fsgid": "root",
  "tty": "pts0",
  "comm": "vi",
  "exe": "/usr/bin/vim.basic",
  "key": "dns_zone_change"
}

Leveraging schema flexibility

The ability to store related security events from different log sources and from different platforms can only be achieved with a schemaless database engine. NXLog agents running on diverse platforms can be configured to process events such that they will have some fields in common before they are sent to Raijin Server. This strategy to normalize log data and maintain a set of common fields from different but similar logs sources can be achieved through:

Enriching log records with additional fields having consistent names across similar log sources
Concatenating each respective field name from different (but similar) log sources that represents a common field among them at query time

The following example shows how a set of authentication-related services from different logging facilities, such as syslog, Linux Audit, and Apple ULS, can be linked together in a single Raijin Database table for a broader view of specific types of security events. For demonstration purposes, the following examples will focus on this narrow set of processes and platforms:

Operating System Logging Facilities NXLog Module(s) Processes

Operating System	Logging Facilities	NXLog Module(s)	Processes
Apple macOS 11.2.3	Apple ULS	macOS ULS (im_maculs)	`ssh` and `sudo`
Ubuntu 20.04 LTS	`syslog`, Linux Audit	Syslog (xm_syslog), Files (im_file), Linux Audit System (im_linuxaudit)	`ssh` and `sudo`

Apple macOS 11.2.3

Apple ULS

macOS ULS (im_maculs)

ssh and sudo

Ubuntu 20.04 LTS

syslog, Linux Audit

Syslog (xm_syslog), Files (im_file), Linux Audit System (im_linuxaudit)

ssh and sudo

Preparing logs for a multi-source, multi-OS Raijin Database table

The following examples are meant to show how schema flexibility can be used to implement partial normalization of log data from different but similar log sources. Some of these events will be logged simultaneously by different logging facilities or processes listed in the table above, each with its own schema. However, by leveraging NXLog’s ability to enrich log records with additional fields, they will be given a common set of fields that can be used for analyzing the aggregated logs as a whole: $Process (either ssh or sudo), $Hostname, $OS, and $OSversion.

Example 3. Collecting and sending ssh and sudo events from macOS ULS to Raijin Server

This configuration uses macOS ULS (im_maculs) for collecting Apple ULS events on macOS. The ULS schema is fairly strict across all log sources. The $processImagePath can be used for determining which system process is being logged. The conditional block is used to filter only ssh and sudo events. All other types of events are dropped. For each ssh or sudo event collected, the event record is enriched with a new field that will be shared with its Linux counterparts: $Process.

Additional fields like $OS and $OSversion that could be useful in queries are also part of this enrichment process. $Hostname is an important field to log because it is missing from the ULS schema and cannot be derived from any other field. The core function hostname() provides the value needed for this new field.

Finally, the to_json() procedure is called to add the new fields along with the NXLog core fields to the event record.

The Raijin (om_raijin) module receives the processed JSON events and sends them to the Raijin Server as defined by the URL directive, i.e. the endpoint, where the records will be inserted into the Raijin database and table defined by the DBName and DBTable directives respectively.

nxlog.conf

<Extension _json>
    Module        xm_json
</Extension>

<Input m1_uls_auth>
    Module        im_maculs
    UUIDTextPath  "/var/db/uuidtext"
    TimeSyncPath  "/var/db/diagnostics/timesync"
    TraceV3Path   "/var/db/diagnostics"
    <Exec>
#       Process only ssh or sudo events
        if $processImagePath == '/usr/bin/ssh'
        {
            $Process = 'ssh';
        }
        else if $processImagePath == '/usr/bin/sudo'
        {
            $Process = 'sudo';
        }
        else
        {
            drop();
        }
#       The "Retrieve Group by ID" event is of no value
        if $eventMessage == 'Retrieve Group by ID'  drop();

        $Hostname = hostname();
        $OS = 'macOS';
        $OSversion = '11.2.3';
        to_json();
    </Exec>
</Input>

<Output raijin1_auditdb_auth>
    Module        om_raijin
    URL           http://raijin-1.example.com:2500
    DBName        auditdb
    DBTable       auth
</Output>

An enriched ssh event from ULS prior to being sent to Raijin Server (displayed as pretty-printed JSON)

{
  "eventMessage": "nw_connection_report_state_with_handler_on_nw_queue [C1] reporting state ready",
  "formatString": "%{public}s [C%u] reporting state %{public}s",
  "activityIdentifier": 0,
  "subsystem": "com.apple.network",
  "category": "connection",
  "threadID": 12580538,
  "senderImageUUID": "07ec53a0-293c-3403-8394-755ae0bddfa4",
  "bootUUID": "80243e89-2bee-44e5-b841-736299e7c5e3",
  "processImagePath": "/usr/bin/ssh",
  "senderImagePath": "/usr/lib/libnetwork.dylib",
  "EventTime": "2021-04-07T13:05:18.807648-05:00",
  "machTimestamp": 48909128392309,
  "messageType": "Default",
  "processImageUUID": "0a7536c1-ab16-31cc-a7c6-be89530f9e6c",
  "processID": 56919,
  "senderProgramCounter": 643704,
  "parentActivityIdentifier": 0,
  "TTL": 0,
  "EventReceivedTime": "2021-04-07T13:06:38.814108-05:00",
  "SourceModuleName": "m1_uls_auth",
  "SourceModuleType": "im_maculs",
  "Process": "ssh",
  "Hostname": "mm37",
  "OS": "macOS",
  "OSversion": "11.2.3"
}

Example 4. Collecting and sending ssh and sudo events from syslog and Linux Audit to Raijin Server

Unlike its macOS counterpart, this configuration example for Linux uses two different log sources, Linux Audit and the syslog information being logged to the /var/log/*log files. It then needs to enrich the events captured by each input instance with the same field name being used for the ssh and sudo events in the macOS configuration.

The parse_syslog() procedure provided by the Syslog (xm_syslog) module transforms the unstructured raw events from the /var/log/*log files specified by the Files directive of the var_log instance, into structured data that can be queried. One of the fields it parses is $SourceName, which returns the name of the application or process that produced the event.

With $SourceName available, the conditional block can filter only sshd and sudo events. All other types of events are dropped. Each event record is then enriched with the same field name used by the macOS configuration for recording the process that logged the event: $Process. For sshd events, the conditional block also normalizes the data to ssh to match what the macOS configuration uses and what the Linux Audit rules will be configured to use.

Finally, the to_json() procedure is called to add the new fields along with the NXLog core fields to the event record.

nxlog.conf

<Extension _json>
    Module        xm_json
</Extension>

<Input var_log>
    Module        im_file
    File          '/var/log/*log'
    <Exec>
        parse_syslog();
#       Process only ssh or sudo events
        if $SourceName == 'sshd'
        {
            $Process = 'ssh';
        }
        else if $SourceName == 'sudo'
        {
            $Process = 'sudo';
        }
        else
        {
            drop();
        }
        $OS = 'Ubuntu';
        $OSversion = '20.04 LTS';
        to_json();
    </Exec>
</Input>

<Extension _resolver>
    Module        xm_resolver
</Extension>

<Input LinuxAudit_auth>
    Module        im_linuxaudit
    FlowControl   FALSE
    LoadRule      '/opt/nxlog/etc/im_linuxaudit_auth.rules'
    ResolveValues TRUE
    <Exec>
        # Process only ssh or sudo events
        if $key == 'ssh'
        {
            $Process = 'ssh';
        }
        else if $key == 'sudo'
        {
            $Process = 'sudo';
        }
        else
        {
            drop();
        }
        $OS = 'Ubuntu';
        $OSversion = '20.04 LTS';
        to_json();
    </Exec>
</Input>

<Output raijin1_auditdb_auth>
    Module        om_raijin
    URL           http://raijin-1.example.com:2500
    DBName        auditdb
    DBTable       auth
</Output>

For the LinuxAudit_auth input instance it is necessary to create a more specific set of rules to gather all related ssh and sudo events into only two sets of events. This is achieved with the -k option that defines the value to be assigned to the $key field once the rule has been matched. The im_linuxaudit module captures events that match any of the rules it has loaded and parses their fields. The following Linux Audit rules are the only ones pertinent to this example. They are part of the ruleset contained in the external rules file defined by the LoadRule directive.

## SSH
-w /etc/ssh/sshd_config -k ssh
-w /usr/bin/ssh -p x -k ssh

## sudo
-w /etc/sudoers -p wa -k sudo
-w /etc/sudoers -p rw -k sudo
-w /bin/su -p x -k sudo
-w /usr/bin/sudo -p x -k sudo

The conditional block in the LinuxAudit_auth input instance takes the field responsible for determining the type of event, in this case $key, and filters only ssh and sudo events while dropping the other events. Each matched event is then enriched with a new $Process field. Finally, the to_json() procedure is called to add the new fields along with the NXLog core fields to the event record.

The Raijin (om_raijin) module receives the processed JSON events and sends them to Raijin Server as defined by the URL directive, i.e. the endpoint, where the records will be inserted into the Raijin database and table defined by the DBName and DBTable directives respectively.

The $Process, $Hostname, $OS, and $OSversion fields will now be available in the event logs from all three log sources being sent to Raijin Server.

An ssh event captured by syslog on Linux after processing and field enrichment (displayed as pretty-printed JSON)

{
  "EventReceivedTime": "2021-04-07T17:43:42.144177+00:00",
  "SourceModuleName": "var_log",
  "SourceModuleType": "im_file",
  "SyslogFacilityValue": 1,
  "SyslogFacility": "USER",
  "SyslogSeverityValue": 5,
  "SyslogSeverity": "NOTICE",
  "SeverityValue": 2,
  "Severity": "INFO",
  "Hostname": "u20server-1",
  "EventTime": "2021-04-07T17:22:37.000000+00:00",
  "SourceName": "sshd",
  "ProcessID": "46325",
  "Message": "pam_unix(sshd:session): session closed for user root",
  "Process": "ssh",
  "OS": "Ubuntu",
  "OSversion": "20.04 LTS"
}

An ssh event captured by Linux Audit after processing and field enrichment (displayed as pretty-printed JSON)

{
  "type": "CONFIG_CHANGE",
  "time": "2021-04-07T17:43:41.695000+00:00",
  "seq": 7036,
  "auid": "unset",
  "ses": "4294967295",
  "op": 0,
  "key": "ssh",
  "list": "4",
  "res": "1",
  "Hostname": "u20server-1",
  "EventReceivedTime": "2021-04-07T17:43:42.154010+00:00",
  "SourceModuleName": "LinuxAudit_auth",
  "SourceModuleType": "im_linuxaudit",
  "Process": "ssh",
  "OS": "Ubuntu",
  "OSversion": "20.04 LTS"
}

Querying the multi-source, multi-OS Raijin Database table

To reiterate the two techniques mentioned in the Leveraging schema flexibility section, you can partially normalize diverse schemas by adding additional fields that are common to a select set of schemas being stored within the same table. And, if the schemas already have some fields that represent distinct attributes that are common among them, i.e., the data itself is already normalized, but the names of the fields/columns are not, you can concatenate the column/field names at query time to achieve a normalized column from diverse records within a schemaless table.

The following queries and their results are based on the configuration examples presented above.

Example 5. Collecting all related events with details

Query

select "EventTime",
       "Hostname",
       "OS",
       "OSversion",
       "Process",
       CONCAT("eventMessage","Message") as "Message"
from auditdb.auth
order by "EventTime" desc
limit 30
offset 300;

A small sample of the results returned (displayed as pretty-printed JSON)

{
  "EventTime": "2021-04-07T16:09:16.000000+00:00",
  "Hostname": "u20server-1",
  "OS": "Ubuntu",
  "OSversion": "20.04 LTS",
  "Process": "ssh",
  "Message": "pam_unix(sshd:session): session opened for user root by (uid=0)"
}
{
  "EventTime": "2021-04-07T16:09:16.000000+00:00",
  "Hostname": "u20server-1",
  "OS": "Ubuntu",
  "OSversion": "20.04 LTS",
  "Process": "ssh",
  "Message": "Accepted publickey for root from 192.168.1.7 port 40310 ssh2: RSA SHA256:hyxG8VslYTlmqxFedvCXQYIWRa8Bcl3uf30rmGiuC6c"
}
{
  "EventTime": "2021-04-07T16:03:12.000000+00:00",
  "Hostname": "u20server-1",
  "OS": "Ubuntu",
  "OSversion": "20.04 LTS",
  "Process": "ssh",
  "Message": "Disconnected from user root 192.168.1.7 port 40302"
}
{
  "EventTime": "2021-04-07T13:05:35.244269-05:00",
  "Hostname": "mm37",
  "OS": "macOS",
  "OSversion": "11.2.3",
  "Process": "sudo",
  "Message": "      jk : TTY=ttys000 ; PWD=/Users/jk ; USER=root ; COMMAND=/usr/bin/su - root"
}
{
  "EventTime": "2021-04-07T13:05:18.807659-05:00",
  "Hostname": "mm37",
  "OS": "macOS",
  "OSversion": "11.2.3",
  "Process": "ssh",
  "Message": "[C1.1 192.168.1.51:22 ready socket-flow (satisfied (Path is satisfied), viable, interface: en0, ipv4, ipv6, dns)] event: flow:changed_viability @0.003s"
}
{
  "EventTime": "2021-04-07T12:52:56.000000-05:00",
  "Hostname": "PN61",
  "OS": "Ubuntu",
  "OSversion": "20.04 LTS",
  "Process": "ssh",
  "Message": "Accepted password for jk from 192.168.1.160 port 63037 ssh2"
}

Example 6. Gathering statistics

GROUP BY query

select "Process",
       count("Process") as "Event Count",
       "Hostname",
       "OS"
from auditdb.auth
where "Process" NotNull
group by "Process","Hostname","OS"
order by "Process","Event Count" desc,"Hostname","OS";

Results

{
  "Process": "ssh",
  "Event Count": 567,
  "Hostname": "mm37",
  "OS": "macOS"
}
{
  "Process": "ssh",
  "Event Count": 119,
  "Hostname": "u20server-1",
  "OS": "Ubuntu"
}
{
  "Process": "ssh",
  "Event Count": 110,
  "Hostname": "PN61",
  "OS": "Ubuntu"
}
{
  "Process": "sudo",
  "Event Count": 564,
  "Hostname": "mm37",
  "OS": "macOS"
}
{
  "Process": "sudo",
  "Event Count": 113,
  "Hostname": "PN61",
  "OS": "Ubuntu"
}
{
  "Process": "sudo",
  "Event Count": 12,
  "Hostname": "u20server-1",
  "OS": "Ubuntu"
}

Disclaimer

While we endeavor to keep the information in this topic up to date and correct, NXLog makes no representations or warranties of any kind, express or implied about the completeness, accuracy, reliability, suitability, or availability of the content represented here. We update our screenshots and instructions on a best-effort basis.

The accurateness of the content was tested and proved to be working in our lab environment at the time of the last revision with the following software versions:

Raijin Server 0.9.3462
NXLog 5.2.6634 & 5.2.6677
Apple macOS 11.2.3 (Big Sur)
Ubuntu 20.04 LTS

Last revision: 16 April 2021