Raijin Database Engine
The Raijin Database engine was designed to work with high-volume log data. It supports semi-structured data and has been optimized for aggregation queries. Its server component accepts SQL queries over a secure REST API and can operate in cluster mode. Raijin databases can be used to aggregate security logs from diverse sources as well as provide a foundation for developing customized data analytics frameworks.
NXLog Agent can send logs directly to Raijin Server using the Raijin (om_raijin) output module.
Installing Raijin Server
Currently, Raijin Server supports only 64-bit, Debian-based or RedHat-based Linux distributions. The Raijin Quickstart page has step-by-step instructions for deploying and querying Raijin Server.
The following section covers only the basics for administering a single
Raijin Server deployment. For more comprehensive documentation, see the
Raijin Database User Manual which is bundled with the Raijin installation
package and is located in /opt/raijin/doc
if you use the default installation
path.
Administering Raijin Server
To start Raijin Server:
# systemctl start raijin-server
To check the status of Raijin Server:
# systemctl status raijin-server
To stop Raijin Server
# systemctl stop raijin-server
Managing Raijin databases and tables
All supported SQL commands — including those used for creating, listing, and
dropping databases and tables — can be issued using the web interface at
http://<raijin-host>:2500
, in the input area
labeled Enter query: which supports multiple SQL statements terminated
with semicolons (;
).
auditDB
that contains a table with an undefined schema called LinuxAudit
Enter query:
create database auditDB;
use auditDB;
create table LinuxAudit();
Then click the Execute Query button.
Sending events to Raijin Server using NXLog Agent
In this example, NXLog Agent has been deployed on a primary DNS server. The Linux Audit System im_linuxaudit module is used with a rules file containing over 100 rules to capture security-related events. Two of these rules are for monitoring changes to DNS configuration files:
-w /etc/bind/named.conf -p wa -k dns_conf_change
-w /etc/bind/zones/db.example.com -p wa -k dns_zone_change
The xm_resolver module is needed for the ResolveValues
directive in the audit
input instance, where it is used for resolving some of
the numeric values to human readable string values.
Log records are then converted to JSON using the to_json() procedure of the xm_json module which enriches each record with the NXLog Agent core fields before they are forwarded to Raijin Server.
This configuration is just one example of how mission-critical servers can send their audit logs to Raijin for centralized logging and for aggregating common log sources from multiple hosts.
<Extension _json>
Module xm_json
</Extension>
<Extension _resolver>
Module xm_resolver
</Extension>
<Input LinuxAudit>
Module im_linuxaudit
FlowControl FALSE
LoadRule '/opt/nxlog/etc/im_linuxaudit.rules'
ResolveValues TRUE
Exec to_json();
</Input>
<Output raijin>
Module om_raijin
URL http://raijin-1.example.com:2500
DBName auditDB
DBTable LinuxAudit
</Output>
These sample Linux Audit events sent to Raijin represent a change to a DNS zone file. Raijin returns query results as newline-delimited JSON (NDJSON) records:
{"_id":4016,"type":"CONFIG_CHANGE","time":"2021-04-05T15:13:22.981000+00:00","seq":5480,"auid":"root","ses":"3op=updated_rules","res":"1","Hostname":"u20server-1","EventReceivedTime":"2021-04-05T15:13:22.987380+00:00","SourceModuleName":"LinuxAudit","SourceModuleType":"im_linuxaudit","key":"dns_zone_change","list":"4","path":"/etc/bind/zones/db.example.com"}
{"_id":4017,"type":"SYSCALL","time":"2021-04-05T15:13:22.981000+00:00","seq":5480,"auid":"root","ses":"3","Hostname":"u20server-1","EventReceivedTime":"2021-04-05T15:13:22.988169+00:00","SourceModuleName":"LinuxAudit","SourceModuleType":"im_linuxaudit","arch":"x86_64","syscall":"rename","success":"yes","exit":0,"a0":"56304f629410","a1":"56304f87b770","a2":"fffffffffffffe98","a3":"81a4","items":4,"ppid":1306,"pid":32973,"uid":"root","gid":"root","euid":"root","suid":"root","fsuid":"root","egid":"root","sgid":"root","fsgid":"root","tty":"pts0","comm":"vi","exe":"/usr/bin/vim.basic","key":"dns_zone_change"}
{
"_id": 4016,
"type": "CONFIG_CHANGE",
"time": "2021-04-05T15:13:22.981000+00:00",
"seq": 5480,
"auid": "root",
"ses": "3op=updated_rules",
"res": "1",
"Hostname": "u20server-1",
"EventReceivedTime": "2021-04-05T15:13:22.987380+00:00",
"SourceModuleName": "LinuxAudit",
"SourceModuleType": "im_linuxaudit",
"key": "dns_zone_change",
"list": "4",
"path": "/etc/bind/zones/db.example.com"
}
{
"_id": 4017,
"type": "SYSCALL",
"time": "2021-04-05T15:13:22.981000+00:00",
"seq": 5480,
"auid": "root",
"ses": "3",
"Hostname": "u20server-1",
"EventReceivedTime": "2021-04-05T15:13:22.988169+00:00",
"SourceModuleName": "LinuxAudit",
"SourceModuleType": "im_linuxaudit",
"arch": "x86_64",
"syscall": "rename",
"success": "yes",
"exit": 0,
"a0": "56304f629410",
"a1": "56304f87b770",
"a2": "fffffffffffffe98",
"a3": "81a4",
"items": 4,
"ppid": 1306,
"pid": 32973,
"uid": "root",
"gid": "root",
"euid": "root",
"suid": "root",
"fsuid": "root",
"egid": "root",
"sgid": "root",
"fsgid": "root",
"tty": "pts0",
"comm": "vi",
"exe": "/usr/bin/vim.basic",
"key": "dns_zone_change"
}
Leveraging schema flexibility
The ability to store related security events from different log sources and from different platforms can only be achieved with a schemaless database engine. NXLog Agent instances running on diverse platforms can be configured to process events such that they will have some fields in common before they are sent to Raijin Server. This strategy to normalize log data and maintain a set of common fields from different but similar log sources can be achieved through:
-
Enriching log records with additional fields having consistent names across similar log sources
-
Concatenating each respective field name from different (but similar) log sources that represent a common field among them at query time
The following example shows how a set of authentication-related services from different logging facilities, such as syslog, Linux Audit, and Apple ULS, can be linked together in a single Raijin Database table for a broader view of specific types of security events. For demonstration purposes, the following examples will focus on this narrow set of processes and platforms:
Operating System | Logging Facilities | NXLog Agent Module(s) | Processes |
---|---|---|---|
Apple macOS 11.2.3 |
Apple ULS |
|
|
Ubuntu 20.04 LTS |
|
Syslog (xm_syslog), Files (im_file), Linux Audit System (im_linuxaudit) |
|
Preparing logs for a multi-source, multi-OS Raijin Database table
The following examples are meant to show how schema flexibility can be used to
implement partial normalization of log data from different but similar log
sources. Some of these events will be logged simultaneously by different
logging facilities or processes listed in the table above, each with its own
schema. However, by leveraging NXLog Agent’s ability to enrich log records
with additional fields, they will be given a common set of fields that can be
used for analyzing the aggregated logs as a whole: $Process
(either ssh
or
sudo
), $Hostname
, $OS
, and $OSversion
.
This configuration uses macOS ULS (im_maculs) for collecting Apple ULS events on macOS.
The ULS schema is fairly strict across all log sources. The $processImagePath
can be used for determining which system process is being logged. The
conditional block is used to filter only ssh
and sudo
events. All other
types of events are dropped. For each ssh
or sudo
event collected, the event
record is enriched with a new field that will be shared with its Linux
counterparts: $Process
.
Additional fields like $OS
and $OSversion
that could be useful in queries
are also part of this enrichment process.
$Hostname
is an important field to log because it is missing from the ULS
schema and cannot be derived from any other field. The core function
hostname() provides the value needed for this new field.
Finally, the to_json() procedure is called to add the new fields along with the NXLog Agent core fields to the event record.
The Raijin (om_raijin) module receives the processed JSON events and sends them to
the Raijin Server as defined by the URL
directive, i.e. the endpoint, where
the records will be inserted into the Raijin database and table defined by the
DBName
and DBTable
directives respectively.
<Extension _json>
Module xm_json
</Extension>
<Input m1_uls_auth>
Module im_maculs
UUIDTextPath "/var/db/uuidtext"
TimeSyncPath "/var/db/diagnostics/timesync"
TraceV3Path "/var/db/diagnostics"
<Exec>
# Process only ssh or sudo events
if $processImagePath == '/usr/bin/ssh'
{
$Process = 'ssh';
}
else if $processImagePath == '/usr/bin/sudo'
{
$Process = 'sudo';
}
else
{
drop();
}
# The "Retrieve Group by ID" event is of no value
if $eventMessage == 'Retrieve Group by ID' drop();
$Hostname = hostname();
$OS = 'macOS';
$OSversion = '11.2.3';
to_json();
</Exec>
</Input>
<Output raijin1_auditdb_auth>
Module om_raijin
URL http://raijin-1.example.com:2500
DBName auditdb
DBTable auth
</Output>
{
"eventMessage": "nw_connection_report_state_with_handler_on_nw_queue [C1] reporting state ready",
"formatString": "%{public}s [C%u] reporting state %{public}s",
"activityIdentifier": 0,
"subsystem": "com.apple.network",
"category": "connection",
"threadID": 12580538,
"senderImageUUID": "07ec53a0-293c-3403-8394-755ae0bddfa4",
"bootUUID": "80243e89-2bee-44e5-b841-736299e7c5e3",
"processImagePath": "/usr/bin/ssh",
"senderImagePath": "/usr/lib/libnetwork.dylib",
"EventTime": "2021-04-07T13:05:18.807648-05:00",
"machTimestamp": 48909128392309,
"messageType": "Default",
"processImageUUID": "0a7536c1-ab16-31cc-a7c6-be89530f9e6c",
"processID": 56919,
"senderProgramCounter": 643704,
"parentActivityIdentifier": 0,
"TTL": 0,
"EventReceivedTime": "2021-04-07T13:06:38.814108-05:00",
"SourceModuleName": "m1_uls_auth",
"SourceModuleType": "im_maculs",
"Process": "ssh",
"Hostname": "mm37",
"OS": "macOS",
"OSversion": "11.2.3"
}
Unlike its macOS counterpart, this configuration example for Linux uses two
different log sources, Linux Audit and the syslog information being logged to
the /var/log/*log
files. It then needs to enrich the events captured by each
input instance with the same field name being used for the ssh
and sudo
events in the macOS configuration.
The parse_syslog() procedure provided by the Syslog (xm_syslog)
module transforms the unstructured raw events from the /var/log/*log
files
specified by the Files
directive of the var_log
instance, into structured
data that can be queried. One of the fields it parses is
$SourceName, which returns the name of the
application or process that produced the event.
With $SourceName
available, the conditional block can filter only sshd
and
sudo
events. All other types of events are dropped. Each event record is then
enriched with the same field name used by the macOS configuration for recording
the process that logged the event: $Process
. For sshd
events, the
conditional block also normalizes the data to ssh
to match what the macOS
configuration uses and what the Linux Audit rules will be configured to use.
Finally, the to_json() procedure is called to add the new fields along with the NXLog Agent core fields to the event record.
<Extension _json>
Module xm_json
</Extension>
<Input var_log>
Module im_file
File '/var/log/*log'
<Exec>
parse_syslog();
# Process only ssh or sudo events
if $SourceName == 'sshd'
{
$Process = 'ssh';
}
else if $SourceName == 'sudo'
{
$Process = 'sudo';
}
else
{
drop();
}
$OS = 'Ubuntu';
$OSversion = '20.04 LTS';
to_json();
</Exec>
</Input>
<Extension _resolver>
Module xm_resolver
</Extension>
<Input LinuxAudit_auth>
Module im_linuxaudit
FlowControl FALSE
LoadRule '/opt/nxlog/etc/im_linuxaudit_auth.rules'
ResolveValues TRUE
<Exec>
# Process only ssh or sudo events
if $key == 'ssh'
{
$Process = 'ssh';
}
else if $key == 'sudo'
{
$Process = 'sudo';
}
else
{
drop();
}
$OS = 'Ubuntu';
$OSversion = '20.04 LTS';
to_json();
</Exec>
</Input>
<Output raijin1_auditdb_auth>
Module om_raijin
URL http://raijin-1.example.com:2500
DBName auditdb
DBTable auth
</Output>
For the LinuxAudit_auth
input instance it is necessary to create a more
specific set of rules to gather all related ssh
and sudo
events into only
two sets of events. This is achieved with the -k
option that defines the
value to be assigned to the $key
field once the rule has been matched. The
im_linuxaudit module captures events that match any of the rules it has
loaded and parses their fields. The following Linux Audit rules are the only
ones pertinent to this example. They are part of the ruleset contained in the
external rules file defined by the LoadRule
directive.
## SSH
-w /etc/ssh/sshd_config -k ssh
-w /usr/bin/ssh -p x -k ssh
## sudo
-w /etc/sudoers -p wa -k sudo
-w /etc/sudoers -p rw -k sudo
-w /bin/su -p x -k sudo
-w /usr/bin/sudo -p x -k sudo
The conditional block in the LinuxAudit_auth
input instance takes the field
responsible for determining the type of event, in this case, $key
, and filters
only ssh
and sudo
events while dropping the other events.
Each matched event is then enriched with a new $Process
field.
Finally, the to_json() procedure is called to add the
new fields along with the NXLog Agent core fields to the
event record.
The Raijin (om_raijin) module receives the processed JSON events and sends them to
Raijin Server as defined by the URL
directive, i.e. the endpoint, where the
records will be inserted into the Raijin database and table defined by the
DBName
and DBTable
directives respectively.
The $Process
, $Hostname
, $OS
, and $OSversion
fields will now be
available in the event logs from all three log sources being sent to Raijin
Server.
{
"EventReceivedTime": "2021-04-07T17:43:42.144177+00:00",
"SourceModuleName": "var_log",
"SourceModuleType": "im_file",
"SyslogFacilityValue": 1,
"SyslogFacility": "USER",
"SyslogSeverityValue": 5,
"SyslogSeverity": "NOTICE",
"SeverityValue": 2,
"Severity": "INFO",
"Hostname": "u20server-1",
"EventTime": "2021-04-07T17:22:37.000000+00:00",
"SourceName": "sshd",
"ProcessID": "46325",
"Message": "pam_unix(sshd:session): session closed for user root",
"Process": "ssh",
"OS": "Ubuntu",
"OSversion": "20.04 LTS"
}
{
"type": "CONFIG_CHANGE",
"time": "2021-04-07T17:43:41.695000+00:00",
"seq": 7036,
"auid": "unset",
"ses": "4294967295",
"op": 0,
"key": "ssh",
"list": "4",
"res": "1",
"Hostname": "u20server-1",
"EventReceivedTime": "2021-04-07T17:43:42.154010+00:00",
"SourceModuleName": "LinuxAudit_auth",
"SourceModuleType": "im_linuxaudit",
"Process": "ssh",
"OS": "Ubuntu",
"OSversion": "20.04 LTS"
}
Querying the multi-source, multi-OS Raijin Database table
To reiterate the two techniques mentioned in the Leveraging schema flexibility section, you can partially normalize diverse schemas by adding additional fields that are common to a select set of schemas being stored within the same table. And, if the schemas already have some fields that represent distinct attributes that are common among them, i.e., the data itself is already normalized, but the names of the fields/columns are not, you can concatenate the column/field names at query time to achieve a normalized column from diverse records within a schemaless table.
The following queries and their results are based on the configuration examples presented above.
select "EventTime",
"Hostname",
"OS",
"OSversion",
"Process",
CONCAT("eventMessage","Message") as "Message"
from auditdb.auth
order by "EventTime" desc
limit 30
offset 300;
{
"EventTime": "2021-04-07T16:09:16.000000+00:00",
"Hostname": "u20server-1",
"OS": "Ubuntu",
"OSversion": "20.04 LTS",
"Process": "ssh",
"Message": "pam_unix(sshd:session): session opened for user root by (uid=0)"
}
{
"EventTime": "2021-04-07T16:09:16.000000+00:00",
"Hostname": "u20server-1",
"OS": "Ubuntu",
"OSversion": "20.04 LTS",
"Process": "ssh",
"Message": "Accepted publickey for root from 192.168.1.7 port 40310 ssh2: RSA SHA256:hyxG8VslYTlmqxFedvCXQYIWRa8Bcl3uf30rmGiuC6c"
}
{
"EventTime": "2021-04-07T16:03:12.000000+00:00",
"Hostname": "u20server-1",
"OS": "Ubuntu",
"OSversion": "20.04 LTS",
"Process": "ssh",
"Message": "Disconnected from user root 192.168.1.7 port 40302"
}
{
"EventTime": "2021-04-07T13:05:35.244269-05:00",
"Hostname": "mm37",
"OS": "macOS",
"OSversion": "11.2.3",
"Process": "sudo",
"Message": " jk : TTY=ttys000 ; PWD=/Users/jk ; USER=root ; COMMAND=/usr/bin/su - root"
}
{
"EventTime": "2021-04-07T13:05:18.807659-05:00",
"Hostname": "mm37",
"OS": "macOS",
"OSversion": "11.2.3",
"Process": "ssh",
"Message": "[C1.1 192.168.1.51:22 ready socket-flow (satisfied (Path is satisfied), viable, interface: en0, ipv4, ipv6, dns)] event: flow:changed_viability @0.003s"
}
{
"EventTime": "2021-04-07T12:52:56.000000-05:00",
"Hostname": "PN61",
"OS": "Ubuntu",
"OSversion": "20.04 LTS",
"Process": "ssh",
"Message": "Accepted password for jk from 192.168.1.160 port 63037 ssh2"
}
select "Process",
count("Process") as "Event Count",
"Hostname",
"OS"
from auditdb.auth
where "Process" NotNull
group by "Process","Hostname","OS"
order by "Process","Event Count" desc,"Hostname","OS";
{
"Process": "ssh",
"Event Count": 567,
"Hostname": "mm37",
"OS": "macOS"
}
{
"Process": "ssh",
"Event Count": 119,
"Hostname": "u20server-1",
"OS": "Ubuntu"
}
{
"Process": "ssh",
"Event Count": 110,
"Hostname": "PN61",
"OS": "Ubuntu"
}
{
"Process": "sudo",
"Event Count": 564,
"Hostname": "mm37",
"OS": "macOS"
}
{
"Process": "sudo",
"Event Count": 113,
"Hostname": "PN61",
"OS": "Ubuntu"
}
{
"Process": "sudo",
"Event Count": 12,
"Hostname": "u20server-1",
"OS": "Ubuntu"
}