NXLog Docs

About NXLog

Modern IT infrastructure produces large volumes of event logging data. In a single organization, hundreds or thousands of different devices, applications, and appliances generate event log messages. These messages require many log processing tasks, including filtration, classification, correlation, forwarding, and storage. In most organizations these requirements are met with a collection of scripts and programs, each with its custom format and configuration. NXLog provides a single, high-performance, multi-platform product for solving all of these tasks and achieving consistent results.

At NXLog’s inception, there were various logging solutions available, but none with the required features. Most were single-threaded and Syslog-oriented, without native support for Windows. Work on NXLog began with the goal of building a modern logger with a multi-threaded design, a clear configuration syntax, multi-platform support, and clean source code. NXLog was born in 2009 as a closed source product heavily used in several production deployments. The source code of NXLog Community Edition was released in November 2011.

NXLog can process event logs from thousands of different sources with volumes over 100,000 events per second. It can accept event logs over TCP, TLS/SSL, and UDP; from files and databases; and in Syslog, Windows Event Log, and JSON formats. NXLog can also perform advanced processing on log messages, such as rewriting, correlating, alerting, pattern matching, scheduling, and log file rotation. It supports prioritized processing of certain log messages, and can buffer messages on disk or in memory to work around problems with input latency or network congestion. After processing, NXLog can store or forward event logs in any of many supported formats. Inputs, outputs, log formats, and complex processing are implemented with a modular architecture and a powerful configuration language.

NXLog features

This section gives an overview of some of the key advantages of NXLog over alternative systems.

Multi-platform deployment

Installer packages are provided for multiple platforms, including Linux and Windows. You can use NXLog across your entire infrastructure, without resorting to different tools for different platforms.

Client or server operation

Depending on the configuration, NXLog will run as a server, a client, or a combination of both. You have the freedom to choose the deployment architecture that best meets your needs. For example, NXLog can collect local log data and forward it, relay data without storing it locally, or save incoming log data to disk.

Many input and output types and formats

NXLog can accept data from many different sources, convert the data internally, and output it to other destinations. You can use NXLog as a single tool to process all of the different types of logs in your organization. For example, logs can be collected from files, databases, Unix domain sockets, network connections, and other sources. BSD Syslog, IETF Syslog, the Snare Agent format, Windows Event Log, JSON, and other formats are supported. NXLog can likely be configured to read or write logs in your custom application format, using the NXLog language and provided extension modules.

High performance, scalable architecture

With an event-based architecture for processing tasks in parallel, non-blocking input and output where possible, and a worker thread pool for incoming log messages, NXLog is designed for high performance on modern multi-core and multi-processor systems. The input/output readiness notifications provided by most operating systems are used to efficiently handle large numbers of open files and network connections.

Security

NXLog provides features throughout the application to maintain the security of your log data and systems. The core can be configured to run as an unprivileged user, and special privileges (such as binding to ports below 1024) are accessed through Linux capabilities rather than requiring the application to run as root. TLS/SSL is supported for encrypted, authenticated communications and to prevent data interception or alteration during transmission.

Modular architecture

NXLog has a lightweight, modular architecture, providing a reduced memory footprint and increased flexibility for different uses. The core handles files, events, and sockets, and provides the configuration language; modules provide the input, output, and processing capabilities. Because modules use a common API, you can write new modules to extend the features of NXLog.

Message buffering

Log messages can be buffered in memory or on disk. This increases reliability by holding messages in a temporary cache when a network connectivity issue or dropout occurs. Conditional buffering can be configured by using the NXLog language to define relevant conditions. For example, UDP messages may arrive faster than they can be processed, and NXLog can buffer the messages to disk for processing when the system is under less load. Conditional buffering can be used to explicitly buffer log messages during certain hours of the day or when the system load is high.

Prioritized processing

NXLog can be configured to separate high-priority log processing from low-priority log processing, ensuring that it processes the most important data first. When the system is experiencing high load, NXLog will avoid dropping important incoming messages. For example, incoming UDP messages can be prioritized to prevent dropped logs if a high volume of TCP messages overloads the system.

Message durability

Built-in flow control ensures that a blocked output does not cause dropped log messages when buffers are full. In combination with the previously mentioned parallel processing, buffering, and prioritization, the possibility of message loss is greatly reduced.

Familiar and powerful configuration syntax

NXLog uses an Apache-style configuration syntax that is easy to read and can be parsed or generated with scripts. The NXLog language supports advanced scripting and processing capabilities that are usually only found in full-fledged scripting languages. The syntax is similar to Perl, so users familiar with that language can learn it easily. It supports polymorphic functions and procedures and regular expressions with captured sub-strings. Modules can register additional functions and procedures to further extend the capabilities of the language.

Scheduled tasks and log rotation

NXLog includes a scheduler service. A task can be scheduled from any module without requiring an external tool such as Cron. Log files can be rotated automatically, on a time-based schedule or according to file size. The file reader and writer modules in NXLog can detect when an input or output file moves or changes its name, and re-open the file automatically.

Advanced message processing

NXLog can perform advanced processing actions on log messages, in addition to the core features already mentioned. By using additional modules, NXLog can solve many related tasks such as message classification, event correlation, pattern matching, message filtering, message rewriting, and conditional alerting. You can use a single tool for all log processing functionality.

Offline processing

Sometimes log messages need to be processed in batches for conversion, filtering, or analysis. NXLog provides an offline mode in which it processes all input and then exits. Because NXLog does not assume that the event time and processing time are identical, time-based correlation features can be used even during offline log processing.

International character and encoding support

NXLog supports explicit character set conversion and automatic character set detection. Log messages received in different character sets can be automatically normalized to a common standard, allowing messages to be compared across different sources.

NXLog Enterprise Edition features

While the NXLog Community Edition provides all the flexibility and performance of the NXLog engine, the NXLog Enterprise Edition provides additional enhancements, including modules and core features, as well as regular hot-fixes and updates which are crucial in a professional environment. The Enterprise Edition provides the following enhancements.

Additional platform support

In addition to Linux and Windows, installer packages are provided for the BSDs and the major variants of Unix (AIX, Solaris, and macOS) operating systems.

Signed installer packages

Installer packages are digitally signed to ensure that the binaries are not corrupted or compromised. In many cases, this is a compliance mandate.

On-the-wire compression

Log data can be transferred in compressed batches with the im_batchcompress and om_batchcompress input/output modules. This can help in limited bandwidth scenarios.

Better control over SSL and TLS

Due to vulnerabilities discovered in the SSL protocols, some protocols may need to be disabled. The various SSL/TLS networking modules in NXLog Enterprise Edition can be configured to allow only specific protocols via the SSLProtocol directive. On Windows, NXLog Enterprise Edition can utilize TLSv1.2 while NXLog Community Edition supports TLSv1.0 only.

ODBC input and output

The ODBC input and output modules, im_odbc and om_odbc, allow log data to be read from or inserted into any ODBC-compliant database. The primary purpose of the im_odbc module is native Windows MSSQL support to enable log collection from Windows applications that write logs to MSSQL. The om_odbc output module can be used to insert data into an ODBC database. These modules are available on Windows and Linux too.

Remote management

The dedicated xm_admin extension module enables NXLog agents to be managed remotely over a secure SOAP/JSON SSL connection or to be integrated with existing monitoring and management tools. The configurations, correlation rules, patterns, and certificates can all be updated remotely from the NXLog Manager web interface or from scripts. In addition, the NXLog agents and the individual modules can be stopped/started and log collection statistics can be queried for real-time statistics.

Crash recovery

Additional functionality is provided to guarantee a clean recovery in the case of a system crash, ensuring that no messages are lost or duplicated.

Event correlation

The pm_evcorr processor module can efficiently solve complex event correlation tasks, with capabilities similar to what the open-source SEC tool provides.

HTTP/HTTPS support

The im_http and om_http input and output modules make it possible to send or receive log message data over HTTP or HTTPS.

TCP and UDP support

Accepting and initiating TCP and UDP connections to a remote server can be achieved using the dedicated protocol-specific im_tcp and om_tcp modules, as well as the im_udp and om_udp modules respectively.

UDP source IP address spoofing

Some SIEM and log collection systems use the IP address of the UDP Syslog packet sent by the client. As a server or relay, the om_udpspoof output module can be configured to retain the original IP address of the sender.

File integrity monitoring

Several compliance standards mandate file integrity monitoring. With the im_fim input module, NXLog Enterprise Edition can be used to detect modifications to files or directories. This module is available on Windows as well as Linux.

Registry monitoring

The im_regmon module facilitates the monitoring of the Windows Registry and generates event records in case of changes in the monitored registry entries.

Network monitoring

The passive monitoring of network traffic can be implemented via utilizing the im_pcap module.

XML support

The xm_xml extension module can parse nested XML and data stored in XML attributes.

JSON support

Parsing of nested JSON is implemented in the xm_json module. UTF-8 character encoding validation can be enforced to avoid parser failures caused by invalid UTF-8 encoding from other tools.

Support of key-value pairs

Parsing and generation of key-value formatted data can be accomplished by using the xm_kvp module.

Parsing with patterns
  • The xm_grok module can parse data using Grok patterns.

  • The database of patterns in XML format can be applied using the xm_pattern module.

Parsing multi-line logs

Messages which span multiple lines can be processed with the xm_multiline module. The flexible configuration of the module is reached through the specification of the header line, the footer line, and the number of lines in a message.

Native W3C parser

The W3C format is widely used in various Microsoft products and perhaps IIS is the most well-known producer. Parsing of W3C is possible with the xm_csv extension module, but that requires defining the fields in the configuration and adjustment when the IIS configuration is changed. The xm_w3c extension module can automatically parse the logs using the field information stored in the headers. It also supports automatic parsing of the data format produced by BRO.

More support for SIEM products

The xm_cef and xm_leef modules provide parsing and generation of CEF and LEEF formatted data. CEF (Common Event Format) was introduced by HP ArcSight and LEEF (Log Event Extended Format) is used by IBM Security QRadar.

Simplified data processing configuration

Two extension modules help simplify data processing. The xm_rewrite module allows fields to be renamed, kept (whitelisted), or deleted (blacklisted). It also supports the Exec directive so log processing logic can be localized to avoid duplicated statements. The xm_filelist module provides two functions, contains() and matches(), which can be invoked to check whether a string is present in a text file. This can be a username or an IP address for example. The files are cached in memory and any changes are automatically picked up without the need to reload NXLog.

Custom input, output, and extension modules

The Enterprise Edition of NXLog supports custom modules which can be developed in the following programming languages:

Name resolution

The xm_resolver extension module provides cached DNS lookup functions for translating between IP addresses and host names. User and group names can also be mapped to/from user and group ids.

Elasticsearch integration

The om_elasticsearch output module allows log data to be loaded directly into an Elasticsearch server without requiring Logstash.

Check Point LEA input

The im_checkpoint input module enables the remote collection of Check Point firewall logs over the OPSEC/LEA protocol. This feature is only available in the Linux version.

Redis Support

Redis is often used as an intermediate queue for log data. Two native modules, im_redis and om_redis, are available to push and pull data to and from Redis servers.

SNMP input

The xm_snmp extension module can be used to parse SNMP traps. The traps can then be handled like regular log messages: converted to Syslog, stored, forwarded, etc.

Multi-platform support for Windows Event Forwarding

The im_wseventing input module can be used to collect forwarded events from Windows hosts. The Windows clients can be configured from Group Policy to send Windows Event Log using Windows Event Forwarding. While NXLog Enterprise Edition can collect Windows Event Log remotely over WMI and MSRPC, this module provides improved security for collecting from Windows machines in agentless mode, with support for both Kerberos and HTTPS data transfer. The im_wseventing module is platform independent and available on Linux as well as Windows.

HDFS output

The om_webhdfs output module is available to support the Hadoop ecosystem.

Windows Performance Counters

The im_winperfcount input module can collect metrics from Windows Performance Counters such as CPU, disk, and memory statistics.

Reading Windows Event Log files directly

The im_msvistalog module can read .evt, .evtx, and .etl event log files directly; this is particularly useful for forensics purposes.

Additional Windows Event Log data

The im_msvistalog module retrieves the EventData and UserData parts which can contain important data in some log sources. In addition, SID values in the Windows Event Log record can be resolved to account names to produce the same output that Windows Event Viewer gives.

Event Tracing for Windows

The im_etw module provides direct reading of event tracing data of both kernel and user-mode applications.

Netflow support

The xm_netflow extension module can parse Netflow packets received over UDP. It supports Netflow v1, v5, v7, v9, and IPFIX.

ZeroMQ support

ZeroMQ is a popular high performance messaging library. The im_zmq and om_zmq modules provide input and output support for the ZeroMQ protocol.

Named pipes and domain sockets

The Enterprise Edition supports communicating logs via:

Testing facilities

Simple test events can be generated using the im_testgen module according to the specified counter. The im_null and om_null module can be used for testing purposes as well.

The om_blocker module can be used for blocking messages to simulate a blocked route.

Mark messages

Via the im_mark module, NXLog can send mark messages confirming its correct operation.

Data transformation

NXLog Enterprise Edition supports conversion of message strings between various character sets. This may be useful when the encoding of accepted messages differs from UTF-8.
Compression and encryption operations with logs are available via the xm_zlib and xm_crypto modules.

Manipulation with files

NXLog Enterprise Edition provides retention and rotation policies which can be applied to files, such as log retention based on the file size and age, and cyclic rotation and retention of files. All these features are available via the xm_fileop module.

Execution of external scripts

External scripts can be run on startup at the input, output, and extension levels using the im_exec, om_exec, and xm_exec modules respectively.

Support of various log sources

NXLog Enterprise Edition supports a wider variety of log sources and provides additional functionalities to the modules which exist in both the Enterprise and Community versions of NXLog.

Buffering of messages

Using the pm_buffer prevents from losing messages which arrive over UDP.

Regular hot fixes

Unlike NXLog Community Edition which is a volunteer effort, NXLog Enterprise Edition receives regular hot fixes and enhancements.

World class support

For NXLog Enterprise Edition, a dedicated professionally trained support team is available and ready to act at request. They can be available 24/7 with a world-class, 4-hour SLA.

Extensive documentation

Our constantly updated, ever-growing documentation, already well above 1500 pages, is a stand-alone product in itself. It is complete with configuration samples, real-world examples, and integration guides offering much more than a generic manual.

Comparison of features

This section compares the features of the NXLog Community Edition and the NXLog Enterprise Edition as well as highlights the differences between them.

There are three possible scenarios in how the features overlap and differentiate between the two editions:

  • The feature is present only in NXLog Enterprise Edition.

  • The feature is present in both editions of NXLog with the exact same functionality.

  • The feature is present in both editions of NXLog, but NXLog Enterprise Edition provides additional or enhanced functionality. In such cases, a link to a more detailed description of these differences is provided in the first column of the matrix table.

For easier interpretation, all NXLog features are grouped into the following logical subsections:

An NXLog Community Edition installation can be upgraded to NXLog Enterprise Edition. See the Deployment guide for your platform for more information.

Supported formats

One of the most important aspects of logs is the format. It is crucial for log files that need to be read by other applications. Above all, it is best if logs are stored as structured data, rather than unstructured text. The format affects availability, readability, manageability, and size.

This subsection compares the data formats NXLog supports. In addition to these formats, NXLog can read and process log data using pattern matching to transform unstructured data into normalized, structured data.

Table 1. Supported formats and patterns
Format and Pattern Name NXLog Community Edition NXLog Enterprise Edition Feature advantage

The existing log formats

AIX Audit Format (xm_aixaudit)

20

20

In combination with the im_file module, this module can collect and parse AIX Audit events.

Apple System Log (xm_asl)

20

20

Apple system logs stored as ASL files and can be read and parsed using this module.

Solaris OS Basic Security Module (BSM) Auditing API (im_bsm, xm_bsm)

20

20

Reads and processes BSM audit events either from file (xm_bsm) or directly from the kernel (im_bsm).

ArcSight Common Event Format (xm_cef)

20

20

Using this feature, the ArcSight CEF-formatted events can be both generated and read.

CSV (xm_csv)

20

20

Generation and parsing of delimiter-separated data is an essential feature of the xm_csv module.

Graylog Extended Log Format (xm_gelf)

20

20

This module enables receiving data in GELF format over the network as well as formatting data to GELF while forwarding it over the network.

JSON (xm_json)

20

20

Converts raw events to JSON format as well as parses events from received JSON logs.

Key-Value Pairs (xm_kvp)

20

20

Convenient method for distilling data formatted as key-value pairs (KVPs).

Log Event Extended Format (xm_leef)

20

20

Generates and reads the Log Event Extended Format (LEEF) commonly used by IBM Security QRadar products.

DNS Server Logs (xm_msdns)

20

20

Reads Windows DNS Server logs and parses over 20 fields commonly used with these events.

Multiline Log Messages (xm_multiline)

20

20

A complete solution for processing multiline log events. Supports any combination of header lines, footer lines, and fixed line counts.

NetFlow Payload (xm_netflow)

20

20

In combination with the im_udp module, this feature provides convenient parsing of Netflow protocol versions v1, v5, v7, v9, and IPFIX.

Radius NPS (xm_nps)

20

20

Allows for parsing file-based data in NPS Database format across all implementations, i.e. both IAS and NPS formatted data.

SNMP Trap Messages (xm_snmp)

20

20

Provides capabilities for reading SNMP v1, v2c, and v3 trap messages.

Syslog (xm_syslog)

20

20

Support for reading and writing all Syslog formats: BSD, IETF, and Snare.

W3C Extended Log File Format (xm_w3c)

20

20

Supports parsing of data in the W3C Extended Log File Format, Zeek format, and Microsoft Exchange Message Tracking logs.

Binary WTMP Files (xm_wtmp)

20

20

Enables processing of binary wtmp files when used in conjunction with the im_file module.

XML (xm_xml)

21

21

Enables parsing and writing XML-formatted data.

Parsing with patterns

Parsing with Grok Patterns (xm_grok)

20

20

Functionality for reading any kind of text log messages using Grok patterns.

Parsing with the Pattern Database (xm_pattern)

20

20

Efficient, non-linear pattern matching using a pattern database in XML format. Each pattern definition can contain multiple field definitions thus allowing multi-dimensional pattern matching.

The following features are available in both editions of NXLog, but NXLog Enterprise Edition provides additional or enhanced functionality.

CSV

Although both versions of NXLog support processing CSV data, only NXLog Enterprise Edition provides Field Count Enforcement with its StrictMode directive that accepts a boolean value. When set to TRUE, the parser will ignore any CSV line not having the same number of fields as those listed in the Fields directive.

Graylog Extended Log Format (GELF)

NXLog Enterprise Edition enables these additional options for processing data in the Graylog Extended Log Format:

TCP input

When set to GELF_TCP and used in conjunction with the TCP (im_tcp) input module, the InputType directive allows GELF data to be received over TCP.

Hidden fields

The IncludeHiddenFields directive, with a default boolean value of TRUE, supports inclusion of fields with names having a leading dot (.) or underscore (_) which would otherwise be discarded.

Event enrichment

GELF events are enriched with some additional fields: $EventTime, $FullMessage, $Hostname, $SeverityValue, $ShortMessage, $SourceLine, $SyslogFacility, and $version.

JSON

The xm_json module of NXLog Enterprise Edition has several directives for customizing how JSON data is parsed and prepared for output:

Custom datetime formats

The xm_json DateFormat overrides the global DateFormat.

Nested structure detection

By default, DetectNestedJSON is set to TRUE and preserves nested structures. FALSE will turn detection off and convert any values containing a JSON key-value pair to a flat string, e.g. {"key":{"subkey":42}} would be converted to {"key":"{\"subkey\":42}"}.

Flatten nested structures

The Flatten directive takes nested JSON fields and creates field names with dot notation to flatten the structure. For example, the following JSON structure:

{"event":{"time":"2015-01-01T00:00:00.000Z","severity":"ERROR"}}

will be converted to

{"event.time":"2015-01-01T00:00:00.000Z","event.severity":"ERROR"}
ForceUTF8

Converts the output to validated UTF-8 encoding regardless of the input encoding.

IncludeHiddenFields

Preserves hidden fields, i.e. those having names with a leading dot (.) or underscore (_).

ParseDate

Enables automatic parsing of strings beginning with a 4-digit year as timestamps.

PrettyPrint

Takes single-line JSON records, puts each key-value pair on a new line, and indents elements based on the depth of nesting for creating a more human-readable data record.

UnFlatten

Is the inverse of the Flatten process when using to_json() to recreate nested structures on output.

Multiline log messages

The xm_multiline module of NXLog Enterprise Edition supports the AutoFlush directive to automatically flush its read buffer once the PollInterval time has elapsed. This enables a more timely processing of the last event while it waits for new events to be read.

Syslog

NXLog Enterprise Edition can improve readability of Syslog data by replacing line breaks inside messages with other characters using the ReplaceLineBreaks directive. This option can be used with the to_syslog_bsd(), to_syslog_ietf(), and to_syslog_snare() procedures. The to_syslog_ietf() procedure’s local time information can be replaced with UTC/GMT timestamps using the UTCTimestamp directive.

XML

NXLog Enterprise Edition additionally provides features for working with XML, such as:

IgnoreRootTag

Results in fields being "flattened" by not having the XML root tag Event prefixed to them with dot notation. For example, $Event.timestamp will be simply $timestamp.

IncludeHiddenFields

Preserves hidden fields, i.e. those having names with a leading dot (.) or underscore (_).

ParseAttributes

Provides the ability to parse an XML field’s attributes in addition to its value. Given this XML sample

<Msg time='2014-06-27T00:27:38' type='ERROR'>foo</Msg>

will be parsed as $Msg.time, $Msg.type, and $Msg fields. Otherwise, only $Msg with its string value foo would be added to event record.

PrefixWinEvent

In the context of XML-formatted Windows events, this is the inverse process of IgnoreRootTag, in which events gleaned from parse_windows_eventlog_xml() will have their fields prefixed with either EventData or UserData using dot notation depending on which XML section they belong to.

Programming languages support

Supporting programming languages means that events can be read, processed, and forwarded using methods and procedures written in any of the supported languages. Being able to choose from a list of popular programming languages for developing custom modules offers a great deal of flexibility.

NXLog Enterprise Edition supports programming languages via the following types of modules:

  • Input modules to replace the existing NXLog input modules

  • Extension modules to process data coming from the input modules

  • Output modules to replace the existing NXLog output modules

The table below lists supported programming languages.

Table 2. Support for programming languages
Supported Language NXLog Community Edition NXLog Enterprise Edition Feature Advantage

Go ( im_go, om_go, xm_go)

20

20

Enables developing log processing modules using the Go or Golang programming language

Java (im_java, om_java, xm_java)

20

20

Facilitates developing custom Java applications that can easily integrate with the NXLog infrastructure

Perl (im_perl, om_perl, xm_perl)

20

20

Perl, with its native regular expression capabilities, allows rapid development of terse scripts for high performance, complex parsing of logs.

Python (im_python, om_python, xm_python)

20

20

Python scripts can be developed to customize how logs are read, processed, and output.

Ruby (im_ruby, om_ruby, xm_ruby)

20

20

Ruby, with its native support for structured data formats like XML and JSON, can be used to develop applications for custom log processing.

Data transformation

The ability to transform captured log data is an essential feature of any log collection strategy.

The following table shows the features that can be used to transform data according to each edition of NXLog.

Table 3. Support for data transformation
Supported Feature NXLog Community Edition NXLog Enterprise Edition Feature Advantage

Converting via Character Sets (xm_charconv)

20

20

Log data can be converted between character sets (codepages) prior to parsing.

Blacklisting and Whitelisting of Log Entries (xm_filelist)

20

20

Based on the contents of each file, files can be whitelisted or blacklisted using this feature.

File Operations (xm_fileop)

20

20

Provides convenient methods for implementing log rotation and retention policies for the specified log files.

Manipulating Fields (xm_rewrite)

20

20

Enables renaming and dropping of specific fields, as well as the enrichment of events with new fields defined in Exec statements.

Encryption and Decryption of Logs (xm_crypto)

20

20

Enables log files to be encrypted and decrypted using AES symmetric encryption for secure storage and/or transport over a network.

Compression and Decompression of Logs (xm_zlib)

20

20

Allows compressing log files using either gzip or zlib prior to transport.

Resolving IP addresses to Hostnames (xm_resolver)

20

20

When log data needs to be human readable, enriching log events with the hostnames, user names, and groups names is preferable to IP addresses, user IDs, and group IDs.

The following features are available in both editions of NXLog, but NXLog Enterprise Edition provides additional or enhanced functionality.

Converting via character sets

The following optional directives are only available with NXLog Enterprise Edition:

File operations

NXLog Enterprise Edition has a file_hash() function to return the calculated hash of the specified file.

Administration and monitoring

NXLog provides several administration and monitoring features, such as remote administration of agents as well as monitoring network traffic and file systems.

Table 4. Administration and monitoring support
Supported Feature NXLog Community Edition NXLog Enterprise Edition Feature Advantage

Secure Remote Administration (xm_admin)

20

20

Enables secure remote administration for NXLog agents and various monitoring tools using JSON or SOAP over HTTP/HTTPS.

Execution of External Scripts (im_exec, om_exec, xm_exec)

20

20

Supports the execution of an external program or script on startup.

Changes in Files and Directories (im_fim)

20

20

This feature enables the monitoring of changes in specific files and directories for integrity monitoring.

Passive Monitoring of Network Traffic (im_pcap)

20

20

Enables NXLog to passively monitor and log network traffic for various protocols.

Windows Registry Monitoring (im_regmon)

20

20

Provides scanning of Windows registry and generates event records when changes are detected.

The following features are available in both editions of NXLog, but NXLog Enterprise Edition provides additional or enhanced functionality.

Execution of external scripts

The xm_exec module of NXLog Enterprise Edition additionally supports execution of commands as a function.

The Restart directive of NXLog Enterprise Edition can restart a process if it terminates unexpectedly.

Log sources

NXLog supports log collection and processing from a wide variety of log sources. Practically any operating system found in enterprise computing environments is supported, as well as their native log sources. In most cases, the native modules that ship with NXLog for collecting logs suffice without any need for any additional software. The following table lists the modules designed for specific log sources.

Table 5. Log source types supported
Source Name NXLog Community Edition NXLog Enterprise Edition Feature Advantage

Accounting Logs From a Linux or BSD Kernel (im_acct)

20

20

Enables collecting and processing logs from a Linux or BSD kernel.

AIX Audit (im_aixaudit)

20

20

Along with the xm_aixaudit module, AIX Audit logs can be read and parsed directly from the kernel.

macOS Endpoint Security (im_maces)

20

20

Collects logs from Apple’s Endpoint Security auditing system.

Microsoft Azure Application Logs (im_azure)

20

20

Parses logs from Microsoft Azure applications received over various TLS and SSL protocols.

Check Point Device Logs (im_checkpoint)

20

20

Remote collection of logs from Check Point devices using the OPSEC LEA protocol.

Database Logs via the libdbi library (im_dbi, om_dbi)

20

20

Convenient method for pulling and saving data to external databases using the libdbi database abstraction library.

Kernel Application Logs Using Event Tracing for Windows (im_etw)

20

20

High-performance log collection using Event Tracing for Windows (ETW).

Internal NXLog Logs (im_internal)

21

21

Enables reading internal NXLog logs.

Elasticsearch Server (om_elasticsearch)

20

20

Enables forwarding logs to an Elasticsearch server.

Kafka Server Logs (im_kafka, om_kafka)

20

20

Enables the collecting of event records from a Kafka topic as well as publishing event records to a Kafka topic.

Logs From the Kernel Log Buffer (im_kernel)

20

20

Collects events from the kernel log buffer on Unix-like operating systems.

Windows Event Log (im_mseventlog, im_msvistalog)

20

20

Collects Windows Event Log messages using its native API.

Windows Event Collector (im_wseventing)

20

20

Using the Windows Event Forwarding infrastructure, this feature enables collecting events from Windows Event Log.

Kernel Logs via Audit Rules (im_linuxaudit)

20

20

Collects logs directly from the kernel using custom rules without the need for installing auditd or other userspace software.

Database Table Logs via ODBC (im_odbc, om_odbc)

20

20

Uses ODBC for storing logs in ODBC-compliant databases.

Forwarding Logs to Raijin Server ( om_raijin)

20

20

Sends batched JSON records to Raijin Server for further analysis and archival.

Redis Server Log Tranfers (im_redis, om_redis)

20

20

Enables the retrieval of data stored in a Redis server as well as populating a Redis server with log data.

Systemd Journal Logs (im_systemd)

20

20

Reads, parses, and processes events from the systemd journal.

Windows Performance Counters (im_winperfcount)

20

20

Enables the polling of various Windows Performance Counters to create event records.

The following features are available in both editions of NXLog, but NXLog Enterprise Edition provides additional or enhanced functionality.

Internal NXLog logs

LogqueueSize

Sets the number messages which are queued if delivery is delayed or blocked.

Event log enrichment

Additional fields which contain the module name and type.

Logs from the kernel log buffer

NXLog Enterprise Edition supports collecting kernel logs from various Linux, BSD, and macOS systems. However, NXLog Community Edition supports only Linux systems.

NXLog Enterprise Edition also provides the following directives:

DeviceFile

For accessing the device file to read events on non-Linux systems.

PollInterval

Sets the interval to check for new events.

Windows Event Log

The Enterprise Edition’s im_msvistalog module supports the following optional directives not found in NXLog Community Edition:

AddPrefix

Adds the EventData prefix to fields which are parsed from the EventData section of the event.

ReadBatchSize

Specifies the number of event records the Windows Event Log API will pass to NXLog for further processing.

CaptureEventXML

Enables the $EventXML field to store the raw XML-formatted event data.

File

Enables NXLog to read directly from the .evt, .evtx, or .etl log files.

Language

Sets the locale to use for rendering language/region-specific event fields such as date/time, currency, etc.

RemoteAuthMethod

Sets the authentication method.

RemoteDomain

Specifies the authentication domain of the remote server for collecting event logs.

RemotePassword

Accepts the user password of the remote server for collecting event logs.

RemoteServer

Sets the name of the remote server from which event logs will be captured.

RemoteUser

Sets the user name on the remote server for collecting event logs.

ResolveGUID

Enables GUID to resolve values of the object names in the $Message field.

ResolveSID

Enables SID to resolve values of the object names in the $Message field.

TolerateQueryErrors

Will prevent the module from starting if any source is invalid when set to FALSE (the default value).

Protocols

Both editions of NXLog provide support for various protocols for working with logs, see the table below.

Table 6. Supported protocols
Protocol Name NXLog Community Edition NXLog Enterprise Edition Feature Advantage

Batched Compression (im_batchcompress, om_batchcompress)

20

20

Enables sending/receiving batches of log messages that are compressed and optionally encrypted to/from a remote NXLog agent.

Files (im_file, om_file)

20

20

Facilities reading log messages from files and writing processed data to them.

WebHDFS (om_webhdfs)

20

20

Using the WebHDFS protocol, this feature can be used to store logs in Hadoop HDFS.

HTTP (im_http, om_http)

20

20

Convenient log message communication using over the HTTP protocol, both inbound and outbound.

Named pipes (im_pipe, om_pipe)

20

20

With this feature, named pipes of Unix-like operating systems can be utilized to send and receive log messages.

TLS/SSL ( im_ssl, om_ssl)

20

20

Provides TLS/SSL transportation capabilities for secure forwarding and retrieval of logs.

TCP (im_tcp, om_tcp)

20

20

Accepts TCP connections for receiving event data and can send events to remote hosts over TCP.

UDP (im_udp, om_udp)

20

20

Enables log data to be sent and received as datagrams using the UDP protocol.

UDP With Spoofing (om_udpspoof)

20

20

With spoofing enabled, UDP packets will contain the IP address of the originating client that produced the message instead of the forwarding server.

Unix Domain Sockets (im_uds, om_uds)

20

20

Enables log data to be sent or received over a Unix domain socket.

ZeroMQ (im_zmq, om_zmq)

20

20

Supports message transport over ZeroMQ, a scalable high-throughput messaging library.

The following features are available in both editions of NXLog, but NXLog Enterprise Edition provides additional or enhanced functionality.

Files

The im_file module of NXLog Enterprise Edition additionally provides the following optional directives:

Exclude

Specifies a file or a set of files to be excluded.

InputType

Sets the input reader function.

NoEscape

Specifies whether the backslash (``) in file paths should be disabled as an escape sequence.

OnEOF

A group of statements to be executed after a file has been completely read (on end-of-file).

The om_file module provides the optional CacheSize directive to keep files open and reduce the overhead caused by open/close operations.

HTTP

NXLog Community Edition provides only the output HTTP capabilities via the om_http module. NXLog Enterprise Edition supports the input and output capabilities via the im_http and om_http modules.

NXLog Enterprise Edition supports the add_http_header() procedure to dynamically add a custom HTTP header to requests.

The Enterprise Edition’s om_udp module supports multiple definitions of the Host directive which can be used to configure failover.

The URL directive of NXLog Enterprise Edition can be specified multiple times for configuring failover. Its HTTPSCADir and HTTPSCAFile directives provide additional support for self-signed certificates.

NXLog Enterprise Edition supports the following optional directives not found in NXLog Community Edition:

AddHeader

An additional header to be added to each HTTP request.

BatchMode

Toggles whether the data is sent as single event or a batch of events.

ContentType

Specifies the Content-Type HTTP header which can be used in conjunction with the BatchMode directive.

FlushInterval

Sets the period of time after which accumulated data will be sent in batch mode.

FlushLimit

The number of events to be batched together in a single POST request.

HTTPSCAThumbprint

The certificate thumbprint of the certificate authority (CA) which can be used to look up the CA certificate from the Windows certificate store.

HTTPSCertThumbprint

The certificate thumbprint to be used for the SSL handshake.

HTTPSCRLDir

The path to the certificate revocation list. This list is used for verifying the certificate of the remote HTTPS server.

HTTPSSSLCipher

Can be used to set the permitted SSL cipher list, overriding the default.

HTTPSSSLCiphersuites

Can be used to define the permitted SSL cipher list in case the HTTPSSSLProtocol directive is set to TLSv1.3.

HTTPSSSLProtocol

Can be used to set the allowed TLS/SSL protocol(s).

HTTPSSSLCompression

Enables data compression when sending over the network.

ProxyAddress

The IP address of the proxy server.

ProxyPort

The port number of the proxy server.

SNI

The host name for Server Name Indication (SNI) in HTTPS mode.

Named pipes

NXLog Community Edition only supports the named pipes input module, whereas NXLog Enterprise Edition allows logs to be sent to named pipes on UNIX-like operating systems with the om_pipe output module.

TLS/SSL

The im_ssl module of NXLog Enterprise Edition supports self-signed certificates and the following optional directives:

CAThumbprint

The certificate thumbprint of the certificate authority (CA) which can be used to look up the CA certificate from the Windows certificate store.

CertThumbprint

The certificate thumbprint to be used for the SSL handshake.

SSLCipher

Can be used to set the permitted SSL cipher list, overriding the default.

SSLCiphersuites

Can be used to define the permitted SSL cipher list in case the HSSLProtocol directive is set to TLSv1.3.

SSLCompression

Enables data compression when sending over the network.

SSLProtocol

Sets the allowed TLS/SSL protocol(s).

The om_ssl module of NXLog Enterprise Edition provides additional functionalities to the following directives:

Host

Can be specified multiple times to work in a failover configuration.

CADir and CAFile

Support self-signed certificates.

It also supports the following optional directives:

CAThumbprint

The certificate thumbprint of the certificate authority (CA) which can be used to look up the CA certificate from the Windows certificate store.

CertThumbprint

The certificate thumbprint to be used for the SSL handshake.

LocalPort

The local port number of the connection.

SNI

The host name for Server Name Indication (SNI).

SSLCipher

Can be used to set the permitted SSL cipher list, overriding the default.

SSLCiphersuites

Can be used to define the permitted SSL cipher list in case the SSLProtocol directive is set to TLSv1.3.

SSLCompression

Enables data compression when sending over the network.

SSLProtocol

Sets the allowed TLS/SSL protocol(s).

TCPNoDelay

Can be used to turn off the network optimization performed by Nagle’s algorithm.

TCP

The im_tcp module of NXLog Enterprise Edition supports the ReusePort directive to enable synchronous listening on the same port by multiple module instances.

The Host directive of the om_tcp module can be defined several times to work in a failover configuration.

The om_tcp module also supports the following optional directives:

LocalPort

The local port number of the connection.

Reconnect

Sets the reconnect interval in seconds.

TCPNoDelay

Turns off network optimization performed by Nagle’s algorithm.

UDP

The im_udp module of NXLog Enterprise Edition provides the following directives:

ReusePort

Enables synchronous listening on the same port by multiple module instances.

UseRecvmmsg

Specifies that the recvmmsg() system call should be used whenever possible. This improves performance by enabling the reception of multiple messages per call.

The Enterprise Edition’s om_udp module supports multiple definitions of the Host directive which can be used to configure failover.

The om_udp module of NXLog Enterprise Edition also supports the following directives:

LocalPort

The local port number of the connection.

OutputType

Sets the output writer function for UDP datagrams.

Reconnect

Sets the reconnect interval in seconds.

Miscellaneous modules

Table 7. Supported operations
Operation Name NXLog Community Edition NXLog Enterprise Edition Feature Advantage

Collecting Mark (im_mark)

20

20

For indicating periodic activity to ensure the logger is running when there are no log messages being received in from other sources.

Testing NXLog Configuration (im_null, om_null)

20

20

Provides testing capabilities and allows creating dummy routes.

Generating Test Events (im_testgen)

20

20

For generating simple events which can be read and processed before running the system in production.

Simulation of Output Messages Blocking (om_blocker)

20

20

Additional testing capability which can block messages to simulate a blocked route.

Buffering of Log Messages (pm_buffer)

20

20

Eliminates loss of log data by configuring log message buffers.

Event Correlation (pm_evcorr)

20

20

Provides event correlation functionality in addition to available NXLog features, such as variables and statistical counters.

ASLR (address space layout randomization)

20

20

Binaries built using ASLR are more resistant to memory allocation exploits. (Windows only).

The following features are available in both editions of NXLog, but NXLog Enterprise Edition provides additional or enhanced functionality.

Buffering of log messages

NXLog Enterprise Edition supports the CreateDir directive to create an output directory before opening the output file for writing.

Event correlation

NXLog Enterprise Edition supports the Group rule to group messages based on the specified correlation context.

Supported core functionalities

As opposed to module instance directives which do not extend beyond the scope of their respective instances, general and global directives can be applied to the entire NXLog configuration. Both of these directives are compared in this section:

General directives

General directives provide configuration reusability. This reduces the amount of configuration needed, increases flexibility, and eliminates errors by using well-tested configuration components.

This subsection compares general directives which are supported by both editions of NXLog.

Table 8. Supported general directives
Directive Name NXLog Community Edition NXLog Enterprise Edition Feature Advantage

define

20

20

This feature allows you to define a constant or macro which can be used as directory name, code snippet, or regular expression. Once defined, you can insert it in a uniform manner within any of the module instances.

envvar

20

20

Retrieves environment values to make them available for use with various NXLog components or in any module instance.

include

20

20

Specifies a file or set of files which can extend the current NXLog configuration. After being included, the file configuration can be reused within the scope of the configuration.

include_stdout

20

20

Using this directive, the configuration content can be read through external commands. This may be useful when a configuration is generated dynamically and cannot be known beforehand.

Global directives

Global directives control the overall behavior of NXLog and provide additional features which can improve its functionality.

Caching and batching

Caching adjusts the NXLog performance to specific needs. Batching log messages saves resources and optimizes the network performance by grouping messages together before forwarding them.

The following global directives can be used to fine-tune NXLog’s caching and batching settings.

Table 9. Caching and batching directives
Directive Name NXLog Community Edition NXLog Enterprise Edition Feature Advantage

Caching directives

CacheDir

20

20

Specifies the directory where the cache file will be stored.

CacheFlushInterval

20

20

This option adjusts logging performance by specifying the interval for flushing the in-memory position cache to the cache file.

CacheInvalidationTime

20

20

Prevents the cache from growing indefinitely by discarding cache entries after their storage period exceeds this directive’s setting.

CacheSync

20

20

Guarantees crash-safe operation by delaying the in-memory cache flush to disk.

NoCache

20

20

Globally disables NXLog’s caching mechanism.

Batching directives

BatchSize

20

20

Specifies the maximum number of records to collect in a batch before forwarding them to the next instance(s) in the route.

BatchFlushInterval

20

20

Sets a timeout before forwarding each batch of messages to the next instance(s) in the route.

Preventing data loss

Persistent queuing provides the capability of temporarily writing the incoming log data to disk in case the in-memory queue overflows. Using flow control, NXLog operations can be suspended in case any module instance (input, processor, or output) becomes blocked or unable to process events. Both persistent queuing and flow control help prevent data loss.

The following general directives provide options for customizing NXLog’s persistent queuing and flow control features.

Table 10. Persistent queuing directives
Directive Name NXLog Community Edition NXLog Enterprise Edition Feature Advantage

Persistent queuing

LogqueueDir

20

20

Specifies the directory where files of the persistent queue will be stored. Persistent storage prevents data loss and thus improves logging reliability.

LogqueueSize

20

20

In addition to the LogqueueDir, this directive offers flexibility in controlling the log queue size needed for persistent queuing.

PersistLogqueue

20

20

This directive determines whether or not log queues should be saved to disk when necessary.

SyncLogqueue

20

20

To make the logging process more reliable and crash-safe, this option provides syncing capabilities for disk-based queues of processor and output module instances.

Flow control

FlowControl

20

20

This feature specifies whether the input-to-output flow should be controlled by NXLog to prevent data loss in case of failure at any stage.

FlowControlFIFO

20

20

A convenient way to enable FIFO mode for modules with disabled flow control functionality.

Processing dates

Customization of date formatting can be important when date/time information needs to be normalized and indexed for enabling it to used in time-based queries.

Table 11. Supported date-related directives
Directive Name NXLog Community Edition NXLog Enterprise Edition Feature Advantage

DateFormat

20

20

Overrides the default date/time pattern which is used by the internal NXLog logging mechanism for converting datetime value to string value.

GenerateDateInUTC

20

20

Allows using UTC instead of local time when generating dates in the YYYY-MM-DD hh:mm:ss format.

ParseDateInUTC

20

20

Enables parsing the date in the YYYY-MM-DD hh:mm:ss format as UTC instead of processing it as local time.

Memory and performance

These NXLog directives can be used to adjust performance and memory consumption.

Table 12. Supported directives
Directive Name NXLog Community Edition NXLog Enterprise Edition Feature Advantage

StringLimit

20

20

To avoid memory exhaustion, each message string can be limited to a maximum size. This will guard NXLog against denial-of-service scenarios.

SuppressRepeatingLogs

20

20

To avoid excessive consumption of disk space, this directive restricts the number of messages the internal logger can accept.

Threads

20

20

For fine-tuning NXLog’s performance, its number of threads can be explicitly set. If not specified, the number of threads is selected automatically.

Debugging capabilities

The following directives are available for setting NXLog’s debugging options.

Table 13. Supported debugging directives
Directive Name NXLog Community Edition NXLog Enterprise Edition Feature Advantage

LogFile

20

20

Provides self-logging capabilities for convenient monitoring and debugging of NXLog-related problems.

LogLevel

20

20

Specifies the logging level for internal NXLog messages set by the LogFile directive. It features a granularity of five levels: either CRITICAL, ERROR, WARNING, INFO, or DEBUG.

NoFreeOnExit

20

20

Additional debugging option which allows showing proper stack trace in module function calls.

Other global directives

These remaining directives offer diverse features and options available with NXLog that do not fit exactly in any the previous categories covered in this section.

Table 14. List of global directives
Directive Name NXLog Community Edition NXLog Enterprise Edition Feature Advantage

Capabilities

20

20

Allows specifying the required Linux capabilities to enable privileged kernel calls.

EscapeGlobPatterns

20

20

Specifies the backslash (\) character to escape glob patterns or wildcarded entries; this measure provides convenience and unification with other company-wide approaches, standards, or other requirements.

Group

20

20

Linux-only directive to specify the group ID that NXLog run as.

IgnoreErrors

20

20

With this directive enabled, NXLog will continue to run even if it encounters configuration-related problems. If set to FALSE, NXLog will stop if errors are encountered.

ModuleDir

20

20

This option allows you to override the standard location where NXLog loadable modules are stored by default.

Panic

20

20

This directive provides a choice of three options, HARD, SOFT, or OFF, that will determine how NXLog should react when it finds itself in a panic condition (a critical state).

PidFile

20

20

To make configuration more convenient, this feature allows overriding the default PID file name which NXLog uses during its operation.

ReadTimeout

20

20

Using this feature, the default exit condition timeout of the nxlog-processor(8) software can be overridden.

RootDir

20

20

As with the SpoolDir directive, this feature offers flexibility in setting a custom root directory location for NXLog.

SpoolDir

20

20

As with the RootDir directive, this feature offers flexibility in setting a custom working directory location for NXLog.

User

20

20

For Unix-like operating systems this directive specifies the user NXLog will run as and consequently which permissions it will have.

What NXLog is not

NXLog provides a broad range of features for collecting, processing, forwarding, and storing log data. However, NXLog is not a SIEM product and does not provide:

  • a graphical interface (or "dashboard") for searching logs and displaying reports,

  • vulnerability detection or integration with external threat data,

  • automatic analysis and correlation algorithms, or

  • pre-configured compliance and retention policies.

NXLog does provide processing features that can be used to set up analysis, correlation, retention, and alerting; NXLog can be integrated with many other products to provide a complete solution for aggregation, analysis, and storage of log data.