Apache NiFi

Apache NiFi is a platform for real-time data ingestion and distribution among various source and destination systems supporting a wide variety of data formats and protocols. NXLog can be configured to send and receive log data from Apache NiFi, supporting several different protocols.

Installation and configuration of Apache NiFi

Apache NiFi runs within a JVM on a host connected to your local network.

For details on the Apache NiFi installation process, visit the How to install and start NiFi section of the Apache NiFi System Administrator’s Guide.

After Apache NiFi has been started, it can be accessed at http://localhost:8080/nifi.

Configuring Apache NiFi processors

To work with data, Apache NiFi uses processors.

Processors are categorized by their names. For example, the names of data ingestion processors start with Get or Listen, such as GetHTTP or ListenTCP. The names of data sending processors start with Put, like PutUDP or PutSyslog.

The following steps lay out the procedure of configuring Apache NiFi to exchange log data from NXLog. For this example, the configuration of the ListenTCP processor is used. These configuration steps are carried out in the Apache NiFi environment by placing components on the canvas.

To add and configure a new processor, follow these steps:

From the toolbar, drag and drop a Processor component onto the canvas.

This will open the Add Processor dialog window.
Find the required processor from the list and click ADD, then right-click the processor pictogram and select Configure.
In the Configure Processor dialog window under the SETTINGS tab, configure the automatic termination of relationships according to your needs.
Under the PROPERTIES tab, configure all required fields in bold.
Once the configuration is complete, click APPLY.

For an in-depth explanation of all options, visit the Configuring a Processor section in the Apache NiFi documentation.

Processors are the basic blocks for creating dataflows. After the processor configuration has been finished, they should be arranged and bound to create dataflows. For the details of creating data flows, visit the Building a DataFlow section in the Apache NiFi documentation.

Sending logs to Apache NiFi

This example demonstrates how to configure NXLog to forward logs to Apache NiFi via TCP. In this instance, the ListenTCP processor is used on the Apache NiFi side to receive the data.

Example 1. Forwarding kernel logs to Apache NiFi with NXLog

The configuration below reads log messages using the im_kernel module from a Unix-like operating system. After reading, the data is parsed and converted to JSON, and the output is sent to Apache NiFi using the om_tcp module.

nxlog.conf

<Extension json>
    Module    xm_json
</Extension>

<Extension syslog>
    Module    xm_syslog
</Extension>

<Input from_kernel>
    Module    im_kernel
</Input>

<Output to_tcp>
    Module    om_tcp
    Host      192.168.31.53:10000
    <Exec>
       parse_syslog();
       to_json();
    </Exec>
</Output>

This is the sample data formatted to JSON before sending it to Apache NiFi.

Output sample

{
  "SourceName": "kernel",
  "EventReceivedTime": "2021-03-10T18:53:56.082022+03:00",
  "SourceModuleName": "from_kernel",
  "SourceModuleType": "im_kernel",
  "SyslogFacilityValue": 0,
  "SyslogFacility": "KERN",
  "SyslogSeverityValue": 7,
  "SyslogSeverity": "DEBUG",
  "SeverityValue": 1,
  "Severity": "DEBUG",
  "EventTime": "2021-03-10T18:57:45.064529+03:00",
  "Hostname": "Ubuntu-VM",
  "Message": "[   29.055026] ISO 9660 Extensions: RRIP_1991A"
}

To forward log data over SSL/TLS, both NXLog and Apache NiFi need additional configuration to work with certificates and private keys. NXLog uses separate directives to specify certificates and keys, while Apache NiFi can be configured to work with keystores and truststores. For more information about the SSL/TLS configuration of Apache NiFI, see the Security Configuration section of the Apache NiFi Documentation.

To establish a secure connection, Apache NiFi can use the ListenTCP processor with the Port property configured to listen on port 10000. Additionally, you should set SSL Context Service to work with the keystore. Also, you need to set the Client Auth property to the correct client authentication policy.

Example 2. Forwarding logs to Apache NiFi over SSL with NXLog

NXLog Enterprise Edition exclusive feature

The NXLog configuration below uses the im_etw module to collect DNS traffic data. Using the xm_json module, all data are converted to JSON, and the om_ssl module is applied to send data to Apache NiFi over SSL/TLS.

nxlog.conf

<Extension json>
    Module            xm_json
</Extension>

<Input from_dns>
    Module            im_etw
    Provider          Microsoft-Windows-DNS-Client
</Input>

<Output to_nifi_ssl>
    Module            om_ssl
    Host              192.168.31.53:10000
    Exec              to_json();
    AllowUntrusted    TRUE
    CertFile          C:\certs\myCertificate.crt
    CertKeyFile       C:\certs\myPrivateKey.pem
</Output>

Below is the output sample of the data forwarded to Apache NiFi.

Output sample

{
  "SourceName": "Microsoft-Windows-DNS-Client",
  "ProviderGuid": "{1C95126E-7EEA-49A9-A3FE-A378B03DDB4D}",
  "EventID": 1019,
  "Version": 0,
  "ChannelID": 16,
  "OpcodeValue": 0,
  "TaskValue": 1019,
  "Keywords": "9223372036854775808",
  "EventTime": "2021-03-09T07:28:14.084075-08:00",
  "ExecutionProcessID": 104,
  "ExecutionThreadID": 1228,
  "EventType": "INFO",
  "SeverityValue": 2,
  "Severity": "INFO",
  "Hostname": "WIN-ET85AK2E1J1",
  "Domain": "NT AUTHORITY",
  "AccountName": "NETWORK SERVICE",
  "UserID": "S-1-5-20",
  "AccountType": "Well Known Group",
  "Flags": "EXTENDED_INFO|IS_64_BIT_HEADER|PROCESSOR_INDEX (577)",
  "Location": "0",
  "Context": "0",
  "EventReceivedTime": "2021-03-09T07:28:15.084417-08:00",
  "SourceModuleName": "from_dns",
  "SourceModuleType": "im_etw"
}

Collecting logs from Apache NiFi

While Apache NiFi can send logs using multiple processors to NXLog, NXLog can receive data using almost any network protocol so there several ways to send data from Apache NiFi to NXLog.

One way to send data to NXLog from Apache NiFi is to utilize the PutSyslog processor with its Hostname and Protocol properties specified. This type of NiFi configuration is the basis for the following example.

Example 3. Collecting syslog-formatted data from Apache NiFi over UDP with NXLog

This configuration uses the im_udp module to collect data from Apache NiFi. Once received, logs can be saved to file or forwarded over the network.

nxlog.conf

<Extension syslog>
    Module        xm_syslog
</Extension>

<Input from_udp>
    Module        im_udp
    ListenAddr    192.168.31.41:10000
    Exec          parse_syslog();
</Input>

Verifying data in Apache NiFi

The user interface of Apache NiFi allows verifying data traffic for the whole flow and by processor.

To see the overall data traffic, click Global Menu in the upper right corner of the NiFi user interface and select Data Provenance.

This will open the Data Provenance dialog window as per below.
To view the input traffic of a specific processor, you can right-click a processor on the NiFi canvas and select View data provenance.

In the NiFi Data Provenance dialog window, you can observe the data traffic details for the selected processor.

Disclaimer

While we endeavor to keep the information in this topic up to date and correct, NXLog makes no representations or warranties of any kind, express or implied about the completeness, accuracy, reliability, suitability, or availability of the content represented here. We update our screenshots and instructions on a best-effort basis.

The accurateness of the content was tested and proved to be working in our lab environment at the time of the last revision with the following software versions:

NXLog version 5.6.7727
Apache NiFi version 1.18.0
Ubuntu version 20.04.4 LTS
Windows Server 2016

Last revision: 01 November 2022