Apache NiFi

Apache NiFi is a platform for real-time data ingestion and distribution among various source and destination systems supporting a wide variety of data formats and protocols. NXLog can be configured to send and receive log data from Apache NiFi, supporting several different protocols.

Installation and configuration of Apache NiFi

Apache NiFi runs within a JVM on a host connected to your local network.

For the details of the Apache NiFi installation process, visit the How to install and start NiFi section of the Apache NiFi System Administrator’s Guide.

After Apache NiFi has been started, it can be accessed at http://localhost:8080/nifi.

Configuring Apache NiFi processors

To work with data, Apache NiFi uses processors.

Processors are categorized by their names. For example, the names of data ingestion processors start with Get or Listen, such as GetHTTP or ListenTCP. The names of data sending processors start with Put, like PutUDP or PutSyslog.

The following steps lay out the procedure of configuring Apache NiFi to exchange log data from NXLog. For this example, the configuration of the ListenTCP processor is used. These configuration steps are carried out in the Apache NiFi environment by placing components on the canvas.

To add and configure a new processor, follow these steps:

  1. From the toolbar, drag and drop a Processor component onto the canvas.

    The Processor component

    This will open the Add Processor dialog window.

    The Add Processor window
  2. Find the required processor from the list and click ADD, then right-click the processor pictogram and select Configure.

    The Configure menu
  3. In the Configure Processor dialog window under the SETTINGS tab, configure the automatic termination of relationships according to your needs.

    The Configure Processor window
  4. Under the PROPERTIES tab, configure all required fields in bold.

    The PROPERTIES tab
  5. Once the configuration is complete, click APPLY.

For an in-depth explanation of all options, visit the Configuring a Processor section in the Apache NiFi documentation.

Processors are the basic blocks for creating dataflows. After the processor configuration has been finished, they should be arranged and bound to create dataflows. For the details of creating data flows, visit the Building a DataFlow section in the Apache NiFi documentation.

Sending logs to Apache NiFi

This example demonstrates how to configure NXLog to forward log events to Apache NiFi via TCP. In this instance, the ListenTCP processor is used on the Apache NiFi side to receive the data.

Example 1. Forwarding kernel logs to Apache NiFi

The configuration below reads log messages using the im_kernel module from a Unix-like operating system. After reading, the data is parsed and converted to JSON, and the output is sent to Apache NiFi using the om_tcp module.

nxlog.conf
<Extension json>
    Module              xm_json
</Extension>

<Extension syslog>
    Module              xm_syslog
</Extension>

<Input from_kernel>
    Module              im_kernel
</Input>

<Output to_tcp>
    Module              om_tcp
    Host                192.168.31.53:10000
    <Exec>
       parse_syslog();
       to_json();
    </Exec>
</Output>

This is the sample data formatted to JSON before sending it to Apache NiFi.

Output sample
{
  "SourceName": "kernel",
  "EventReceivedTime": "2021-03-10T18:53:56.082022+03:00",
  "SourceModuleName": "from_kernel",
  "SourceModuleType": "im_kernel",
  "SyslogFacilityValue": 0,
  "SyslogFacility": "KERN",
  "SyslogSeverityValue": 7,
  "SyslogSeverity": "DEBUG",
  "SeverityValue": 1,
  "Severity": "DEBUG",
  "EventTime": "2021-03-10T18:57:45.064529+03:00",
  "Hostname": "Ubuntu-VM",
  "Message": "[   29.055026] ISO 9660 Extensions: RRIP_1991A"
}

To forward log data over SSL/TLS, both NXLog and Apache NiFi need additional configuration to work with certificates and private keys. NXLog uses separate directives to specify certificates and keys, while Apache NiFi can be configured to work with keystores and truststores. For more information about the SSL/TLS configuration of Apache NiFI, see the Security Configuration section of the Apache NiFi Documentation.

To establish a secure connection, Apache NiFi can use the ListenTCP processor with the Port property being configured to listen on port 10000. Additionally, you should set SSL Context Service to work with the keystore. Also, you need to set the Client Auth property to the correct client authentication policy.

Example 2. Forwarding logs to Apache NiFi over SSL

The NXLog configuration below uses the im_etw module to collect DNS traffic data. Using the xm_json module, all data are being converted to JSON, and the om_ssl module is applied to send data to Apache NiFi over SSL/TLS.

nxlog.conf
<Extension json>
    Module   xm_json
</Extension>

<Input from_dns>
    Module              im_etw
    Provider            Microsoft-Windows-DNS-Client
</Input>

<Output to_nifi_ssl>
    Module              om_ssl
    Host                192.168.31.53:10000
    Exec                to_json();
    AllowUntrusted      TRUE
    CertFile            C:\certs\myCertificate.crt
    CertKeyFile         C:\certs\myPrivateKey.pem
</Output>

Below is the output sample of the data forwarded to Apache NiFi.

Output sample
{
  "SourceName": "Microsoft-Windows-DNS-Client",
  "ProviderGuid": "{1C95126E-7EEA-49A9-A3FE-A378B03DDB4D}",
  "EventID": 1019,
  "Version": 0,
  "ChannelID": 16,
  "OpcodeValue": 0,
  "TaskValue": 1019,
  "Keywords": "9223372036854775808",
  "EventTime": "2021-03-09T07:28:14.084075-08:00",
  "ExecutionProcessID": 104,
  "ExecutionThreadID": 1228,
  "EventType": "INFO",
  "SeverityValue": 2,
  "Severity": "INFO",
  "Hostname": "WIN-ET85AK2E1J1",
  "Domain": "NT AUTHORITY",
  "AccountName": "NETWORK SERVICE",
  "UserID": "S-1-5-20",
  "AccountType": "Well Known Group",
  "Flags": "EXTENDED_INFO|IS_64_BIT_HEADER|PROCESSOR_INDEX (577)",
  "Location": "0",
  "Context": "0",
  "EventReceivedTime": "2021-03-09T07:28:15.084417-08:00",
  "SourceModuleName": "from_dns",
  "SourceModuleType": "im_etw"
}

Collecting logs from Apache NiFi

While Apache NiFi can send logs using multiple processors to NXLog, NXLog can receive data using almost any network protocol so there are a number ways to send data from Apache NiFi to NXLog.

One way to send data to NXLog from Apache NiFi is to utilize the PutSyslog processor with its Hostname and Protocol properties specified. This type of NiFi configuration is the basis for the following example.

Example 3. Collecting syslog-formatted data from Apache NiFi over UDP

This configuration uses the im_udp module to collect data from Apache NiFi. Once received, logs can be saved to file or forwarded over the network.

nxlog.conf
<Extension syslog>
    Module           xm_syslog
</Extension>

<Input from_udp>
    Module           im_udp
    ListenAddr       192.168.31.41:10000
    Exec             parse_syslog();
</Input>

Verifying data in Apache NiFi

The user interface of Apache NiFi allows verifying data traffic for the whole flow and by processor.

  1. To see the overall data traffic, click Global Menu in the upper right corner of the NiFi user interface and select Data Provenance.

    Global Menu of the Apache NiFi UI

    This will open the Data Provenance dialog window as per below.

    The Data Provenance dialog window
  2. To view the input traffic of a specific processor, you can right-click a processor on the NiFi canvas and select View data provenance.

    The View data provenance menu point

    In the NiFi Data Provenance dialog window, you can observe the data traffic details for the selected processor.

    The NiFi Data Provenance window
Disclaimer

While we endeavor to keep the information in this topic up to date and correct, NXLog makes no representations or warranties of any kind, express or implied about the completeness, accuracy, reliability, suitability, or availability of the content represented here.

The accurateness of the content was tested and proved to be working in our lab environment at the time of the last revision with the following software versions:

NXLog 5.2.6388 on Linux (Ubuntu 18.04) and Windows Server 2016
Apache NiFi 1.13.0 running on Ubuntu 18.04

Last revision: 20 April 2021