Logstash

Logstash is a data collection and processing tool. It is often used as part of the ELK stack (Elasticsearch, Logstash, Kibana) to collect logs from various sources, normalize them, and forward them to Elasticsearch. NXLog Agent and Logstash can be configured to exchange log data between each other in various formats including JSON and syslog, and over different transport protocols such as TCP, UDP, and HTTP(S).

NXLog Agent as a data shipper for Logstash

Logstash can be resource-intensive and requires Java to run. For this reason, it is often used as an ingester, with different data shippers forwarding logs to it. NXLog Agent, with its small resource footprint, can act as a data shipper for Logstash with the benefit of being an all-in-one solution. For example, when using Elastic Beats, different Beats need to be installed for each log source, such as Filebeat for file-based logs, Winlogbeat for Windows Event Log and so on. An NXLog Agent installation can collect and process logs from different sources on the same host, all configured within a single configuration file.

NXLog Agent can also be used as a data shipper when it provides support for log sources that are not directly supported by Logstash. This includes collecting logs using Event Tracing for Windows or kernel log messages for Linux, BSD, and macOS. NXLog Agent also provides additional log processing power through custom scripts with support for Go, Java, Perl, Python, and Ruby.

NXLog Agent as an alternative to Logstash

NXLog Agent is a powerful log processing tool that is capable of ingesting, processing, and forwarding log data without the use of third-party tools. This includes the ability to forward data directly to Elasticsearch, and thus it can be used as a Logstash alternative. NXLog Agent also provides additional benefits over Logstash, some of which include:

  • Logstash depends on Java to run, requiring additional system resources. This also increases administrative work to deploy Java on production servers and keep installations up-to-date with security updates. On the other hand, NXLog Agent does not require Java and has a small resource footprint.

  • When using Logstash with different data shippers, each type of log source requires a special agent, each with its own configuration method. An NXLog Agent installation includes full functionality to collect, process, and forward logs. It can be configured as an end-to-end solution, or as an agent, forwarding logs to another NXLog Agent installation acting as a relay. This facilitates a more streamlined approach to deploying and managing your installations since it employs a single type of agent using a standardized, familiar method of configuration.

  • Logstash is an open-source software with a large repository of official and unofficial plugins. While this has its benefits, it also presents challenges when things don’t work as expected, or when plugins stop being maintained. All components in the NXLog Agent software, including input and output modules, extensions, and add-ons are developed, maintained, and supported by NXLog.

For more information and examples of how NXLog Agent can replace Logstash in the ELK stack, refer to the documentation on integrating with Elasticsearch and Kibana.

Sending logs to Logstash with NXLog Agent

Logstash supports data ingestion from various sources through its many input plugins and can process data in various formats by using filters and codec plugins. Likewise, NXLog Agent offers a vast selection of output modules and extensions that can format and send data in many of the formats and ingestion methods supported by Logstash. This section provides configuration examples for sending data from NXLog Agent to Logstash over TCP, UDP, and HTTPS. The examples show how to send data in two log formats commonly used with Logstash, JSON and syslog.

Example 1. Sending logs to Logstash over TCP in JSON format

One of the easiest ways for Logstash to ingest data is to feed it logs in JSON format. This simplifies the Logstash configuration since logs can be parsed as structured data without any need to write complex grok filters. NXLog Agent provides support for JSON through its xm_json extension module, which can be used with any of the available input and output modules.

In this example, NXLog Agent is configured to collect logs from Event Tracing for Windows using the im_etw input module. Events are converted to JSON and then forwarded to Logstash over TCP using the om_tcp output module.

This configuration uses the Logstash tcp input plugin to listen for connections on port 5444. The json_lines codec is specified to process log records as JSON delimited by newline.

logstash.conf
input {
  tcp {
    port  => 5444
    codec => json_lines
  }
}

The NXLog Agent configuration below uses the im_etw module to collect Microsoft Windows DNS Server audit events. A Type field is added to each record to tag it as a DNS Server event. This enables them to be readily found in searches on Elasticsearch/Kibana or other log analytics platforms. Events are then converted to JSON and sent to Logstash over TCP.

nxlog.conf
<Extension json>
    Module      xm_json
</Extension>

<Input etw_dns>
    Module      im_etw
    Provider    Microsoft-Windows-DNSServer
    Exec        $Type = "DNS-Server";
    Exec        to_json();
</Input>

<Output logstash>
    Module      om_tcp
    Host        192.168.0.123:5444
</Output>
Input sample

The following is a sample DNS Server event as it is received by NXLog Agent.

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-DNSServer" Guid="{EB79061A-A566-4698-9119-3ED2807060E7}" />
    <EventID>516</EventID>
    <Version>0</Version>
    <Level>4</Level>
    <Task>5</Task>
    <Opcode>0</Opcode>
    <Keywords>0x4000000000100000</Keywords>
    <TimeCreated SystemTime="2021-03-02T08:59:30.763237500Z" />
    <EventRecordID>198</EventRecordID>
    <Correlation />
    <Execution ProcessID="1540" ThreadID="1696" />
    <Channel>Microsoft-Windows-DNSServer/Audit</Channel>
    <Computer>dns-1.example.com</Computer>
    <Security UserID="S-1-5-21-3379025547-2756628915-3927403441-500" />
  </System>
  <EventData>
    <Data Name="Type">1</Data>
    <Data Name="NAME">test-1.example.com</Data>
    <Data Name="TTL">0</Data>
    <Data Name="BufferSize">4</Data>
    <Data Name="RDATA">0A000007</Data>
    <Data Name="Zone">example.com</Data>
    <Data Name="ZoneScope">Default</Data>
    <Data Name="VirtualizationID">.</Data>
  </EventData>
</Event>
Output sample

The following JSON shows the above event after being processed by Logstash.

{
  "EventID": 516,
  "Severity": "INFO",
  "SourceModuleName": "etw_dns",
  "Domain": "EXAMPLE",
  "Type": "DNS-Server",
  "@timestamp": "2021-03-02T08:59:31.064Z",
  "Version": 0,
  "AccountType": "User",
  "EventTime": "2021-03-02T09:59:30.763240+01:00",
  "TTL": "0",
  "EventType": "INFO",
  "UserID": "S-1-5-21-3379025547-2756628915-3927403441-500",
  "SourceModuleType": "im_etw",
  "ChannelID": 17,
  "Keywords": "4611686018428436480",
  "Hostname": "WIN2016-AD",
  "BufferSize": "4",
  "RDATA": "0x0A000007",
  "port": 61258,
  "Zone": "example.com",
  "Flags": "EXTENDED_INFO|IS_64_BIT_HEADER|PROCESSOR_INDEX (577)",
  "OpcodeValue": 0,
  "NAME": "test-1.example.com",
  "VirtualizationID": ".",
  "EventReceivedTime": "2021-03-02T09:59:31.754431+01:00",
  "ExecutionProcessID": 1540,
  "AccountName": "Administrator",
  "ZoneScope": "Default",
  "host": "dns-1.example.com",
  "SourceName": "Microsoft-Windows-DNSServer",
  "@version": "1",
  "ExecutionThreadID": 1696,
  "SeverityValue": 2,
  "ProviderGuid": "{EB79061A-A566-4698-9119-3ED2807060E7}",
  "TaskValue": 5
}
Example 2. Sending logs to Logstash over TCP using TLS

TCP communication between NXLog Agent and Logstash can be secured by using TLS. In this configuration, certificate-based authentication is enabled using self-signed certificates. See Generating self-signed digital certificates for instructions on generating the required certificates.

This Logstash configuration uses the tcp input plugin to listen for connections on port 5444. TLS/SSL is enabled which requires clients to present a certificate that can be verified by any certificate authority listed in the array assigned to the ssl_certificate_authorities key in the configuration. In this example, the array contains only one certificate authority.

logstash.conf
input {
  tcp {
    port                        => 5444
    ssl_enable                  => true
    ssl_certificate_authorities => ["/path/to/root-CA.pem"]
    ssl_cert                    => "/path/to/root-CA.pem"
    ssl_key                     => "/path/to/root-CA.key"
    ssl_verify                  => true
  }
}

The NXLog Agent configuration below uses the om_ssl output module to send logs to Logstash using TLS. The AllowUntrusted directive is set to FALSE, which means that Logstash must present a valid certificate that can be verified by the certificate authority specified in the CAFile directive.

nxlog.conf
<Output logstash>
    Module            om_ssl
    Host              192.168.0.123:5444
    CAFile            /path/to/root-CA.pem
    CertFile          /path/to/client-cert.pem
    CertKeyFile       /path/to/client-cert.key
    AllowUntrusted    FALSE
</Output>
Example 3. Sending logs to Logstash in syslog format

Syslog is a widely used log format, especially in UNIX/Linux environments which has led many vendors to choose this log format for their appliances. NXLog Agent provides support for syslog through its xm_syslog extension module. The xm_syslog module can be used in combination with the om_tcp or om_udp output modules to forward syslog messages to Logstash.

This configuration uses the Logstash udp input plugin to listen for connections on port 1514. A grok filter is used to parse the log records as syslog messages. It is important to note that if logs do not conform to this pattern, they will fail to be parsed as structured data and will be tagged with _grokparsefailure.

logstash.config
input {
  udp {
    port  => 1514
  }
}

filter {
  grok {
    match => {
      "message" => "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}"
    }
  }
}

The NXLog Agent configuration below uses the om_udp module to forward logs to Logstash. The to_syslog_bsd() procedure provided by the xm_syslog module is used to send syslog messages in the RFC 3164 format expected by Logstash.

nxlog.config
<Extension syslog>
    Module    xm_syslog
</Extension>

<Output logstash>
    Module    om_udp
    Host      192.168.0.123:1514
    Exec      to_syslog_bsd();
</Output>
Input sample

The following is a sample of a syslog message as sent from NXLog Agent to Logstash.

<13>Mar  8 12:25:35 server01 systemd[1]: Started NXLog daemon.
Output sample

The following JSON shows the above log sample after being processed by Logstash. Note that the @version, @timestamp, and host fields are added by Logstash.

{
  "syslog_program": "systemd",
  "syslog_pid": "1",
  "syslog_message": "Started NXLog daemon.",
  "syslog_timestamp": "Mar  8 12:25:35",
  "syslog_pri": "13",
  "syslog_hostname": "server01",
  "message": "<13>Mar  8 12:25:35 server01 systemd[1]: Started NXLog daemon.",
  "@version": "1",
  "@timestamp": "2021-03-08T11:33:17.919Z",
  "host": "192.168.0.100"
}
Example 4. Sending logs to Logstash over HTTPS

Logstash can receive logs over HTTP(S) using the http input plugin and NXLog Agent can be configured to send logs to it using the om_http output module.

In this configuration, the Logstash http input plugin listens for connections on port 8080. Certificate-based authentication is enabled using self-signed certificates. See Generating self-signed digital certificates for instructions on generating the required certificates. The json_lines codec is specified to process log records as JSON delimited by newline.

logstash.conf
input {
  http {
    port                        => 8080
    codec                       => json_lines
    ssl                         => true
    ssl_certificate_authorities => ["/path/to/root-CA.pem"]
    ssl_certificate             => "/path/to/root-CA.pem"
    ssl_key                     => "/path/to/pkcs8-root-CA.key"
    ssl_verify_mode             => "force_peer"
  }
}

The NXLog Agent configuration below processes Apache access logs in the Combined Log Format. The log data is parsed into fields and a Type field is added to each record. Log records are converted to JSON using the xm_json module and forwarded to Logstash over HTTPS. The BatchMode directive in the output instance is set to multiline to specify that log records should be sent in batches one record per line.

nxlog.conf
<Extension json>
    Module                 xm_json
</Extension>

<Input apache_access>
    Module                 im_file
    File                   "path/to/file"
    <Exec>
        if $raw_event =~ /(?x)^(\S+)\ \S+\ (\S+)\ \[([^\]]+)\]\ \"(\S+)\ (.+)
                          \ HTTP\/\d\.\d\"\ (\S+)\ (\S+)\ \"([^\"]+)\"
                          \ \"([^\"]+)\"/
        {
            $Hostname = $1;
            if $2 != '-' $AccountName = $2;
            $EventTime = parsedate($3);
            $HTTPMethod = $4;
            $HTTPURL = $5;
            $HTTPResponseStatus = $6;
            if $7 != '-' $FileSize = $7;
            if $8 != '-' $HTTPReferer = $8;
            if $9 != '-' $HTTPUserAgent = $9;
            $Type = "Apache-Access";
        }
    </Exec>
</Input>

<Output logstash>
    Module                 om_http
    URL                    https://192.168.0.123:8080
    BatchMode              multiline
    HTTPSCAFile            /path/to/root-CA.pem
    HTTPSCertFile          /path/to/client-cert.pem
    HTTPSCertKeyFile       /path/to/client-cert.key
    HTTPSAllowUntrusted    FALSE
    Exec                   to_json();
</Output>
Input sample

The following is an Apache access log sample received by NXLog Agent.

83.149.9.216 - - [02/Mar/2021:16:13:44 +0000] "GET /presentations/logstash/plugin/zoom-js/zoom.js HTTP/1.1" 200 7697 "http://example.com/presentations/logstash/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
Output sample

The following JSON shows the above log sample as output by Logstash. Note that the @version, @timestamp, host, and headers fields are added by Logstash.

{
  "HTTPUserAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36",
  "HTTPReferer": "http://example.com/presentations/logstash/",
  "HTTPMethod": "GET",
  "Hostname": "83.149.9.216",
  "HTTPResponseStatus": "200",
  "EventReceivedTime": "2021-03-02T16:18:26.139421+01:00",
  "SourceModuleType": "im_file",
  "EventTime": "2021-03-02T16:13:44.000000+01:00",
  "SourceModuleName": "apache_access",
  "HTTPURL": "/presentations/logstash/plugin/zoom-js/zoom.js",
  "FileSize": "7697",
  "Type": "Apache-Access",
  "@version": "1",
  "@timestamp": "2021-03-02T15:18:29.568Z",
  "host": "192.168.1.100",
  "headers": {
    "content_length": "2212",
    "keep_alive": "timeout=300",
    "request_path": "/",
    "content_type": "text/plain",
    "http_accept": null,
    "http_user_agent": "nxlog",
    "http_version": "HTTP/1.1",
    "connection": "keep-alive",
    "request_method": "POST",
    "http_host": "localhost:8080"
  }
}
NXLog Agent can parse log records in any format using the available extension modules or through the use of regular expressions and other string manipulation functions.

See Parse standard log formats in the NXLog Platform User Guide for more information and examples.

Receiving logs from Logstash with NXLog Agent

NXLog Agent can receive and process data in many of the output methods and data formats supported by Logstash. See the Reference Manual for a complete list of input modules and extensions.

Example 5. Receiving logs over TCP with NXLog Agent

In this configuration, NXLog Agent uses the im_tcp input module to listen for connections on port 5445. It expects to receive log records as JSON delimited by newline and uses the xm_json module to parse log records into fields.

nxlog.conf
<Extension json>
    Module        xm_json
</Extension>

<Input tcp_listen>
    Module        im_tcp
    ListenAddr    0.0.0.0:5445
    Exec          parse_json();
</Input>

The Logstash configuration below uses the tcp output plugin to send logs to NXLog Agent. Although the tcp plugin documentation states that each JSON event is separated by a newline, this is currently not the case thus the json_lines codec is required as a workaround.

logstash.conf
output {
  tcp {
    host  => "192.168.0.100"
    port  => 5445
    codec => json_lines
  }
}
Example 6. Receiving syslog over UDP with NXLog Agent

In this configuration, NXLog Agent uses the im_udp input module to listen for connections on port 1514. It expects to receive syslog messages in RFC 5424 format and uses the xm_syslog module to parse log records into fields.

nxlog.conf
<Extension syslog>
    Module        xm_syslog
</Extension>

<Input udp_listen>
    Module        im_udp
    ListenAddr    0.0.0.0:1514
    Exec          parse_syslog_ietf();
</Input>

The Logstash configuration below uses the syslog output plugin to send logs to NXLog Agent. This plugin is not installed by default. To install it, execute the following command within the Logstash home directory:

# bin/logstash-plugin install logstash-output-syslog
logstash.conf
output {
  syslog {
    host     => "192.168.0.100"
    port     => 1514
    protocol => "rfc5424"
  }
}

Verifying data output from Logstash

Logstash can output data in several ways thus the method to verify the output depends on the Logstash configuration. For testing purposes, it can be configured to output log records to the shell’s standard output (stdout) where Logstash is running or to a local file. Multiple output methods can be specified simultaneously. If Logstash is running as part of the ELK stack, see Verifying data in Elasticsearch/Kibana.

Example 7. Output logs to the standard output and to file

This Logstash configuration uses the stdout plugin to output logs to standard output (stdout), and the file output plugin to write log records to file.

logstash.config
output {
  stdout {
    codec => rubydebug
  }

  file {
    path  => "/path/to/file"
  }
}

Verifying data in Elasticsearch/Kibana

When Logstash is used as part of the ELK stack it is configured to forward logs to Elasticsearch while Kibana handles data visualization. One way to verify the reception of log data is to launch the Kibana web interface and navigate to Analytics > Discover.

Kibana Navigation Menu

By default, only log events received during the last 15 minutes are displayed. This image shows the Kibana web interface configured to display log events received during the last day.

Viewing logs in Kibana

Filters can be defined to show only log records of interest. The following image shows a filter where only records having the value DNS-Server for the Type field will be displayed.

Filtering logs in Kibana

Kibana offers two viewing modes for log records. This image depicts a Microsoft Windows DNS Server event in Table view mode.

Kibana table view

The following image depicts the same DNS Server event in JSON view mode.

Kibana JSON view

Generating self-signed digital certificates

You can follow these steps that use the OpenSSL command line tool to create self-signed certificates for Logstash with NXLog Agent as a client.

  1. Create the Certificate Authority (CA) certificate. These commands will create a root CA private key file for signing certificates (root-CA.key) and a self-signed root CA certificate file (root-CA.pem).

    $ openssl genrsa -out root-CA.key 2048
    $ openssl req -x509 -new -nodes -key root-CA.key -sha256 -days 1024 -out root-CA.pem
  2. Create an OpenSSL configuration file for the NXLog Agent client certificate and name it client_options.conf. Replace the values under [req_distinguished_name] and [alt_names] accordingly.

    client_options.conf
    [req]
    distinguished_name = req_distinguished_name
    x509_extensions = v3_req
    prompt = no
    [req_distinguished_name]
    CN = nxlog-host.example.com
    [v3_req]
    keyUsage = keyEncipherment, dataEncipherment
    extendedKeyUsage = serverAuth
    subjectAltName = @alt_names
    [alt_names]
    DNS.1 = 192.168.1.100
  3. Generate the client key and certificate signing request:

    $ openssl genrsa -out client-cert.key 2048
    $ openssl req -new -key client-cert.key -out client-cert.csr -config client_options.conf
  4. Generate the client certificate to be used by NXLog Agent:

    $ openssl x509 -req -in client-cert.csr -CA root-CA.pem -CAkey root-CA.key -CAcreateserial -out client-cert.pem -days 1024 -sha256
  5. If using the Logstash http input module the CA key must be in the PKCS8 format. The following command converts the previously created PKCS1 key to the required PKCS8 format:

    $ openssl pkcs8 -topk8 -inform PEM -outform PEM -nocrypt -in root-CA.key -out pkcs8-root-CA.key
Disclaimer

While we endeavor to keep the information in our guides up to date and correct, NXLog makes no representations or warranties of any kind, express or implied about the completeness, accuracy, reliability, suitability, or availability of the content represented here. We update our screenshots and instructions on a best-effort basis.

The accurateness of the content was tested and proved to be working in our lab environment at the time of the last revision with the following software versions:

NXLog Agent version 5.6.7727
Logstash version 8.5.0
Kibana version 8.5.0
Windows Server 2016

Last revision: 7 November 2022