NXLog Legacy Documentation

Sumo Logic

Sumo Logic is a cloud-based Application Performance Monitoring service that collects, manages, and analyzes log data. NXLog can forward logs to Sumo Logic in syslog format over TCP or via a custom HTTP endpoint and collect and send host metrics via HTTP.

Sumo Logic offers different plans for their service, including a free plan. Some of the methods described here may require a paid plan.

NXLog as a Sumo Logic collector alternative

Sumo Logic accepts data from two types of collectors, installed or hosted. Installed collectors are set up by installing agent software provided by Sumo Logic, whereas hosted collectors are used to send data over TCP or HTTP(S) from agents like NXLog. This section presents some scenarios in which using an NXLog agent with a Sumo Logic hosted collector has an advantage over using a Sumo Logic installed collector.

Flexible log processing

NXLog provides more control at the log processing stage before forwarding the events, which is especially useful since Sumo Logic charges per ingested byte.

Although installed collectors support processing rules to drop or keep certain events, these are pretty basic compared to the flexibility an NXLog agent provides. With NXLog, you can selectively trim event data rather than drop entire events and have complete control over the output format. See Reducing Bandwidth and Data Size for some configuration options available when using NXLog.

Support for more log sources

Sumo Logic collectors are designed to work with standard log formats primarily stored in flat files. Therefore, processing data from unsupported log sources such as databases, network packet captures, or other events not in a standard log format may not be possible. On the other hand, NXLog can easily fill this gap with its vast selection of input modules. It can also process log data using custom scripts with support for Go, Java, Perl, Python, and Ruby. See the NXLog documentation on Available Modules for a complete list of input, processing, and extension modules.

Platform requirements

The Sumo Logic collector is a Java-based agent that requires substantial system resources. NXLog, with its smaller footprint, is a powerful Sumo Logic collector alternative for use cases where you need to eliminate the dependency on Java or prefer a less resource-intensive agent with more flexibility. Case in point: when installed on a Microsoft Windows 10 system and processing only Application events from Windows Event Log, we observed that the Sumo Logic collector consumes 200+ MB of memory, compared with 4-6 MB of memory usage by the NXLog agent when tested under the same conditions. See the table below for the minimum system requirements for the NXLog and Sumo Logic agents. Furthermore, the Sumo Logic collector does not support collecting events in the older Microsoft Windows Event Log format, as documented in this Sumo Logic article on Windows 2003 Event Logs ingestion. NXLog, on the other hand, explicitly supports the Windows XP/2000/2003 event log format through its im_mseventlog input module and can be installed on systems running such older Microsoft Windows releases.

Table 1. NXLog and Sumo Logic minimum system requirements
	NXLog	Sumo Logic Collector
Processor Cores	1	1
Memory	60 MB	512 MB
Disk Space	50 MB	8 GB
Pre-requisites	-	Java 1.8+

Setting up a Sumo Logic hosted collector

For Sumo Logic to receive data over TCP or HTTP(S), you must create a hosted collector from the Sumo Logic web interface. To create a hosted collector for data coming from NXLog:

Log in to the Sumo Logic web interface.
Navigate to Manage Data > Collection.
On the Collection tab, click on Add Collector.
Choose Hosted Collector, fill in the required details, and click Save.

For more details, see Configure a Hosted Collector on Sumo Logic Docs.

Setting up the Sumo Logic digital certificate

To be able to send data to Sumo Logic, a PEM-encoded DigiCert Certification Authority certificate is required. This section describes preparing the certificate on Linux and Microsoft Windows for use with NXLog.

Linux

On Linux, you first need to download the certificate from the DigiCert website and then use the OpenSSL tool to convert it.

Download the DigiCert DER encoded certificate:

$ sudo wget -O digicert_ca.der https://www.digicert.com/CACerts/DigiCertHighAssuranceEVRootCA.crt

Convert the certificate to a PEM-encoded certificate:

$ sudo openssl x509 -inform der -in digicert_ca.der -out digicert_ca.crt

Copy digicert_ca.crt to a location accessible by NXLog.

Microsoft Windows

The certificate can be exported on Microsoft Windows using the Certificates MMC snap-in.

Go to the Windows Start menu, type certmgr.msc, and click Enter.
Expand Trusted Root Certification Authorities > Certificates.
Right-click on DigiCert High Assurance EV Root CA from the list of certificates and select Open.
Go to the Details tab and click the Copy to File… button.
The Certificate Export Wizard opens; click Next.
Select Base-64 encoded X.509 (.CER) and click Next.
Click the Browse… button and select a location accessible by NXLog.
Enter a filename, e.g., digicert_ca.cer, and click Save.
Click Next and then Finish to complete the export.

Sending data to a Sumo Logic Cloud Syslog source

Sumo Logic accepts log data as syslog messages in IETF (RFC 5424) format and requires TLS v1.2 over TCP. To use this method, you must create a Cloud Syslog Source in Sumo Logic.

From the Sumo Logic web interface, navigate to Manage Data > Collection.
Click on Add Source next to the previously created hosted collector.
Select Cloud Syslog and fill in the required details. We recommend specifying a Source Category to make events from this source easily searchable in Sumo Logic.
Click Save to finish creating the syslog source.
After creating the new syslog source, a dialog containing the Token, Host, and TCP Port is displayed. The NXLog configuration will need these details for sending log data to Sumo Logic.

For further details on configuring a syslog source, refer to the Cloud Syslog Source documentation on Sumo Logic Docs.

Syslog messages must comply with RFC 5424, or they will be dropped by Sumo Logic. Messages over 64KB will be truncated.

Example 1. Forwarding syslog messages to Sumo Logic using TLS encryption

This NXLog configuration reads syslog messages from a file using the im_file input module, converts the records to JSON format, and sends the syslog-encapsulated data to Sumo Logic using the om_ssl output module.

define SUMO_TOKEN    xxxxxxxxxxxxxxxxxxxx@41123 (1)
define SUMO_HOST     syslog.collection.<YOUR_DEPLOYMENT>.sumologic.com (2)
define SUMO_PORT     6514 (3)
define CA_FILE       /opt/nxlog/cert/digicert_ca.crt (4)

<Extension json>
    Module           xm_json
</Extension>

<Extension syslog>
    Module           xm_syslog
</Extension>

<Input file>
    Module           im_file
    File             "/var/log/file"
    Exec             parse_syslog(); (5)
    Exec             $Message = to_json(); (6)
</Input>

<Output sumo_tcp>
    Module           om_ssl
    Host             %SUMO_HOST%:%SUMO_PORT%
    CAFile           %CA_FILE%
    Exec             to_syslog_ietf(); (7)
    Exec             $raw_event =~ s/(\[NXLOG@14506.*?\])//g; \
                     $raw_event = replace($raw_event, \
                     '{', '[%SUMO_TOKEN%] {', 1);
</Output>

1	Replace the `SUMO_TOKEN` value with an actual Sumo Logic syslog source token. The token is used in the output instance to replace the NXLog ID in the structured data of the syslog message.
2	Replace the `SUMO_HOST` value with an actual Sumo Logic syslog source host URL.
3	Replace the `SUMO_PORT` value with the correct port for the host defined by the `SUMO_HOST` constant.
4	Replace the `CA_FILE` value with the absolute path to the DigiCert Certification Authority certificate.
5	Parses the syslog messages to structured data using the parse_syslog() procedure of the xm_syslog module.
6	Converts the record to JSON format using the to_json() function of the xm_json module and stores the data in the `$Message` field.
7	Encapsulates the JSON data in a syslog message using the to_syslog_ietf() procedure of the xm_syslog module.

Output sample

This syslog header sample will be sent to Sumo Logic. The structured data contains the Sumo Logic token specified in the NXLog configuration.

<13>1 2020-12-04T11:33:47 NXLog-Ubuntu-1 systemd 1 -  [xxxxxxxxxxxxxxxxxxxx@41123]

The following is a sample JSON-formatted event that will be sent to the Sumo Logic service as part of the syslog message.

{
  "EventReceivedTime": "2020-12-04 13:37:47",
  "SourceModuleName": "file",
  "SourceModuleType": "im_file",
  "SyslogFacilityValue": 1,
  "SyslogFacility": "USER",
  "SyslogSeverityValue": 5,
  "SyslogSeverity": "NOTICE",
  "SeverityValue": 2,
  "Severity": "INFO",
  "Hostname": "NXLog-Ubuntu-1",
  "EventTime": "2020-12-04 11:33:47",
  "SourceName": "systemd",
  "ProcessID": "1",
  "Message": "Started Run anacron jobs."
}

Example 2. Forwarding Windows events to Sumo Logic as syslog messages

This NXLog configuration collects Windows events using the im_msvistalog input module, converts the records to JSON, and sends the syslog-encapsulated data to Sumo Logic using the xm_syslog module, and sends it to Sumo Logic using the om_ssl output module.

define SUMO_TOKEN    xxxxxxxxxxxxxxxxxxxx@41123 (1)
define SUMO_HOST     syslog.collection.<YOUR_DEPLOYMENT>.sumologic.com (2)
define SUMO_PORT     6514 (3)
define CA_FILE       C:\Program Files\nxlog\cert\digicert_ca.cer (4)

<Extension json>
    Module           xm_json
</Extension>

<Extension syslog>
    Module           xm_syslog
</Extension>

<Input eventlog>
    Module           im_msvistalog
    <QueryXML>
        <QueryList>
            <Query Id='0'>
                <Select Path="System">*[System[(Level=1  or Level=2)]]</Select>
            </Query>
        </QueryList>
    </QueryXML>
    Exec             $Message = to_json(); (5)
</Input>

<Output sumo_tcp>
    Module           om_ssl
    Host             %SUMO_HOST%:%SUMO_PORT%
    CAFile           %CA_FILE%
    Exec             to_syslog_ietf(); (6)
    Exec             $raw_event =~ s/(\[.*])//g; \
                     $raw_event = replace($raw_event, \
                     '{', '[%SUMO_TOKEN%] {', 1);
</Output>

1	Replace the `SUMO_TOKEN` value with an actual Sumo Logic syslog source token. The token is used in the output instance to replace the NXLog ID in the structured data of the syslog message.
2	Replace the `SUMO_HOST` value with an actual Sumo Logic syslog source host URL.
3	Replace the `SUMO_PORT` value with the correct port for the host defined by the `SUMO_HOST` constant.
4	Replace the `CA_FILE` value with the absolute path to the DigiCert Certification Authority certificate.
5	Converts the record to JSON format using the to_json() function of the xm_json module and stores the data in the `$Message` field.
6	Encapsulates the JSON data in a syslog message using the to_syslog_ietf() procedure of the xm_syslog module.

Output sample

This syslog header sample will be sent to Sumo Logic. The structured data contains the Sumo Logic token specified in the NXLog configuration.

<11>1 2020-12-04T11:31:38.031887Z SERVER-1 VBoxNetLwf 0 -  [xxxxxxxxxxxxxxxxxxxx@41123]

The following is a sample JSON-formatted event that will be sent to the Sumo Logic service as part of the syslog message.

{
  "EventTime": "2020-12-07 11:30:19",
  "Hostname": "SERVER-1",
  "Keywords": "36028797018963968",
  "EventType": "ERROR",
  "SeverityValue": 4,
  "Severity": "ERROR",
  "EventID": 12,
  "SourceName": "VBoxNetLwf",
  "TaskValue": 0,
  "RecordNumber": 44758,
  "ExecutionProcessID": 0,
  "ExecutionThreadID": 0,
  "Channel": "System",
  "Message": "The driver detected an error on \\Device\\VBoxNetLwf.",
  "Data": "\\Device\\VBoxNetLwf",
  "EventData.Binary": "00000C0001000000000000000C0004",
  "EventReceivedTime": "2020-12-07 11:31:38",
  "SourceModuleName": "eventlog",
  "SourceModuleType": "im_msvistalog"
}

Sending data to a Sumo Logic HTTP Logs & Metrics source

NXLog Enterprise Edition exclusive feature

Logs and host metrics can be sent to Sumo Logic over HTTP(S) using a unique URL generated for each source. To use this method, you must create an HTTP Logs & Metrics Source in Sumo Logic.

From the Sumo Logic web interface, navigate to Manage Data > Collection.
Click on Add Source next to the previously created hosted collector.
Select HTTP Logs & Metrics and fill in the required details. We recommend specifying a Source Category to make events from this source easily searchable in Sumo Logic.
Click Save to finish creating the HTTP source.
After creating the new HTTP source, a dialog containing the HTTP Source Address is displayed. The NXLog configuration will need this URL for sending log data to Sumo Logic.

For further details on configuring an HTTP source, see Configure HTTP Source for Logs and Metrics on Sumo Logic Docs.

Sending event logs to Sumo Logic

Example 3. Forwarding events to Sumo Logic in batches over HTTPS

This NXLog configuration forwards log data to Sumo Logic with the om_http output module using the POST method.

Sumo Logic accepts batched requests up to 1MB of uncompressed data. For simplicity, this example uses the default NXLog batch settings. For further configuration options, see the BatchSize and BatchFlushInterval directives.

Data can be sent to Sumo Logic as plain uncompressed or compressed by the deflate or gzip method. The om_http module supports data compression based on the zlib compression library (deflate). The configuration below enables the HTTPSSSLCompression directive to send compressed data to Sumo Logic. Data will be sent uncompressed if this directive is not specified or set to FALSE.

When sending data over HTTP(S), Sumo Logic accepts additional optional headers to configure custom settings related to the log records. See Supported HTTP Headers on Sumo Logic Docs for more information. In turn, the om_http module supports specifying additional headers using the AddHeader directive. For example, the configuration below adds the X-Sumo-Category header with the value my-category.

define SUMO_URL            https://<YOUR_SUMO_ENDPOINT>/receiver/v1/http/<UNIQUE_CODE> (1)
define CA_FILE             /opt/nxlog/cert/digicert_ca.crt (2)

<Output sumo_http>
    Module                 om_http
    URL                    %SUMO_URL%
    BatchMode              multiline (3)
    HTTPSSSLCompression    TRUE
    AddHeader              X-Sumo-Category: my-category
    HTTPSCAFile            %CA_FILE%
</Output>

1	Replace the `SUMO_URL` value with an actual Sumo Logic HTTP source URL.
2	Replace the `CA_FILE` value with the absolute path to the DigiCert Certification Authority certificate.
3	Sets the BatchMode directive in the output instance to `multiline` to specify that it should send data in batches, with each record separated by a new line.

It may take a few minutes for data to be shown in the Sumo Logic web interface. If not, see Troubleshooting HTTP Sources on Sumo Logic Docs.

Sending host metrics to Sumo Logic

In Sumo Logic terms, host metrics are a set of data points that measure the value of a property over time. For example, you can use host metrics to monitor an application’s availability and performance or a host’s resource usage. For more information, see Introduction to Metrics on Sumo Logic Docs.

Sumo Logic supports metrics in the Graphite, Carbon 2.0, and Prometheus formats. NXLog can send data in any of these formats by reading preformatted records from a file or using an Exec block to output the data in the desired format.

For more information on the requirements of host metrics, refer to Upload Metrics to an HTTP Source on Sumo Logic Docs.

Example 4. Forwarding host metrics to Sumo Logic in the Carbon 2.0 format

This configuration illustrates how NXLog can retrieve Microsoft Windows performance counters using the im_winperfcount module and send the data to Sumo Logic in the Carbon 2.0 format using the om_http module.

The Exec block in the output instance builds the metric in the required Carbon 2.0 format. Sumo Logic accepts timestamps in seconds or milliseconds; therefore, in this example, the NXLog timestamp is converted from microseconds to milliseconds.

define SUMO_URL      https://<YOUR_SUMO_ENDPOINT>/receiver/v1/http/<UNIQUE_CODE> (1)
define CA_FILE       C:\Program Files\nxlog\cert\digicert_ca.cer (2)

<Input memory_count>
    Module           im_winperfcount
    Counter          \Memory\Available Bytes
    PollInterval     30 (3)
    Exec             rename_field("\\Memory\\Available Bytes", "MemFreeBytes"); (4)
</Input>

<Output sumo_http>
    Module           om_http
    URL              %SUMO_URL%
    BatchMode        multiline
    HTTPSCAFile      %CA_FILE%
    ContentType      application/vnd.sumologic.carbon2
    <Exec>
        $raw_event = "metric=Mem_Free " + "host=" + $Hostname + \
                     "  " + $MemFreeBytes + " " + \
                     integer($EventTime)*1/1000;
    </Exec>
</Output>

1	Replace the `SUMO_URL` value with an actual Sumo Logic HTTP source URL.
2	Replace the `CA_FILE` value with the absolute path to the DigiCert Certification Authority certificate.
3	Polls the host’s available memory every 30 seconds.
4	Renames the field containing the retrieved value to `MemFreeBytes` to simplify referencing it from the output instance.

Output sample

The following is a sample of the data that will be sent to Sumo Logic. In this example, the metric is named Mem_Free and represents the available free memory in bytes.

metric=Mem_Free host=Hopper  1008345088 1608040330345
metric=Mem_Free host=Hopper  1017278464 1608040340346
metric=Mem_Free host=Hopper  1056428032 1608040350346

It may take a few minutes for data to appear in the Sumo Logic web interface. If not, see Troubleshooting HTTP Sources on Sumo Logic Docs.

Verifying data in Sumo Logic

The reception of log data can be verified using the Sumo Logic web interface. One way to do this is to search for events using the collector’s name and source by navigating to Manage Data > Collection, and on the Collection tab, click on the Open in Log Search icon next to the respective log source.

This image displays the NXLog hosted collector and a list of sources in the Sumo Logic web interface. In addition, it highlights the icon to open the log search.

This image displays events filtered by the collector and source.

This image displays a syslog event in JSON format sent to Sumo Logic via HTTPS.

This image displays a Windows event in JSON format sent to Sumo Logic as a syslog message.

You can export events from Sumo Logic in CSV format, in which event fields are output as comma-separated values. For example, the following is a sample syslog message exported from Sumo Logic. NXLog sent the original event via HTTPS.

"_messagetimems","_messagetime","_raw","_collector","_size","_source","_sourcecategory","_sourcehost","_sourcename"
"1607341156001","12/07/2020 12:39:16.001 +0100","{""EventReceivedTime"":""2020-12-07T12:39:16.001994+01:00"",""SourceModuleName"":""file"",""SourceModuleType"":""im_file"",""SyslogFacilityValue"":1,""SyslogFacility"":""USER"",""SyslogSeverityValue"":5,""SyslogSeverity"":""NOTICE"",""SeverityValue"":2,""Severity"":""INFO"",""Hostname"":""NXLog-Ubuntu-1"",""EventTime"":""2020-12-07T12:30:37.000000+01:00"",""SourceName"":""systemd"",""ProcessID"":""1"",""Message"":""Started Fingerprint Authentication Daemon.""}","NXLog","414","NXLog-Ubuntu1","linux-http","19.168.0.100","Http Input"

You can view Host Metrics in the Sumo Logic web interface by clicking on the + New button to open a Metrics tab and adding a metric query for the desired property. For example, the image below shows host metrics sent by NXLog for available memory (Bytes) over time. When sending the data, the metric was named Mem_Free.

Disclaimer

While we endeavor to keep the information in this topic up to date and correct, NXLog makes no representations or warranties of any kind, express or implied about the completeness, accuracy, reliability, suitability, or availability of the content represented here. We update our screenshots and instructions on a best-effort basis.

The accurateness of the content was tested and proved to be working in our lab environment at the time of the last revision with the following software versions:

Ubuntu 20.04.5 LTS
NXLog 5.5.7535

Last revision: 3 October 2022