Generating test data

You may need to generate and send test data to your NXLog instance for performance testing, to reproduce an issue, or simply to verify that your setup is configured correctly. There are several tools available for generating test data, some of which are outlined below.

Using the Test Generator input module

The Test Generator (im_testgen) input module is a simple log generator available as part of NXLog. This module generates log records up to a specified number or until the module is stopped. Just like any other NXLog input module, it supports the Common Module Directives and can be used in conjunction with the available Extension Modules.

Example 1. Generating JSON log records with im_testgen

This configuration uses the im_testgen input module to generate 10 events. Log records are converted to JSON using the xm_json extension and saved to file using the om_file output module.

<Extension json>
    Module      xm_json
</Extension>

<Input generate_data>
    Module      im_testgen
    MaxCount    10
    Exec        to_json();
</Input>

<Output output_file>
    Module      om_file
    File        "/var/log/file"
</Output>

Output sample

The following log record in JSON format was generated by the im_testgen input module.

{
  "SeverityValue": 2,
  "EventTime": "2021-05-03T11:47:43.118684+02:00",
  "SourceName": "nxlog",
  "ProcessID": 8812,
  "EventReceivedTime": "2021-05-03T11:47:43.118686+02:00",
  "SourceModuleName": "generate_data",
  "SourceModuleType": "im_testgen"
}

im_testgen can also be used to test the configuration of another NXLog installation. This is especially useful for testing input modules that receive logs over the network such as the TCP (im_tcp) and UDP (im_udp) modules.

Example 2. Sending generated syslog messages over TCP

This configuration uses the im_testgen input module to generate events until NXLog is stopped. Log records are converted to syslog (RFC 3164) format using the xm_syslog extension and forwarded over TCP using the om_tcp output module.

<Extension syslog>
    Module      xm_syslog
</Extension>

<Input generate_data>
    Module      im_testgen
    Exec        to_syslog_bsd();
</Input>

<Output output_tcp>
    Module      om_tcp
    Host        192.168.0.123:514
</Output>

Output sample

The following syslog record was generated by the im_testgen input module.

<14>May  3 12:38:18 server-1 nxlog[13049]: 0@Mon May 03 12:38:18 2021

The log messges generated by im_testgen are sequentially numbered in most formats. This may come in useful for troubleshooting and can for example help in detecting packet loss or flow control issues.

Using netcat

netcat is a utility for network communication over TCP or UDP and is available on most Linux distributions. It can be used as a traffic generator for an NXLog input instance that has been configured to listen for connections using the TCP (im_tcp) or UDP (im_udp) modules.

netcat can send log records from file over TCP using the following command:

$ netcat [IP] [PORT] <[INPUT_FILE]

Where [IP] and [PORT] are the destination IP address and port number and [INPUT_FILE] is the path to a text file containing log records separated by newline. Example:

$ netcat 192.168.0.123 514 </path/to/log/file

Other useful arguments:

-i: Interval in seconds between sending lines of text.
-u: Use UDP instead of the default TCP.
-w: Specifies an idle timeout in seconds after which the connection will be closed.

Using a log generator script

NXLog can execute external scripts using the External programs (im_exec) input module. In addition it provides language-specific input modules that support the execution of scripts and programs written in the Perl, the Python, Java, Ruby, and Go. This enables you to use a custom event generator that can output structured logs in a way that mimics a real log source.

An example of such a script is the Log Generator tool available in our public contrib repository. The tool is written in Perl and is designed to generate logs based on variadic templates. Several templates are readily available and can be used to start generating data. You can also generate log records in a custom format by creating your own JSON template and specifying the variable data in the script accordingly.

Example 3. Generating log data with the Log Generator script

This configuration uses the im_exec input module to execute the log-generator.pl script. It specifies two arguments that will be passed to the script:

-delay is the interval in microseconds between each event
-datadir specifies the path to the directory containing the template(s) and other data files

Log records are parsed as JSON using the xm_json extension and saved to file using the om_file output module.

<Extension json>
    Module      xm_json
</Extension>

<Input generate_data>
    Module      im_exec
    Command     /path/to/log-generator.pl
    Arg         -delay=5000000
    Arg         -datadir=/opt/nxlog/etc/log-generator
    Exec        parse_json(); $raw_event = $rawEvent; delete($rawEvent);
    InputType   LineBased
</Input>

<Output output_file>
    Module      om_file
    File        "/var/log/file"
</Output>

Input sample

The following log record in JSON format was generated by the Log Generator tool using the template_cisco_acs.json template.

{
  "MessageSourceAddress": "67.0.49.113",
  "EventReceivedTime": "2021-05-04 10:56:05",
  "SourceModuleName": "mod_test3",
  "SourceModuleType": "im_file",
  "SyslogFacilityValue": 22,
  "SyslogFacility": "LOCAL6",
  "SyslogSeverityValue": 6,
  "SyslogSeverity": "CRITICAL",
  "SeverityValue": 2,
  "Severity": "WARNING",
  "Hostname": "heilbronn-emh1",
  "EventTime": "2021-05-04 10:56:05",
  "Message": "User-Name=Hunter,NAS-IP-Address=67.0.49.113,Login-IP-Host=67.0.49.113,AAA Server=srvcourbet,Access Device=nac,",
  "ACSCategoryNumber": "235",
  "ACSCategoryName": "FIXME",
  "ACSMessageId": "512206847",
  "ACSTotalSegments": "235",
  "ACSSegmentNumber": "235",
  "AccountName": "Hunter",
  "ACSNASIPAddress": "67.0.49.113",
  "ACSAAAServer": "srvcourbet",
  "rawEvent": "<31>May 04 10:56:05 heilbronn-emh1 CisACS_235_FIXME 512206847 235 235 User-Name=Hunter,NAS-IP-Address=67.0.49.113,Login-IP-Host=67.0.49.113,AAA Server=srvcourbet,Access Device=nac,"
}

Output sample

The following is the same log record after it was parsed by NXLog and saved to file.

<31>May 04 10:56:05 heilbronn-emh1 CisACS_235_FIXME 512206847 235 235 User-Name=Hunter,NAS-IP-Address=67.0.49.113,Login-IP-Host=67.0.49.113,AAA Server=srvcourbet,Access Device=nac,

Load testing

Sometimes you may need to perform load testing on NXLog, for example to determine data throughput and resource usage. A stress testing tool is available as part of the publicly available NXLog Community Edition repository. This tool generates log data and sends it over the network using TCP. NXLog can receive these logs using the TCP (im_tcp) input module.

Execute the tool by specifying one or more destinations in the format IP:PORT example:

$ ./stresstest 192.16.0.123:514

The tool stops sending logs when the connection is closed by the server or its execution is interrupted from the console.

The stress test tool may need to be built from source as part of NXLog Community Edition.