Generating test data

You may need to generate and send test data to an NXLog Agent instance to test performance, reproduce an issue, or verify that your configuration is correct. Several tools are available for generating test data, some of which are outlined below.

Using the Test Generator input module

NXLog Agent provides the Test Generator (im_testgen) input module. This module generates log records up to a specified number or until the module stops. Like any other NXLog Agent input module, it supports the common module settings and can be used with extension modules.

Example 1. Generating JSON log records

This configuration uses the im_testgen module to generate ten events. It converts log records to JSON using the xm_json extension and saves them to a file using the om_file output module.

nxlog.conf

<Extension json>
    Module      xm_json
</Extension>

<Input test>
    Module      im_testgen
    MaxCount    10
    Exec        to_json();
</Input>

<Output output_file>
    Module      om_file
    File        '/var/log/file'
</Output>

The configuration above generates JSON log records like the following.

Output sample

{
  "SeverityValue": 2,
  "EventTime": "2024-06-27T11:47:43.118684+02:00",
  "SourceName": "nxlog",
  "ProcessID": 8812,
  "EventReceivedTime": "2024-06-27T11:47:43.118686+02:00",
  "SourceModuleName": "test",
  "SourceModuleType": "im_testgen"
}

You can also use im_testgen to test another NXLog Agent installation. It helps in testing input modules that receive logs over the network, such as the TCP (im_tcp) and UDP (im_udp) modules.

Example 2. Generating syslog messages

This configuration uses the im_testgen module to generate events until NXLog Agent stops. It converts log records to syslog (RFC 3164) format using the xm_syslog extension and forwards them over TCP using the om_tcp output module.

nxlog.conf

<Extension syslog>
    Module      xm_syslog
</Extension>

<Input test>
    Module      im_testgen
    Exec        to_syslog_bsd();
</Input>

<Output tcp_output>
    Module      om_tcp
    Host        192.168.0.123:514
</Output>

The configuration above generates syslog messages like the following.

Output sample

<14>May  3 12:38:18 server-1 nxlog[13049]: 0@Mon May 03 12:38:18 2021

im_testgen generates logs numbered sequentially in most formats. Numbering helps troubleshoot issues such as packet loss or flow control not working as expected.

Using netcat

netcat is a utility for network communication over TCP or UDP that is available on most Linux distributions. It can be used as a network traffic generator for an NXLog Agent input instance configured to listen for connections using the TCP (im_tcp) or UDP (im_udp) modules.

The following netcat command sends log records from a file over TCP, where [IP] and [PORT] are the destination IP address and port number, and [INPUT_FILE] is the path to a text file containing log records separated by a newline:

$ netcat [IP] [PORT] <[INPUT_FILE]

For example:

$ netcat 192.168.0.123 514 </path/to/log/file

Other useful arguments:

-i: Interval in seconds between sending lines of text.
-u: Use UDP instead of the default TCP.
-w: Specifies an idle timeout in seconds, after which the connection will be closed.

Generating logs from a script

NXLog Agent can execute external scripts using the External programs (im_exec) module. In addition, it provides language-specific input modules that support executing scripts and programs written in Perl, Python, Java, Ruby, and Go. These modules enable you to use a custom event generator that can output structured logs in a way that mimics a real log source.

An example of such a script is the Log Generator tool in our public Git repository. The tool is written in Perl and designed to generate logs based on templates. You can use one of the available templates to generate data or create a custom template to generate log records in a specific format.

Example 3. Executing a log generator script

This configuration uses the im_exec input module to execute the log-generator.pl script, specifying two arguments to pass on to the script:

-delay is the interval in microseconds between each event
-datadir specifies the path to the directory containing the template(s) and other data files

It parses the JSON log records using the xm_json extension and saves them to a file using the om_file output module.

nxlog.conf

<Extension json>
    Module      xm_json
</Extension>

<Input test>
    Module      im_exec
    Command     /path/to/log-generator.pl
    Arg         -delay=5000000
    Arg         -datadir=/opt/nxlog/etc/log-generator
    InputType   LineBased
    <Exec>
        parse_json(); (1)
        $raw_event = $rawEvent; (2)
        delete($rawEvent); (3)
    </Exec>
</Input>

<Output output_file>
    Module      om_file
    File        '/var/log/file'
</Output>

1	The parse_json() procedure parses the JSON log records into fields.
2	Copies the value of the templates’s `$rawEvent` field to the NXLog Agent `$raw_event` core field. The om_file module writes the value of `$raw_event` to the output file. If the field is not set or does not exist, it will write an empty line.
3	The delete() procedure drops the duplicate field.

The Log Generator Tool produced the following log record using the template_cisco_acs.json template.

Input sample

{
  "MessageSourceAddress": "67.0.49.113",
  "EventReceivedTime": "2024-06-27 10:56:05",
  "SourceModuleName": "mod_test3",
  "SourceModuleType": "im_file",
  "SyslogFacilityValue": 22,
  "SyslogFacility": "LOCAL6",
  "SyslogSeverityValue": 6,
  "SyslogSeverity": "CRITICAL",
  "SeverityValue": 2,
  "Severity": "WARNING",
  "Hostname": "heilbronn-emh1",
  "EventTime": "2024-06-27 10:56:05",
  "Message": "User-Name=Hunter,NAS-IP-Address=67.0.49.113,Login-IP-Host=67.0.49.113,AAA Server=srvcourbet,Access Device=nac,",
  "ACSCategoryNumber": "235",
  "ACSCategoryName": "FIXME",
  "ACSMessageId": "512206847",
  "ACSTotalSegments": "235",
  "ACSSegmentNumber": "235",
  "AccountName": "Hunter",
  "ACSNASIPAddress": "67.0.49.113",
  "ACSAAAServer": "srvcourbet",
  "rawEvent": "<31>Jun 27 10:56:05 heilbronn-emh1 CisACS_235_FIXME 512206847 235 235 User-Name=Hunter,NAS-IP-Address=67.0.49.113,Login-IP-Host=67.0.49.113,AAA Server=srvcourbet,Access Device=nac,"
}

The following is the same log record after NXLog Agent processed it.

Output sample

<31>Jun 27 10:56:05 heilbronn-emh1 CisACS_235_FIXME 512206847 235 235 User-Name=Hunter,NAS-IP-Address=67.0.49.113,Login-IP-Host=67.0.49.113,AAA Server=srvcourbet,Access Device=nac,

Load testing

Sometimes, you may need to perform load testing on NXLog Agent to determine data throughput and resource usage. A stress testing tool is available in the public NXLog Community Edition repository. This tool generates logs and sends them over the network via TCP. NXLog Agent can receive these logs with the TCP (im_tcp) input module.

Execute the tool by specifying one or more destinations in the format IP:PORT, for example:

$ ./stresstest 192.16.0.123:514

The tool stops sending logs when the remote server closes the connection or its execution is interrupted.

You may need to build the stress test tool from the source as part of NXLog Community Edition.