Generating test data
You may need to generate and send test data to an NXLog Agent instance to test performance, reproduce an issue, or verify that your configuration is correct. Several tools are available for generating test data, some of which are outlined below.
Using the Test Generator input module
NXLog Agent provides the Test Generator (im_testgen) input module. This module generates log records up to a specified number or until the module stops. Like any other NXLog Agent input module, it supports the common module settings and can be used with extension modules.
This configuration uses the im_testgen module to generate ten events. It converts log records to JSON using the xm_json extension and saves them to a file using the om_file output module.
<Extension json>
Module xm_json
</Extension>
<Input test>
Module im_testgen
MaxCount 10
Exec to_json();
</Input>
<Output output_file>
Module om_file
File '/var/log/file'
</Output>
The configuration above generates JSON log records like the following.
{
"SeverityValue": 2,
"EventTime": "2024-06-27T11:47:43.118684+02:00",
"SourceName": "nxlog",
"ProcessID": 8812,
"EventReceivedTime": "2024-06-27T11:47:43.118686+02:00",
"SourceModuleName": "test",
"SourceModuleType": "im_testgen"
}
You can also use im_testgen to test another NXLog Agent installation. It helps in testing input modules that receive logs over the network, such as the TCP (im_tcp) and UDP (im_udp) modules.
This configuration uses the im_testgen module to generate events until NXLog Agent stops. It converts log records to syslog (RFC 3164) format using the xm_syslog extension and forwards them over TCP using the om_tcp output module.
<Extension syslog>
Module xm_syslog
</Extension>
<Input test>
Module im_testgen
Exec to_syslog_bsd();
</Input>
<Output tcp_output>
Module om_tcp
Host 192.168.0.123:514
</Output>
The configuration above generates syslog messages like the following.
<14>May 3 12:38:18 server-1 nxlog[13049]: 0@Mon May 03 12:38:18 2021
im_testgen generates logs numbered sequentially in most formats. Numbering helps troubleshoot issues such as packet loss or flow control not working as expected. |
Using netcat
netcat is a utility for network communication over TCP or UDP that is available on most Linux distributions. It can be used as a network traffic generator for an NXLog Agent input instance configured to listen for connections using the TCP (im_tcp) or UDP (im_udp) modules.
The following netcat command sends log records from a file over TCP, where [IP]
and [PORT]
are the destination IP address and port number, and [INPUT_FILE]
is the path to a text file containing log records separated by a newline:
$ netcat [IP] [PORT] <[INPUT_FILE]
For example:
$ netcat 192.168.0.123 514 </path/to/log/file
Other useful arguments:
- -i
-
Interval in seconds between sending lines of text.
- -u
-
Use UDP instead of the default TCP.
- -w
-
Specifies an idle timeout in seconds, after which the connection will be closed.
Generating logs from a script
NXLog Agent can execute external scripts using the External programs (im_exec) module. In addition, it provides language-specific input modules that support executing scripts and programs written in Perl, Python, Java, Ruby, and Go. These modules enable you to use a custom event generator that can output structured logs in a way that mimics a real log source.
An example of such a script is the Log Generator tool in our public Git repository. The tool is written in Perl and designed to generate logs based on templates. You can use one of the available templates to generate data or create a custom template to generate log records in a specific format.
This configuration uses the im_exec input module to execute the log-generator.pl
script, specifying two arguments to pass on to the script:
-
-delay
is the interval in microseconds between each event -
-datadir
specifies the path to the directory containing the template(s) and other data files
It parses the JSON log records using the xm_json extension and saves them to a file using the om_file output module.
<Extension json>
Module xm_json
</Extension>
<Input test>
Module im_exec
Command /path/to/log-generator.pl
Arg -delay=5000000
Arg -datadir=/opt/nxlog/etc/log-generator
InputType LineBased
<Exec>
parse_json(); (1)
$raw_event = $rawEvent; (2)
delete($rawEvent); (3)
</Exec>
</Input>
<Output output_file>
Module om_file
File '/var/log/file'
</Output>
1 | The parse_json() procedure parses the JSON log records into fields. |
2 | Copies the value of the templates’s $rawEvent field to the NXLog Agent $raw_event core field.
The om_file module writes the value of $raw_event to the output file.
If the field is not set or does not exist, it will write an empty line. |
3 | The delete() procedure drops the duplicate field. |
The Log Generator Tool produced the following log record using the template_cisco_acs.json
template.
{
"MessageSourceAddress": "67.0.49.113",
"EventReceivedTime": "2024-06-27 10:56:05",
"SourceModuleName": "mod_test3",
"SourceModuleType": "im_file",
"SyslogFacilityValue": 22,
"SyslogFacility": "LOCAL6",
"SyslogSeverityValue": 6,
"SyslogSeverity": "CRITICAL",
"SeverityValue": 2,
"Severity": "WARNING",
"Hostname": "heilbronn-emh1",
"EventTime": "2024-06-27 10:56:05",
"Message": "User-Name=Hunter,NAS-IP-Address=67.0.49.113,Login-IP-Host=67.0.49.113,AAA Server=srvcourbet,Access Device=nac,",
"ACSCategoryNumber": "235",
"ACSCategoryName": "FIXME",
"ACSMessageId": "512206847",
"ACSTotalSegments": "235",
"ACSSegmentNumber": "235",
"AccountName": "Hunter",
"ACSNASIPAddress": "67.0.49.113",
"ACSAAAServer": "srvcourbet",
"rawEvent": "<31>Jun 27 10:56:05 heilbronn-emh1 CisACS_235_FIXME 512206847 235 235 User-Name=Hunter,NAS-IP-Address=67.0.49.113,Login-IP-Host=67.0.49.113,AAA Server=srvcourbet,Access Device=nac,"
}
The following is the same log record after NXLog Agent processed it.
<31>Jun 27 10:56:05 heilbronn-emh1 CisACS_235_FIXME 512206847 235 235 User-Name=Hunter,NAS-IP-Address=67.0.49.113,Login-IP-Host=67.0.49.113,AAA Server=srvcourbet,Access Device=nac,
Load testing
Sometimes, you may need to perform load testing on NXLog Agent to determine data throughput and resource usage. A stress testing tool is available in the public NXLog Community Edition repository. This tool generates logs and sends them over the network via TCP. NXLog Agent can receive these logs with the TCP (im_tcp) input module.
Execute the tool by specifying one or more destinations in the format IP:PORT
, for example:
$ ./stresstest 192.16.0.123:514
The tool stops sending logs when the remote server closes the connection or its execution is interrupted.
You may need to build the stress test tool from the source as part of NXLog Community Edition. |