Statistical counters

Statistical counters in the NXLog language are specialized types for aggregating and analyzing log data. These counters enable detailed statistical analysis over specified intervals, helping understand trends and patterns in the logged data.

You must first create a statistical counter using the create_stat() procedure. Once you create it, you can update the statistical counter’s value with the add_stat() procedure and retrieve it with the get_stat() function.

The statistical counter’s value is recalculated every time you update or read it. This can result in a slight distortion of the calculated value if you don’t update or read the value frequently.

You can also specify a time value when creating, updating, and reading a statistical counter to use them with offline log processing.

NXLog Agent supports the following statistical counters:

COUNT: Calculates the sum of the values. It accepts positive and negative integers.

COUNTMIN: Stores the minimum value recorded.

COUNTMAX: Stores the maxium value recorded.

AVG: Calculates the average over the specified interval.

AVGMIN: Calculates the average over the specified interval and stores the lowest average recorded.

AVGMAX: Calculates the average over the specified interval and stores the highest average recorded.

RATE: Calculates the value over the specified interval. You can use this to calculate the events per second (EPS) rate.

RATEMIN: Calculates the value over the specified interval and stores the lowest rate recorded.

RATEMAX: Calculates the value over the specified interval and stores the highest rate recorded.

GRAD: Calculates the change in the rate of the counter over the specified interval, which is the gradient.

GRADMIN: Calculates the gradient and returns the lowest value recorded.

GRADMAX: Calculates the gradient and returns the highest value recorded.

Examples

Example 1. Using COUNT with positive and negative values

This configuration demonstrates how the COUNT statistical counter works. It initializes sum on startup. It then adds 1 to it every second and subtracts 5 every 20 seconds.

nxlog.conf

<Input null_input>
    Module    im_null
</Input>

<Output null_output>
    Module    om_null
   <Schedule>
      Exec    log_info("******* @ STARTUP ********");      
      When    @startup
      Exec    create_stat('sum', 'COUNT', 10);     
   </Schedule>
   <Schedule>
      Every   1 sec
      Exec    add_stat('sum', 1);
      Exec    log_info("Sum: " + get_stat('sum'));       
   </Schedule>
   <Schedule>
      Every   20 sec
      Exec    log_info("******* EVERY 20 SEC subtract 5 ********");       
      Exec    add_stat('sum', -5);      
   </Schedule>   
</Output>

<Route r1>
    Path     null_input => null_output
</Route>

Output sample

2024-11-22 15:40:04 INFO [om_null|null_output] ******* @ STARTUP ********
2024-11-22 15:40:04 INFO [om_null|null_output] Sum: 1
2024-11-22 15:40:04 INFO [om_null|null_output] ******* EVERY 20 SEC subtract 5 ********
2024-11-22 15:40:05 INFO [om_null|null_output] Sum: -3
2024-11-22 15:40:06 INFO [om_null|null_output] Sum: -2
2024-11-22 15:40:07 INFO [om_null|null_output] Sum: -1
2024-11-22 15:40:08 INFO [om_null|null_output] Sum: 0
2024-11-22 15:40:09 INFO [om_null|null_output] Sum: 1
2024-11-22 15:40:10 INFO [om_null|null_output] Sum: 2
2024-11-22 15:40:11 INFO [om_null|null_output] Sum: 3
2024-11-22 15:40:12 INFO [om_null|null_output] Sum: 4
2024-11-22 15:40:13 INFO [om_null|null_output] Sum: 5
2024-11-22 15:40:14 INFO [om_null|null_output] Sum: 6
2024-11-22 15:40:15 INFO [om_null|null_output] Sum: 7
2024-11-22 15:40:16 INFO [om_null|null_output] Sum: 8
2024-11-22 15:40:17 INFO [om_null|null_output] Sum: 9
2024-11-22 15:40:18 INFO [om_null|null_output] Sum: 10
2024-11-22 15:40:19 INFO [om_null|null_output] Sum: 11
2024-11-22 15:40:20 INFO [om_null|null_output] Sum: 12
2024-11-22 15:40:21 INFO [om_null|null_output] Sum: 13
2024-11-22 15:40:22 INFO [om_null|null_output] Sum: 14
2024-11-22 15:40:23 INFO [om_null|null_output] Sum: 15
2024-11-22 15:40:24 INFO [om_null|null_output] ******* EVERY 20 SEC subtract 5 ********
2024-11-22 15:40:24 INFO [om_null|null_output] Sum: 11
2024-11-22 15:40:25 INFO [om_null|null_output] Sum: 12
2024-11-22 15:40:26 INFO [om_null|null_output] Sum: 13
2024-11-22 15:40:27 INFO [om_null|null_output] Sum: 14

Example 2. Using AVG to calculate EPS

This configuration demonstrates how to calculate the processing rate of a module instance in events per second. It creates the sum and avg counters on startup. The avg counter has an interval of 60 seconds. It then checks the value of the counters every minute.

nxlog.conf

<Input test>
    Module    im_testgen
</Input>

<Output null_output>
    Module    om_null
    Exec      add_stat("sum", 1); (1)
    Exec      add_stat("avg", 1); (2)

    <Schedule>
      Exec    log_info("******* @ STARTUP ********");      
      When    @startup
      Exec    create_stat("sum", "COUNT");
      Exec    create_stat("avg", "AVG", 60); 
    </Schedule>
    <Schedule>
        Every   1 min
        Exec    log_info("SUM: " + get_stat("sum"));
        Exec    log_info("AVG: " + get_stat("avg"));
    </Schedule>
</Output>

<Route r1>
    Path     test => null_output
</Route>

1	Increases `sum` by 1 for every event processed.
2	Adds 1 to the `avg` counter for every event processed.

Ouput sample

2024-11-22 16:04:51 INFO [om_null|null_output] ******* @ STARTUP ********
2024-11-22 16:04:51 INFO [om_null|null_output] SUM:0
2024-11-22 16:04:51 INFO [om_null|null_output] AVG: 0
2024-11-22 16:05:51 INFO [om_null|null_output] SUM:12562050
2024-11-22 16:05:51 INFO [om_null|null_output] AVG: 12561907
2024-11-22 16:06:51 INFO [om_null|null_output] SUM:25721900
2024-11-22 16:06:51 INFO [om_null|null_output] AVG: 12860859
2024-11-22 16:07:51 INFO [om_null|null_output] SUM:38399200
2024-11-22 16:07:51 INFO [om_null|null_output] AVG: 12799685
2024-11-22 16:08:51 INFO [om_null|null_output] SUM:50173900
2024-11-22 16:08:51 INFO [om_null|null_output] AVG: 12543448
2024-11-22 16:09:51 INFO [om_null|null_output] SUM:62157450
2024-11-22 16:09:51 INFO [om_null|null_output] AVG: 12431473