Filter metrics with NXLog Agent

Collecting metrics from servers and applications often generates excess data, including information from test or staging environments, and attributes that produce large numbers of unique time series. Retaining this data unnecessarily increases storage usage, network traffic, and query complexity.

Filtering metrics at the agent level helps you store only data relevant to monitoring and troubleshooting. By discarding unnecessary data early in the pipeline, you can reduce overhead and keep your observability data focused and manageable.

Drop metrics from non-production infrastructure

Metrics from development, testing, or staging systems may not be relevant for production monitoring and can unnecessarily increase data volume. In such cases, you can configure NXLog Agent to discard metrics originating from non-production environments before forwarding them to the observability platform.

Example 1. Discarding metrics from non-production environments

This configuration uses the TCP input module to listen for JSON-formatted metrics. It parses records using the JSON extension and discards records from non-production environments.

nxlog.conf

<Extension json>
    Module        xm_json
    Flatten       TRUE (1)
</Extension>

<Input metrics>
    Module        im_tcp
    ListenAddr    0.0.0.0:1514
    <Exec>
        parse_json(); (2)

        if ($service.environment != "production") { (3)
            drop();
        }
    </Exec>
</Input>

1	The Flatten directive is set to `TRUE` so that the module converts fields in nested JSON objects to dot notation when parsing JSON data.
2	The parse_json() procedure parses the value of the $raw_event core field into fields.
3	Discards the record if the value of the `$service.environment` field is not equal to `production`.

The following is a sample of a metric from a frontend application. The NXLog Agent configuration above expects records in this format.

Input sample

{
  "timestamp": "2026-03-06T12:41:10Z",
  "metric": "http_requests_total",
  "value": 1245,
  "unit": "requests",
  "type": "counter",
  "service": {
    "name": "web-frontend",
    "version": "2.3.1",
    "environment": "production"
  },
  "host": {
    "name": "web-01",
    "region": "eu-west"
  },
  "labels": {
    "method": "GET",
    "route": "/products",
    "status_code": 200
  }
}

Remove high-cardinality fields

Metrics sometimes have attributes with many unique values, such as process IDs or file paths. Removing these high-cardinality fields reduces the number of metric series and improves the efficiency of your monitoring platform. NXLog Agent provides several features to remove fields depending on the type and structure of the data.

Example 2. Deleting unnecessary fields

This configuration uses the OpenTelemetry Collector input module to collect telemetry data over OTLP and removes high-cardinality and unnecessary fields from metric records.

nxlog.conf

<Extension rewrite>
    Module    xm_rewrite (1)
    Delete    EventReceivedTime, SourceModuleName, SourceModuleType, Hostname, RecordType (2)
</Extension>

<Input otel>
    Module    im_otel
    <Exec>
        if ($RecordType == "Metric") {
            hash_remove($Resource.Attributes, "host.id"); (3)
            hash_remove($Resource.Attributes, "process.pid");
            hash_remove($Resource.Attributes, "process.command");
            hash_remove($Resource.Attributes, "process.command_args");
            hash_remove($Resource.Attributes, "process.executable.path");
            hash_remove($Resource.Attributes, "process.runtime.description");

            rewrite->process();
        }
    </Exec>
</Input>

1	The Rewrite extension provides the functionality to normalize data records, including discarding unwanted fields.
2	Deletes fields added by NXLog Agent.
3	The hash_remove() procedure removes elements from a hash value.

The following is a sample of an OpenTelemetry metric. The NXLog Agent configuration above expects records in this format.

Input sample

{
  "resourceMetrics": [
    {
      "resource": {
        "attributes": [
          {
            "key": "host.name",
            "value": {
              "stringValue": "web-01"
            }
          },
          {
            "key": "host.arch",
            "value": {
              "stringValue": "amd64"
            }
          },
          {
            "key": "host.id",
            "value": {
              "stringValue": "7994bd4cc9744ab78846106097c30ff6"
            }
          },
          {
            "key": "process.pid",
            "value": {
              "intValue": 121652
            }
          },
          {
            "key": "process.executable.name",
            "value": {
              "stringValue": "node"
            }
          },
          {
            "key": "process.executable.path",
            "value": {
              "stringValue": "/opt/demo-webapp/.nvm/versions/node/v22.13.0/bin/node"
            }
          },
          {
            "key": "process.command_args",
            "value": {
              "arrayValue": {
                "values": [
                  {
                    "stringValue": "/opt/demo-webapp/.nvm/versions/node/v22.13.0/bin/node"
                  },
                  {
                    "stringValue": "/opt/demo-webapp/otel-node-demo/app.js"
                  }
                ]
              }
            }
          },
          {
            "key": "process.runtime.version",
            "value": {
              "stringValue": "22.13.0"
            }
          },
          {
            "key": "process.runtime.name",
            "value": {
              "stringValue": "nodejs"
            }
          },
          {
            "key": "process.runtime.description",
            "value": {
              "stringValue": "Node.js"
            }
          },
          {
            "key": "process.command",
            "value": {
              "stringValue": "/opt/webapp/otel-node-demo/app.js"
            }
          },
          {
            "key": "process.owner",
            "value": {
              "stringValue": "adm"
            }
          },
          {
            "key": "service.name",
            "value": {
              "stringValue": "demo-webapp"
            }
          },
          {
            "key": "service.version",
            "value": {
              "stringValue": "1.0.0"
            }
          },
          {
            "key": "deployment.environment",
            "value": {
              "stringValue": "staging"
            }
          }
        ],
        "droppedAttributesCount": 0
      },
      "scopeMetrics": [
        {
          "scope": {
            "name": "webapp.metrics",
            "version": ""
          },
          "metrics": [
            {
              "name": "web_requests_total",
              "description": "",
              "unit": "",
              "sum": {
                "aggregationTemporality": 2,
                "isMonotonic": true,
                "dataPoints": [
                  {
                    "attributes": [
                      {
                        "key": "status",
                        "value": {
                          "stringValue": "success"
                        }
                      }
                    ],
                    "startTimeUnixNano": "1773082415440000000",
                    "timeUnixNano": "1773082419441000000",
                    "asDouble": 26
                  }
                ]
              }
            }
          ]
        }
      ]
    }
  ]
}