Prometheus (om_prometheus)

This module exposes Prometheus-compatible metrics via an HTTP endpoint. It supports counters and gauges extracted from structured records.

Output data format

The module converts incoming records into a set of Prometheus-compatible key-value pairs. It processes events according to the specified mapping. It supports:

Counters — Values that accumulate over time.
Gauges — Values that represent the current state.

The module treats any numeric field without a corresponding mapping as a Gauge.

The module exposes metrics over HTTP in standard Prometheus exposition format. The following endpoints are available:

/metrics: Returns the current set of metrics in Prometheus format
/healthy: Always returns HTTP 200. Used for checking the probe health.
/ready: Always returns HTTP 200. Used for checking the probe readiness.

Nested structures in incoming records are automatically flattened using _ as the delimiter. For example, { "system": { "cpu": { "usage": 42.5 }}} becomes system_cpu_usage. The name and labels fields in the MappingFile are sanitized as well and must match the fully flattened keys.

Consider the behavior of floating-point number representation—Incoming integer numbers greater than 2^53 will lose precision, and their output will be in scientific format.

Configuration

The om_prometheus module accepts the following directives in addition to the common module directives.

HTTPS directives

HTTPSCACertFile

The path of the certificate authority (CA) certificate that will be used to verify the certificate presented by the remote host. A remote host’s self-signed certificate (which is not signed by a CA) can be trusted by specifying the remote host certificate itself. In the case of certificates signed by an intermediate CA, the certificate specified must contain the complete certificate chain (certificate bundle).

HTTPSCertFile

The path of the certificate file that will be presented to the remote host during the HTTPS handshake.

HTTPSCertKeyFile

The path of the private key file that was used to generate the certificate specified by the HTTPSCertFile directive. This is used for the HTTPS handshake.

The key must be in PKCS#8 format. Outdated PKCS#1 RSA keys, for example, those starting with -----BEGIN RSA PRIVATE KEY-----, are not supported and will cause the module to fail.

Optional directives

ListenAddr

The IP address and port that the module will bind to and accept connections on. The default is 0.0.0.0:9400.

MappingFile

The path to a JSON schema file specifying how to interpret fields. This file should define a metrics object containing counter and gauge arrays as follows:

{
  "metrics": {
    "counter": [
      {
        "name": "field_1",
        "labels": ["label_1", "label_2"]
      }
    ],
    "gauge": [
      {
        "name": "field_2",
        "labels": ["label_1", "label_3"]
      }
    ]
  }
}

The module treats fields in log records without a corresponding mapping as gauges.

Each field in the incoming records can either be used as a metric or as a label, but not both.

If a field name appears as a label in the MappingFile, that field will not be used as a metric—even if it holds a numeric value in some records.

Sanitize

Prometheus allows only [A-Za-z0-9:_] characters in metric and label names. Use this directive to automatically sanitize all keys, including those defined in the mapping file, to ensure compliance with Prometheus naming rules.

The default is TRUE, the module sanitizes keys in incoming structured data, for example, JSON field names, to comply with Prometheus naming according to these rules:

Replace invalid characters, including the period (.), with an underscore (_).
Prefix keys that start with a digit or non-alphabetic character with nx_.

When FALSE, invalid metric names may cause Prometheus to reject the data.

For more information about Prometheus naming requirements, refer to the Prometheus naming guidelines.

When using this module, keep this sanitization behavior in mind and ensure that duplicate metrics are not created unintentionally.

For example, the keys 1metric and nx_1metric will both be sanitized to nx_1metric, causing a duplicate metric.

Examples

Example 1. Exposing metrics for Prometheus scraping

This configuration processes JSON records and exports them as metrics for Prometheus scraping. See the Prometheus <scrape_config> documentation for information on scraping targets.

nxlog.conf

<Output prometheus_export>
    Module         om_prometheus
    ListenAddr     0.0.0.0:9464
    MappingFile    /opt/nxlog/etc/prometheus-mapping.json
</Output>

The following mapping defines a counter and several gauge fields and their labels.

prometheus-mapping.json

{
  "metrics": {
    "counter": [
      {
        "name": "http_request_count",
        "labels": ["method", "region"]
      }
    ],
    "gauge": [
      {
        "name": "cpu_usage",
        "labels": ["host", "region"]
      },
      {
        "name": "memory_usage",
        "labels": ["host", "region"]
      },
      {
        "name": "disk_io",
        "labels": ["host", "region"]
      }
    ]
  }
}

Input sample

{"cpu_usage": 38.6, "memory_usage": 65.0, "disk_io": 78, "host": "server3", "region": "eu-central"}
{"cpu_usage": 38.7, "http_request_count": 33, "disk_io": 110, "new_parameter": 333.33, "method": "SET", "region": "eu-central"}

The following is a sample response to a request to http://localhost:9464/metrics.

Output sample

# TYPE cpu_usage gauge
cpu_usage{region="eu-central"} 38.7
# TYPE disk_io gauge
disk_io{region="eu-central"} 110
# TYPE http_request_count counter
http_request_count{method="SET",region="eu-central"} 33
# TYPE memory_usage gauge
memory_usage{host="server3",region="eu-central"} 65
# TYPE new_parameter gauge
new_parameter{region="eu-central"} 333.33