Monitoring & Metrics
RESTHeartArchitecture
RESTHeart includes a built-in metrics system that collects HTTP request data and optionally JVM metrics, exposes them in Prometheus format via a REST endpoint, and provides a real-time browser dashboard (v9.5.0+).
ββββββββββββββββ Prometheus ββββββββββββββββββββ HTML/JS βββββββββββββββββββ
β RESTHeart ββββ text format βββΆβ GET /metrics βββββββββββββββββββ /metrics-ui β
β (interceptorβ on demand β (MetricsService)β Chart.js β (embedded SPA) β
β collects) β ββββββββββββββββββββ βββββββββββββββββββ
ββββββββββββββββ
The metrics module has four components:
| Component | Type | Description |
|---|---|---|
| RequestsMetricsCollector | Interceptor (REQUEST_BEFORE_AUTH) |
Records timing and count for every HTTP request, grouped by method + status code + path template |
| JvmMetricsCollector | Initializer | Optionally registers JVM memory and GC metrics (disabled by default) |
| MetricsService | Service (GET /metrics) |
Serves aggregated metrics in Prometheus exposition format |
| Metrics UI | Embedded static page (v9.5.0+) | Self-contained dashboard at /metrics-ui that visualizes metrics in real time |
Configuration
Metrics configuration is in restheart.yml (or overridden via the RHO environment variable).
Metrics Service
metrics:
enabled: true # enable/disable the metrics service
uri: /metrics # endpoint path
missing-registry-status-code: 404 # HTTP status when no registries exist (404 or 200)
Request Metrics Collector
requestsMetricsCollector:
enabled: true
include: ["/*"] # path patterns to track
exclude: ["/metrics", "/metrics/*"] # path patterns to skip (avoids feedback loop)
Metrics will be gathered for requests that match the path templates specified in the include criteria and do not match those listed in the exclude criteria.
Note that when using the variable {tenant} in the include path templates, the metrics will be tagged with path_template_param_tenant=<value>. This tagging does not apply when using wildcards in path templates.
JVM Metrics Collector
jvmMetricsCollector:
enabled: false # set to true to expose JVM memory and GC metrics
Additionally, RESTHeart captures JVM metrics such as memory usage and garbage collector data.
Metrics UI (Static Resources)
Note: The embedded Metrics UI dashboard is available starting from RESTHeart v9.5.0.
The dashboard is served as an embedded static resource:
static-resources:
- what: static/metrics
where: /metrics-ui
welcome-file: restheart-metrics.html
embedded: true
No additional configuration is needed β the UI is built into the restheart-metrics JAR.
Metrics UI
Note: The Metrics UI is available starting from RESTHeart v9.5.0.
Open /metrics-ui in a browser while RESTHeart is running. The dashboard connects to the RESTHeart server and displays real-time metrics.
Connection
- Enter the RESTHeart server URL (e.g.
http://localhost:8080) in the server field - Provide authentication credentials (default:
admin/secret) - Click connect
The dashboard auto-discovers available metric registries by calling GET /metrics.
Dashboard Panels
| Panel | Description |
|---|---|
| KPI Row | Total requests (since startup), request rate (1-min and 5-min rolling), latency percentiles (p50, p95, p99) |
| Request Rate Chart | Time-series line chart of requests/sec per method+status combination |
| Request Count Chart | Bar chart of cumulative request counts per method+status |
| Latency Panels | Distribution bars showing p50, p75, p95, p98, p99, p999 latency in milliseconds |
Features
- Auto-refresh: Configurable interval (5s, 10s, 15s, 30s, 1m). Enabled by default at 30s.
- Filter chips: Click to toggle visibility of specific method+status combinations. Preferences are persisted in
localStorageper host. - Dark theme: Optimized for monitoring screens and low-light environments.
Tutorial
Run RESTHeart with metrics enabled and specify a configuration:
$ docker run --rm -p "8080:8080" -e RHO="/http-listener/host->'0.0.0.0';/mclient/connection-string->'mongodb://host.docker.internal';/ping/uri->'/acme/ping';/requestsMetricsCollector/enabled->true;/jvmMetricsCollector/enabled->true;/requestsMetricsCollector/include->['/{tenant}/*']" softinstigate/restheart
With the given RHO env variable, the configuration is:
ping:
uri: /acme/ping # change the ping service uri for testing purposes
metrics:
enabled: true
uri: /metrics
requestsMetricsCollector:
enabled: true
include:
- /{tenant}/*
exclude:
- /metrics
- /metrics/*
jvmMetricsCollector:
enabled: true
Now, make a few requests to /acme/ping using httpie.
$ http -b :8080/acme/ping
{
"client_ip": "127.0.0.1",
"host": "localhost:8080",
"message": "Greetings from RESTHeart!",
"version": "8.4.0"
}
$ http -b :8080/acme/ping
{
"client_ip": "127.0.0.1",
"host": "localhost:8080",
"message": "Greetings from RESTHeart!",
"version": "8.4.0"
}
$ http -b :8080/acme/ping
{
"client_ip": "127.0.0.1",
"host": "localhost:8080",
"message": "Greetings from RESTHeart!",
"version": "8.4.0"
}
Now we can ask for available metrics:
$ http -b -a admin:secret :8080/metrics
[
"/jvm",
"/{tenant}/ping"
]
Letβs get the metrics for requests matching "/{tenant}/*":
$ http -b -a admin:secret :8080/metrics/{tenant}/\*
(omitting many rows)
http_requests_count{request_method="GET",path_template="/{tenant}/*",response_status_code="200",path_template_param_tenant="acme",} 3.0
The response is in prometheus format. The highlighted row is the metrics http_requests_count with value 3 and the following tags:
request_method="GET"
path_template="/{tenant}/*"
response_status_code="200",
path_template_param_tenant="acme"
Prometheus Endpoint
The /metrics endpoint returns metrics in Prometheus exposition format. You can scrape it with Prometheus or any compatible tool.
List Registries
$ curl -u admin:secret http://localhost:8080/metrics
Returns a JSON array of available registries:
["/requests"]
If jvmMetricsCollector is enabled, youβll also see /jvm.
Get Registry Metrics
$ curl -u admin:secret http://localhost:8080/metrics/requests
Returns Prometheus text format:
# HELP requests_total Total number of HTTP requests
# TYPE requests_total counter
requests_total{response_status_code="200",request_method="GET",path_template="/{db}/{coll}"} 142
# HELP requests_rate_1m Request rate (1-minute moving average)
# TYPE requests_rate_1m gauge
requests_rate_1m{response_status_code="200",request_method="GET",path_template="/{db}/{coll}"} 2.35
Prometheus Scrape Config
Define the following prometheus configuration file prometheus.yml:
global:
scrape_interval: 5s
evaluation_interval: 5s
scrape_configs:
- job_name: 'restheart'
metrics_path: '/metrics/requests'
basic_auth:
username: 'admin'
password: 'secret'
static_configs:
- targets: ['localhost:8080']
# If you also want JVM metrics:
- job_name: 'restheart jvm'
metrics_path: '/metrics/jvm'
basic_auth:
username: 'admin'
password: 'secret'
static_configs:
- targets: ['localhost:8080']
Run prometheus with:
$ docker run --rm --name prometheus -p 9090:9090 -v ./prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus --config.file=/etc/prometheus/prometheus.yml
Prometheus will start scraping restheart metrics. Note that given the default exclude path templates, metrics for prometheus requests are not collected.
Open localhost:9090 with your browser and check the metrics:
Handling Missing Metrics Registries
added in RESTHeart v. 8.9.0
By default, RESTHeart returns HTTP 404 when a metrics registry for a requested path does not exist (i.e., no matching traffic has occurred yet). This behavior is compatible with legacy clients that expect a 404 for non-existent metrics.
However, some monitoring tools (such as Prometheus) expect a 200 OK response from metrics endpoints, even if no metrics are available yet. To support this, you can configure RESTHeart to return HTTP 200 with an empty body for missing registries.
To enable Prometheus compatibility, starting from RESTHeart v 8.9.0 you can add the following to your configuration:
metrics:
enabled: true
uri: /metrics
missing-registry-status-code: 200 # Return 200 OK with empty body for missing registries
Note: If
missing-registry-status-codeis not set, RESTHeart will return 404 by default.
Summary of behaviors:
missing-registry-status-code: 404(default): Returns 404 for missing registries (legacy clients)missing-registry-status-code: 200: Returns 200 OK with empty body (Prometheus-friendly)
Metric Labels
HTTP request metrics include the following labels:
| Label | Description | Example |
|---|---|---|
request_method |
HTTP method | GET, POST, PUT, DELETE |
response_status_code |
HTTP status code | 200, 404, 500 |
path_template |
Matched path template | /{db}/{coll}, /{db}/{coll}/{id} |
Custom Labels
The org.restheart.metrics.Metrics.attachMetricLabels(Request<?> request, List<MetricLabel> labels) method provides the capability to include custom labels in the metrics that are being collected.
For example, the GraphQLService utilizes this method to include the query label in the metrics, which corresponds to the name of the executed GraphQL query.
Plugins can also attach custom labels using the Metrics API:
Metrics.addMetricLabel(request, "tenant", "acme-corp");
These labels will appear in the /metrics output and in the Metrics UI filter panel.
Custom Metrics (RESTHeart v9)
RESTHeart v9 introduces a programmatic API for registering and collecting custom application-specific metrics. This allows you to track business metrics, performance indicators, and domain-specific measurements alongside the built-in HTTP and JVM metrics.
Custom metrics are automatically exposed alongside built-in metrics at /metrics in Prometheus format.
Supported Metric Types
RESTHeart supports four standard Prometheus metric types:
| Type | Description | Use Case |
|---|---|---|
| Counter | Monotonically increasing value (can only go up) | Request counts, error counts, orders processed |
| Gauge | Value that can increase or decrease | Active connections, queue size, temperature |
| Histogram | Samples observations and counts them in configurable buckets | Request durations, response sizes |
| Summary | Similar to histogram, with configurable quantiles | Request latencies, SLA measurements |
Registering Custom Metrics
Custom metrics must be registered during application startup using an Initializer plugin.
Example: Counter Metric
@RegisterPlugin(
name = "customMetricsInit",
description = "Registers custom metrics",
enabledByDefault = true)
public class CustomMetricsInitializer implements Initializer {
@Override
public void init() {
// Register a counter with labels
Metrics.registerCounter(
"custom_requests_total",
"Total number of custom requests",
"endpoint", // Label names
"method"
);
}
}
Example: Gauge Metric
@RegisterPlugin(
name = "cacheMetricsInit",
description = "Registers cache size gauge",
enabledByDefault = true)
public class CacheMetricsInitializer implements Initializer {
private final CacheService cacheService;
@Inject("cacheService")
public CacheMetricsInitializer(CacheService cacheService) {
this.cacheService = cacheService;
}
@Override
public void init() {
// Register a gauge with a supplier function
Metrics.registerGauge(
"cache_size",
"Current cache size",
() -> cacheService.getCacheSize()
);
}
}
Example: Histogram Metric
@Override
public void init() {
// Register a histogram with custom buckets
Metrics.registerHistogram(
"order_value_dollars",
"Distribution of order values in USD",
new double[]{10, 50, 100, 500, 1000, 5000} // Bucket boundaries
);
}
Example: Summary Metric
@Override
public void init() {
// Register a summary with quantiles
Metrics.registerSummary(
"api_latency_seconds",
"API request latency",
0.5, 0.9, 0.99 // 50th, 90th, 99th percentiles
);
}
Updating Custom Metrics
Once registered, you can update metrics from Services or Interceptors during request handling.
Updating Counters
@RegisterPlugin(
name = "orderService",
description = "Order processing service")
public class OrderService implements JsonService {
@Override
public void handle(JsonRequest req, JsonResponse res) {
// Process order...
// Increment counter with labels
Metrics.counter("custom_requests_total")
.labels("orders", req.getMethod().toString())
.inc();
if (orderSuccessful) {
Metrics.counter("orders_processed_total").inc();
}
}
}
Updating Gauges
// Increment gauge
Metrics.gauge("active_connections").inc();
// Decrement gauge
Metrics.gauge("active_connections").dec();
// Set to specific value
Metrics.gauge("queue_size").set(42);
Recording Histogram/Summary Values
@Override
public void handle(JsonRequest req, JsonResponse res) {
long startTime = System.nanoTime();
// Process request...
long duration = System.nanoTime() - startTime;
// Record observation in histogram
Metrics.histogram("request_duration_seconds")
.observe(duration / 1_000_000_000.0);
// Record observation in summary
Metrics.summary("api_latency_seconds")
.observe(duration / 1_000_000_000.0);
}
Complete Example: Business Metrics
This example shows how to track business metrics for an e-commerce application:
@RegisterPlugin(
name = "businessMetricsInit",
description = "Registers business metrics",
enabledByDefault = true)
public class BusinessMetricsInitializer implements Initializer {
@Override
public void init() {
// Order metrics
Metrics.registerCounter(
"orders_total",
"Total number of orders",
"status" // pending, completed, failed
);
Metrics.registerHistogram(
"order_value_dollars",
"Order value distribution",
new double[]{10, 50, 100, 500, 1000}
);
// Cart metrics
Metrics.registerGauge(
"active_carts",
"Number of active shopping carts"
);
// Performance metrics
Metrics.registerSummary(
"checkout_duration_seconds",
"Time to complete checkout",
0.5, 0.95, 0.99
);
}
}
Using the metrics:
@RegisterPlugin(name = "orderService")
public class OrderService implements JsonService {
@Override
public void handle(JsonRequest req, JsonResponse res) {
var order = req.getContent();
double orderValue = order.get("total").asDouble();
long startTime = System.nanoTime();
try {
// Process order...
processOrder(order);
// Track successful order
Metrics.counter("orders_total")
.labels("completed")
.inc();
Metrics.histogram("order_value_dollars")
.observe(orderValue);
} catch (Exception e) {
// Track failed order
Metrics.counter("orders_total")
.labels("failed")
.inc();
} finally {
// Track checkout duration
long duration = System.nanoTime() - startTime;
Metrics.summary("checkout_duration_seconds")
.observe(duration / 1_000_000_000.0);
}
}
}
Querying Custom Metrics
Custom metrics are exposed at the same /metrics endpoint as built-in metrics:
# Get all metrics
$ http -a admin:secret :8080/metrics
# Custom metrics appear in the list
[
"/jvm",
"/{tenant}/ping",
"/custom" # Your custom metrics
]
# Query custom metrics
$ http -a admin:secret :8080/metrics/custom
# TYPE orders_total counter
orders_total{status="completed",} 1523.0
orders_total{status="failed",} 12.0
# TYPE order_value_dollars histogram
order_value_dollars_bucket{le="10.0",} 45.0
order_value_dollars_bucket{le="50.0",} 234.0
order_value_dollars_bucket{le="100.0",} 456.0
order_value_dollars_sum 45678.90
order_value_dollars_count 789.0
# TYPE active_carts gauge
active_carts 42.0
# TYPE checkout_duration_seconds summary
checkout_duration_seconds{quantile="0.5",} 0.234
checkout_duration_seconds{quantile="0.95",} 1.456
checkout_duration_seconds{quantile="0.99",} 2.789
checkout_duration_seconds_sum 1234.567
checkout_duration_seconds_count 789.0
Disabling Metrics
To disable the metrics system entirely:
metrics:
enabled: false
requestsMetricsCollector:
enabled: false
jvmMetricsCollector:
enabled: false
Or via the RHO environment variable:
RHO='/metrics/enabled->false;/requestsMetricsCollector/enabled->false'
See Also
- Static Resources β How RESTHeart serves static files
- Custom Metrics Example β Full working plugin with custom metrics
- Prometheus Documentation β Setting up Prometheus scraping