aster.Web Data / Docs Aster UI ↗

Aster Web Data / Reference

Metrics and logs

Every metric the server exports, with its labels, and the fields and targets of its logs.

Metrics

The server exports its metrics through OpenTelemetry when an exporter is configured. It also serves them as Prometheus text at GET /metrics. The /metrics endpoint is on the OData port, or on the health port when health.port is set. It needs no authentication, so protect it by where it is bound or at the proxy.

Instruments

Names are the OpenTelemetry names. Attribute values come from fixed sets, or are _OTHER, so a caller cannot create new series from the request path.

Name Kind Unit Attributes Meaning
aster_webdata.http.server.requests counter aster_webdata.service, http.request.method, http.route, http.response.status_class, aster_webdata.entity_set OData HTTP requests completed.
aster_webdata.http.server.duration histogram seconds aster_webdata.service, http.request.method, http.route, http.response.status_class Time from the request to the first byte of the response.
aster_webdata.http.server.in_flight up-down counter aster_webdata.service Requests being served now.
aster_webdata.pushdown.ratio histogram aster_webdata.service The share of a read's plan nodes that ran on a database.
aster_webdata.pushdown.residual_nodes counter aster_webdata.service, aster_webdata.reason Plan nodes that ran in the engine, or were split, by reason.
aster_webdata.database.up gauge database 1 while the database is serving and answers its ping, else 0.
aster_webdata.database.connect.attempts counter database Attempts to open, check or ping the database that failed.
aster_webdata.unbounded_residual counter set, option Residual $skip and $count operations that run over every row of an unbounded fetch. See limits.unbounded_residual.
aster_webdata.config.reloads counter outcome (ok or failed) Configuration reloads on SIGHUP. Both series exist from start-up.
aster_webdata.config.last_reload_success gauge seconds none Unix time of the last successful reload. Absent until one succeeds.

Operations and virtual data sources

Operations, hooks and virtual data sources, which run your own code, add these instruments. The handler label is the handler's name from the definitions; past 256 names, further ones are counted as _OTHER.

Name Kind Unit Attributes Meaning
aster_webdata.handler.runs counter aster_webdata.handler, aster_webdata.handler.kind, aster_webdata.outcome Handler runs.
aster_webdata.handler.duration histogram seconds the same Time a run took, host work included.
aster_webdata.handler.host_calls histogram aster_webdata.handler, aster_webdata.handler.kind Host calls (callbacks and log calls) in one run.
aster_webdata.handler.log.records counter level Records handlers logged.
aster_webdata.handler.log.dropped counter reason (cap, fields or key) Log records or fields the log sink dropped.

aster_webdata.handler.kind is operation, hook or virtual_source. aster_webdata.outcome is ok, error, timeout, memory or refused. The duration buckets, in seconds, are 0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5 and 5.

Outbound HTTP calls

A script that calls an API through an HTTP connection adds these instruments. One requests point is recorded per call, retries included.

Name Kind Unit Attributes Meaning
aster_webdata.http.requests counter aster_webdata.http.connection, aster_webdata.http.status_class Outbound calls, by how each ended.
aster_webdata.http.duration histogram seconds the same Time a call took, retries and waits included.
aster_webdata.http.retries counter aster_webdata.http.connection Attempts after the first.
aster_webdata.http.rate_limit.waits counter aster_webdata.http.connection Calls that waited for the connection's rate limit.
aster_webdata.http.rate_limit.wait_duration histogram seconds aster_webdata.http.connection Time calls waited for the rate limit.

aster_webdata.http.connection is the name of a connection in the configuration. aster_webdata.http.status_class is 1xx, 2xx, 3xx, 4xx or 5xx, or error when the call got no response: the server was unreachable, the call timed out, the response was too large, or the request was cancelled. Both sets of values are closed, so the labels stay bounded. The buckets of both histograms, in seconds, are 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10 and 30.

Each call also records a span named aster_webdata.http, a child of the handler's span. It carries connection, method, path as written with its {name} placeholders, status, retries, duration_ms and error. It never carries a query value, a header or a body.

On /metrics the names follow the usual rule: aster_webdata_http_requests_total, aster_webdata_http_duration_seconds, aster_webdata_http_retries_total, aster_webdata_http_rate_limit_waits_total and aster_webdata_http_rate_limit_wait_duration_seconds.

Attribute values

Attribute Values
aster_webdata.service The name of a configured service, or _OTHER for a request below no service root.
http.request.method GET, POST, PATCH, PUT, DELETE, HEAD, OPTIONS, or _OTHER.
http.route service_root, metadata, batch, read, create, update, delete, operation, health (/healthz, /readyz and /metrics on the OData port), or other.
http.response.status_class 2xx, 3xx, 4xx, 5xx, or _OTHER.
aster_webdata.entity_set The name of a set in the loaded definitions, or _OTHER. Only on aster_webdata.http.server.requests.
aster_webdata.reason A planner reason code (below).
database The name of a configured database.

Reason codes for aster_webdata.pushdown.residual_nodes:

Code Meaning
missing:<capability> The database does not declare the capability the node needs. The capability name follows the colon.
input_residual An input runs in the engine.
subplan_residual A sub-plan runs in the engine.
mixed_backends Inputs are on different databases.
no_backend No database is below the node.
residual_source The source is owned by the engine.
server_operation The node is a server operation.
nested_output The output has nested rows, so the plan is cut below it.
correlated_path The node is on a correlation path of the engine, so the plan is cut below it.
keyset_order A sorted input of an engine keyset page.
expand_counts The node carries expand counts that a node above would lose.
unresolved An unresolved collation or enum literal.

Past 64 distinct missing: capability names, further names are counted as missing:_OTHER.

Plan shapes

Reads are also counted by their plan shape: the plan with its literal values removed. Requests that differ only in values share a shape. No value is recorded. These metrics appear only on /metrics, not in the OpenTelemetry export.

Labels Meaning
service, set, shape The service, the entity set, and the first 12 hex digits of the shape's ID.
service, set, shape, phase phase is planning, database or engine.
_other as shape Shapes past the cap of observe.max_plan_shapes, counted together.

At most observe.max_plan_shapes shapes are tracked for each service. The default is 1,000. The shape's text is not exported. Use GET /shapes on the separate health listener (health.port) for the shapes with the most total time, with their texts: 20 by default, or ?n= up to 1,000.

Names on /metrics

The Prometheus text uses the OpenTelemetry names with dots changed to underscores. Counters take a _total suffix, and a histogram in seconds takes _seconds. Plan-shape metrics are written by the server.

Name on /metrics Type Source
aster_webdata_http_server_requests_total counter aster_webdata.http.server.requests
aster_webdata_http_server_duration_seconds histogram aster_webdata.http.server.duration
aster_webdata_http_server_in_flight gauge aster_webdata.http.server.in_flight
aster_webdata_pushdown_ratio histogram aster_webdata.pushdown.ratio
aster_webdata_pushdown_residual_nodes_total counter aster_webdata.pushdown.residual_nodes
aster_webdata_database_up gauge aster_webdata.database.up
aster_webdata_database_connect_attempts_total counter aster_webdata.database.connect.attempts
aster_webdata_unbounded_residual_total counter aster_webdata.unbounded_residual
aster_webdata_config_reloads_total counter aster_webdata.config.reloads
aster_webdata_config_last_reload_success_seconds gauge aster_webdata.config.last_reload_success
aster_webdata_plan_shape_requests_total counter Plan shapes
aster_webdata_plan_shape_rows_returned_total counter Plan shapes
aster_webdata_plan_shape_rows_read_total counter Plan shapes
aster_webdata_plan_shape_duration_seconds histogram, phase label Plan shapes
aster_webdata_plan_shapes_tracked gauge Plan shapes
aster_webdata_plan_shapes_max gauge Plan shapes, the cap

Request-duration buckets, in seconds, are 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5 and 10. Plan-shape duration buckets, in seconds, are 0.0005, 0.002, 0.005, 0.01, 0.025, 0.1, 0.25, 1 and 5.

Logs

The server writes logs to standard error, as one JSON object per line by default. Set log.format: text for text. Log fields and targets below are the ones the server writes.

Request log

One line per request, with target aster_webdata::request and the message request.

Field Meaning
request_id The client's X-Request-ID, or one the server generated. The server returns it in the response.
trace_id, span_id Set when the request is part of a trace.
method The HTTP method.
path The request path.
status The HTTP status.
user The caller's user ID. Empty for an anonymous caller.
streamed true when the response body is streamed.
ms_to_first_byte Milliseconds from the request to the first byte of the response.

Start-up and shutdown

Event Fields
service loaded (one per service) service, service_root, definition_files, entity_sets, server_rules, rules_not_enforced, metadata_diagnostics, seeded, locales
server built listen, services, health_listener, databases (name, kind and whether it is up), authenticator, seeded
A database is down at start-up (warning) service, service_root, databases_down. The service's sets on those databases answer 503 until they are reached.
drain summary in_flight_at_start, completed, cut, elapsed_ms
stopped none

The drain summary reports the requests in flight when shutdown began, how many finished, and how many were cut when the grace period ended. See Configuration for shutdown.grace_seconds.

Warnings

The server writes a warning in these cases, among others:

Targets and filters

The log targets are aster_webdata::request, aster_webdata::error, aster_webdata::unbounded_residual, aster_webdata::engine and aster_webdata::batch. Spans carry the names aster_webdata.execute.fragment (one per fragment, on a database) and aster_webdata.encode (the response body). Each request opens the root span http.server.request.

The filter is ASTER_LOG if set, else RUST_LOG, else log.level (one level for every target). Without any of them, the default directives are warn, with Aster's targets at info. See Configuration.