Aster Web Data / Operate

Observability

How to see what each request did, in your own tracing, metrics and logging tools, and why part of a request ran outside the database.

When a screen is slow, you want to know whether the time went to the database, to the engine or to the network, and which kind of query caused it. Aster Web Data reports what each request did: how long it took, which database ran which part of it, how many rows it read, and whether each database is reachable. Traces, metrics and logs go to the tools you already run, through OpenTelemetry or a Prometheus endpoint. The Metrics and logs reference lists every name with its labels.

Traces

Each request starts a server span named http.server.request. A request that carries a W3C traceparent header becomes a child of the caller's trace, and a request without one starts a new trace. Under that span, the engine records one span per query fragment and backend, named aster_webdata.execute.fragment, and one for writing the response, named aster_webdata.encode.

The request log line carries the trace_id and span_id, so a log entry links to its trace.

Spans leave the process only when a trace exporter is on. Turn one on with OTEL_TRACES_EXPORTER=otlp and OTEL_EXPORTER_OTLP_ENDPOINT, or with observability.traces_exporter and observability.otlp_endpoint in the configuration file. With no exporter, the server opens no telemetry connection. OTEL_SDK_DISABLED=true turns the whole telemetry pipeline off.

The span filter is the log filter. Setting ASTER_LOG=warn also stops these spans being exported.

Metrics

The server publishes metrics at GET /metrics, in Prometheus text format. The endpoint is on the OData port, or on the health listener when health.port is set. It needs no authentication, so bind it to the health listener or protect it at the proxy.

The OpenTelemetry names use dots. The Prometheus names replace each dot with an underscore, add _total to counters and use _seconds for durations, so aster_webdata.database.up is aster_webdata_database_up.

The ones to watch first:

aster_webdata_http_server_requests_total and aster_webdata_http_server_duration_seconds: requests and their time to the first byte, by service, route and status class.
aster_webdata_database_up: 1 while a database serves and answers its ping, else 0. Alert on it.
aster_webdata_pushdown_ratio: the share of each read's plan that ran in a database. A ratio of 1 means all of it did. Below 1, aster_webdata_pushdown_residual_nodes_total counts why the rest ran in the engine.
aster_webdata_unbounded_residual_total: a $skip or $count that ran in memory over every row of a fetch. See Reads that would run in memory.
aster_webdata_config_reloads_total: reloads on SIGHUP, by outcome, so a failed reload is visible.

Operations and virtual data sources, which run your own code, have metrics of their own, by handler and outcome, and so do outbound HTTP calls. Metrics and logs lists them all.

A scrape of a service that has served a few requests returns this, including the plan shape metrics:

GET /metrics

As a URL

GET /metrics

HTTP 200
content-type: text/plain; version=0.0.4; charset=utf-8

# HELP aster_webdata_config_reloads_total Configuration reloads on SIGHUP, by outcome
# TYPE aster_webdata_config_reloads_total counter
aster_webdata_config_reloads_total{outcome="failed",otel_scope_name="aster-webdata"} 0
aster_webdata_config_reloads_total{outcome="ok",otel_scope_name="aster-webdata"} 0
# HELP aster_webdata_database_connect_attempts_total Attempts to open, check or ping the database that failed
# TYPE aster_webdata_database_connect_attempts_total counter
aster_webdata_database_connect_attempts_total{database="shop",otel_scope_name="aster-webdata"} 0
# HELP aster_webdata_database_up 1 while the database is serving and answers its ping, else 0
# TYPE aster_webdata_database_up gauge
aster_webdata_database_up{database="shop",otel_scope_name="aster-webdata"} 1
# HELP aster_webdata_http_server_duration_seconds Time from the request to the first byte of the response
# TYPE aster_webdata_http_server_duration_seconds histogram
aster_webdata_http_server_duration_seconds_sum{aster_webdata_service="_OTHER",http_request_method="GET",http_response_status_class="2xx",http_route="health",otel_scope_name="aster-webdata"} 0.000076791
aster_webdata_http_server_duration_seconds_count{aster_webdata_service="_OTHER",http_request_method="GET",http_response_status_class="2xx",http_route="health",otel_scope_name="aster-webdata"} 1
aster_webdata_http_server_duration_seconds_sum{aster_webdata_service="shop",http_request_method="GET",http_response_status_class="2xx",http_route="metadata",otel_scope_name="aster-webdata"} 0.000628291
aster_webdata_http_server_duration_seconds_count{aster_webdata_service="shop",http_request_method="GET",http_response_status_class="2xx",http_route="metadata",otel_scope_name="aster-webdata"} 3
aster_webdata_http_server_duration_seconds_sum{aster_webdata_service="shop",http_request_method="GET",http_response_status_class="2xx",http_route="read",otel_scope_name="aster-webdata"} 0.042929209999999995
aster_webdata_http_server_duration_seconds_count{aster_webdata_service="shop",http_request_method="GET",http_response_status_class="2xx",http_route="read",otel_scope_name="aster-webdata"} 22
aster_webdata_http_server_duration_seconds_sum{aster_webdata_service="shop",http_request_method="GET",http_response_status_class="2xx",http_route="service_root",otel_scope_name="aster-webdata"} 0.000087292
aster_webdata_http_server_duration_seconds_count{aster_webdata_service="shop",http_request_method="GET",http_response_status_class="2xx",http_route="service_root",otel_scope_name="aster-webdata"} 1
aster_webdata_http_server_duration_seconds_sum{aster_webdata_service="shop",http_request_method="PATCH",http_response_status_class="2xx",http_route="update",otel_scope_name="aster-webdata"} 0.001579125
aster_webdata_http_server_duration_seconds_count{aster_webdata_service="shop",http_request_method="PATCH",http_response_status_class="2xx",http_route="update",otel_scope_name="aster-webdata"} 1
aster_webdata_http_server_duration_seconds_sum{aster_webdata_service="shop",http_request_method="PATCH",http_response_status_class="4xx",http_route="update",otel_scope_name="aster-webdata"} 0.00380325
aster_webdata_http_server_duration_seconds_count{aster_webdata_service="shop",http_request_method="PATCH",http_response_status_class="4xx",http_route="update",otel_scope_name="aster-webdata"} 1
aster_webdata_http_server_duration_seconds_sum{aster_webdata_service="shop",http_request_method="POST",http_response_status_class="2xx",http_route="create",otel_scope_name="aster-webdata"} 0.004364792
aster_webdata_http_server_duration_seconds_count{aster_webdata_service="shop",http_request_method="POST",http_response_status_class="2xx",http_route="create",otel_scope_name="aster-webdata"} 2
aster_webdata_http_server_duration_seconds_sum{aster_webdata_service="shop",http_request_method="POST",http_response_status_class="4xx",http_route="create",otel_scope_name="aster-webdata"} 0.001028041
aster_webdata_http_server_duration_seconds_count{aster_webdata_service="shop",http_request_method="POST",http_response_status_class="4xx",http_route="create",otel_scope_name="aster-webdata"} 1
# HELP aster_webdata_http_server_in_flight Requests being served
# TYPE aster_webdata_http_server_in_flight gauge
aster_webdata_http_server_in_flight{aster_webdata_service="_OTHER",otel_scope_name="aster-webdata"} 1
aster_webdata_http_server_in_flight{aster_webdata_service="shop",otel_scope_name="aster-webdata"} 0
# HELP aster_webdata_http_server_requests_total OData HTTP requests completed
# TYPE aster_webdata_http_server_requests_total counter
aster_webdata_http_server_requests_total{aster_webdata_service="_OTHER",http_request_method="GET",http_response_status_class="2xx",http_route="health",otel_scope_name="aster-webdata"} 1
aster_webdata_http_server_requests_total{aster_webdata_service="shop",http_request_method="GET",http_response_status_class="2xx",http_route="metadata",otel_scope_name="aster-webdata"} 3
aster_webdata_http_server_requests_total{aster_webdata_service="shop",http_request_method="GET",http_response_status_class="2xx",http_route="service_root",otel_scope_name="aster-webdata"} 1
aster_webdata_http_server_requests_total{aster_webdata_entity_set="Customer",aster_webdata_service="shop",http_request_method="GET",http_response_status_class="2xx",http_route="read",otel_scope_name="aster-webdata"} 10
aster_webdata_http_server_requests_total{aster_webdata_entity_set="Customer",aster_webdata_service="shop",http_request_method="PATCH",http_response_status_class="2xx",http_route="update",otel_scope_name="aster-webdata"} 1
aster_webdata_http_server_requests_total{aster_webdata_entity_set="Customer",aster_webdata_service="shop",http_request_method="POST",http_response_status_class="2xx",http_route="create",otel_scope_name="aster-webdata"} 2
aster_webdata_http_server_requests_total{aster_webdata_entity_set="Invoice",aster_webdata_service="shop",http_request_method="GET",http_response_status_class="2xx",http_route="read",otel_scope_name="aster-webdata"} 3
aster_webdata_http_server_requests_total{aster_webdata_entity_set="Invoice",aster_webdata_service="shop",http_request_method="PATCH",http_response_status_class="4xx",http_route="update",otel_scope_name="aster-webdata"} 1
aster_webdata_http_server_requests_total{aster_webdata_entity_set="Invoice",aster_webdata_service="shop",http_request_method="POST",http_response_status_class="4xx",http_route="create",otel_scope_name="aster-webdata"} 1
aster_webdata_http_server_requests_total{aster_webdata_entity_set="Product",aster_webdata_service="shop",http_request_method="GET",http_response_status_class="2xx",http_route="read",otel_scope_name="aster-webdata"} 9
# HELP aster_webdata_pushdown_ratio Share of a read's plan nodes that ran on a backend
# TYPE aster_webdata_pushdown_ratio histogram
aster_webdata_pushdown_ratio_sum{aster_webdata_service="shop",otel_scope_name="aster-webdata"} 22
aster_webdata_pushdown_ratio_count{aster_webdata_service="shop",otel_scope_name="aster-webdata"} 22
# HELP aster_webdata_plan_shapes_tracked Plan shapes tracked separately (the cap is aster_webdata_plan_shapes_max).
# TYPE aster_webdata_plan_shapes_tracked gauge
aster_webdata_plan_shapes_tracked{service="shop"} 21
# TYPE aster_webdata_plan_shapes_max gauge
aster_webdata_plan_shapes_max{service="shop"} 1000
# HELP aster_webdata_plan_shape_requests_total Reads by plan shape.
# TYPE aster_webdata_plan_shape_requests_total counter
aster_webdata_plan_shape_requests_total{service="shop",set="Customer",shape="46cc58218e7c"} 1
aster_webdata_plan_shape_requests_total{service="shop",set="Product",shape="0ba56266ac61"} 1
aster_webdata_plan_shape_requests_total{service="shop",set="Customer",shape="37284415172d"} 1
aster_webdata_plan_shape_requests_total{service="shop",set="Customer",shape="4f6e3bcdb4f7"} 1
aster_webdata_plan_shape_requests_total{service="shop",set="Invoice",shape="dfcc02902c97"} 1
aster_webdata_plan_shape_requests_total{service="shop",set="Product",shape="a83788a87d84"} 1
aster_webdata_plan_shape_requests_total{service="shop",set="Product",shape="fdd841936ba9"} 2
aster_webdata_plan_shape_requests_total{service="shop",set="Customer",shape="3a71c5c0a05f"} 1
aster_webdata_plan_shape_requests_total{service="shop",set="Product",shape="586f19993bac"} 1
aster_webdata_plan_shape_requests_total{service="shop",set="Customer",shape="06321bd1ca0a"} 1
aster_webdata_plan_shape_requests_total{service="shop",set="Product",shape="ccf7c3acbb2a"} 1
aster_webdata_plan_shape_requests_total{service="shop",set="Invoice",shape="e3270c54573c"} 1
aster_webdata_plan_shape_requests_total{service="shop",set="Customer",shape="aaf69aeed341"} 1
aster_webdata_plan_shape_requests_total{service="shop",set="Customer",shape="8684d0813eeb"} 1
aster_webdata_plan_shape_requests_total{service="shop",set="Product",shape="9e1190aba9ac"} 1
aster_webdata_plan_shape_requests_total{service="shop",set="Invoice",shape="9417b943055e"} 1
aster_webdata_plan_shape_requests_total{service="shop",set="Product",shape="cf3fc1dfae60"} 1
aster_webdata_plan_shape_requests_total{service="shop",set="Customer",shape="13a83d3cef14"} 1
aster_webdata_plan_shape_requests_total{service="shop",set="Customer",shape="382e62367c66"} 1
aster_webdata_plan_shape_requests_total{service="shop",set="Product",shape="88f88e8d82bb"} 1
aster_webdata_plan_shape_requests_total{service="shop",set="Invoice",shape="b03dcb576c6f"} 1
# HELP aster_webdata_plan_shape_rows_returned_total Rows returned to the client by plan shape.
# TYPE aster_webdata_plan_shape_rows_returned_total counter
aster_webdata_plan_shape_rows_returned_total{service="shop",set="Customer",shape="46cc58218e7c"} 5
aster_webdata_plan_shape_rows_returned_total{service="shop",set="Product",shape="0ba56266ac61"} 7
aster_webdata_plan_shape_rows_returned_total{service="shop",set="Customer",shape="37284415172d"} 11
aster_webdata_plan_shape_rows_returned_total{service="shop",set="Customer",shape="4f6e3bcdb4f7"} 11
aster_webdata_plan_shape_rows_returned_total{service="shop",set="Invoice",shape="dfcc02902c97"} 0
aster_webdata_plan_shape_rows_returned_total{service="shop",set="Product",shape="a83788a87d84"} 1
aster_webdata_plan_shape_rows_returned_total{service="shop",set="Product",shape="fdd841936ba9"} 9
aster_webdata_plan_shape_rows_returned_total{service="shop",set="Customer",shape="3a71c5c0a05f"} 20
aster_webdata_plan_shape_rows_returned_total{service="shop",set="Product",shape="586f19993bac"} 2
aster_webdata_plan_shape_rows_returned_total{service="shop",set="Customer",shape="06321bd1ca0a"} 5
aster_webdata_plan_shape_rows_returned_total{service="shop",set="Product",shape="ccf7c3acbb2a"} 2
aster_webdata_plan_shape_rows_returned_total{service="shop",set="Invoice",shape="e3270c54573c"} 1
aster_webdata_plan_shape_rows_returned_total{service="shop",set="Customer",shape="aaf69aeed341"} 1
aster_webdata_plan_shape_rows_returned_total{service="shop",set="Customer",shape="8684d0813eeb"} 5
aster_webdata_plan_shape_rows_returned_total{service="shop",set="Product",shape="9e1190aba9ac"} 7
aster_webdata_plan_shape_rows_returned_total{service="shop",set="Invoice",shape="9417b943055e"} 3
aster_webdata_plan_shape_rows_returned_total{service="shop",set="Product",shape="cf3fc1dfae60"} 2
aster_webdata_plan_shape_rows_returned_total{service="shop",set="Customer",shape="13a83d3cef14"} 1
aster_webdata_plan_shape_rows_returned_total{service="shop",set="Customer",shape="382e62367c66"} 29
aster_webdata_plan_shape_rows_returned_total{service="shop",set="Product",shape="88f88e8d82bb"} 7
aster_webdata_plan_shape_rows_returned_total{service="shop",set="Invoice",shape="b03dcb576c6f"} 155
# HELP aster_webdata_plan_shape_rows_read_total Rows read into the engine by plan shape.
# TYPE aster_webdata_plan_shape_rows_read_total counter
aster_webdata_plan_shape_rows_read_total{service="shop",set="Customer",shape="46cc58218e7c"} 5
aster_webdata_plan_shape_rows_read_total{service="shop",set="Product",shape="0ba56266ac61"} 7
aster_webdata_plan_shape_rows_read_total{service="shop",set="Customer",shape="37284415172d"} 11
aster_webdata_plan_shape_rows_read_total{service="shop",set="Customer",shape="4f6e3bcdb4f7"} 11
aster_webdata_plan_shape_rows_read_total{service="shop",set="Invoice",shape="dfcc02902c97"} 0
aster_webdata_plan_shape_rows_read_total{service="shop",set="Product",shape="a83788a87d84"} 1
aster_webdata_plan_shape_rows_read_total{service="shop",set="Product",shape="fdd841936ba9"} 9
aster_webdata_plan_shape_rows_read_total{service="shop",set="Customer",shape="3a71c5c0a05f"} 20
aster_webdata_plan_shape_rows_read_total{service="shop",set="Product",shape="586f19993bac"} 2
aster_webdata_plan_shape_rows_read_total{service="shop",set="Customer",shape="06321bd1ca0a"} 5
aster_webdata_plan_shape_rows_read_total{service="shop",set="Product",shape="ccf7c3acbb2a"} 2
aster_webdata_plan_shape_rows_read_total{service="shop",set="Invoice",shape="e3270c54573c"} 1
aster_webdata_plan_shape_rows_read_total{service="shop",set="Customer",shape="aaf69aeed341"} 1
aster_webdata_plan_shape_rows_read_total{service="shop",set="Customer",shape="8684d0813eeb"} 5
aster_webdata_plan_shape_rows_read_total{service="shop",set="Product",shape="9e1190aba9ac"} 7
aster_webdata_plan_shape_rows_read_total{service="shop",set="Invoice",shape="9417b943055e"} 3
aster_webdata_plan_shape_rows_read_total{service="shop",set="Product",shape="cf3fc1dfae60"} 2
aster_webdata_plan_shape_rows_read_total{service="shop",set="Customer",shape="13a83d3cef14"} 1
aster_webdata_plan_shape_rows_read_total{service="shop",set="Customer",shape="382e62367c66"} 29
aster_webdata_plan_shape_rows_read_total{service="shop",set="Product",shape="88f88e8d82bb"} 7
aster_webdata_plan_shape_rows_read_total{service="shop",set="Invoice",shape="b03dcb576c6f"} 155
# HELP aster_webdata_plan_shape_duration_seconds Read time by plan shape and phase (planning, database, engine).
# TYPE aster_webdata_plan_shape_duration_seconds histogram
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Customer",shape="46cc58218e7c",phase="planning"} 0.001198542
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Customer",shape="46cc58218e7c",phase="planning"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Customer",shape="46cc58218e7c",phase="database"} 0.001643916
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Customer",shape="46cc58218e7c",phase="database"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Customer",shape="46cc58218e7c",phase="engine"} 0.000352125
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Customer",shape="46cc58218e7c",phase="engine"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Product",shape="0ba56266ac61",phase="planning"} 0.000414958
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Product",shape="0ba56266ac61",phase="planning"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Product",shape="0ba56266ac61",phase="database"} 0.000495917
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Product",shape="0ba56266ac61",phase="database"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Product",shape="0ba56266ac61",phase="engine"} 0.000071916
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Product",shape="0ba56266ac61",phase="engine"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Customer",shape="37284415172d",phase="planning"} 0.000413416
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Customer",shape="37284415172d",phase="planning"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Customer",shape="37284415172d",phase="database"} 0.000715334
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Customer",shape="37284415172d",phase="database"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Customer",shape="37284415172d",phase="engine"} 0.000119458
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Customer",shape="37284415172d",phase="engine"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Customer",shape="4f6e3bcdb4f7",phase="planning"} 0.000694833
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Customer",shape="4f6e3bcdb4f7",phase="planning"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Customer",shape="4f6e3bcdb4f7",phase="database"} 0.00124675
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Customer",shape="4f6e3bcdb4f7",phase="database"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Customer",shape="4f6e3bcdb4f7",phase="engine"} 0.000156083
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Customer",shape="4f6e3bcdb4f7",phase="engine"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Invoice",shape="dfcc02902c97",phase="planning"} 0.000654584
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Invoice",shape="dfcc02902c97",phase="planning"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Invoice",shape="dfcc02902c97",phase="database"} 0.000763125
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Invoice",shape="dfcc02902c97",phase="database"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Invoice",shape="dfcc02902c97",phase="engine"} 0.000016958
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Invoice",shape="dfcc02902c97",phase="engine"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Product",shape="a83788a87d84",phase="planning"} 0.000393458
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Product",shape="a83788a87d84",phase="planning"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Product",shape="a83788a87d84",phase="database"} 0.000696833
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Product",shape="a83788a87d84",phase="database"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Product",shape="a83788a87d84",phase="engine"} 0.000034625
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Product",shape="a83788a87d84",phase="engine"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Product",shape="fdd841936ba9",phase="planning"} 0.000960709
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Product",shape="fdd841936ba9",phase="planning"} 2
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Product",shape="fdd841936ba9",phase="database"} 0.001714
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Product",shape="fdd841936ba9",phase="database"} 2
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Product",shape="fdd841936ba9",phase="engine"} 0.000178916
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Product",shape="fdd841936ba9",phase="engine"} 2
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Customer",shape="3a71c5c0a05f",phase="planning"} 0.000789709
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Customer",shape="3a71c5c0a05f",phase="planning"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Customer",shape="3a71c5c0a05f",phase="database"} 0.001497125
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Customer",shape="3a71c5c0a05f",phase="database"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Customer",shape="3a71c5c0a05f",phase="engine"} 0.00059175
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Customer",shape="3a71c5c0a05f",phase="engine"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Product",shape="586f19993bac",phase="planning"} 0.000362125
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Product",shape="586f19993bac",phase="planning"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Product",shape="586f19993bac",phase="database"} 0.000483875
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Product",shape="586f19993bac",phase="database"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Product",shape="586f19993bac",phase="engine"} 0.000039792
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Product",shape="586f19993bac",phase="engine"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Customer",shape="06321bd1ca0a",phase="planning"} 0.001443625
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Customer",shape="06321bd1ca0a",phase="planning"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Customer",shape="06321bd1ca0a",phase="database"} 0.001996292
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Customer",shape="06321bd1ca0a",phase="database"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Customer",shape="06321bd1ca0a",phase="engine"} 0.000491708
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Customer",shape="06321bd1ca0a",phase="engine"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Product",shape="ccf7c3acbb2a",phase="planning"} 0.0003665
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Product",shape="ccf7c3acbb2a",phase="planning"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Product",shape="ccf7c3acbb2a",phase="database"} 0.000518917
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Product",shape="ccf7c3acbb2a",phase="database"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Product",shape="ccf7c3acbb2a",phase="engine"} 0.000039875
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Product",shape="ccf7c3acbb2a",phase="engine"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Invoice",shape="e3270c54573c",phase="planning"} 0.000433167
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Invoice",shape="e3270c54573c",phase="planning"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Invoice",shape="e3270c54573c",phase="database"} 0.001036875
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Invoice",shape="e3270c54573c",phase="database"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Invoice",shape="e3270c54573c",phase="engine"} 0.000051917
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Invoice",shape="e3270c54573c",phase="engine"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Customer",shape="aaf69aeed341",phase="planning"} 0.000677333
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Customer",shape="aaf69aeed341",phase="planning"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Customer",shape="aaf69aeed341",phase="database"} 0.001346667
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Customer",shape="aaf69aeed341",phase="database"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Customer",shape="aaf69aeed341",phase="engine"} 0.000109583
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Customer",shape="aaf69aeed341",phase="engine"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Customer",shape="8684d0813eeb",phase="planning"} 0.00056925
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Customer",shape="8684d0813eeb",phase="planning"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Customer",shape="8684d0813eeb",phase="database"} 0.000868625
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Customer",shape="8684d0813eeb",phase="database"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Customer",shape="8684d0813eeb",phase="engine"} 0.000076708
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Customer",shape="8684d0813eeb",phase="engine"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Product",shape="9e1190aba9ac",phase="planning"} 0.000447833
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Product",shape="9e1190aba9ac",phase="planning"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Product",shape="9e1190aba9ac",phase="database"} 0.00063975
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Product",shape="9e1190aba9ac",phase="database"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Product",shape="9e1190aba9ac",phase="engine"} 0.000084
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Product",shape="9e1190aba9ac",phase="engine"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Invoice",shape="9417b943055e",phase="planning"} 0.001034917
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Invoice",shape="9417b943055e",phase="planning"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Invoice",shape="9417b943055e",phase="database"} 0.000971
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Invoice",shape="9417b943055e",phase="database"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Invoice",shape="9417b943055e",phase="engine"} 0.00004975
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Invoice",shape="9417b943055e",phase="engine"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Product",shape="cf3fc1dfae60",phase="planning"} 0.000418166
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Product",shape="cf3fc1dfae60",phase="planning"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Product",shape="cf3fc1dfae60",phase="database"} 0.000628875
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Product",shape="cf3fc1dfae60",phase="database"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Product",shape="cf3fc1dfae60",phase="engine"} 0.000048417
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Product",shape="cf3fc1dfae60",phase="engine"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Customer",shape="13a83d3cef14",phase="planning"} 0.00050325
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Customer",shape="13a83d3cef14",phase="planning"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Customer",shape="13a83d3cef14",phase="database"} 0.000546625
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Customer",shape="13a83d3cef14",phase="database"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Customer",shape="13a83d3cef14",phase="engine"} 0.000033375
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Customer",shape="13a83d3cef14",phase="engine"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Customer",shape="382e62367c66",phase="planning"} 0.000490416
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Customer",shape="382e62367c66",phase="planning"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Customer",shape="382e62367c66",phase="database"} 0.000803209
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Customer",shape="382e62367c66",phase="database"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Customer",shape="382e62367c66",phase="engine"} 0.000255166
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Customer",shape="382e62367c66",phase="engine"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Product",shape="88f88e8d82bb",phase="planning"} 0.000482917
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Product",shape="88f88e8d82bb",phase="planning"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Product",shape="88f88e8d82bb",phase="database"} 0.0004895
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Product",shape="88f88e8d82bb",phase="database"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Product",shape="88f88e8d82bb",phase="engine"} 0.000078625
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Product",shape="88f88e8d82bb",phase="engine"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Invoice",shape="b03dcb576c6f",phase="planning"} 0.000480583
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Invoice",shape="b03dcb576c6f",phase="planning"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Invoice",shape="b03dcb576c6f",phase="database"} 0.002215375
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Invoice",shape="b03dcb576c6f",phase="database"} 1
aster_webdata_plan_shape_duration_seconds_sum{service="shop",set="Invoice",shape="b03dcb576c6f",phase="engine"} 0.002171709
aster_webdata_plan_shape_duration_seconds_count{service="shop",set="Invoice",shape="b03dcb576c6f",phase="engine"} 1

Logs

Logs are JSON lines on stderr by default. Set log.format to text for people reading a terminal. Each request writes one aster_webdata::request event with its ID, method, path, status, user, whether the body streamed, the time to the first byte, and the trace and span IDs when the request has a trace. The server also logs its start-up summary and, at shutdown, the drain summary.

The log level comes from ASTER_LOG, then RUST_LOG, then log.level. Without any of them the server logs warnings, and information for the server's own targets. Targets and filters lists the log targets you can filter on.

The server logs a warning when a plaintext or unverified database connection is in use, and logs the cause class of each failed database connection attempt. See Health and outages.

Plan shapes

Each read is counted by its plan shape: the plan with its literal values taken out. Requests that differ only in their values share a shape, so you can see which kinds of query your application sends and how each one performs, without recording any value.

For each shape and service, the server counts requests, rows returned and rows read, and keeps a latency histogram for three phases:

planning: parsing, lowering, the policy step and planning.
database: starting the row stream and reading its first row.
engine: the rest, including writing the response.

The metrics are aster_webdata_plan_shape_requests_total, aster_webdata_plan_shape_rows_returned_total, aster_webdata_plan_shape_rows_read_total and aster_webdata_plan_shape_duration_seconds, which has a phase attribute. Each carries the service, the set, and the first 12 hex digits of the shape ID. The shape text is never a label.

At most observe.max_plan_shapes shapes are tracked for each service, 1000 by default. The rest are counted together as _other. Set observe.plan_shapes: off to turn the counting off.

GET /shapes on the health listener lists the top shapes by total time, with their text and histogram buckets. It is served only there, because the text names sets.

The explain header

The x-aster-explain response header says how a request was planned. For each plan node it records whether the node ran in a database or in the engine and, when it stayed in the engine, why. It names sets, databases and capabilities, so the server never sends it unless you turn it on.

Turn it on in the configuration. role limits it to callers whose roles include that one. With no role, any caller can have it, which suits local development.

explain:
  enabled: true
  role: ops

A caller who may see it asks for it on each request, with the request header x-aster-explain-request: 1 or the query option aster.explain=true:

GET /shop/Customer

top	`1`
aster.explain	`true`

As a URL

GET /shop/Customer?$top=1&aster.explain=true

If enabled is off, or the caller lacks the role, the server ignores both and sends the ordinary response. The server removes the query option before it reads the request, so the option changes neither the query nor the @odata.nextLink. A client that follows the next link adds the option again.

The header is also on an error response, where it describes the plan that was refused, and on a $batch response, where it covers every read in the batch.

Reading the value

The value is v1. followed by the compact JSON of the plan, encoded as base64url without padding. It is at most 4096 bytes. A plan that does not fit loses detail, first the nodes and then the fragments, and if it still does not fit the server leaves the header out. Browsers can read the header through CORS, and x-aster-explain-request is an allowed request header.

To read it, drop the v1. prefix, decode the rest as base64url and parse the JSON. The decoded object has these keys:

Key	Holds
`v`	The format version.
`d`	The detail the value carries: `nodes`, `fragments` or `metrics`.
`m`	The summary. `n` is four counts: all nodes, nodes pushed to a database, nodes that ran in the engine and split nodes. `q` is the share of nodes pushed to a database, rounded to four places. `fr` is `true` when the whole read ran in the engine. `why` counts the reasons nodes stayed in the engine, by reason code.
`ops`	One entry per operation: one for a single request, one per read for a `$batch`. `f` lists the fragments with their database, and `n` lists the nodes in tree order with the fragment each belongs to and its reason code.

Here is the decoded header for the request above:

{
  "d": "nodes",
  "m": {
    "fr": false,
    "n": [
      3,
      3,
      0,
      0
    ],
    "q": 1,
    "why": {}
  },
  "ops": [
    {
      "f": [
        [
          0,
          "shop",
          true,
          []
        ]
      ],
      "n": [
        [
          2,
          "Page",
          "p",
          0,
          ""
        ],
        [
          1,
          "Sort",
          "p",
          0,
          ""
        ],
        [
          0,
          "Scan",
          "p",
          0,
          ""
        ]
      ]
    }
  ],
  "v": 1
}

The reason codes are the ones that label aster_webdata_pushdown_residual_nodes_total. See Metrics and logs. The configuration keys are in the reference.