Aster Web Data / Operate

Health and outages

What the health endpoints report, what clients see while a database is down, and how the server reconnects.

When one database goes down, a service built on several keeps serving the data on the others, and the sets on the lost database answer 503 with a time to retry. The server reconnects on its own, so nothing needs a restart when the database comes back. Your load balancer and orchestrator see this through two health endpoints.

Liveness and readiness

Two endpoints report the server's state:

• /healthz is liveness. It answers 200 with the process uptime while the server runs. It does not check a database, so a database outage does not restart the process.
• /readyz is readiness. It answers 200 when the server can take traffic, and 503 with not_ready otherwise, including while it starts and while it shuts down. It lists each service with its root and state, the databases, and the checks it made:

HTTP 200
content-type: application/json

{
  "checks": {
    "definitions_loaded": true
  },
  "databases": {
    "down": 0,
    "up": 1
  },
  "services": {
    "shop": {
      "root": "/shop",
      "status": "ready"
    }
  },
  "status": "ready"
}

A database that is down does not fail readiness: the instance stays in rotation and serves the sets on its other databases. A probe never queries a database itself. A background task pings every database that is up on an interval, health.ping_interval_seconds (default 5), and a probe reads the last result.

When health.port is set, /healthz, /readyz and /metrics are served only on that listener, which binds to 127.0.0.1 unless health.host says otherwise. Without it, they are served on the OData port. On the OData port, /readyz reports the databases as counts, and on the health listener it names them.

A database goes down

By default the server starts even when a database cannot be reached or fails its start-up check. While a database is unavailable:

• Only the sets stored on it answer 503 with the code ConnectionLost and a Retry-After header, the number of seconds until the next attempt.
• Other sets, other databases and other services serve normally.
• $metadata is served as usual.
• /readyz still answers 200, reports the affected service as degraded, and counts the database as down (on the health listener, it names it).

A request that reads from the unavailable database and from another one is refused as a whole with 503. In a $batch that is not atomic, only the operations that touched the unavailable database report the 503.

A request that meets a connection the database has dropped fails with 503 ConnectionLost and Retry-After: 1. A failed background ping marks the database down and starts its reconnection.

The server also answers 503 in two other cases:

• PoolExhausted, with Retry-After: every connection in the database's pool is in use. Nothing ran.
• Timeout: a database statement ran past its statement timeout. See Limits.

The Error codes reference has the full table.

To stop the process instead, mark the database as required with databases.<name>.required: true, or set startup.require_databases: true. A required database that cannot be opened or fails its check stops start-up, so an orchestrator sees a failed start rather than a running server that cannot serve its data. The environment variable ASTER_WEBDATA_REQUIRE_DATABASES=1 overrides the file.

An invalid configuration always stops start-up, whether or not the database is required. This covers an unknown key, a data source that names a database that is not configured, and a read-only violation.

Reconnection

The server retries a database it cannot reach with exponential backoff and jitter. The first retry comes after retry.initial (default 1 second), the wait doubles on each failure, and it stops growing at retry.max (default 60 seconds). There is no attempt limit, so the server keeps trying until the database returns.

The server logs the outage so you can see what failed:

• the first failure, with its cause class: refused, dns, auth, tls, timeout, schema, credentials or other;
• each change of cause;
• a line every ten failed attempts;
• the recovery, with how long the database was down.

Two metrics show the same state: aster_webdata_database_up, which is 1 while the database serves, and aster_webdata_database_connect_attempts_total, which counts the attempts that failed. See Metrics and logs.

The connection URL is read again on every attempt, so a rotated secret takes effect at the next attempt. See Secrets.

A database that is up is checked by a background ping, and that check is not repeated after start. If a table is dropped while the server runs, the next statement that uses it fails.

Configure and deploy covers readiness during a shutdown, and Observability the rest of the telemetry.