Aster Web Data / Connect your data

Virtual data sources

Serve a set from a REST API or your own code, and query it like a table.

Some of the data an application needs is not in your database. The issues of a project live in a tracker's REST API, exchange rates come from a feed, and an old service answers only its own calls. A virtual data source puts that data in the same API as your tables. You write a short Python or TypeScript script that returns rows, and clients filter, sort, page and expand the set with the same requests they send to any other set. It appears in $metadata and the OpenAPI description like a table.

Aster does the protocol work around the script. It passes the script the part of the query the script says it can handle, checks every row the script returns against the DataSource's fields, and filters, sorts and pages whatever the script left undone. For an API, it also owns the connection: the credential, the timeout, the retries and the rate limit.

A virtual set over a REST API

The example serves the issues of an issue tracker as a set called Issue. The server configuration declares the connection to the tracker once, under http::

http:
  tracker:
    base_url: http://127.0.0.1:8090/api/v1
    # The token is read from the environment; scripts never see it.
    auth:
      bearer_env: TRACKER_TOKEN
    timeout_ms: 10000
    retries: 3
    rate_limit:
      per_second: 10
      burst: 20
    headers: {Accept: application/json}

The DataSource sets serverType: virtual and names the script. The tracker's API can filter by state, so the pushdown: block passes a state eq '...' filter to the script, and Aster does every other filter, the sort and the paging:

dataSource:
  ID: Issue
  serverType: virtual
  script: issues.py
  pushdown:
    filter: {fields: [state], ops: [eq]}
    # The API compares text exactly, as aster's `ordinal` does.
    collation: ordinal
  fields:
    - {name: id, type: integer, primaryKey: true}
    - {name: title, type: text, length: 200}
    - {name: state, type: text, length: 20}

The script's read function pages through the API. The tracker answers each page with an X-Next-Page header, and the script follows it until the header is empty:

def state_of(query):
    # The one filter the pushdown passes: `state eq '...'`, alone or in an `and`.
    f = query.get('filter')
    if f is None:
        return None
    for p in f['args'] if f['op'] == 'and' else [f]:
        if p['op'] == 'eq' and p.get('field') == 'state':
            return p['value']
    return None


def row(issue):
    return {'id': issue['id'], 'title': issue['title'], 'state': issue['state']}


def read(query):
    rows = []
    page = 1
    while page:
        r = http.get('/issues', connection='tracker',
                     params={'state': state_of(query), 'page': page, 'per_page': 50})
        if not r.ok:
            fail('TrackerError', f'the tracker answered {r.status_code}')
        rows.extend([row(issue) for issue in r.json()])
        next_page = r.headers.get('X-Next-Page')
        page = int(next_page) if next_page else 0
    return rows

The same script in TypeScript uses web fetch:

interface ApiIssue {
  id: number;
  title: string;
  state: string;
}

// The one filter the pushdown passes: `state eq '...'`, alone or in an `and`.
function stateOf(query: Query): string | null {
  const f = query.filter;
  const preds = f === undefined ? [] : f.op === 'and' ? f.args : [f];
  for (const p of preds) {
    if (p.op === 'eq' && p.field === 'state') return p.value as string;
  }
  return null;
}

const row = ({ id, title, state }: ApiIssue) => ({ id, title, state });

export async function read(query: Query) {
  const rows: ApiIssue[] = [];
  let page = 1;
  while (page) {
    const r = await fetch('/issues', { connection: 'tracker', query: { state: stateOf(query), page, per_page: 50 } });
    if (!r.ok) aster.fail('TrackerError', `the tracker answered ${r.status}`);
    const issues: ApiIssue[] = await r.json();
    rows.push(...issues.map(row));
    const next = r.headers.get('X-Next-Page');
    page = next ? Number(next) : 0;
  }
  return rows;
}

Clients query the set like any other:

GET /odata/Issue?$count=true
GET /odata/Issue?$filter=state eq 'closed' and contains(title,'a')&$orderby=title

In the second request, state eq 'closed' reaches the API as ?state=closed. Aster applies contains(title,'a') and the order to the rows the script returns. A null query value, such as the state when the request has no state filter, is left out of the URL.

The call

Each language uses the HTTP interface its developers already know. Python has the shape of requests, and TypeScript has web fetch with a few options of Aster's own:

The URL is relative to the connection's base_url. An absolute URL below a configured base_url, such as a next-page link the API returned, needs no connection. Each placeholder value is encoded as one path segment:

r = http.get('/projects/{id}/pipelines', connection='gitlab', path_params={'id': 42}, params={'status': 'failed'})

The answer comes back for every status, so a 404 or a 500 that persists after the retries is an answer to inspect. In Python it has status_code, ok (below 400), headers, text, json() and raise_for_status(). In TypeScript it is a Response, with status, ok (2xx), statusText, headers.get(), text() and json(). Header names match in any case, and the body is parsed as JSON whatever its content type.

A call that gets no response raises an AsterError in Python, and rejects with an aster.AsterError in TypeScript. Its code is one of HttpUnknownConnection, HttpBadRequest, HttpConnection, HttpTimeout and HttpResponseTooLarge. In Python, read the code with aster_error(e)['code'], and in TypeScript with e.code. An error the script does not catch fails the request with 500, and the detail goes to the log.

What the engine does around each call

• Credentials. The configuration names where the credential is (bearer_env, bearer_file, basic or an API-key header) and never holds it. The script cannot read the value or set the credential's header. An absolute URL gets the credential only when it is below base_url, and a call to any other host gets none.
• Retries. A connection error, 429, 502, 503 or 504 is retried up to retries times for GET, HEAD, PUT, DELETE and OPTIONS. A POST or PATCH is retried only when the call asks, so a retry cannot create a second record unasked. The wait doubles from 100 ms up to 10 s with jitter, or follows the API's Retry-After. After the last attempt the script gets the last answer.
• Rate limit. rate_limit is a token bucket shared by every call on the connection. A call waits for a token, and the wait is counted in aster_webdata.http.rate_limit.waits.
• Timeouts. timeout_ms bounds a call, retries and waits included, and a call can set its own. A response over max_response_bytes ends the call with HttpResponseTooLarge.
• The request's deadline. A call never outlives its request. One that would pass the request's deadline is cut off and ends the run with a timeout, even if the script catches the error, and a client that goes away aborts the call.
• Traces and metrics. Each call is a span and counts in the outbound metrics by connection. The span never holds a query value, a header or a body. See Metrics and logs.

An API that returns an unexpected shape fails the request, because Aster checks each row against the DataSource's fields before it reaches the client. The connection's keys and defaults are in Configuration, and Operations in code shows an action that posts to the same API.

Rows from your own code

The smallest virtual set is a read function that returns every row. This DataSource serves exchange rates from a table in the script:

dataSource:
  ID: Rate
  serverType: virtual
  script: rates.py
  fields:
    - {name: pair, type: text, length: 7, primaryKey: true}
    - {name: rate, type: decimal, precision: 18, scale: 6}
    - {name: region, type: text, length: 8}

The script declares no pushdown, so read ignores its query and Aster filters, sorts and pages the rows. The answer is correct, and every request reads every row:

RATES = [
    ('AUDUSD', '0.65', 'NA'),
    ('EURGBP', '0.86', 'EU'),
    ('EURUSD', '1.08', 'EU'),
    ('GBPUSD', '1.27', 'EU'),
    ('USDJPY', '149.5', 'NA'),
]


def read(query):
    log.info('serving every rate', extra={'rows': len(RATES)})
    return [{'pair': pair, 'rate': rate, 'region': region} for pair, rate, region in RATES]

const RATES: [string, string, string][] = [
  ['AUDUSD', '0.65', 'NA'],
  ['EURGBP', '0.86', 'EU'],
  ['EURUSD', '1.08', 'EU'],
  ['GBPUSD', '1.27', 'EU'],
  ['USDJPY', '149.5', 'NA'],
];

export function read(query: Query) {
  aster.log.info('serving every rate', { rows: RATES.length });
  return RATES.map(([pair, rate, region]) => ({ pair, rate, region }));
}

The file's extension picks the language: .py runs as Python and .ts as TypeScript, and the path is relative to the DataSource file. The server strips TypeScript's types without checking them. The declarations of everything a script can call are in aster.d.ts; put it beside your scripts and run tsc --noEmit in your editor or in CI. A function may be async.

To give the function another name, or set limits for this script, use the block form. functions maps the role read to a function in the script:

  script:
    file: rates_enhanced.py
    functions: {read: read}
    limits: {memoryMiB: 16, runMs: 500, hostCalls: 50}

What a script can use

A script cannot open files or import third-party modules. It reaches outside its own code through the names Aster provides:

print and console.log also write to the server's log. Operations in code describes the db calls.

A script is code you write and deploy beside the DataSource file. Aster runs it within the limits below, and it is not a sandbox for code you do not trust.

What read receives

read receives one query object. It always has order, a list of {field, desc}, and skip. It has top when the request asked for a page, filter when the request had a filter, and select when the source declared it honours the field list.

A filter is a tree. A leaf has an op (eq, ne, lt, le, gt, ge, in or isnull), a field, and a value, or values for in. An isnull leaf has negated. An and or or node has args.

Decimals reach the script as text and go back as text, so no value passes through a binary float. Dates and timestamps are ISO text in both directions. An int64 field is a Python int, or a TypeScript bigint.

Declaring what the script honours

By default a script declares nothing, so Aster passes read no filter, order or page and does all of that itself. A pushdown: block moves that work into the script:

  pushdown:
    filter: {fields: [pair, region], ops: [eq, in, ne, isnull]}
    sort: true
    paging: true
    collation: ordinal

Each declaration is a promise the script must keep. The filter Aster passes is a superset of the request's: the script may return rows the request's filter would exclude, because Aster filters them again, but it must never drop a row the filter admits. skip and top are passed only when the whole filter, and any sort, was passed. A script that claims paging but cuts the page before it filters gets a 500, because Aster checks the page it receives.

This read honours the filter, the sort and the page. The matches helper that evaluates a filter tree sits above it in the same file:

def read(query):
    rows = [{'pair': pair, 'rate': rate, 'region': region} for pair, rate, region in RATES]
    f = query.get('filter')
    if f is not None:
        rows = [r for r in rows if matches(f, r)]
    # Stable sorts, last key first.
    for k in reversed(query['order']):
        rows.sort(key=lambda r: r[k['field']], reverse=k['desc'])
    # Paging is only passed when the whole filter and sort were.
    skip, top = query['skip'], query.get('top')
    return rows[skip:skip + top] if top is not None else rows[skip:]

With that pushdown, GET /odata/Rate?$filter=rate gt 1&$orderby=pair desc&$top=2 returns USDJPY and then GBPUSD, as it does from the minimal script.

Aster does the work a script does not handle in memory, and the rows it may hold at once are bounded by the residual_rows limit, so a request too broad to run is refused. See Limits. A set on the far side of an $expand from a SQL set gets key batches only when it declares key_batches. Otherwise it is read whole, within the same limit.

Writes

A virtual set takes writes only for the functions it names. Map the roles add, update and remove in functions: for the writes you want. A write with no function is refused with 405.

dataSource:
  ID: Rate
  serverType: virtual
  script:
    file: rates.py
    functions: {read: read, add: add, update: update, remove: remove}
  fields:
    - {name: pair, type: text, length: 7, primaryKey: true}
    - {name: rate, type: decimal, precision: 18, scale: 6}

add(values) receives the new record and returns the stored row. update(key, values) receives the key and only the fields the request changed, and returns the stored row. remove(key) receives the key.

This script makes Rate a front for a stored set, RateStore. Each function goes through db as the caller, so the stored set's access rules apply to every request:

def to_rate(row):
    return {'pair': row['pair'], 'rate': row['rate']}


def read(query):
    return [to_rate(r) for r in db.fetch('RateStore')]


def add(values):
    return to_rate(db.insert('RateStore', values))


def update(key, values):
    return to_rate(db.update('RateStore', key, values))


def remove(key):
    db.delete('RateStore', key)

A create answers 201 with the stored row. A PATCH or DELETE takes an If-Match header with the ETag of the row read returned (a virtual set declares its token with etag: {columns: [...]}), and a stale one is refused with 412 before the script runs. In a $batch, an atomic group whose only write is to a virtual set runs that write on its own. A group that makes more than one write and writes a virtual set is refused with 501, and nothing in it runs.

Concurrent writes

A virtual set has no transactions, so Aster cannot lock a row while a write decides over it. A rule that reads the stored record, such as a field that is read-only for some records, decides over the row as read returned it, and another writer can change that row before the write reaches the script. The set's concurrency says what happens then:

• lastWriteWins, the default: the write goes ahead as the system behind the script takes it, and the last write wins.
• guarded: every update and delete sends the script the values the decision was made over, and the script writes only if the row still holds them. A row that changed is refused with 412, as a stale If-Match is, and nothing is written.

dataSource:
  ID: Ticket
  serverType: virtual
  concurrency: guarded
  etag: {columns: [version]}
  script:
    file: tickets.py
    functions: {read: read, update: update, updateIf: update_if, remove: remove, removeIf: remove_if}

A guarded set needs conditional forms of its writes: updateIf(key, values, expected) and removeIf(key, expected), which check the values in the same step as the write, for example as a conditional request to the API. expected holds the fields the decision read: the ETag's columns when the set has a token, else every stored field, with null for no value. When the row no longer holds them, raise PreconditionFailed and change nothing:

def update_if(key, values, expected):
    # The API applies the change only if the ticket still has the version the decision read.
    r = http.patch('/tickets/{id}', connection='tracker', path_params={'id': key['id']}, json=values,
                   headers={'If-Match': str(expected['version'])})
    if r.status_code == 412:
        raise PreconditionFailed('the ticket changed')
    r.raise_for_status()
    return r.json()

guarded on a set whose script lacks the conditional forms stops the server at start-up, naming the set and what to add. aster-webdata-server check warns about each virtual set that has no concurrency and whose writes decide over the stored record, naming the rules that read it. Set lastWriteWins to say the last write is meant to win. The rules for If-Match and $batch are on Write data.

Errors and limits

A script ends a call with an error by raising one of the error classes. Aster answers with the matching status and returns the code, target and message the script gives:

Each run has limits. Set them under limits: on the script. The server's scripts: ceilings: caps every script, and a key the script's language does not use is a start-up error.

Each run is counted in aster_webdata.handler.runs, by handler, kind and outcome, and timed in aster_webdata.handler.duration. Log records carry the request's ID. See Metrics and logs.

Next

• Operations in code adds actions and functions to a set, in the same languages.
• Model your data covers the fields, keys and relations a DataSource declares.