Architecture

Overview

The following picture illustrates the architecture of Adama:

The main components are:

  • Adama itself is a Python application running over an Apache web server, providing the REST endpoints and business logic. (see API).
  • A data store (Redis) collects a registry of adapters.
  • A task queue (RabbitMQ) distributes work to the adapters for each request to the query endpoint.
  • The workers are Docker containers that get created through the register endpoint, and get started and shutdown under Adama command, depending on the load of the query endpoint. The workers are continuously consuming work from the task queue.
  • The user module in the picture is the adapter provided by the developer at registration time (see Adapter API).
  • The user module, data sources, and 3rd party protocol (colored blue in the picture) specify the parts under control of the developer role (see Developer Role Quickstart).
  • Elasticsearch intercepts objects generated by the queries and creates an aggregated database that allows extra capabalities for searching and analyzing the data sources. Data sources can opt-in or opt-out of this functionality at any time.

API

Warning

This section is in flux. The API will change before a stable release.

In what follows, assume that $ADAMA is the base url of the Adama services.

The resource $ADAMA/register accepts a GET and POST request. The GET verb returns a list of registered adapters in the form:

{
  "identifier": "foo_v0.1",
  "language": "python",
  "name": "foo",
  "url": "http://example.com",
  "version": "0.1",
  "workers": [
    "ad89eefd01ca4648dc388dd12b95816cc24fe938ca801bd938ef982fa057a489",
    "7ec01fefe8c8a54d92a773569e0dc0b38be8b3a9bbdea7a16da05c5a800117ad",
    "ca88c73a13c17704e348eca468c101c784654dbe90b1156846e34507d0cccd6a",
    "7f3fbb1faadd7493c349dd316ad3d1dcf8775b8851dbf0e9111b00bf64c03612"
  ]
}

The identifier is unique and can be used in the queries to refer to the service provided by this adapter. The workers field contains internal information about the workers currently running to attend this adapter (it may be removed from the public API in the future).

The POST verb allows to register a new adapter. It accepts the parameters described below. The type of the parameter and whether they are mandatory or optional is described besides the paramter name.

name [form, mandatory]
Name of the service provided by this adapter (together with the version they must form a unique identifier)
version [form, mandatory]
Version of the service.
url [form, mandatory]
URL of the data source. The network access for the adapter may be restricted to access only this URL.
description [form, mandatory]
Human readable description of the service.
requirements [form, optional]
Comma separated list of third party modules to be installed inside the workers. They should be accessible in the standard package repository for the language bein used (i.e., pypi for Python, rubygems for Ruby, etc.).
code [file, mandatory]
The user’s code for the adapter. See Adapter API for its API and requirements. The code can be provided in a single file, or in a tarball or zip compressed archive. The type is detected automatically.

Workers are started immediately after registration. The response is the standard (status, message, result) triple (see Agave).

The verbs PUT and DELETE will be implemented in the future to allow administration of already registered adapters.

The resource $ADAMA/query accepts POST requests to perform queries to a selected list of services. The parameter is a JSON encoded in the body with the schema:

{
  "serviceName": "foo_v0.1",
  "query": "...Araport Language query..."
}

The serviceName field can also be a list of multiple services. The query will be delivered to all of them, and responses will be collected together. See Araport Language for the schema of the queries.

Adapter API

Warning

This section is in flux. The API will change before a stable release.

An adapter can be written in any of the programming languages supported by Adama. The list initially includes:

Python, Javascript (node.js), Ruby, Java, Lua, Perl.

Other languages can be added in the future by request.

The description that follows is generic, and details for each language will be provided in newer revisions of this document.

An adapter is a module called main. It contains a function named process which accepts a string and returns nothing. The string argument is a JSON encoded object that will be passed by Adama, and it will contain the query from the user (in the Araport Language).

The task of the function process is:

  • Convert the Araport Language query to the proper format for the 3rd party service.

  • Send the query to the 3rd party service and retrieve the results.

  • For each result, convert it to Araport Language and print it to screen (as a JSON encoded value).

    The output may use several lines with no restriction. Print the line --- to separate results. Many results can be generated from every result from the data source.

The function process in the module main can be tested by the developer by simply running it in his or her own system, with no access to Adama or Araport. As long as the adapter follows the protocol to print to standard output as JSON, and to separate the objects with ---, Adama will be able to capture the results.

Note

By printing to standard output, the adapter is effectively using an asynchronous output model, allowing Adama to start delivering results to the clients as soon as possible. It does not require any effort from the developer. And it is actually easier to code than collecting the results in a temporary container and returning them.

The main module can have dependencies of two types:

  • It can depend on other modules provided by the developer. In such case, the developer can choose to register the adapter as tarball or zip compressed archive. The only requirement is that the main module has to be at the root level of the compressed archive. Other files or assets needed by the module can be at the same level or in subdirectories.
  • Or it can depend on modules or packages from the standard package repository for the corresponding language. In this case, the extra modules will be installed at registration time in the workers (see Registration API).

The language of the adapter is detected automatically by looking at the module main uploaded during registration.

See Developer Role Quickstart for an example of this API.

Araport Language

Todo

To be defined and to be written.