Architecture¶
Overview¶
The following picture illustrates the architecture of Adama:
The main components are:
- Adama itself is a Python application running over an Apache web server, providing the REST endpoints and business logic. (see API).
- A data store (Redis) collects a registry of adapters.
- A task queue (RabbitMQ) distributes work to the adapters for each
request to the
query
endpoint. - The workers are Docker containers that get created through the
register
endpoint, and get started and shutdown under Adama command, depending on the load of thequery
endpoint. The workers are continuously consuming work from the task queue. - The user module in the picture is the adapter provided by the developer at registration time (see Adapter API).
- The user module, data sources, and 3rd party protocol (colored blue in the picture) specify the parts under control of the developer role (see Developer Role Quickstart).
- Elasticsearch intercepts objects generated by the queries and creates an aggregated database that allows extra capabalities for searching and analyzing the data sources. Data sources can opt-in or opt-out of this functionality at any time.
API¶
Warning
This section is in flux. The API will change before a stable release.
In what follows, assume that $ADAMA
is the base url of the Adama
services.
The resource $ADAMA/register
accepts a GET
and POST
request. The GET
verb returns a list of registered adapters in
the form:
{
"identifier": "foo_v0.1",
"language": "python",
"name": "foo",
"url": "http://example.com",
"version": "0.1",
"workers": [
"ad89eefd01ca4648dc388dd12b95816cc24fe938ca801bd938ef982fa057a489",
"7ec01fefe8c8a54d92a773569e0dc0b38be8b3a9bbdea7a16da05c5a800117ad",
"ca88c73a13c17704e348eca468c101c784654dbe90b1156846e34507d0cccd6a",
"7f3fbb1faadd7493c349dd316ad3d1dcf8775b8851dbf0e9111b00bf64c03612"
]
}
The identifier is unique and can be used in the queries to refer to
the service provided by this adapter. The workers
field contains
internal information about the workers currently running to attend
this adapter (it may be removed from the public API in the
future).
The POST
verb allows to register a new adapter. It accepts the
parameters described below. The type of the parameter and whether
they are mandatory or optional is described besides the paramter name.
name
[form, mandatory]- Name of the service provided by this adapter (together with the version they must form a unique identifier)
version
[form, mandatory]- Version of the service.
url
[form, mandatory]- URL of the data source. The network access for the adapter may be restricted to access only this URL.
description
[form, mandatory]- Human readable description of the service.
requirements
[form, optional]- Comma separated list of third party modules to be installed inside
the workers. They should be accessible in the standard package
repository for the language bein used (i.e.,
pypi
for Python,rubygems
for Ruby, etc.). code
[file, mandatory]- The user’s code for the adapter. See Adapter API for its API and requirements. The code can be provided in a single file, or in a tarball or zip compressed archive. The type is detected automatically.
Workers are started immediately after registration. The response is
the standard (status
, message
, result
) triple (see Agave).
The verbs PUT
and DELETE
will be implemented in the future to
allow administration of already registered adapters.
The resource $ADAMA/query
accepts POST
requests to perform
queries to a selected list of services. The parameter is a JSON
encoded in the body with the schema:
{
"serviceName": "foo_v0.1",
"query": "...Araport Language query..."
}
The serviceName
field can also be a list of multiple services. The
query will be delivered to all of them, and responses will be
collected together. See Araport Language for the schema of the
queries.
Adapter API¶
Warning
This section is in flux. The API will change before a stable release.
An adapter can be written in any of the programming languages supported by Adama. The list initially includes:
Python, Javascript (node.js), Ruby, Java, Lua, Perl.
Other languages can be added in the future by request.
The description that follows is generic, and details for each language will be provided in newer revisions of this document.
An adapter is a module called main
. It contains a function named
process
which accepts a string and returns nothing. The string
argument is a JSON encoded object that will be passed by Adama, and it
will contain the query from the user (in the Araport Language).
The task of the function process
is:
Convert the Araport Language query to the proper format for the 3rd party service.
Send the query to the 3rd party service and retrieve the results.
For each result, convert it to Araport Language and print it to screen (as a JSON encoded value).
The output may use several lines with no restriction. Print the line
---
to separate results. Many results can be generated from every result from the data source.
The function process
in the module main
can be tested by the
developer by simply running it in his or her own system, with no
access to Adama or Araport. As long as the adapter follows the
protocol to print to standard output as JSON, and to separate the
objects with ---
, Adama will be able to capture the results.
Note
By printing to standard output, the adapter is effectively using an asynchronous output model, allowing Adama to start delivering results to the clients as soon as possible. It does not require any effort from the developer. And it is actually easier to code than collecting the results in a temporary container and returning them.
The main
module can have dependencies of two types:
- It can depend on other modules provided by the developer. In such
case, the developer can choose to register the adapter as tarball or
zip compressed archive. The only requirement is that the
main
module has to be at the root level of the compressed archive. Other files or assets needed by the module can be at the same level or in subdirectories. - Or it can depend on modules or packages from the standard package repository for the corresponding language. In this case, the extra modules will be installed at registration time in the workers (see Registration API).
The language of the adapter is detected automatically by looking at
the module main
uploaded during registration.
See Developer Role Quickstart for an example of this API.
Araport Language¶
Todo
To be defined and to be written.