Adama¶

Problem¶

There is a huge amount of data related to Arabidopsis being produced and being served to the internet. However, the way the data is presented varies wildly across each service. The main aspects in which services vary are:

• The format the data is presented: anything from CSV to binary formats;
• The API used to access the service: anything from a RESTful service to SOAP based service;
• The vocabulary used to denote the objects.

Trying to collect and to unify these aspects is difficult given their dynamic nature. Data formats and interfaces change quickly. A central service would need to be continuously updating its software. Adama tries to solve this problem by engaging the developers of each service in a distributed and federated architecture.

Strategy¶

Adama is a RESTful service that provides two main endpoints:

• Registration interface: a provider of a data source (see Roles) can register a service to be accessible through Adama. The provider’s work (see Developer Role Quickstart) consists in defining a conversion procedure between the Araport language (see Araport Language) and the native format of the data source.
• Query interface: an user of Araport (see User Role Quickstart) can perform queries in the Araport language and direct them to one or more registered services.

In addition to this interface or API, the architecture provides the following features that aim to solve the points in Problem:

• Unified language and API. Araport provides access through a RESTful service, accepting and returning JSON objects. The schema for these objects is defined in the Araport Language, a rationalized and extensible format designed to express queries for all the registered services. This allows a query to be spread to multiple data sources simultaneously and the results to be collected through a single endpoint.
• Distributed responsability. The task of maintaining an up-to-date conversion between the Araport Language and the third party services is distributed among the developers of the latter. Adama provides an extremely easy way to develop the adapters and to test them in complete isolation of the Araport site (see Developer Role Quickstart).
• Security and isolation. The adapters from each developer and for each service are run fully isolated and with a restricted access to the network. This allows to control the security in execution, and to control the network usage (including throttling, caching, etc.).
• Scalability. Adama takes care of the horizontal scaling of the adapters. The scalability can be performed in a dynamic way, depending on the load to particular services. In addition, load balancing to the full Adama API is performed automatically.
• Extra Services. Adama provides extra services for each data source: pagination, count, and caching. These services do not require support from the data source and they do not require extra work from the developer of the adapter.

Overview¶

The following picture illustrates the architecture of Adama:

The main components are:

• Adama itself is a Python application running over an Apache web server, providing the REST endpoints and business logic. (see API).
• A data store (Redis) collects a registry of adapters.
• A task queue (RabbitMQ) distributes work to the adapters for each request to the query endpoint.
• The workers are Docker containers that get created through the register endpoint, and get started and shutdown under Adama command, depending on the load of the query endpoint. The workers are continuously consuming work from the task queue.
• The user module in the picture is the adapter provided by the developer at registration time (see Adapter API).
• The user module, data sources, and 3rd party protocol (colored blue in the picture) specify the parts under control of the developer role (see Developer Role Quickstart).
• Elasticsearch intercepts objects generated by the queries and creates an aggregated database that allows extra capabalities for searching and analyzing the data sources. Data sources can opt-in or opt-out of this functionality at any time.

API¶

In what follows, assume that $ADAMA is the base url of the Adama services.

The resource $ADAMA/register accepts a GET and POST request. The GET verb returns a list of registered adapters in the form:

{
  "identifier": "foo_v0.1",
  "language": "python",
  "name": "foo",
  "url": "http://example.com",
  "version": "0.1",
  "workers": [
    "ad89eefd01ca4648dc388dd12b95816cc24fe938ca801bd938ef982fa057a489",
    "7ec01fefe8c8a54d92a773569e0dc0b38be8b3a9bbdea7a16da05c5a800117ad",
    "ca88c73a13c17704e348eca468c101c784654dbe90b1156846e34507d0cccd6a",
    "7f3fbb1faadd7493c349dd316ad3d1dcf8775b8851dbf0e9111b00bf64c03612"
  ]
}

The identifier is unique and can be used in the queries to refer to the service provided by this adapter. The workers field contains internal information about the workers currently running to attend this adapter (it may be removed from the public API in the future).

The POST verb allows to register a new adapter. It accepts the parameters described below. The type of the parameter and whether they are mandatory or optional is described besides the paramter name.

name [form, mandatory]

Name of the service provided by this adapter (together with the version they must form a unique identifier)

version [form, mandatory]

Version of the service.

url [form, mandatory]

URL of the data source. The network access for the adapter may be restricted to access only this URL.

description [form, mandatory]

Human readable description of the service.

requirements [form, optional]

Comma separated list of third party modules to be installed inside the workers. They should be accessible in the standard package repository for the language bein used (i.e., pypi for Python, rubygems for Ruby, etc.).

code [file, mandatory]

The user’s code for the adapter. See Adapter API for its API and requirements. The code can be provided in a single file, or in a tarball or zip compressed archive. The type is detected automatically.

Workers are started immediately after registration. The response is the standard (status, message, result) triple (see Agave).

The verbs PUT and DELETE will be implemented in the future to allow administration of already registered adapters.

The resource $ADAMA/query accepts POST requests to perform queries to a selected list of services. The parameter is a JSON encoded in the body with the schema:

{
  "serviceName": "foo_v0.1",
  "query": "...Araport Language query..."
}

The serviceName field can also be a list of multiple services. The query will be delivered to all of them, and responses will be collected together. See Araport Language for the schema of the queries.

Adapter API¶

An adapter can be written in any of the programming languages supported by Adama. The list initially includes:

Python, Javascript (node.js), Ruby, Java, Lua, Perl.

Other languages can be added in the future by request.

The description that follows is generic, and details for each language will be provided in newer revisions of this document.

An adapter is a module called main. It contains a function named process which accepts a string and returns nothing. The string argument is a JSON encoded object that will be passed by Adama, and it will contain the query from the user (in the Araport Language).

The task of the function process is:

• Convert the Araport Language query to the proper format for the 3rd party service.

• Send the query to the 3rd party service and retrieve the results.

• For each result, convert it to Araport Language and print it to screen (as a JSON encoded value).

  The output may use several lines with no restriction. Print the line --- to separate results. Many results can be generated from every result from the data source.

The function process in the module main can be tested by the developer by simply running it in his or her own system, with no access to Adama or Araport. As long as the adapter follows the protocol to print to standard output as JSON, and to separate the objects with ---, Adama will be able to capture the results.

The main module can have dependencies of two types:

• It can depend on other modules provided by the developer. In such case, the developer can choose to register the adapter as tarball or zip compressed archive. The only requirement is that the main module has to be at the root level of the compressed archive. Other files or assets needed by the module can be at the same level or in subdirectories.
• Or it can depend on modules or packages from the standard package repository for the corresponding language. In this case, the extra modules will be installed at registration time in the workers (see Registration API).

The language of the adapter is detected automatically by looking at the module main uploaded during registration.

See Developer Role Quickstart for an example of this API.

Araport Language¶

Roles¶

We called developer an individual who has access or knowledge of an existing data source, and he or she wants to register it into Adama. The developer does not need special access to the third party data source, other than the standard API access it already provides. He or she needs an account on Araport with permissions to register adapters. A developer is expected to have a minimum of skills on some programming language (from the list supported by Adama, see programming languages).

We called user an individual who is using Adama through the query interface. No special permissions are required. Knowledge to access a RESTful API is recommended, although command line and web tools are also planned in the future.

Developer Role Quickstart¶

In this setion we register an adapter for the service Expressolog by locus at bar.utoronto.ca.

The service accepts queries as GET requests to the URL:

http://bar.utoronto.ca/webservices/get_expressologs.php

with parameter request=[{"gene": "...""}].

The native output of the service for the gene At2g26230 looks like :

{"At2g26230": [
  {"probeset_A": "267374_at",
   "gene_B": "Glyma20g17440",
   "probeset_B": "Glyma20g17440",
   "correlation_coefficient": "0.5264",
   "seq_similarity":"67",
   "efp_link": "http://bar.utoronto.ca/efp_soybean/cgi-bin
                /efpWeb.cgi?dataSource=soybean&primaryGene
                =Glyma20g17440&modeInput=Absolute"
  },
  {"probeset_A": "267374_at",
   "gene_B":"Solyc11g006550",
   "probeset_B": "Solyc11g006550",
   "correlation_coefficient": "0.1768",
   "seq_similarity": "68",
   "efp_link":"http://bar.utoronto.ca/efp_tomato/cgi-bin
               /efpWeb.cgi?dataSource=tomato&primaryGene
               =Solyc11g006550&modeInput=Absolute"
  }
 ]
}

The Araport Language query would look like (see Araport Language):

{
   "query": {"locus": "At2g26230"},
   "countOnly": false,
   "pageSize": 100,
   "page": 1
}

The transformed Araport Language output will contain two records for each of the previous records:

{
    "status": "success",
    "message": "",
    "result": [
        {
            "class": "locus_relationship",
            "reference": "TAIR10",
            "locus": "At2g26230",
            "type": "coexpression",
            "related_entity": "Solyc11g006550",
            "direction": "undirected",
            "score": [
                {
                    "correlation_coefficient": 0.1768
                }
            ],
            "source": "tomato"
        },
        {
            "class": "locus_relationship",
            "reference": "TAIR10",
            "locus": "At2g26230",
            "type": "similarity",
            "related_entity": "Solyc11g006550",
            "direction": "undirected",
            "score": [
                {
                    "similarity_percentage": 68
                }
            ],
            "source": "tomato"
        },
        {
            "class": "locus_relationship",
            "reference": "TAIR10",
            "locus": "At2g26230",
            "type": "coexpression",
            "related_entity": "Glyma20g17440",
            "direction": "undirected",
            "score": [
                {
                    "correlation_coefficient": 0.5264
                }
            ],
            "source": "soybean"
        },
        {
            "class": "locus_relationship",
            "reference": "TAIR10",
            "locus": "At2g26230",
            "type": "similarity",
            "related_entity": "Glyma20g17440",
            "direction": "undirected",
            "weight": [
                {
                    "similarity_percentage": 67
                }
            ],
            "source": "soybean"
        }
    ]
}

The complete code (in Python) for the adapter can be seen at Adama github repository.

In pseudo-code, the function process of the module main.py can be described as:

def process(args):
    # <extract 'locus' from the JSON object 'args'>
    # <send request to the expressologs service>
    for result in # <results from expressologs>:
        obj = # <convert result to Araport format (for type "similarity")>
        print obj
        print '---'

        obj = # <convert result to Araport format (for type "coexpression")>
        print obj
        print '---'

    print 'END'

This is the code which the developer of the adapter for the third party data source will create. The developer can test this function interactively in the Python interpreter, independently of any interaction with Adama, as in:

>>> import main, json
>>> main.process(json.dumps(
...   {"query": {"locus": "At2g26230"},
...    "countOnly": false,
...    "pageSize": 100,
...    "page": 1}))
...

If successful, the function will print to standard output a sequence of JSON objects separated by the lines '---', and it will print the 'END' string when the stream of results is exhausted. After a successful testing, the developer will upload this module to Adama by posting to $ADAMA/register, for example with the (Python) command:

>>> requests.post('$ADAMA/register',
...               data={'name': 'expressologs_by_locus',
...                     'version': '0.1',
...                     'url': 'http://bar.utoronto.ca/webservices/get_expressologs.php',
...                     'requirements': 'requests'},
...               files={'code': ('main.py', open('main.py'))})
...

Note that we are assuming the main.py module (as in the example on the github repository) is using the module requests to access the expressolog service. For such reason, we include it in the requirements field, so it is properly installed in the Adama workers. This example can be found already built in the examples directory of Adama github repository.

On success, Adama will return the full identifier for this service (expressologs_by_locus_v0.1), and it will start workers ready to attend queries.

User Role Quickstart¶

After a developer registered the expressolog service as in Developer Role Quickstart, a user can query the service by posting to $ADAMA/query:

>>> requests.post('$ADAMA/query',
...               data=json.dumps({
...                    'serviceName': 'expressologs_by_locus_v0.1',
...                    'query': {'locus': 'At2g26230'}}))
...

On success, the response will return a JSON object with the results in the format described in Developer Role Quickstart. As mentioned in Adapter API, Adama will start streaming the results as soon as they are available from the data source. Users can start reading the results using any of the available increamental JSON parsers.