Overview

Introduction

The SPARC metadata APIs are provided via an ElasticSearch endpoint. The metadata is extracted from K-Core's metadata export and processed via the FDI Lab's Foundry - a message-oriented, horizontally scalable ETL system for scientific data integration and enhancement (Ozyurt and Grethe 2018).

Access to the metadata APIs

Access to the metadata API is provided via an Elasticsearch pass-through.

General Elasticsearch

GET https://scicrunch.org/api/1/elastic/SPARC_PortalDatasets_pr/_search

The pass-through is accessible at https://scicrunch.org/api/1/elastic. Similar to standard Elasticsearch APIs you must then supply an index and an action. In this case the SPARC Portal Dataset metadata (SPARC_PortalDatasets_pr) and the search command (_search). Documentation on the Elasticsearch Search API is available at https://www.elastic.co/guide/en/elasticsearch/reference/6.8/search.html

Path Parameters

Name	Type	Description
api_key	string	Your API key to access the services

200

{
    "took": 35,
    "timed_out": false,
    "_shards": {
        "total": 2,
        "successful": 2,
        "skipped": 0,
        "failed": 0
    },
    "hits": {...

SPARC metadata indices

Management of indices is accomplished via Elasticsearch aliases. Three index aliases are provided:

Index Alias	Description
SPARC_PortalDatasets_pr	Production metadata index for SPARC Portal datasets
SPARC_PortalDatasets_dev	Development metadata index for SPARC Portal datasets
SPARC_PortalDatasets_old	Prior production metadata index for SPARC Portal datasets

Using aliases will allow for testing on updates and enhancements to the metadata index structure. If needed additional aliases can be constructed for specialized testing.

References

Ozyurt, I. B., & Grethe, J. S. (2018). Foundry: a message-oriented, horizontally scalable ETL system for scientific data integration and enhancement. Database : the journal of biological databases and curation, 2018, bay130. https://doi.org/10.1093/database/bay130