EUDAT launches the Generic Execution Framework

Primary tabs

The EUDAT team is pleased to announce the release of the beta version of the Generic Execution Framework (GEF). The trend towards larger data volumes exacerbates the problem of costly and time consuming data transfers to off-site data processing locations. The EUDAT GEF enables you to deploy your containerised scientific tools to computational resources in close proximity to the input data in order to minimise overall data transfer costs for your research community.

Move the tools not the data to avoid costly data transfers!

The EUDAT GEF enables execution of containerized software tools on data stored in the EUDAT Collaborative Data Infrastructure (CDI). You can avoid or minimise data transfers by manually containerising the software tools required for a specific data processing job and enacting them as close to the input data as the infrastructure permits, minimising overall data transfer costs. This is the idea behind the EUDAT GEF. This approach also fosters a high degree of reproducibility as the same containerised tools and their configuration can be run repeatedly on different sites without requiring new installation or configuration procedures.

The GEF employs the Docker containerisation technology and uses specifically annotated Docker container images called GEF services for scientific data processing. You can configure the GEF to interface with Docker Server and Docker Swarm installations on various platforms. If you wish to employ the GEF you are asked to containerise your software tools by uploading a set of containerisation instructions in the form of a Dockerfile to the GEF. Input data to GEF services can be specified via Persistent Identifiers (PID) or URL and are automatically transferred to the processing location. The GEF is integrated with the EUDAT AAI and requires users who wish to build GEF services and run them to have a B2ACCESS account. Note that the GEF beta is only integrated with the B2ACCESS development instance and you require a separate account there. The aim of the GEF development is full integration with all EUDAT data services of the CDI and it currently supports B2SHARE and B2DROP.

Is the EUDAT GEF useful for you?

The EUDAT GEF service is for all scientists who wish to minimise data transfers associated with their processing jobs, have a Docker installation available in proximity of their input data and are willing to familiarise themselves with containerisation. One use case for the EUDAT GEF is extraction and computation of domain-specific metadata from seismological waveform data deposited in the EUDAT CDI by the EPOS community, the European Plate Observation System, a research infrastructure on solid Earth science. Another use case is data analytics on climate data stored in the EUDAT CDI by the ENES community, the European Network on Earth System Modelling.

Try it now and send us your feedback!

You can visit our GEF testing instance at https://eudat-gef.mpimet.mpg.de where you will find a preinstalled GEF server. Use your browser to test a few example services that we have provided for this purpose. We will provide more example GEF services in the next few months to illustrate several use cases. These use cases will also represent GEF applications in different fields of science.

If you want to install the GEF and build you own service, go to https://github.com/EUDAT-GEF/GEF and follow the installation instructions. You can either use the 64-bit Linux binary provided or build the GEF from sources. Your own installation will allow you to build your own GEF services from Dockerfiles and execute them.

Please contact us for any questions or feedback. GEF development is still continuing and we are counting on community feedback to provide us with some important pointers and requirements. The beta version also needs to be scrutinised and tested by you for it to grow into a fully stable and well-rounded piece of software.

We are:

Alexandr Chernov (alexandr.chernov@uni-tuebingen.de)

Emanuel Dima (emanuel.dima@uni-tuebingen.de)

Asela Rajapakse (asela.rajapakse@mpimet.mpg.de)

Or use the EUDAT helpdesk at: https://eudat.eu/contact-support-request

Useful links:

For GEF documentation and installation instructions go to: https://github.com/EUDAT-GEF/GEF

For information on Docker visit: https://docker.com/