Configure B2SAFE

About

Documentation how to configure the B2SAFE service for Site Administrators and Community Data Managers.

Modified: 12 February 2018

Synopsis

B2SAFE, the EUDAT Safe Replication service, is the cornerstone of EUDAT's service for safe replication. In addition to replication, the service also allows communities to implement data management policies on their research data across multiple administrative domains in a trustworthy manner.

This document describes how to implement and configure B2SAFE such that a community data centre can join EUDAT.

The document applies to B2SAFE version 4.0.1.

Acronyms

PID: Persistent identifier associated to a file or iRODS collection, usually an EUDAT Handle.

ROR: Repository of Records, (persistent) identifier to the original data object. Can be of any identifier type. If the community does not assign an own identifier the B2SAFE PID will be used.

FIO: First ingested object, the persistent identifier associated to the very first object in in the EUDAT domain. If the the chain has only two elements, the master copy and the first replica, then the PARENT = FIO.

PARENT: Parent PID, the persistent identifier associated to the source object in a replication chain. If the chain has only two elements, the master copy and the first replica, then the PARENT = FIO.

Digital entity: Files and folders

Digital object: Files and folders that carry a persistent identifier and possibly some metadata.

What is B2SAFE

The EUDAT B2SAFE Service offers the functionality to replicate datasets across different data centres in a safe and efficient way while maintaining all information required to easily find and query information about the replica locations. To this end information on locations along with other important information is stored in PID records. The B2SAFE Service is implemented as an iRODS module providing a set of iRODS rules or policies to interface with either the Handle HTTP JSON REST API or the EPIC Handle API to create and manage PIDs and uses the iRODS middleware to replicate datasets from a source data (or community) centre to a destination data centre. B2SAFE offers the possibility to replicate single files (also called data objects) and set of data objects organised in iRODS collections. The organisation of data is left to the communities. While B2SAFE is internally using the handle HTTP JSON REST API or the EPIC Handle API, communities have the choice to use any PID system they prefer to assign PIDs to their digital objects at the ROR. In any case, replicas in the EUDAT domain will receive a handle upon replication by the B2SAFE service. The PIDs are returned by the B2SAFE service and can be employed to reference datasets. For more information of B2SAFE please read our B2SAFE user documentation page.

Be aware that the EPIC API is being fazed out in favor of the handle HTTP JSON REST API.

Communities can elect to ask an EUDAT site to run B2SAFE for them; please see our Using B2SAFE guide for more information. This guide is for communities that have elected to run their own B2SAFE service, and it explains how to achieve this. 

In the following we will provide a comprehensive guide to install the basic features of B2SAFE. For more information and further configurations please see the B2SAFE wiki.

Configure B2SAFE 

This section explains the required steps for a Community Centre Administrator to configure the B2SAFE Service at their site. 

Prerequisites

Before installing and configuring the EUDAT B2SAFE module, make sure you meet the following conditions:

• You are running an iRODS server, either version 4.0.X, 4.1.X (preferred) or 4.2.X  (User Documentation - iRODS Deployment)
• You have a Handle prefix (User Documentation - PIDs in EUDAT) with installed private public keys and certificates (see B2HANDLE documentation); you can create and update persistent identifiers with this prefix (please test before installing B2SAFE)
• You have access to the EUDAT git repository
• You have access to the EUDAT B2HANDLE git repository to create and install the B2HANDLE library
• You have enabled access to ports 1247, 1248 and 20000-20199 that iRODS uses to communicate.

Definitions

• <irods> denotes the absolute path to your iRODS directory. e.g /var/lib/irods/iRODS
• <irods user> unix service account which runs iRODS; iRODS creates such an account and calls it by default "irods"
• <b2safe-download> denotes the absolute path to your B2SAFE module downloaded from git. e.g /home/<user>/B2SAFE-core or /var/lib/irods/B2SAFE-core
• <B2SAFE-core> Configuration folder, usually /opt/eudat/b2safe

iRODS Versions Supported

B2SAFE module release 3.X.X needs to be installed with iRODS 4.0.X or 4.1.X. In the following we describe the installation of B2SAFE through packages first.

The installation through packages is the preferred way of installing B2SAFE. 

Steps

0. Get and install the B2HANDLE library

B2SAFE uses Handles as Persistent Identifiers and it manages them using the B2HANDLE python library.

To install the B2HANDLE python library download the latest version from github and follow the installation guide (installation via a python egg file). You may also create an rpm as follows and install it:

python setup.py bdist_rpm
rpm -ivh <path>/dist/b2handle-<version>.noarch.rpm

Please also make sure you have installed the following python packages:

queuelib, lxml, defusedxml, httplib2 and simplejson

If you have not done so beforehand, please generate private/public keys for your Handle prefix following the instructions (you will need to download the Handle software) and install or let the system administrator of your local Handle server install them for you.

Before you proceed with the installation of B2SAFE, please make sure that you can create and update Persistent Identifiers with your prefix like demonstrated in the tutorial.

1. Get the B2SAFE module

From https://github.com/EUDAT-B2SAFE/B2SAFE-core/releases you may download a B2SAFE release that is suitable for you. When you download the B2SAFE zip file and unzip it (as a NOT root user), it creates a directory named "B2SAFE-core-4.X.X". You may rename this directory, but in this documentation we refer to it as <b2safe-download> . The directory name must be different to any iRODS directories.

cd <b2safe-download>/packaging

./create_rpm_package.sh

rpm -ivh /home/<user>/rpmbuild/RPMS/noarch/irods-eudat-b2safe-3.X-X.noarch.rpm
  Preparing...                ########################################### [100%]
     1:irods-eudat-b2safe     ########################################### [100%]

cd <b2safe-download>/packaging

./create_deb_package.sh 

dpkg-deb: building package `irods-eudat-b2safe' in `/home/<user>/debbuild/irods-eudat-b2safe_3.X-X.deb’

root# dpkg -i /home/<user>/debbuild/irods-eudat-b2safe_3.X-X.deb
Selecting previously unselected package irods-eudat-b2safe.
(Reading database ... 48938 files and directories currently installed.)
Unpacking irods-eudat-b2safe (from .../irods-eudat-b2safe_3.X-X.deb) ...
Setting up irods-eudat-b2safe (3.X-X) ...

sudo vi /opt/eudat/b2safe/packaging/install.conf

AUTHZ_ENABLED=<false, true>
MSG_QUEUE_ENABLED=<false, true>

source /etc/irods/service_account.config

cd /opt/eudat/b2safe/packaging
./install.sh

queuelib, lxml, defusedxml, httplib2 and simplejson

eudat-b2safe:~/B2SAFE-core$ irule -F rules/eudatGetV.r
*version=4.1-0

eudat-b2safe:~/B2SAFE-core$ echo "test" > testfile.txt
eudat-b2safe:~/B2SAFE-core$ iput testfile.txt

eudatPidsColl{
    # Create PIDs for all collections and objects in the collection recursively
    # Parameters: *parent_pid, *source, *ror, *fio, "true"(fixed content) , *newPID
    EUDATCreatePID("None", *coll_path, "None", "None", "true", *newPID);
}
INPUT *coll_path=‘/<zone>/home/<user>/<file or collection>’
OUTPUT *newPID,ruleExecOut

eudat-b2safe:~/B2SAFE-core$ irule -F rules/eudatPidsColl.r
*newPID = 846/839b44e8-7678-11e6-b7b4-0200c0a809d6
eudat-b2safe:~/B2SAFE-core$ imeta ls -d testfile.txt

eudatRepl{ 
    # Data set replication 
    # registered data (with PID registration) (3rd argument - 1st bool("true")) 
    # recursive (4th argument 2nd bool("true")) 
    EUDATReplication(*source, *destination, bool("true"), bool("true"), *response) 
} 
INPUT *source=‘/<your zone>/home/<user>/<collection>’, *destination=‘/<remote zone>/<user>#<your zone>/<collection>’ 
OUTPUT ruleExecOut 

eudat-b2safe:~/B2SAFE-core$ irule -F rules/eudatRepl.r 
eudat-b2safe:~/B2SAFE-core$ ils /<remote zone>/<user>#<your zone>/<collection> 
eudat-b2safe:~/B2SAFE-core$ imeta ls -C /<remote zone>/<user>#<your zone>/<collection>

"federation": [
    {
        "icat_host": “”,
        "zone_name": “”,
        "zone_key": “”,
        "negotiation_key": “”
    }]

B2SAFE and other Persistent Identifier Systems

B2SAFE uses by default Handles and the Handle System to label data in B2SAFE and track replicas across the different iRODS instances.

Some communities already have an own persistent identifier system and might want to replace the usage of Handles in B2SAFE with their own System. In such a case the community is responsible for creating scripts that introduce the relation, i.e. the links to replicas and parents, for the data objects. To this end they will need to device new iRODS rules. In order to set this up you need to create an iRODS rule which gets a sourcePID and replicaPID as parameters. The logic required to implement this policy depends on the community requirements, the repository and the persistent identifier system. This is a community effort, but EUDAT will provide support if needed.

Additional Tools and Feature Preview

Repository Package

The Repository Package allows communities to ingest data from their own repositories into B2SAFE. The source code and documentation can be found on the EUDAT B2SAFE github page. For now, it is possible to ingest data stored on a file system or in the DSPACE repository. The connectors for FEDORA and ePRINTS are planned to be implemented.

Messaging

Messaging is an experimental feature that provides the results in case of asynchronous (server side triggered) replication process (see the Workflows section). The messages are posted to a queue which can be accessed via an HTTP interface. In the current implementation the queue is provided by dweet.io. The messaging is disabled by default. In order to enabled it, just change from "false" to "true" the parameter "*enabled" in the "getMessageParameters" rule in the local.re rule set as follows:

getMessageParameters(*msgConfPath, *enabled) {
    *msgConfPath="/var/log/irods/msgManager.conf";
    *enabled=bool("false");
}

When enabled the results of the PID registration and data replication workflows will be posted to a dweet url based on the iRODS zone and the username as follows: http://dweet.io/iRODS_zone_name/username

Metadata

Metadata management is an experimental feature. It is disabled by default. When enabled it provides a set of metadata properties for each data object, storing them in a file located as follows: given the object /path/to/myobject, the metadata file is stored in /path/to/.metadata/myobject_metadata.json. In order to enable metadata, just edit the "getMetaParameters" rule in the local.re rule set and change the parameter "*enabled" from "false" to "true".

getMetaParameters(*metaConfPath, *enabled) {
    *metaConfPath="/srv/irods/current/modules/B2SAFE/cmd/metadataManager.conf";
    *enabled=bool("false");
}

The expected content is as follows:

{
  "checksum": "sha2:nmDjK/7k1D5jjMUFoWHjX5qZmke9vpQbR6FaY9sk6eI=",
  "ror": "None",
  "pid": "842/6cff9eb8-47ef-11e5-a889-fa163e62896a",
  "checksum_timestamp": "01440065311"
}

and it is triggered by the rules EUDATCreatePID and EUDATeCHECKSUMupdate.

Support

Our B2SAFE presentation discusses data replication.

You can access B2SAFE hands-on training material from our github; note in particular the sessions dedicated to configuring B2SAFE and using policies to manage data across iRODS zones.

Support for B2SAFE is available via the EUDAT ticketing system through the webform.

If you have comments on this page, please submit them though the EUDAT ticketing system.

Document Data

Version: 3.9

Authors:

Elena Erastova, elena.erastova@mpcdf.mpg.de

Claudio Cacciari, c.cacciari@cineca.it

Robert Verkerk, robert.verkerk@surfsara.nl

Pascal Dugénie dugenie@cines.fr

Editors:

Kostas Kavoussanakis, k.kavoussanakis@epcc.ed.ac.uk

Themis Zamani, themis@admin.grnet.gr

Christine Staiger, christine.staiger@surfsara.nl