baton Python Wrapper

Introduction

Python 3 Wrapper for baton, superseding a previous implementation in metadata-check.

The wrapper provides access to most of baton's functionality.

How to use

Prerequisites

• Python >= 3.5.2
• baton >= 0.16.4
• iRODS >= 4.1.9

Note: Although older version of baton/iRODS will probably work, the library is only aimed at the versions specified above.

Installation

Stable releases can be installed via PyPI: bash $ pip3 install baton

Bleeding edge versions can be installed directly from GitHub: bash $ pip3 install git+https://github.com/wtsi-hgi/python-baton-wrapper.git@<commit_id_or_branch_or_tag>#egg=baton

To declare this library as a dependency of your project, add it to your requirement.txt file.

API

Setup

To use the iRODS API, you must first define a "connection" to an iRODS server: ```python from baton.api import connecttoirodswithbaton, Connection

Setup connection to iRODS using baton

irods = connecttoirodswithbaton("/where/baton/binaries/are/installed/", skipbatonbinaries_validation=False) # type: Connection ```

Data Objects and Collections

The API provides the ability to retrieve models of the data objects and collections stored on an iRODS server. Similarly to the JSON that baton provides, the models do not contain the payloads. They do however provide access to all of the information that baton can retrieve about an entity, including Access Control Lists (ACLs), custom metadata (AVUs), the content of collections and information about data object replicas. All methods provide the option to not load AVUs. ```python from baton.models import DataObject, Collection, SearchCriterion, ComparisonOperator

Get models of data objects or collections at the given path(s) in iRODS

irods.dataobject.getbypath("/collection/dataobject", loadmetadata=False) # type: DataObject: irods.collection.getbypath(["/collection", "/othercollection"]) # type: Sequence[Collection]:

Setup search for data objects or collections based on their metadata

searchcriterion1 = SearchCriterion("attribute", "matchvalue", ComparisonOperator.EQUALS) searchcriterion2 = SearchCriterion("otherattribute", "othermatchvalue", ComparisonOperator.LESS_THAN)

Do search to get models of data objects or collections

irods.dataobject.getbymetadata(searchcriterion1, zone="OptionalZoneRestriction") # type: Sequence[DataObject] irods.collection.getbymetadata([searchcriterion1, searchcriterion2], loadmetadata=False) # type: Sequence[Collection]

Get models of data objects or collections contained within a collection(s)

irods.collection.getallincollection("/collection", loadmetadata=False) # type: Sequence[Collection] irods.dataobject.getallincollection(["/collection", "/other_collection"]) # type: Sequence[DataObject] ```

Metadata (AVUs)

The API provides the ability to both retrieve and manipulate the custom metadata (AVUs) associated with data objects and collections.

Warning: there is currently no support for reading/writing the unit property of AVUs.

Although the type of metadata is the same for both data objects and collections, due to the way iRODS works, it is necessary to know the type of entity that a path corresponds to in order to retrieve metadata. ```python from baton.collections import IrodsMetadata

metadata1 = IrodsMetadata({"key": {"value1"}}) metadata2 = IrodsMetadata({"anotherkey": {"value1", "value2"}})

Metadata (methods available for both data_object and collection)

irods.dataobject.metadata.getall("/collection/dataobject") # type: IrodsMetadata irods.collection.metadata.getall(["/collection", "/other_collection"]) # type: Sequence[IrodsMetadata]

metadata_1 is added to the data object with the first path in the list and metadata_2 is added to the second

irods.dataobject.metadata.add(["/collection/dataobject", "/otherdataobject"], [metadata1, metadata2]) irods.collection.metadata.add("/collection", metadata_1)

irods.dataobject.metadata.set("/collection/dataobject", metadata_1)

metadata_1 is added to both collections in the list

irods.collection.metadata.set(["/collection", "/othercollection"], metadata1)

irods.dataobject.metadata.remove(["/collection/dataobject", "/otherdataobject"], [metadata1, metadata2]) irods.collection.metadata.remove("/collection", metadata_1)

irods.dataobject.metadata.removeall("/collection/dataobject") irods.collection.metadata.removeall(["/collection", "/other_collection"]) ```

Access Control Lists (ACLs)

The API provides the ability to both retrieve and manipulate the access control lists (ACLs) associated with data objects and collections. ```python from baton.models import AccessControl

ACLs. Note: it is implied that the owner is in the same zone as the entity to which the access control is applied

aclexamples = [ AccessControl(User("user1", "zoneuserisin"), AccessControl.Level.READ), AccessControl(User("group1", "zonegroupisin"), AccessControl.Level.WRITE), AccessControl("user1#zoneuseris_in", AccessControl.Level.OWN) ]

irods.dataobject.accesscontrol.getall("/collection/dataobject") # type: Set[AccessControl] irods.collection.accesscontrol.getall(["/collection", "/another/collection"]) # type: List[Set[AccessControl]]

irods.dataobject.accesscontrol.addorreplace(["/collection/dataobject", "/another/dataobject"], aclexamples[0]) irods.collection.accesscontrol.addorreplace("/collection", acl_examples, recursive=True)

irods.dataobject.accesscontrol.set("/collection/dataobject", aclexamples[1]) irods.collection.accesscontrol.set(["/collection", "/another/collection"], aclexamples[0], recursive=False)

irods.dataobject.accesscontrol.revoke(["/collection/dataobject", "/another/dataobject"], aclexamples) irods.collection.accesscontrol.revoke("/collection", acl_examples[1], recursive=True)

irods.dataobject.accesscontrol.revokeall(["/collection/dataobject", "/another/dataobject"]) irods.collection.accesscontrol.revoke_all("/collection", recursive=True) ```

Custom objects via specific queries

iRODS supports specific queries which return new types of object. In order to use such custom objects in iRODS via this library, a custom model of the object should to be made. Then, a subclass of BatonCustomObjectMapper needs to be defined to specify how a specific query (or number of specific queries) can be used to retrieve from and/or modify the object in iRODS.

The API provides the ability to retrieve the queries that are installed on an iRODS server (ironically, by use of a specific query!): ```python from baton.models import SpecificQuery

Get specific queries that have been installed on the iRODS server

irods.specificquery.getall(zone="OptionalZoneRestriction") # type: Sequence[SpecificQuery] ```

JSON Serialization/Deserialization

There are JSON encoders and decoders for nearly all iRODS object models in this library. These can be used to convert models to/from their baton defined JSON representations. All serializers/deserializers extend JSONEncoder and JSONDecoder (most through use of the hgijson library) meaning that they can be used with Python's built in json package: ```python import json from baton.json import DataObjectJSONEncoder, DataObjectJSONDecoder, CollectionJSONEncoder, CollectionJSONDecoder, IrodsMetadataJSONEncoder, IrodsMetadataJSONDecoder, AccessControlJSONEncoder, AccessControlJSONDecoder

dataobjectasjsonstring = json.dumps(dataobject, cls=DataObjectJSONEncoder) # type: str dataobject = json.loads(dataobjectasjsonstring, cls=DataObjectJSONDecoder) # type: DataObject

collectionasjsonstring = json.dumps(collection, cls=CollectionJSONEncoder) # type: str collection = json.loads(collectionasjsonstring, cls=CollectionJSONDecoder) # type: Collection

metadataasjsonstring = json.dumps(metadata, cls=IrodsMetadataJSONEncoder) # type: str metadata = json.loads(metadataasjsonstring, cls=IrodsMetadataJSONDecoder) # type: IrodsMetadata

aclasjsonstring = json.dumps(metadata, cls=AccessControlJSONEncoder) # type: str acl = json.loads(aclasjsonstring, cls=AccessControlJSONDecoder) # type: List[AccessControl] ```

Development

Setup

Install both library dependencies and the dependencies needed for testing: bash $ pip3 install -q -r requirements.txt $ pip3 install -q -r test_requirements.txt A baton installation is not required.

Some tests use Docker therefore a Docker daemon must be running on the test machine, with the environment variables DOCKER_TLS_VERIFY, DOCKER_HOST and DOCKER_CERT_PATH set.

Testing

Using nosetests, in the project directory, run: bash $ nosetests -v --cover-inclusive --tests baton/tests, baton/tests/_baton

To generate a test coverage report with nosetests: bash $ nosetests -v --with-coverage --cover-package=baton --cover-inclusive --tests baton/tests, baton/tests/_baton

License

LGPL license.

Copyright (c) 2015, 2016 Genome Research Limited