dserver-notification-plugin

Listen to S3 event notifications from an S3-compatible storage backend

https://github.com/livmats/dserver-notification-plugin

Repository

Listen to S3 event notifications from an S3-compatible storage backend

README.rst

dserver Notification Plugin
===========================

.. |dtool| image:: https://github.com/livMatS/dserver-notification-plugin/blob/main/icons/22x22/dtool_logo.png?raw=True
    :height: 20px
    :target: https://github.com/livMatS/dserver-notification-plugin
.. |pypi| image:: https://img.shields.io/pypi/v/dserver-notification-plugin
    :target: https://pypi.org/project/dserver-notification-plugin/
.. |tag| image:: https://img.shields.io/github/v/tag/livMatS/dserver-notification-plugin
    :target: https://github.com/livMatS/dserver-notification-plugin/tags
.. |test| image:: https://img.shields.io/github/actions/workflow/status/livMatS/dserver-notification-plugin/test.yml?branch=main&label=tests
    :target: https://github.com/livMatS/dserver-notification-plugin/actions/workflows/test.yml
.. |zenodo| image:: https://zenodo.org/badge/365692008.svg
    :target: https://zenodo.org/doi/10.5281/zenodo.12702039

|dtool| |pypi| |tag| |test| |zenodo|

- GitHub: https://github.com/livMatS/dserver-notification-plugin
- PyPI: https://pypi.python.org/pypi/dserver-notification-plugin
- Free software: MIT License


Features
--------

- Listen to `S3 event notifications `_
  from an S3-compatible storage backend


Introduction
------------

`dtool `_ is a command line tool for packaging
data and metadata into a dataset. A dtool dataset manages data and metadata
without the need for a central database.

However, if one has to manage more than a hundred datasets it can be helpful
to have the datasets' metadata stored in a central server to enable one to
quickly find datasets of interest.

`dservercore `_
provides a web API for registering datasets' metadata
and provides functionality to lookup, list and search for datasets.

This plugin enables the dserver to listen to
notifications for the registration and deregistration of datasets.


Installation
------------

Install the dtool lookup server dependency graph plugin

.. code-block:: bash

    $ pip install dserver-notification-plugin

Setup and configuration
-----------------------

Configure plugin behavior
^^^^^^^^^^^^^^^^^^^^^^^^^

The plugin needs to know how to convert a bucket name into a base URI. The
environment variable ``DSERVER_NOTIFY_BUCKET_TO_BASE_URI`` is used
to specify that conversion, e.g.::

    DSERVER_NOTIFY_BUCKET_TO_BASE_URI={"bucket": "ecs://bucket"}

It is also advisable to limit access to the notification listener to a certain
IP range. Use::

    DSERVER_NOTIFY_ALLOW_ACCESS_FROM=192.168.0.0/16

to specify the allowed remote network. To specify a single IP, use::

    DSERVER_NOTIFY_ALLOW_ACCESS_FROM=192.168.1.1/32

Configure webhook in minio
^^^^^^^^^^^^^^^^^^^^^^^^^^

The `Publish Events to Webhook minio docs
`_
walks through the configuration for sending S3 event notifications to a webhook.
Assuming a *dserver* with this plugin activated running at
``http://dserver:5000``, and your minio instance with a
bucket ``test-bucket`` at ``https://s3server:9000``, use

.. code-block:: bash

    # mc: minio client
    mc config host add s3server http://s3server:9000 {admin_user} {admin_password}

    # Note that the endpoint must be reachable when configuring, otherwise minio will reject
    mc admin config set s3server/ notify_webhook:dtool  endpoint="http://dserver:5000/webhook/notify"
    mc admin service restart s3server  # restart is necessary

    # Activate the actual notifications
    mc event add s3server/test-bucket arn:minio:sqs::testbucket:dtool --event "put,delete"

to configure a webhook endpoint identified by ``dtool`` and activate ``put`` and
``delete`` event notification.
Choose the parameters for ``--event "put,delete"`` from minio's
`Supported Bucket Evenets `_.

Note that minio is very strict on whom they talk to. If your `dserver`
communicates via `https`, make sure that the server certificate uses `SANs
`_ and that the
signing authority's root certificate is available to minio. See
`Install Certificates from Third-party CAs
`_
in the minio docs. Also assure all services are reachable by valid hostnames.
Within a containerized environment such as launched with `docker-compose` ,
host names containing underscores ``_`` may occur, but minio refuses to speak with
such.

Configure webhook in NetApp StorageGRID
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

NetAPP StorageGRID is able to submit S3 event notifications when configured to
communicate with an SNS endpoint, refer to the according sections of the 
`NetApp StorageGRID docs `_ for 
`creating service endpoints `_
and
`configuring event notifications `_.

In short, create an endpoint ``http://dserver:5000/webhook/notify``
with a suitable URN, i.e. ``urn:dserver:sns:region:notify:all``,
where you may pick all fields freely except ``urn`` and ``sns``. 

Next, enable event notifications for the desired bucket, i.e. for object creation events with a policy snippet like this:

.. code-block:: xml

    
      
        Object created
        urn:dserver:sns:region:notify:all
        s3:ObjectCreated:*
      
    


Testing
-------

Launch a minimal mongodb instance with

.. code-block:: bash

    $ cd tests/container && docker-compose up -d

and run tests from within repository root using

.. code-block:: bash

    pytest --log-cli-level=DEBUG

Refer to ``.github/workflows/test.yml`` for the recommended testing environment.

Related repositories
--------------------

- `dtool-s3 `_ - storage broker interface to S3 object storage
- `dtool-ecs `_ - storage broker interface to ECS S3 object storage

Owner

Citation (CITATION.cff)

cff-version: 1.2.0
authors:
- family-names: Hörmann
  given-names: Johannes Laurin
  orcid: 0000-0001-5867-695X
- family-names: Pastewka
  given-names: Lars
  orcid:  0000-0001-8351-7336
title: dserver-notification-plugin
version: 0.4.1
date-released: 2024-07-09

GitHub Events

Packages