hyp3-flood-monitoring

⚠️ This repository has been permanently archived. ⚠️

This was a successful proof-of-concept. If the project were to resume in the future, it would need to be rewritten to avoid relying on the now-obsolete HyP3 Subscriptions feature.

Architecture overview

The Pacific Disaster Center (PDC) provides a Hazard API (see PDC Hazard API). The purpose of this project is to provide RTC products for areas of interest affected by active flood hazards. This is accomplished by maintaining a HyP3 subscription for each active flood hazard.

An AWS Lambda function runs periodically and executes the following steps:

1. Query the PDC Hazard API for a list of active flood hazards.
2. For each hazard:
   1. If there is no existing HyP3 subscription for the hazard, create one, and set its end datetime for a few hours into the future.
   2. Otherwise, update the existing HyP3 subscription with any parameters that have changed (e.g. AOI), and set its end datetime for a few hours into the future.

Note that when a hazard expires (becomes inactive), the following steps occur automatically:

1. The hazard disappears from the list of active hazards returned by the PDC Hazard API.
2. As a result, our system does not update the end datetime for the hazard's HyP3 subscription, and the subscription is soon disabled.

Note that a HyP3 subscription remains enabled for a few days beyond its end datetime, in case any new data becomes available that was acquired within the subscription's start and end datetime range. No jobs will be submitted for data acquired after the subscription's end datetime.

Additionally, a second AWS Lambda function runs periodically and copies any new RTC products that have been created by flood monitoring subscriptions into an S3 bucket for permanent archival.

Important constants

There are some important global constants defined in hyp3_floods.py:

• HAZARD_START_DATE_DELTA allows us to set a HyP3 subscription start date for slightly before the hazard start date, in case the hazard start date has an error margin.
• HAZARD_START_DATE_MINIMUM prevents setting a HyP3 subscription start date before the minimum date, so that we don't back-process a ton of data.

We attempted to choose reasonable values for these constants, but we are open to changing them on request from the PDC team or other key stakeholders.

Developer setup

conda env create -f environment.yml conda activate hyp3-flood-monitoring

Environment variables

You'll need to create a .env file to specify environment variables required for local development. You can create a single file named .env or multiple *.env files in the env/ directory, depending on whether you need to specify multiple different environments. Each line of the file should be of the form KEY='value'.

Below is a non-exhaustive list of some environment variables that you may want to set in your .env file. (Of course, you can leave your .env file blank for now, and simply add variables as needed. For example, you can try to run a particular script and allow it to complain about missing environment variables, and then add those variables to your .env file.)

• PDC_HAZARDS_AUTH_TOKEN: Authorization token for the PDC Hazard API (production); see PDC Hazard API.
• HYP3_URL: URL for the HyP3 API that you want to query, e.g. https://hyp3-test-api.asf.alaska.edu.
• EARTHDATA_USERNAME: Available in the tools_user_accounts secret in AWS Secrets Manager (in the HyP3 AWS account), via the secret key hyp3-flood-monitoring-edl-username.
• EARTHDATA_PASSWORD: Available in the tools_user_accounts secret in AWS Secrets Manager (in the HyP3 AWS account), via the secret key hyp3-flood-monitoring-edl-password.

PDC Hazard API

PDC provides a Hazard API:

• PDC Hazard API (production): https://sentry.pdc.org/hp_srv/
• PDC Hazard API (test): https://testsentry.pdc.org/hp_srv/

The automated flood monitoring system (including our test deployment) only interacts with the production Hazard API. When running queries manually, we as developers also interact only with the production Hazard API, unless the PDC team asks us to query their test API for a specific reason.

The Hazard API authorization tokens are available in the tools_user_accounts secret in AWS Secrets Manager (in the HyP3 AWS account), via the secret keys PDC Hazard API auth token (prod) and PDC Hazard API auth token (test).

Running the Lambda functions locally

You can locally execute the source code for the AWS Lambda functions, e.g. for debugging purposes. Each Lambda function is provided as a Python script with a command-line interface.

To show the help text for the hyp3-floods Lambda function (for creating and updating HyP3 subscriptions):

python hyp3-floods/src/hyp3_floods.py -h

To show the help text for the transfer-products Lambda function (for archiving products by copying them to an S3 bucket):

python transfer-products/src/transfer_products.py -h

Each Lambda function takes a .env file as a command-line argument (see Environment variables).

Additional scripts

Additional scripts are provided on the scripts branch.

Before running these scripts, create a .env file as described in Environment variables.

Also, make sure you have an AWS config profile for the HyP3 account.

You may want to run check_subscriptions.py periodically in order to verify that the flood monitoring system is working as expected:

AWS_PROFILE=hyp3 PYTHONPATH=${PWD}/hyp3-floods/src python scripts/check_subscriptions.py <dotenv_path>

The get_stats.py script is intended to run as a GitHub Actions workflow, though you can run it locally if you wish. Its purpose is to generate certain statistics describing the operation of the flood monitoring system, which are made available to key stakeholders.

To run the script via GitHub Actions, navigate to the Actions console for this repo, select the "Generate Stats" workflow for either test or prod (depending on whether you wish to generate stats for the test or prod flood monitoring system), and then manually run the workflow.

The script will make the stats available by publishing a message to an AWS SNS topic, which has been created manually (not via CloudFormation) in the HyP3 AWS account.