connectivity search backend

This django application powers the API available at https://search-api.het.io/.

Environment

This repository uses conda to manage its environment as specified in environment.yml. Install the environment with:

shell conda env create --file=environment.yml

Then use conda activate hetmech-backend and conda deactivate to activate or deactivate the environment.

Secrets

Users must supply dj_hetmech/secrets.yml with the database connection information and two optional parameters for Django settings. See dj_hetmech/secrets-template.yml for what fields should be defined. These secrets will determine whether django connects to a local database or a remote database and other security settings in Django.

Notebooks

Use the following command to launch Jupyter Notebook in your browser for interactive development:

shell python manage.py shell_plus --notebook

Server

A local development server can be started with the command:

shell python manage.py runserver

This exposes the API at http://localhost:8000/v1/.

Database

This project uses a PostgreSQL database. The deployed version of this application uses a remote database. Public read-only access is available with the following configuration:

yaml name: connectivity_db user: read_only_user password: tm8ut9uzqx7628swwkb9 host: search-db.het.io port: 5432

To erect a new database locally for development, run:

```shell

https://docs.docker.com/samples/library/postgres/

docker run \ --name connectivitydb \ --env POSTGRESDB=connectivitydb \ --env POSTGRESUSER=djhetmech \ --env POSTGRESPASSWORD=not_secure \ --volume "$(pwd)"/database:/var/lib/postgresql/data \ --publish 5432:5432 \ --detach \ postgres:12.4 ```

Populating the database

To populate the database from scratch, use the populate_database management command (source). Here is an example workflow:

```shell

migrate database to the current Django models

python manage.py makemigrations python manage.py migrate --run-syncdb

view the populate_database usage docs

python manage.py populate_database --help

wipe the existing database (populate_database assumes empty tables)

python manage.py flush --no-input

populate the database (will take a long time)

python manage.py populate_database --max-metapath-length=3 --reduced-metapaths --batch-size=12000

output database information and table summaries

python manage.py database_info ```

Another option to load the database is to import it from the connectivity-search-pg_dump.sql.gz database dump, which will save time if you are interested in loading the full database (i.e. without --reduced-metapaths). This 5 GB file is available on Zenodo (TODO: update latest database dump to Zenodo).

To load connectivity-search-pg_dump.sql.gz into a new database, modify the following command:

shell zcat hetmech-pg_dump.sql.gz | psql --user=dj_hetmech --dbname=connectivity_db --host=HOST

connectivity-search-pg_dump.sql.gz was exported from the development Docker database with the command:

```shell docker exec connectivitydb \ pgdump \ --host=localhost --username=djhetmech --dbname=connectivitydb \ --create --clean \ --compress=8 \

connectivity-search-pg_dump.sql.gz ```