Data

Tools

Indexes

Applications

Experiments

Open Source Science

Recent Releases of Council Data Project

Council Data Project - Whisper and GCP Compute Runners

CouncilDataProject cdp-backend v4.0.0

:warning: :warning: This is a major breaking release. Instance maintainers should update the instance with just update-from-cookiecutter. :warning: :warning:

You should re-read through the SETUP/README.md document as there is some new minor configuration required. Specifically the new PERSONAL_ACCESS_TOKEN and Quote Increase request should be the only things that need to be updated for existing instances.

You should also lower how often your CRON event gather runs prior to running just update-from-cookiecutter. All of the instances maintained by the CDP Core Team will be lowered to running only once per day.

Council Data Project is a backend, frontend, and cookiecutter deployment for creating a whole database, storage system, and website, for archiving, exploring, and tracking municipal council action.

This library, cookiecutter-cdp-deployment ties together multiple projects to make a single deployable infrastructure.

v4.0.0

There are two main changes for this release.

  1. We are swapping out Google Speech-to-Text for OpenAIs Whisper.

Specifically, we are using a forked version called faster-whisper. This new speech-to-text model performs much better (ranging from ~3.6% word-error-rate to ~9% word-error-rate on long audio files).

To use this new model efficiently, we need access to a GPU. Since GitHub Actions do not have GPUs available, we are using a system which spins up a Google Cloud Compute Engine instance, connects to it, runs our job, and then tears it down all in the course of a single GitHub Action workflow. From multiple tests, this should be a reduction in cost and processing time however with this release we will do more testing to get a better estimate.

  1. We have switched from MIT to MPLv2 License.

Unless you are trying to fork our code and take it private, this won't affect you.

Scientific Software - Peer-reviewed - Python
Published by evamaxfield about 3 years ago

Council Data Project - Parametrizable Event Gather CRON and Default Timedelta

Council Data Project v3.2.1

Council Data Project is a backend, frontend, and cookiecutter deployment for creating a whole database, storage system, and website, for archiving, exploring, and tracking municipal council action.

This library, cookiecutter-cdp-deployment is a cookiecutter template to deploy a whole new CDP infrastructure.

There are no new library changes from cdp-backend or cdp-frontend this release. This is a release that simply adds new parameters to the cookiecutter to make it easier to randomize CRON event gather schedules (via deployment bot), or provide a specific CRON from cookiecutter CLI, as well as set the default event gather timedelta lookback (how many days are checked for events each time the pipeline runs).

Those changes mean that the template also needed to be updated to utilize the new infrastructure deployment strategy.

:warning: :warning:

This release is a "breaking" change for instances.

Instance maintainers will want to do the following:

  • Add a value to the .cookiecutter.yaml file called: event_gather_timedelta_lookback_days with an integer value for the number of days to check during the event gather. The default (and current value prior to this update to make it parametrizable) is 2 for two days. To decide what this parameter should be, a general rule of thumb is: "how long does it take for the municipality to post events"
  • Add a value to the .cookiecutter.yaml file called: event_gather_cron with a CRON string for how often to run the event gather pipeline. The default (and current value prior to this update to make it parametrizable) is "26 0,6,12,18 * * *". Which means: "run the pipeline at 12:26am, 6:26am, 12:26pm and 6:26pm UTC everyday". A general rule of thumb is: "if the event gather timedelta is large, the pipeline doesn't need to run as often."
  • Run just update-from-cookiecutter or make update-from-cookiecutter

:warning: :warning:

Full Changelog: https://github.com/CouncilDataProject/cookiecutter-cdp-deployment/compare/v3.2.0...v3.2.1

Scientific Software - Peer-reviewed - Python
Published by evamaxfield over 3 years ago

Council Data Project - New Infrastructure Management and Admin Updates

Council Data Project v3.2.0

Council Data Project is a backend, frontend, and cookiecutter deployment for creating a whole database, storage system, and website, for archiving, exploring, and tracking municipal council action.

This library, cookiecutter-cdp-deployment is a cookiecutter template to deploy a whole new CDP infrastructure.

The recent changes to cdp-backend is meant to fix the currently broken infrastructure deployment system. We are ripping out all of Pulumi and using gcloud, gsutil, and firebase CLI tooling to achieve the same result (which are all first-party tools so should be better supported).

Those changes mean that the template also needed to be updated to utilize the new infrastructure deployment strategy.

:warning: :warning: This release is a "breaking" change for instances. Instance maintainers will want to update the instance with make update-from-cookiecutter. :warning: :warning:

What's Changed

  • bugfix/infra-stack by @evamaxfield in https://github.com/CouncilDataProject/cookiecutter-cdp-deployment/pull/111

Full Changelog: https://github.com/CouncilDataProject/cookiecutter-cdp-deployment/compare/v3.1.1...v3.2.0

Scientific Software - Peer-reviewed - Python
Published by evamaxfield over 3 years ago

Council Data Project - New Indexing Workflow

This change utilizes the new indexing pipelines for events! Instance maintainers will need to run make update-from-cookiecutter to upgrade.

What's Changed

  • feature/split-indexing-into-multi-step-workflow by @JacksonMaxfield in https://github.com/CouncilDataProject/cookiecutter-cdp-deployment/pull/108

Full Changelog: https://github.com/CouncilDataProject/cookiecutter-cdp-deployment/compare/v3.1.0...v3.1.1

Scientific Software - Peer-reviewed - Python
Published by evamaxfield over 3 years ago

Council Data Project - Legislation Tracking and Easier Deployment Bots!

The main reason for this version bump is due to https://github.com/CouncilDataProject/cdp-frontend/releases/tag/v3.1.0 Which introduces quite a lot but mainly the legislation tracking page!

There is a breaking change introduced in this release but it is minor and should be easily handlable. See the release notes in cdp-frontend linked above for details.

There are also many changes to this repo itself but they are largely administrative. Specifically the GitHub Action deployment bots now require timezone and many bugs have been fixed.

Full Changelog: https://github.com/CouncilDataProject/cookiecutter-cdp-deployment/compare/v3.0.6...v3.1.0

Scientific Software - Peer-reviewed - Python
Published by evamaxfield over 3 years ago

Council Data Project - More Docs and Frontend Optimizations

This release brings some better docs to the cookiecutter from Smai! Thanks Smai!

Additionally, this brings a lot of frontend optimizations but with those optimizations comes a requirement on instance maintainers to rerun the cookiecutter. The main culprit here is that we have upgraded the frontend from node v12 to node v16.

Instead of listening to dependabot, please run make update-from-cookiecutter to pull in these changes.

What's Changed

  • docs/new-deployment-clarifications-around-event-gathering-and-cost by @smai-f in https://github.com/CouncilDataProject/cookiecutter-cdp-deployment/pull/81
  • admin/upgrade-deps-and-build-eng by @JacksonMaxfield in https://github.com/CouncilDataProject/cookiecutter-cdp-deployment/pull/84

New Contributors

  • @smai-f made their first contribution in https://github.com/CouncilDataProject/cookiecutter-cdp-deployment/pull/81

Full Changelog: https://github.com/CouncilDataProject/cookiecutter-cdp-deployment/compare/v3.0.5...v3.0.6

Scientific Software - Peer-reviewed - Python
Published by evamaxfield almost 4 years ago

Council Data Project - Process YouTube Videos, Embed YouTube URLs for Video Source, and Paginate Voting Records

Contains minimum version bumps for fresh CDP instance to backend v3.0.9 and frontend v3.0.7 which contain features for handling YouTube video URLs and on the frontend pagination of voting records due to simply having more data.

Some additional bug fixes in this repo (related to GitHub Issue Bot) and in both libraries (Index Creation, Localization, etc.) are additionally added in these updates.

Scientific Software - Peer-reviewed - Python
Published by evamaxfield almost 4 years ago

Council Data Project - Person Page and Database Patches for Programmatic UX

See changelog for backend here: https://github.com/CouncilDataProject/cdp-backend/releases/tag/v3.0.4

See changelog for frontend here: https://github.com/CouncilDataProject/cdp-frontend/releases/tag/v3.0.4

There were also changes to the deployment instance cookiecutter that document how to rollout the backend changes. These are optional but recommended.

Full Cookiecutter Changelog: https://github.com/CouncilDataProject/cookiecutter-cdp-deployment/compare/v3.0.3...v3.0.4

Scientific Software - Peer-reviewed - Python
Published by evamaxfield about 4 years ago

Council Data Project - GSR Improvements and Frontend Bugfixes

Scientific Software - Peer-reviewed - Python
Published by evamaxfield about 4 years ago

Council Data Project - Event Gather Pipeline Fix and Web App Build Fix

Event Gather Pipeline bugfix addresses a bug found when trying to download a picture for person or seat and the download being rejected by the host. Web app build fix solves the whole web app build and deploy due to upstream deps / warnings from react-scripts.

Scientific Software - Peer-reviewed - Python
Published by evamaxfield about 4 years ago

Council Data Project - Hashing Fix and Frontend UX Cleanup

See cdp-backend notes: https://github.com/CouncilDataProject/cdp-backend/releases/tag/v3.0.1 See cdp-frontend notes: https://github.com/CouncilDataProject/cdp-frontend/releases/tag/v3.0.1

Scientific Software - Peer-reviewed - Python
Published by evamaxfield about 4 years ago

Council Data Project - An Early Release

This is an early release of v3!

This release is the culmination of multiple years of work in expanding and making our tools and infrastructure more easily deployable and ship with more features.

While not everything is completed yet (v3.0.1 just around the corner) this release has everything needed to get an instance up-and-running and generally usable for archival and event discovery.

Notably the features this release is shipping with are: * Full infrastructure setup and management through cdp-backend * Pipeline for scraping and processing events (with option of providing custom datetime to GitHub) * Pipeline for indexing processed events * Pipeline for processing a special event such as a debate or forum (with option of providing custom datetime to GitHub) * A web application for searching and browsing of events with attached minutes items and voting record

And all of this is wrapped up into this cookiecutter for easy deployment and management.

Thanks to everyone who has contributed to the project along the way. * @JacksonMaxfield * @tohuynh * @isaacna * @nniiicc * @hawkticehurst * @BrianL3 * @sarahjliu * @dphoria * @Shak2000 * Melanie White * Neha Singh * Steve Stremsterfer * @zmbc * @andrewlubrino * @trungvngo * @isometimescode * @dhanya-shraddha * Emily Gilles * Katlyn M. F. Greene * Madeleine Farrer * All the many previous volunteers from DemocracyLab Hackathons, Open Seattle, and Textio

More soon!

Scientific Software - Peer-reviewed - Python
Published by evamaxfield about 4 years ago

Council Data Project - Paper Fixes, Zenodo Fixes, Dependabot Fixes

Fixes the titles of papers referenced in the paper bibliography, fixes the Zenodo integration by adding the authors from the paper as contributors, fixes the dependabot for generated repositories to include monitoring for infra deps.

Full Changelog: https://github.com/CouncilDataProject/cookiecutter-cdp-deployment/compare/v0.0.4...v0.0.5

Scientific Software - Peer-reviewed - Python
Published by evamaxfield about 4 years ago

Council Data Project - Better Dependabot, Frontend Updates, and Paper Release

This release includes fixes for dependabot autofixes for generated instance repositories, minor frontend updates, and the inclusion of the JOSS paper!

What's Changed

  • admin/fix-og-metadata by @JacksonMaxfield in https://github.com/CouncilDataProject/cookiecutter-cdp-deployment/pull/64
  • bugfix/quickstart-readme-after-model-change by @JacksonMaxfield in https://github.com/CouncilDataProject/cookiecutter-cdp-deployment/pull/63
  • admin/joss-paper by @JacksonMaxfield in https://github.com/CouncilDataProject/cookiecutter-cdp-deployment/pull/59
  • admin/resolve-joss-reviewer-comments by @JacksonMaxfield in https://github.com/CouncilDataProject/cookiecutter-cdp-deployment/pull/67
  • admin/explicit-version-pin-for-dependabot by @isaacna in https://github.com/CouncilDataProject/cookiecutter-cdp-deployment/pull/70
  • admin/update-acknowledgements-in-joss-paper by @JacksonMaxfield in https://github.com/CouncilDataProject/cookiecutter-cdp-deployment/pull/68

Full Changelog: https://github.com/CouncilDataProject/cookiecutter-cdp-deployment/compare/v0.0.3...v0.0.4

Scientific Software - Peer-reviewed - Python
Published by evamaxfield about 4 years ago

Council Data Project - Web App In-Place

The web app from https://github.com/CouncilDataProject/cdp-frontend is now imported and used in the generated repository!

There are very few dependencies listed in the web/package.json to get this to work which was the original goal.

Minor bugs remain but a large checkpoint has been reached in cdp v3!

Scientific Software - Peer-reviewed - Python
Published by evamaxfield over 4 years ago

Council Data Project - Infra Update: Set Default Object Access Control

Updates the min infrastructure cdp-backend version to pull in the addition of default object access control.

Because of the above and with changes here to the deployment process we now also have a section on the generated README for Python API access to data and storage.

Scientific Software - Peer-reviewed - Python
Published by evamaxfield over 4 years ago

Council Data Project - Relatively Stable Backend

All major backend processing pipelines and features have made it into cdp-backend and have implementations of the pipelines have been added to this release.

Scientific Software - Peer-reviewed - Python
Published by evamaxfield over 4 years ago