cdp-data

Data Utilities and Processing Generalized for All CDP Instances

Installation

Stable Release: pip install cdp-data
Development Head: pip install git+https://github.com/CouncilDataProject/cdp-data.git

Documentation

For full package documentation please visit councildataproject.github.io/cdp-data.

Quickstart

Pulling Datasets

Install basics: pip install cdp-data

Transcripts and Session Data

```python from cdp_data import CDPInstances, datasets

ds = datasets.getsessiondataset( infrastructureslug=CDPInstances.Seattle, startdatetime="2021-01-01", store_transcript=True, ) ```

Transcript Schema and Usage

It may be useful to look at our transcript model documentation.

Transcripts can be read into memory and processed as an object:

```python from cdpbackend.pipeline.transcriptmodel import Transcript

Read the file as a Transcript object

with open("transcript.json", "r") as openf: transcript = Transcript.fromjson(open_f.read())

Navigate the object

for sentence in transcript.sentences: if "clerk" in sentence.text.lower(): print(f"{sentence.index}, {sentence.start_time}: '{sentence.text}') ```

If you do not want to do this processing in Python or prefer to work with a DataFrame, you can convert transcripts to DataFrames like so:

```python from cdp_data import datasets

assume that transcript is the same transcript as the prior code snippet

sentences = datasets.converttranscriptto_dataframe(transcript) ```

You can also do this conversion (and storage of the coverted transcript) for all transcripts in a session dataset during dataset construction with the store_transcript_as_csv parameter.

```python from cdp_data import CDPInstances, datasets

ds = datasets.getsessiondataset( infrastructureslug=CDPInstances.Seattle, startdatetime="2021-01-01", storetranscript=True, storetranscriptascsv=True, ) ```

This will store the transcript for each session as both JSON and CSV.

Voting Data

```python from cdp_data import CDPInstances, datasets

ds = dataset.getvotedataset( infrastructureslug=CDPInstances.Seattle, startdatetime="2021-01-01", ) ```

Data Definitions and Schema

Please refer to our database schema and our database model definitions for more information on CDP generated and archived data is structured.

Saving Datasets

Because we heavily rely on our database models for database interaction, in many cases, we default to returning the full fireo.models.Model object as column values.

These objects cannot be immediately stored to disk so we provide a helper to replace all model objects with their database IDs for storage.

This can be done directly if you already have a dataset you have been working with:

```python from cdp_data import datasets

data should be a pandas dataframe

dataset.save_dataset(data, "data.csv") ```

Or this can be premptively be done during dataset construction:

```python from cdp_data import CDPInstances, dataset

both getsessiondataset and getvotedataset

have a replace_py_objects parameter

sessions = datasets.getsessiondataset( infrastructureslug=CDPInstances.Seattle, replacepy_objects=True, )

votes = datasets.getvotedataset( infrastructureslug=CDPInstances.Seattle, replacepy_objects=True, ) ```

Plotting and Analysis

Install plotting support: pip install cdp-data[plot]

Ngram Usage over Time

```python from cdp_data import CDPInstances, keywords, plots

ngramusage = keywords.computengramusagehistory( CDPInstances.Seattle, startdatetime="2022-03-01", enddatetime="2022-10-01", ) grid = plots.plotngramusagehistories( ["police", "housing", "transportation"], ngramusage, lmplotkws=dict( # extra plotting params col="ngram", hue="ngram", scatterkws={"alpha": 0.2}, aspect=1.6, ), ) grid.savefig("seattle-keywords-over-time.png") ```

Development

See CONTRIBUTING.md for information related to developing the code.

MIT license