data-understand

Motivation

As data scientists and machine learning engineers, we are often required to execute various data science tasks like loading up the dataset into a pandas dataframe, inspecting the columns/rows in the dataset, visualizing the distribution of values, finding feature correlations and determining if there are any sort of imbalances in the dataset. Often these tasks are repetitive and involve creating multiple jupyter notebooks and we have to manage these jupyter notebooks separately with different handles to the location of input dataset. How about you have one tool which could take the directory location of your dataset and generate the boring aforementioned logic for you to execute and learn the same insights about your dataset. All you need to do is to install this tool in your local python environment and then execute the tool from a command line.

Installation

You can install the package data-understand from pypi using the following command:-

pip install data-understand

Usage

Once you have installed the tool locally, you can then look at the various options of the CLI tool:-

```

data_understand -h

======================================================================================================================== usage: data_understand [-h] [-f FILE_NAME] [-t TARGET_COLUMN] [-p] [-j]

data.understand CLI

options: -h, --help show this help message and exit -f FILENAME, --filename FILENAME Directory path to CSV file -t TARGETCOLUMN, --targetcolumn TARGETCOLUMN Target column name -p, --generatepdf Generate PDF file for understanding of data -j, --generatejupyter_notebook Generate jupyter notebook file for understanding of data ```

Notebook and PDF report generation

In order to generate both PDF report and jupyter notebook you can execute the following CLI command:-

```

dataunderstand --filename adultdataset.csv --targetcolumn income --generatepdf --generatejupyter_notebook

======================================================================================================================== The parsed arguments are:- filename: adultdataset.csv targetcolumn: income generatepdf: True generatejupyternotebook: True

Time taken: 0.0 min 0.0012356000000863787 sec

Generating PDF report and jupyter notebook

Generating PDF report for the dataset in adultdataset.csv Successfully generated PDF report for the dataset in adultdataset.csv at adult_dataset.csv.pdf

Time taken: 0.0 min 7.363417799999979 sec

======================================================================================================================== Generating jupyter notebook for the dataset in adultdataset.csv Successfully generated jupyter notebook for the dataset in adultdataset.csv at adult_dataset.csv.ipynb

Time taken: 0.0 min 0.053841799999986506 sec

Successfully generated PDF report and jupyter notebook

Time taken: 0.0 min 7.485209299999951 sec

```

This would generate the jupyter notebook and PDF report in the same directory location as your dataset. You can execute the cells in the jupyter notebook to generate various insights and graphs on the fly or you can read through the PDF report to learn about various aspects of your dataset.

Repos using data-understand to generate notebooks and PDF reports

• understanding-datasets