Azimuth, an open-source dataset and error analysis tool for text classification, with love from ServiceNow.

Overview

Azimuth is an open source application that helps AI practitioners and data scientists better understand their dataset and model predictions by performing thorough dataset and error analyses. The application leverages different tools, including robustness tests, semantic similarity analysis and saliency maps, unified by concepts such as smart tags and proposed actions. It currently only supports text classification use cases.

Published Paper at EMNLP

Our paper was accepted as a system demo at EMNLP 2022. It is available here.

Documentation

Our documentation contains all the instructions and references to use Azimuth.

• Getting Started contains all the instructions to install and launch the app.
• Key Concepts explains the different concepts and analyses that are provided in Azimuth to perform dataset and error analysis.
• User Guide goes screen per screen to explain the different interactions and visualizations available.
• Reference details the config file and the different contracts which allow configuring Azimuth with different datasets and pipelines.
• Development guides on how to develop and contribute to the repo.

YouTube Playlist

Run Our Demo

After installing the requirements, as explained in Installation, run the following command to run the demo. More details in Learn Basics.

pip install gdown make download_demo make CFG_PATH=/config/development/clinc/conf.json launch

Once the startup tasks are completed, you will be able to access Azimuth at http://localhost:8080. To learn more about Azimuth's features, refer to our Key Concepts and our User Guide.

To use your own dataset and models, please refer to Run on Your Use Case.

How to Contribute?

Refer to our CONTRIBUTING.md.

Code Documentation

At different places in the code, you'll find README files explaining further how the back end works. Don't hesitate to add other README files in appropriate places, and don't forget to edit them if you change these components.

• In azimuth, explaining the main back-end components.
• In azimuth/modules, explaining how modules work.
• In azimuth/routers, explaining how to add new routes.

Repo Structure

bash ├── azimuth # Back End │   ├── modules │   │   └── Where core capabilities are implemented │   ├── plots │   │   └── Plotting functions │   ├── routers │   │   └── FastAPI routers │   ├── types │   │   └── Specific pydantic types that this application is based on │   ├── utils │   │   └── Extra utilities ├── azimuth_shr │   └── Custom code that is shareable. ├── config │   └── Config files for different data/models ├── docs │   └── User documentation ├── tests │   └── Unit/integration tests ├── webapp │   └── Front End ├── docker-compose.yml # Where the config and images are specified. ├── docker-compose-gpu.yml # Extension to use Azimuth with gpus. └── runner.py # Application entrypoint.

Contact

To contact us, join our slack or send us an email!

License

The package is licensed by ServiceNow, Inc. under the Apache 2.0 license. See LICENSE for more details.

Vulnerability Reporting

Please notify psirt-oss@servicenow.com regarding any vulnerability reports in addition to following current reporting procedure.