ml-hierarchical-confusion-matrix

Neo: Hierarchical Confusion Matrix Visualization (CHI 2022)

https://github.com/apple/ml-hierarchical-confusion-matrix

Science Score: 67.0%

This score indicates how likely this project is to be science-related based on various indicators:

✓
CITATION.cff file
Found CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
✓
.zenodo.json file
Found .zenodo.json file
✓
DOI references
Found 1 DOI reference(s) in README
✓
Academic publication links
Links to: arxiv.org
○
Committers with academic emails
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (8.3%) to scientific vocabulary

Keywords

confusion-matrix data-visualization machine-learning visual-analytics

Keywords from Contributors

interaction embedding numeric mesh interpretability profiles sequences generic projection standardization

Last synced: 6 months ago · JSON representation ·

Repository

Neo: Hierarchical Confusion Matrix Visualization (CHI 2022)

Basic Info

Host: GitHub
Owner: apple
License: other
Language: TypeScript
Default Branch: main
Homepage: https://apple.github.io/ml-hierarchical-confusion-matrix/
Size: 2.31 MB

Statistics

Stars: 312
Watchers: 12
Forks: 33
Open Issues: 5
Releases: 4

Topics

confusion-matrix data-visualization machine-learning visual-analytics

Created almost 4 years ago · Last pushed 7 months ago

Metadata Files

Readme Contributing License Code of conduct Citation

Neo: Hierarchical Confusion Matrix

The confusion matrix, a ubiquitous visualization for helping people evaluate machine learning models, is a tabular layout that compares predicted class labels against actual class labels over all data instances. Neo is a visual analytics system that enables practitioners to flexibly author and interact with hierarchical and multi-output confusion matrices, visualize derived metrics, renormalize confusions, and share matrix specifications.

This code accompanies the research paper:

Neo: Generalizing Confusion Matrix Visualization to Hierarchical and Multi-Output Labels
Jochen Görtler, Fred Hohman, Dominik Moritz, Kanit Wongsuphasawat, Donghao Ren, Rahul Nair, Marc Kirchner, Kayur Patel
ACM Conference on Human Factors in Computing Systems (CHI), 2022.
Paper, Live demo, Video, Video Preview, Code

Documentation

You can embed our confusion matrix visualization into your own project. There are two ways to use it.

NPM

Install with npm install --save @apple/hierarchical-confusion-matrix or yarn add @apple/hierarchical-confusion-matrix.

Then you can import the module in your project

```js import confMat from '@apple/hierarchical-confusion-matrix';

const spec = { classes: ['root'], };

const confusions = [ { actual: ['root:a'], observed: ['root:a'], count: 1, }, { actual: ['root:a'], observed: ['root:b'], count: 2, }, { actual: ['root:b'], observed: ['root:a'], count: 3, }, { actual: ['root:b'], observed: ['root:b'], count: 4, }, ];

confMat.embed('matContainer', spec, confusions); ```

Embed the Compiled File

If you prefer to load the compiled JavaScript directly, you have to compile it. To do this, run yarn install and copy the public/confMat.js into your project. Here is a simple example of a small confusion matrix:

```html <!DOCTYPE html> Neo: Hierarchical Confusion Matrix

<body>
    <div id="matContainer"></div>
    <script src="confMat.js"></script>
    <script>
        const spec = {
            classes: ['root'],
        };

        const confusions = [
            {
                actual: ['root:a'],
                observed: ['root:a'],
                count: 1,
            },
            {
                actual: ['root:a'],
                observed: ['root:b'],
                count: 2,
            },
            {
                actual: ['root:b'],
                observed: ['root:a'],
                count: 3,
            },
            {
                actual: ['root:b'],
                observed: ['root:b'],
                count: 4,
            },
        ];

        confMat.embed('matContainer', spec, confusions);
    </script>
</body>

```

Specification

You can find all the options that you can pass via the spec argument in src/specification.ts.

Loaders

The different loaders can be found in src/loaders, which include loading data from json, csv, vega, and a synthetic example synth for testing.

Confusion Data Format Examples

Example 1: Conventional Confusions

The confusions for data with actual labels of fruit:lemon that are incorrectly predicted as fruit:apple, of which there are count 1 of them.

json { "actual": ["fruit:lemon"], "observed": ["fruit:apple"], "count": 1 }

Example 2: Hierarchical Confusions

The confusions for hierarchical data with actual labels of fruit:citrus:lemon that are incorrectly predicted as fruit:pome:apple, of which there are count 2 of them. Note : denotes hierarchies.

json { "actual": ["fruit:citrus:lemon"], "observed": ["fruit:pome:apple"], "count": 2 }

Example 3: Multi-output Confusions

The confusions for multi-output data with actual labels of fruit:lemon,taste:sweet that are incorrectly predicted as fruit:apple,taste:sour, of which there are count 3 of them. Note , denotes multi-ouput labels.

json { "actual": ["fruit:lemon", "taste:sweet"], "observed": ["fruit:apple", "taste:sour"], "count": 3 }

Example 4: Hierarchical and Multi-output Confusions

The confusions for hierarchical and multi-output data with actual labels of fruit:citrus:lemon,taste:sweet,ripeness:ripe that are incorrectly predicted as fruit:pome:apple,taste:sour,ripeness:not-ripe, of which there are count 4 of them.

json { "actual": [ "fruit:citrus:lemon", "taste:sweet", "ripeness:ripe" ], "observed": [ "fruit:pome:apple", "taste:sour" "ripeness:not-ripe" ], "count": 4 }

See fruit.json for a complete example of confusions for a hierarchical fruit, taste, and ripeness classification model.

Development

Build:

sh yarn install yarn build

Test:

sh yarn test:unit

Dev Server:

sh yarn dev

Lint & Fix:

sh yarn lint

Contributing

When making contributions, refer to the CONTRIBUTING guidelines and read the CODE OF CONDUCT.

BibTeX

To cite our paper, please use:

bibtex @inproceedings{goertler2022neo, title={Neo: Generalizing Confusion Matrix Visualization to Hierarchical and Multi-Output Labels}, author={Görtler, Jochen and Hohman, Fred and Moritz, Dominik and Wongsuphasawat, Kanit and Ren, Donghao and Nair, Rahul and Kirchner, Marc and Patel, Kayur}, booktitle={Proceedings of the SIGCHI Conference on Human Factors in Computing Systems}, year={2022}, organization={ACM}, doi={10.1145/3491102.3501823} }

License

This code is released under the LICENSE terms.

Owner

Name: Apple
Login: apple
Kind: organization
Location: Cupertino, CA

Website: https://apple.com
Repositories: 305
Profile: https://github.com/apple

Citation (CITATION.cff)

cff-version: 1.2.0
message: 'If you use this software, please cite it as below.'
authors:
    - family-names: 'Görtler'
      given-names: 'Jochen'
    - family-names: 'Hohman'
      given-names: 'Fred'
    - family-names: 'Moritz'
      given-names: 'Dominik'
title: 'Neo: Hierarchical Confusion Matrix'
version: 1.0.0
date-released: 2022-04-18
url: 'https://github.com/apple/ml-hierarchical-confusion-matrix'
preferred-citation:
    type: conference-article
    authors:
        - family-names: 'Görtler'
          given-names: 'Jochen'
        - family-names: 'Hohman'
          given-names: 'Fred'
        - family-names: 'Moritz'
          given-names: 'Dominik'
        - family-names: 'Wongsuphasawat'
          given-names: 'Kanit'
        - family-names: 'Ren'
          given-names: 'Donghao'
        - family-names: 'Nair'
          given-names: 'Rahul'
        - family-names: 'Kirchner'
          given-names: 'Marc'
        - family-names: 'Patel'
          given-names: 'Kayur'
    doi: '10.1145/3491102.3501823'
    journal: 'Proceedings of the SIGCHI Conference on Human Factors in Computing Systems'
    title: 'Neo: Generalizing Confusion Matrix Visualization to Hierarchical and Multi-Output Labels'
    year: 2022
    organization: 'ACM'

GitHub Events

Total

Issues event: 2
Watch event: 10
Delete event: 38
Issue comment event: 1
Push event: 43
Pull request review event: 36
Pull request event: 74
Fork event: 8
Create event: 33

Last Year

Issues event: 2
Watch event: 10
Delete event: 38
Issue comment event: 1
Push event: 43
Pull request review event: 36
Pull request event: 74
Fork event: 8
Create event: 33

Committers

Last synced: 9 months ago

All Time

Total Commits: 215
Total Committers: 4
Avg Commits per committer: 53.75
Development Distribution Score (DDS): 0.135

Past Year

Commits: 50
Committers: 2
Avg Commits per committer: 25.0
Development Distribution Score (DDS): 0.02

Top Committers

Name	Email	Commits
dependabot[bot]	4****]	186
Dominik Moritz	d**z@g**m	15
Jochen Görtler	g****r	8
Fred Hohman	f**n@g**m	6

Issues and Pull Requests

Last synced: 6 months ago

All Time

Total issues: 9
Total pull requests: 235
Average time to close issues: 3 months
Average time to close pull requests: 5 days
Total issue authors: 8
Total pull request authors: 6
Average comments per issue: 1.0
Average comments per pull request: 0.11
Merged pull requests: 210
Bot issues: 1
Bot pull requests: 215

Past Year

Issues: 3
Pull requests: 72
Average time to close issues: 41 minutes
Average time to close pull requests: about 1 hour
Issue authors: 2
Pull request authors: 1
Average comments per issue: 0.0
Average comments per pull request: 0.0
Merged pull requests: 64
Bot issues: 1
Bot pull requests: 72

View more stats

Top Authors

Issue Authors

xnought (2)
ElliotNguyen68 (1)
hanxiaotian (1)
grtlr (1)
dependabot[bot] (1)
bhishanpdl (1)
23pointsNorth (1)
mintrinishant (1)

Pull Request Authors

dependabot[bot] (219)
domoritz (7)
grtlr (7)
fredhohman (4)
mkery (1)
tafsiri (1)

Top Labels

Issue Labels

enhancement (2) bug (1) dependencies (1) javascript (1)

Pull Request Labels

dependencies (219) javascript (203) github_actions (6)

Dependencies

package.json npm

@babel/core ^7.18.10 development
@babel/preset-typescript ^7.18.6 development
@rollup/plugin-commonjs ^22.0.2 development
@rollup/plugin-json ^4.1.0 development
@rollup/plugin-node-resolve ^13.1.3 development
@rollup/plugin-typescript ^8.4.0 development
@tsconfig/svelte ^3.0.0 development
@types/cwise ^1.0.4 development
@types/d3-array ^3.0.2 development
@types/d3-dsv ^3.0.0 development
@types/d3-scale ^4.0.2 development
@types/jest ^27.4.1 development
@types/lodash ^4.14.179 development
@types/ndarray ^1.0.11 development
@types/node ^18.7.14 development
@typescript-eslint/eslint-plugin ^5.32.0 development
@typescript-eslint/parser ^5.30.3 development
babel-jest ^29.0.1 development
eslint ^8.21.0 development
eslint-plugin-jest ^26.5.3 development
eslint-plugin-json ^3.1.0 development
eslint-plugin-svelte3 ^4.0.0 development
gh-pages ^4.0.0 development
jest ^27.5.1 development
npm-run-all ^4.1.5 development
rollup ^2.79.0 development
rollup-plugin-css-only ^3.1.0 development
rollup-plugin-livereload ^2.0.5 development
rollup-plugin-svelte ^7.1.0 development
rollup-plugin-terser ^7.0.2 development
sirv-cli ^2.0.2 development
svelte ^3.46.4 development
svelte-check ^2.8.0 development
svelte-preprocess ^4.10.4 development
ts-jest ^27.1.5 development
tslib ^2.3.1 development
typescript ^4.7.4 development
d3-array ^2.11.0
d3-dsv ^3.0.1
d3-scale ^4.0.2
lodash ^4.17.21
ndarray ^1.0.19
ndarray-ops ^1.2.2
vega-scale ^7.1.1

yarn.lock npm

637 dependencies

.github/workflows/merge-dependabot.yml actions

akheron/dependabot-cron-action v1 composite

.github/workflows/test.yml actions

actions/checkout v3 composite
actions/setup-node v3 composite

.github/workflows/static.yml actions

actions/checkout v4 composite
actions/configure-pages v3 composite
actions/deploy-pages v2 composite
actions/setup-node v3 composite
actions/upload-pages-artifact v2 composite

ml-hierarchical-confusion-matrix

Science Score: 67.0%

Keywords

Keywords from Contributors

Repository

Basic Info

Statistics

Topics

Metadata Files

README.md

Neo: Hierarchical Confusion Matrix

Documentation

NPM

Embed the Compiled File

Specification

Loaders

Confusion Data Format Examples

Example 1: Conventional Confusions

Example 2: Hierarchical Confusions

Example 3: Multi-output Confusions

Example 4: Hierarchical and Multi-output Confusions

Development

Contributing

BibTeX

License

Owner

Citation (CITATION.cff)

GitHub Events

Total

Last Year

Committers

All Time

Past Year

Top Committers

Issues and Pull Requests

All Time

Past Year

Top Authors

Issue Authors

Pull Request Authors

Top Labels

Issue Labels

Pull Request Labels

Dependencies