Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/chaoss/grimoirelab

GrimoireLab: platform for software development analytics and insights

https://github.com/chaoss/grimoirelab

Science Score: 49.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ✓
    DOI references
    Found 7 DOI reference(s) in README
  • ○
    Academic publication links
  • ✓
    Committers with academic emails
    3 of 62 committers (4.8%) from academic institutions
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (14.3%) to scientific vocabulary

Keywords

chaoss data-mining data-visualization grimoirelab insights metrics software-analytics

Keywords from Contributors

orchestration data-fetching data-sources perceval data-enrichment diversity-measurement generic community project-governance mentorship
Last synced: 5 months ago · JSON representation

Repository

GrimoireLab: platform for software development analytics and insights

Basic Info
Statistics
  • Stars: 548
  • Watchers: 21
  • Forks: 208
  • Open Issues: 69
  • Releases: 0
Topics
chaoss data-mining data-visualization grimoirelab insights metrics software-analytics
Created over 8 years ago · Last pushed 6 months ago
Metadata Files
Readme Changelog Contributing License Security Governance Roadmap Authors

README.md

GrimoireLab

grimoirelab-showcase

GrimoireLab is a CHAOSS toolset for software development analytics. It includes a coordinated set of tools to retrieve data from systems used to support software development (repositories), store it in databases, enrich it by computing relevant metrics, and make it easy to run analytics and visualizations on it.

You can learn more about GrimoireLab in the GrimoireLab tutorial, or visit the GrimoireLab website.

Metrics available in GrimoireLab are, in part, developed in the CHAOSS project. For more information regarding CHAOSS metrics, see the latest release at: https://chaoss.community/metrics/

Getting started

To ease the newcomer experience we are providing a default setup to analyze git activity for this repository. For this set up, there are several options to run GrimoireLab:

Using docker-compose

Requirements: * Software: git, docker client and docker compose. An example of working configuration: console root@test-68b8628f:~# git --version git version 2.17.1 root@test-68b8628f:~# docker --version Docker version 19.03.1, build 74b1e89 root@test-68b8628f:~# docker-compose --version docker-compose version 1.22.0, build f46880fe * Hardware: 2 CPU cores, 8GB memory RAM and enough virtual memory for Elasticsearch

Steps: 1. Clone this project: console git clone https://github.com/chaoss/grimoirelab 2. Go to docker-compose folder and run the following command: console cd grimoirelab/docker-compose docker-compose up -d

Your data will be ready after a while. The waiting time depends on the amount of data to fetch from a repo, for small repositories you can expect your data to be available after 10-15 minutes.

To view the data you can access OpenSearch Dashboards at http://localhost:8000 or use the API directly at http://localhost:9200.

To manage the identities you can access SortingHat at http://localhost:8000/identities/.

By default, OpenSearch Dashboards doesn't contain any visualization, but we have a set of default dashboards. To import them, open OpenSearch Dashboards, go to Stack Management > Saved Objects and import the files you want.

More details or troubleshooting in the docker-compose folder.

Using docker run

Requirements: * Software: git and docker client. An example of working configuration: console root@test-68b8628f:~# git --version git version 2.17.1 root@test-68b8628f:~# docker --version Docker version 19.03.1, build 74b1e89 * Hardware: 2 CPU cores, 8GB memory RAM and set * ElasticSearch and Kibana up and running. * SortingHat up and running

Steps: 1. Clone this project: console git clone https://github.com/chaoss/grimoirelab 2. Go to the project folder and run the following command: console cd grimoirelab docker run --net=host \ -v $(pwd)/default-grimoirelab-settings/projects.json:/home/grimoire/conf/projects.json \ -v $(pwd)/default-grimoirelab-settings/setup-docker.cfg:/home/grimoire/conf/setup.cfg \ -t grimoirelab/grimoirelab

Your dashboard will be ready after a while at http://localhost:8000. The waiting time depends on the amount of data to fetch from a repo, for small repositories you can expect your data to be visible in the dashboard after 10-15 minutes.

More details in the docker folder.

Using on FreeBSD

Recently FreeBSD has adopted Grimoirelab for their Bugzilla Analysis and have deployed Grimoirelab in their cluster. FreeBSD Foundation has been kind enough to share the process of running Grimoire on FreeBSD which is available here. This guide will also help who are interested in running Grimoirelab from source instead of deploying them with Docker.

Breaking changes

GrimoireLab 1.3.0. SortingHat permission groups.

Starting from GrimoireLab 1.3.0, creating new users in SortingHat requires assigning them to a permission group. By default, they will have read-only permissions. Please refer to the following documentation for instructions on how to update permissions: assign users to permission groups

GrimoireLab components

Currently, GrimoireLab toolkit is organized in the following repositories:

There are also some components built by the GrimoreLab community, which can be useful for you. Other related repositories are: * GrimoireLab Tutorial * GrimoireLab as a whole (this repository)

Contents of this repository

This repository is for content relevant to GrimoireLab as a whole. For example:

  • Issues for new features or bug reports that affect more than one GrimoireLab module. In this case, let's open an issue here, and when implementing the fix or the feature, let´s comment about the specific tickets in the specific modules that are used. For example, when supporting a new datasource, we will need patches (at least) in Perceval, GrimoireELK and panels. In this case, we would open a feature request (or the user story) for the whole case, an issue (and later a pull request) in Perceval for the data retriever, same for GrimoireELK for the enriching code, and same for panels.

  • Release notes for most GrimoireLab components (directory releases).

  • Docker container for showcasing GrimoireLab (directory docker). Includes a Dockerfile and configuration files for the GrimoireLab containers that can be used to demo the technology, and can be the basis for real deployments. See more information in the docker README.md file.

  • If you feel more comfortable with docker-compose, the docker-compose folder includes instructions and configuration files to deploy GrimoireLab using docker-compose command.

  • Source code of the GrimoireLab components is available in src. Each directory is a Git submodule, so its contents will not be available after cloning the repository. To fetch all the data, and get the latest version, you can run the following command: console git submodule update --init --remote

  • How releases of GrimoireLab are built and tested: Building

Citation

If you use GrimoireLab in your research papers, please refer to GrimoireLab: A toolset for software development analytics:

APA style:

Dueñas S, Cosentino V, Gonzalez-Barahona JM, del Castillo San Felix A, Izquierdo-Cortazar D, Cañas-Díaz L, Pérez García-Plaza A. 2021. GrimoireLab: A toolset for software development analytics. PeerJ Computer Science 7:e601 https://doi.org/10.7717/peerj-cs.601

BibTeX / BibLaTeX:

@Article{duenas2021:grimoirelab, author = {Dueñas, Santiago and Cosentino, Valerio and Gonzalez-Barahona, Jesus M. and del Castillo San Felix, Alvaro and Izquierdo-Cortazar, Daniel and Cañas-Díaz, Luis and Pérez García-Plaza, Alberto}, title = {GrimoireLab: A toolset for software development analytics}, journaltitle = {PeerJ Computer Science}, date = {2021-07-09}, volume = 7, number = {e601}, doi = {10.7717/peerj-cs.601}, url = {https://doi.org/10.7717/peerj-cs.601}}

Contributing

Contributions are welcome, please check the Contributing Guidelines.

Owner

  • Name: CHAOSS
  • Login: chaoss
  • Kind: organization

GitHub Events

Total
  • Create event: 30
  • Release event: 16
  • Issues event: 45
  • Watch event: 55
  • Delete event: 13
  • Issue comment event: 70
  • Push event: 51
  • Pull request review comment event: 2
  • Pull request review event: 29
  • Pull request event: 80
  • Fork event: 30
Last Year
  • Create event: 30
  • Release event: 16
  • Issues event: 45
  • Watch event: 55
  • Delete event: 13
  • Issue comment event: 70
  • Push event: 51
  • Pull request review comment event: 2
  • Pull request review event: 29
  • Pull request event: 80
  • Fork event: 30

Committers

Last synced: 8 months ago

All Time
  • Total Commits: 711
  • Total Committers: 62
  • Avg Commits per committer: 11.468
  • Development Distribution Score (DDS): 0.783
Past Year
  • Commits: 72
  • Committers: 21
  • Avg Commits per committer: 3.429
  • Development Distribution Score (DDS): 0.514
Top Committers
Name Email Commits
Santiago Dueñas s****s@b****m 154
Jesus M. Gonzalez-Barahona j****b@g****s 120
Alvaro del Castillo a****s@b****m 107
Jose Javier Merchante j****e@b****m 106
J. Manrique Lopez de la Fuente j****e@b****m 62
Valerio Cosentino v****s@b****m 36
Manrique Lopez j****e@g****m 15
Quan Zhou q****n@b****m 13
Venu Vardhan Reddy Tekula v****u@c****y 12
Georg J.P. Link l****g@g****m 9
David Moreno d****b@g****m 7
Luis Cañas Díaz l****s@b****m 5
dependabot[bot] 4****] 5
Alberto Martín a****n@b****m 4
Henrik Mitsch h****h@s****m 2
anajs a****5@g****m 2
alpgarcia a****a@g****m 2
Yehui Wang y****h@g****m 2
Ruth Cheesley r****y@a****m 2
Drashti d****7@g****m 2
Brent Beer b****r@g****m 2
Animesh Kumar 3****8 2
Boris Baldassari b****i 1
Benjamin Copeland b****n@c****k 1
Ayberk Tecimer t****k@g****m 1
Andrew McLachlan a****n@m****m 1
Andre Klapper a****r@w****g 1
Alice Sowerby a****y@h****m 1
sophia-IV 6****V 1
Muhammad Moinur Rahman m****n@b****m 1
and 32 more...
Committer Domains (Top 20 + Academic)
bitergia.com: 9 gsyc.es: 1 chaoss.community: 1 sonomotors.com: 1 acquia.com: 1 copeland.me.uk: 1 me.com: 1 wikimedia.org: 1 bofh.im: 1 porsche.de: 1 dischord.org: 1 umt.edu: 1 qq.com: 1 vmware.com: 1 singlemusic.co: 1 ow2.org: 1 seretalabs.com: 1 thunderbird.net: 1 linaker.se: 1 turing.ac.uk: 1 unomaha.edu: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time
  • Total issues: 163
  • Total pull requests: 229
  • Average time to close issues: over 1 year
  • Average time to close pull requests: about 2 months
  • Total issue authors: 67
  • Total pull request authors: 40
  • Average comments per issue: 3.42
  • Average comments per pull request: 0.71
  • Merged pull requests: 162
  • Bot issues: 0
  • Bot pull requests: 38
Past Year
  • Issues: 27
  • Pull requests: 94
  • Average time to close issues: 11 days
  • Average time to close pull requests: 4 days
  • Issue authors: 9
  • Pull request authors: 23
  • Average comments per issue: 0.96
  • Average comments per pull request: 0.69
  • Merged pull requests: 62
  • Bot issues: 0
  • Bot pull requests: 15
View more stats
Top Authors
Issue Authors
  • sduenas (38)
  • GeorgLink (14)
  • jsmanrique (7)
  • maltif (7)
  • canasdiaz (6)
  • loganknecht (5)
  • pilgrimage80 (4)
  • jjmerchante (4)
  • mhow2 (3)
  • krokoko (3)
  • marcofranssen (3)
  • bhack (2)
  • mohlatif227 (2)
  • GrayStranger (2)
  • AnishaM7 (2)
Pull Request Authors
  • jjmerchante (95)
  • dependabot[bot] (39)
  • sduenas (21)
  • GeorgLink (10)
  • andrew-mclachlan (5)
  • canasdiaz (2)
  • natarajan-chidambaram (2)
  • ayberktecimer (2)
  • shaileshvaidya (2)
  • CBID2 (2)
  • PickupRAIN (2)
  • Stefan-Buchner (2)
  • Ranchhand87 (2)
  • borisbaldassari (2)
  • feitosa-daniel (2)
Top Labels
Issue Labels
enhancement (20) 2.x (14) scheduling (12) bug (6) events (4) question (2) data model (2) configuration (1) wontfix (1)
Pull Request Labels
dependencies (39) python (10) events (2) 2.x (2)

Packages

  • Total packages: 3
  • Total downloads:
    • pypi 8,053 last-month
  • Total docker downloads: 138
  • Total dependent packages: 17
    (may contain duplicates)
  • Total dependent repositories: 55
    (may contain duplicates)
  • Total versions: 203
  • Total maintainers: 2
pypi.org: grimoirelab

Tool set for software development analytics

  • Versions: 130
  • Dependent Packages: 1
  • Dependent Repositories: 5
  • Downloads: 1,222 Last month
  • Docker Downloads: 69
Rankings
Docker downloads count: 2.9%
Stargazers count: 3.1%
Forks count: 3.9%
Dependent packages count: 4.8%
Average: 5.1%
Dependent repos count: 6.6%
Downloads: 9.2%
Maintainers (2)
sduenas zhquan
Last synced: 6 months ago
pypi.org: grimoirelab-toolkit

Toolkit of common functions used across GrimoireLab

  • Versions: 72
  • Dependent Packages: 16
  • Dependent Repositories: 50
  • Downloads: 6,820 Last month
  • Docker Downloads: 69
Rankings
Dependent packages count: 0.7%
Dependent repos count: 2.1%
Docker downloads count: 2.4%
Downloads: 3.6%
Forks count: 5.7%
Average: 5.8%
Stargazers count: 20.3%
Maintainers (2)
sduenas zhquan
Last synced: 6 months ago
pypi.org: grimoirelab-chronicler

Generator of GrimoireLab events using Perceval data.

  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 11 Last month
Rankings
Dependent packages count: 9.8%
Average: 32.3%
Dependent repos count: 54.9%
Maintainers (1)
sduenas
Last synced: 6 months ago

Dependencies

poetry.lock pypi
  • astroid 2.11.7
  • bandit 1.7.4
  • beautifulsoup4 4.11.1
  • cereslib 0.3.1
  • certifi 2022.6.15
  • cffi 1.15.1
  • charset-normalizer 2.0.12
  • cheroot 8.6.0
  • cherrypy 18.8.0
  • click 8.1.3
  • cloc 0.2.5
  • colorama 0.4.5
  • colored 1.4.3
  • colorlog 4.1.0
  • cryptography 3.4.8
  • dill 0.3.5.1
  • dulwich 0.20.45
  • elasticsearch 6.3.1
  • elasticsearch-dsl 6.3.1
  • execnet 1.9.0
  • feedparser 6.0.10
  • file-read-backwards 2.0.0
  • flake8 4.0.1
  • geographiclib 1.52
  • geopy 2.2.0
  • gitdb 4.0.9
  • gitpython 3.1.27
  • graal 0.3.1
  • grimoire-elk 0.102.0
  • grimoirelab-panels 0.1.0
  • grimoirelab-toolkit 0.3.0
  • idna 3.3
  • importlib-metadata 4.2.0
  • importlib-resources 5.8.0
  • isort 5.10.1
  • jaraco.classes 3.2.2
  • jaraco.collections 3.5.2
  • jaraco.context 4.1.2
  • jaraco.functools 3.5.1
  • jaraco.text 3.8.1
  • jinja2 3.0.3
  • kidash 0.5.0
  • kingarthur 0.2.1
  • lazy-object-proxy 1.7.1
  • lizard 1.16.6
  • markupsafe 2.1.1
  • mccabe 0.6.1
  • more-itertools 8.13.0
  • networkx 2.6.3
  • numpy 1.18.3
  • packaging 21.3
  • pandas 0.25.3
  • patsy 0.5.2
  • pbr 5.9.0
  • perceval 0.19.1
  • perceval-mozilla 0.3.1
  • perceval-opnfv 0.2.1
  • perceval-puppet 0.2.1
  • perceval-weblate 0.2.1
  • platformdirs 2.5.2
  • portend 3.1.0
  • pycodestyle 2.8.0
  • pycparser 2.21
  • pydot 1.4.2
  • pyflakes 2.4.0
  • pyjwt 2.4.0
  • pylint 2.13.9
  • pymysql 0.9.3
  • pyparsing 3.0.9
  • python-dateutil 2.8.2
  • pytz 2022.1
  • pywin32 304
  • pyyaml 6.0
  • redis 3.0.0
  • requests 2.26.0
  • rq 1.0
  • scipy 1.6.1
  • sgmllib3k 1.0.0
  • sirmordred 0.3.1
  • six 1.16.0
  • smmap 5.0.0
  • sortinghat 0.7.20
  • soupsieve 2.3.2.post1
  • sqlalchemy 1.3.24
  • statsmodels 0.13.2
  • stevedore 3.5.0
  • tempora 5.0.2
  • tomli 2.0.1
  • typed-ast 1.5.4
  • typing-extensions 4.3.0
  • urllib3 1.26.5
  • wrapt 1.14.1
  • zc.lockfile 2.0
  • zipp 3.8.1
pyproject.toml pypi
  • cereslib >=0.3
  • graal >=0.3
  • grimoire-elk >=0.102
  • grimoirelab-panels >=0.1
  • grimoirelab-toolkit >=0.3
  • kidash >=0.5
  • kingarthur >=0.2
  • perceval >=0.19
  • perceval-mozilla >=0.3
  • perceval-opnfv >=0.2
  • perceval-puppet >=0.2
  • perceval-weblate >=0.2
  • python ^3.7
  • sirmordred >=0.3
  • sortinghat >=0.7
  • statsmodels ^0.13.2
requirements.txt pypi
  • cereslib ==0.3.1
  • graal ==0.3.1
  • grimoire-elk ==0.102.0
  • grimoirelab ==0.4.0
  • grimoirelab-panels ==0.1.0
  • grimoirelab-toolkit ==0.3.0
  • kidash ==0.5.0
  • kingarthur ==0.2.1
  • perceval ==0.19.1
  • perceval-mozilla ==0.3.1
  • perceval-opnfv ==0.2.1
  • perceval-puppet ==0.2.1
  • perceval-weblate ==0.2.1
  • sirmordred ==0.3.1
  • sortinghat ==0.7.20
.github/workflows/docker-image.yml actions
  • actions/setup-python 13ae5bb136fac2878aff31522b9efb785519f984 composite
  • docker/build-push-action c56af957549030174b10d6867f20e78cfd7debc5 composite
  • docker/login-action f4ef78c080cd8ba55a85445d5b36e214a81df20a composite
  • docker/metadata-action 57396166ad8aefe6098280995947635806a0e6ea composite
  • docker/setup-buildx-action 95cb08cb2672c73d4ffd2f422e6d11953d2a9c70 composite
  • docker/setup-qemu-action e81a89b1732b9c48d79cd809d8d81d79c4647a18 composite
  • sigstore/cosign-installer 7cc35d7fdbe70d4278a0c96779081e6fac665f88 composite
.github/workflows/grimoirelab-release.yml actions
  • actions/checkout 93ea575cb5d8a053eaa0ac8fa3b40d7e05a33cc8 composite
  • actions/setup-python 13ae5bb136fac2878aff31522b9efb785519f984 composite
.github/workflows/release-grimoirelab-component.yml actions
  • actions/checkout 93ea575cb5d8a053eaa0ac8fa3b40d7e05a33cc8 composite
  • actions/setup-python 13ae5bb136fac2878aff31522b9efb785519f984 composite
.github/workflows/release.yml actions
  • actions/download-artifact fb598a63ae348fa914e94cd0ff38f362e927b741 composite
  • actions/setup-python 13ae5bb136fac2878aff31522b9efb785519f984 composite
  • chaoss/grimoirelab-github-actions/build master composite
  • chaoss/grimoirelab-github-actions/publish master composite
  • chaoss/grimoirelab-github-actions/release master composite
docker/Dockerfile docker
  • python 3.8-slim-bullseye build
docker-compose/docker-compose-opensearch.yml docker
  • grimoirelab/grimoirelab latest
  • grimoirelab/sortinghat latest
  • grimoirelab/sortinghat-worker latest
  • mariadb 10.6
  • nginx latest
  • opensearchproject/opensearch 1.3.7
  • opensearchproject/opensearch-dashboards 1.3.7
  • redis latest
docker-compose/docker-compose-secured.yml docker
  • bitergia/elasticsearch 6.8.6-secured
  • bitergia/kibiter secured-v6.8.6-3
  • grimoirelab/grimoirelab latest
  • grimoirelab/sortinghat latest
  • grimoirelab/sortinghat-worker latest
  • mariadb 10.6
  • nginx latest
  • redis latest
docker-compose/docker-compose.yml docker
  • bitergia/kibiter community-v6.8.6-3
  • docker.elastic.co/elasticsearch/elasticsearch-oss 6.8.6
  • grimoirelab/grimoirelab latest
  • grimoirelab/sortinghat latest
  • grimoirelab/sortinghat-worker latest
  • mariadb 10.6
  • nginx latest
  • redis latest