green-db

The monorepo that powers the GreenDB.

https://github.com/calgo-lab/green-db

Science Score: 67.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • CITATION.cff file
    Found CITATION.cff file
  • codemeta.json file
    Found codemeta.json file
  • .zenodo.json file
    Found .zenodo.json file
  • DOI references
    Found 16 DOI reference(s) in README
  • Academic publication links
    Links to: arxiv.org, zenodo.org
  • Academic email domains
  • Institutional organization owner
  • JOSS paper metadata
  • Scientific vocabulary similarity
    Low similarity (11.3%) to scientific vocabulary
Last synced: 11 months ago · JSON representation ·

Repository

The monorepo that powers the GreenDB.

Basic Info
Statistics
  • Stars: 26
  • Watchers: 3
  • Forks: 2
  • Open Issues: 10
  • Releases: 0
Created over 4 years ago · Last pushed over 2 years ago
Metadata Files
Readme License Citation

README.md

GreenDB

Preprint: arXiv\ Dataset: DOI

This repository contains code and infrastructure components (Helm charts) that powers the Green Database (GreenDB). This development is part of the research project Green Consumption Assistant (GCA).

The publicly available open GreenDB is a product database. It contains classical product attributes, e.g., name, description, colors, etc. On the other hand, it also includes information about the products' sustainability. Moreover, this sustainability information is transparently evaluated so that it is possible to rank products depending on their sustainability. For more information about the GreenDB dataset or technical details see the dataset on zenodo [1] or our preprint on arXiv [2].

The GreenDB enables the implementation of applications such as the prototypical Chrome Extension Koala - Ecosia Assistant Beta - Grün shoppen, which we built as part of the GCA project. The Koala detects when and what users would like to buy online and offers more sustainable alternatives. More information can be found in one of our working papers [3].

With the open GreenDB, we hope to contribute to the fight against climate crises by enabling others to build applications that support more sustainable consumption. Further, we plan to publish datasets to support the research community.

If you would like to participate with code, data, or anything else, reach out to use!

Where do we get product and sustainability information?

Nowadays, some online shops offer sustainability filters. We use scraping technologies to automatically find these sustainable products and integrate them into the GreenDB. Further, we evaluate the sustainability information and use easily accessible scores (0 - 100) for fine-grained aspects of three sustainability dimensions: social, ecological, and their credibility.

However, manual evaluating products does not scale well. Therefore, we leverage that products can be certified with sustainability labels. The German website Siegelklarheit systematically evaluates sustainability labels and makes their results publicly available. We use these to infer a products' sustainability if it is certified with a sustainability label.

GreenDB schema

The GreenDB schema is highly inspired by schema.org. However, this is still a proof of concept implementation and can differ from their definitions (e.g., the brand column of the green-db table). One of our future plans is to tackle this and make the GreenDB fully compatible with schema.org's Product definition. Our ultimate goal is to eventually contribute an extension of schema.org that integrates the products' sustainability information.

Table: green-db

| column name | id | gtin | asin | timestamp | url | source | merchant | country | category | name | description | brand | sustainability_labels | price | currency | image_urls | colors | sizes | gender | consumer_lifestage | | -------------------- | ------ | -------- | -------- | ------------- | ------- | ---------- | ------------ | ----------- | ------------ | -------- | --------------- | --------- | ------------------------- | --------- | ------------ | -------------- | ----------- | ----------- | ---------- | ---------------------- | | column data type | int4 | int8 | text | timestamp | text | text | text | text | text | text | text | text | array[text] | numeric | text | array[text] | array[text] | array[text] | text | text | | column nullable | no | yes | yes | no | no | no | no | no | no | no | no | no | no | no | no | no | yes | yes | yes | yes |

Please note: Currently, we use our own category strings, which are not documented. However, we plan to switch to the GS1 Global Product Classification taxonomy to rely on a public definition of the category column.

Table: sustainability-labels

| column name | id | timestamp | name | description | cred_credibility | eco_chemicals | eco_lifetime | eco_water | eco_inputs | eco_quality | eco_energy | ecowasteair | ecoenvironmentalmanagement | sociallabourrights | socialbusinesspractice | socialsocialrights | socialcompanyresponsibility | | -------------------- | ------ | ------------- | -------- | --------------- | -------------------- | ----------------- | ---------------- | ------------- | -------------- | --------------- | -------------- | ----------------- | -------------------------------- | ------------------------ | ---------------------------- | ------------------------ | --------------------------------- | | column data type | text | timestamp | text | text | int4 | int4 | int4 | int4 | int4 | int4 | int4 | int4 | int4 | int4 | int4 | int4 | int4 | | column nullable? | no | no | no | no | yes | yes | yes | yes | yes | yes | yes | yes | yes | yes | yes | yes | yes |

Cite us

If you are using our code or the GreenDB dataset, please reference the preprint as: @misc{https://doi.org/10.48550/arxiv.2205.02908, doi = {10.48550/ARXIV.2205.02908}, url = {https://arxiv.org/abs/2205.02908}, author = {Jäger, Sebastian and Greene, Jessica and Jakob, Max and Korenke, Ruben and Santarius, Tilman and Biessmann, Felix}, keywords = {Machine Learning (cs.LG), FOS: Computer and information sciences, FOS: Computer and information sciences}, title = {GreenDB: Toward a Product-by-Product Sustainability Database}, publisher = {arXiv}, year = {2022} }

Future plans

  • [x] We are already working on a first publication of the GreenDB dataset.

  • [x] In the near future, we plan to extend the GreenDB to more shops and non-German markets. We hope to drastically increase its product and sustainability label coverage and make it, therefore, more interesting to a wider range of potential users.

  • [ ] As mentioned above, we plan to switch to the GS1 Global Product Classification taxonomy to replace the category column. Further, we work on minor refactorings to be fully compatible to schema.org.

Contact

  • Main developer of the GreenDB: sebastian.jaeger@bht-berlin.de
  • Green Consumption Assistant research project: https://green-consumption-assistant.de/kontakt/

Research

  • [1]: Jäger, Sebastian, Bießmann, Felix, Flick, Alexander, Sanchez Garcia, Jessica Adriana, von den Driesch, Kaspar, & Brendel, Karl. (2022). GreenDB: A Product-by-Product Sustainability Database [Data set]. Zenodo. https://doi.org/10.5281/zenodo.6078038
  • [2]: Jäger, S., Greene, J., Jakob, M., Korenke, R., Santarius, T., & Biessmann, F. (2022). GreenDB: Toward a Product-by-Product Sustainability Database. ArXiv, abs/2205.02908.
  • [3]: Cathérine Lehmann. (2021). Scaling sustainability advice - Options for generating large-scale green consumption recommendations. URL: https://green-consumption-assistant.de/wp-content/uploads/GCA-Working-Paper-I-Scaling-Sustainability-Advice.pdf

Publications that use the GreenDB

  1. Gossen, M., Jäger, S., Hoffmann, M.L., Biessmann, F., Korenke, R., & Santarius, T. (2022). Nudging Sustainable Consumption: A Large-Scale Data Analysis of Sustainability Labels for Fashion in German Online Retail. Frontiers in Sustainability. DOI: https://doi.org/10.3389/frsus.2022.922984
  2. Jäger, S., Flick, A., Garcia, J.A., Driesch, K.V., Brendel, K., & Biessmann, F. (2022). GreenDB - A Dataset and Benchmark for Extraction of Sustainability Information of Consumer Goods. ArXiv, abs/2207.10733. DOI: https://doi.org/10.48550/arXiv.2207.10733
  3. Flick, A., Jäger, S., Trajanovska, I., Biessmann, F. (2023). Automated Extraction of Fine-Grained Standardized Product Information from Unstructured Multilingual Web Data. In: , et al. Advances in Information Retrieval. ECIR 2023. Lecture Notes in Computer Science, vol 13982. Springer, Cham. https://doi.org/10.1007/978-3-031-28241-6_19

Disclaimer!

This is research code and under development and not supposed to be used in production.

Owner

  • Name: Cognitive Algorithms Lab
  • Login: calgo-lab
  • Kind: organization
  • Location: Germany

Citation (CITATION.cff)

preferred-citation:
  type: misc
  authors:
    - given-names: Sebastian
      family-names: Jäger
      email: sebastian.jaeger@bht-berlin.de
      affiliation: Berliner Hochschule für Technik
    - given-names: Jessica
      family-names: Greene
      affiliation: Ecosia GmbH
    - given-names: Max
      family-names: Jakob
      affiliation: Ecosia GmbH
    - given-names: Ruben
      family-names: Korenke
      affiliation: Ecosia GmbH
    - family-names: Santarius
      given-names: Tilman
      affiliation: "Technical University of Berlin, Einstein Center Digital Future"
    - family-names: Bießmann
      given-names: Felix
      affiliation: "Berliner Hochschule für Technik, Einstein Center Digital Future"

  doi: "10.48550/arXiv.2205.02908"
  title: "GreenDB: A Product-by-Product Sustainability Database"
  year: 2022
  note: "Supported by the Federal Ministry for the Environment, Nature Conservation and Nuclear Safety based on a decision of the German Bundestag. Förderkennzeichen: 67KI2022B"
  url: https://arxiv.org/abs/2205.02908
  publisher: arXiv

GitHub Events

Total
  • Watch event: 4
Last Year
  • Watch event: 4

Dependencies

core/poetry.lock pypi
  • atomicwrites 1.4.0 develop
  • attrs 21.4.0 develop
  • black 21.12b0 develop
  • click 8.0.3 develop
  • colorama 0.4.4 develop
  • flake8 3.9.2 develop
  • isort 5.10.1 develop
  • mccabe 0.6.1 develop
  • more-itertools 8.12.0 develop
  • mypy 0.910 develop
  • mypy-extensions 0.4.3 develop
  • packaging 21.3 develop
  • pathspec 0.9.0 develop
  • platformdirs 2.5.0 develop
  • pluggy 0.13.1 develop
  • py 1.11.0 develop
  • pycodestyle 2.7.0 develop
  • pyflakes 2.3.1 develop
  • pyparsing 3.0.7 develop
  • pytest 5.4.3 develop
  • toml 0.10.2 develop
  • tomli 1.2.3 develop
  • wcwidth 0.2.5 develop
  • pydantic 1.9.0
  • typing-extensions 4.0.1
core/pyproject.toml pypi
  • black ^21.12b0 develop
  • flake8 ^3.9.2 develop
  • isort ^5.10.1 develop
  • mypy ^0.910 develop
  • pytest ^5.2 develop
  • pydantic ^1.9.0
  • python ^3.9
database/poetry.lock pypi
  • atomicwrites 1.4.0 develop
  • attrs 21.4.0 develop
  • black 21.12b0 develop
  • click 8.0.3 develop
  • colorama 0.4.4 develop
  • flake8 3.9.2 develop
  • isort 5.10.1 develop
  • mccabe 0.6.1 develop
  • more-itertools 8.12.0 develop
  • mypy 0.910 develop
  • mypy-extensions 0.4.3 develop
  • packaging 21.3 develop
  • pathspec 0.9.0 develop
  • platformdirs 2.5.0 develop
  • pluggy 0.13.1 develop
  • py 1.11.0 develop
  • pycodestyle 2.7.0 develop
  • pyflakes 2.3.1 develop
  • pyparsing 3.0.7 develop
  • pytest 5.4.3 develop
  • toml 0.10.2 develop
  • tomli 1.2.3 develop
  • wcwidth 0.2.5 develop
  • core 0.1.6
  • greenlet 1.1.2
  • psycopg2 2.9.3
  • pydantic 1.9.0
  • sqlalchemy 1.4.31
  • typing-extensions 4.0.1
database/pyproject.toml pypi
  • black ^21.12b0 develop
  • flake8 ^3.9.2 develop
  • isort ^5.10.1 develop
  • mypy ^0.910 develop
  • pytest ^5.2 develop
  • SQLAlchemy ^1.4.29
  • core *
  • psycopg2 ^2.9.3
  • python ^3.10
extract/poetry.lock pypi
  • atomicwrites 1.4.0 develop
  • attrs 21.4.0 develop
  • black 21.12b0 develop
  • click 8.1.3 develop
  • colorama 0.4.4 develop
  • flake8 3.9.2 develop
  • iniconfig 1.1.1 develop
  • isort 5.10.1 develop
  • mccabe 0.6.1 develop
  • mypy 0.910 develop
  • mypy-extensions 0.4.3 develop
  • packaging 21.3 develop
  • pathspec 0.9.0 develop
  • platformdirs 2.5.2 develop
  • pluggy 1.0.0 develop
  • py 1.11.0 develop
  • pycodestyle 2.7.0 develop
  • pyflakes 2.3.1 develop
  • pytest 7.1.2 develop
  • requests-mock 1.9.3 develop
  • toml 0.10.2 develop
  • tomli 1.2.3 develop
  • types-requests 2.27.30 develop
  • types-urllib3 1.26.15 develop
  • beautifulsoup4 4.11.1
  • certifi 2022.5.18.1
  • charset-normalizer 2.0.12
  • chompjs 1.1.6
  • core 0.1.12
  • database 0.1.12
  • extruct 0.13.0
  • greenlet 1.1.2
  • html-text 0.5.2
  • html5lib 1.1
  • idna 3.3
  • isodate 0.6.1
  • jstyleson 0.0.2
  • lxml 4.9.0
  • mf2py 1.1.2
  • psycopg2 2.9.3
  • pydantic 1.9.1
  • pyparsing 3.0.9
  • pyrdfa3 3.5.3
  • rdflib 6.1.1
  • rdflib-jsonld 0.6.2
  • requests 2.28.0
  • six 1.16.0
  • soupsieve 2.3.2.post1
  • sqlalchemy 1.4.37
  • typing-extensions 4.2.0
  • urllib3 1.26.9
  • w3lib 1.22.0
  • webencodings 0.5.1
extract/pyproject.toml pypi
  • black ^21.12b0 develop
  • flake8 ^3.9.2 develop
  • isort ^5.10.1 develop
  • mypy ^0.910 develop
  • pytest ^7.0.1 develop
  • requests-mock ^1.9.3 develop
  • types-requests ^2.27.7 develop
  • beautifulsoup4 ^4.10.0
  • chompjs ^1.1.6
  • core *
  • database *
  • extruct ^0.13.0
  • python ^3.10
message-queue/poetry.lock pypi
  • atomicwrites 1.4.0 develop
  • attrs 21.4.0 develop
  • more-itertools 8.12.0 develop
  • pluggy 0.13.1 develop
  • py 1.11.0 develop
  • pytest 5.4.3 develop
  • wcwidth 0.2.5 develop
  • click 8.0.3
  • colorama 0.4.4
  • core 0.1.0
  • deprecated 1.2.13
  • packaging 21.3
  • pydantic 1.9.0
  • pyparsing 3.0.7
  • redis 4.1.1
  • rq 1.10.1
  • typing-extensions 4.0.1
  • wrapt 1.13.3
message-queue/pyproject.toml pypi
  • pytest ^5.2 develop
  • core *
  • python ^3.9
  • redis ^4.1.1
  • rq ^1.10.1
scraping/poetry.lock pypi
  • atomicwrites 1.4.0 develop
  • black 21.12b0 develop
  • flake8 3.9.2 develop
  • isort 5.10.1 develop
  • mccabe 0.6.1 develop
  • more-itertools 8.13.0 develop
  • mypy 0.910 develop
  • mypy-extensions 0.4.3 develop
  • pathspec 0.9.0 develop
  • platformdirs 2.5.2 develop
  • pluggy 0.13.1 develop
  • py 1.11.0 develop
  • pycodestyle 2.7.0 develop
  • pyflakes 2.3.1 develop
  • pytest 5.4.3 develop
  • scrapyd-client 1.2.2 develop
  • toml 0.10.2 develop
  • tomli 1.2.3 develop
  • uberegg 0.1.1 develop
  • wcwidth 0.2.5 develop
  • async-timeout 4.0.2
  • attrs 21.4.0
  • automat 20.2.0
  • certifi 2022.5.18.1
  • cffi 1.15.0
  • charset-normalizer 2.0.12
  • click 8.1.3
  • colorama 0.4.4
  • constantly 15.1.0
  • core 0.1.12
  • cryptography 37.0.2
  • cssselect 1.1.0
  • deprecated 1.2.13
  • filelock 3.7.1
  • greenlet 1.1.2
  • hyperlink 21.0.0
  • idna 3.3
  • incremental 21.3.0
  • itemadapter 0.6.0
  • itemloaders 1.0.4
  • jmespath 1.0.0
  • lxml 4.9.0
  • message-queue 0.1.12
  • packaging 21.3
  • parsel 1.6.0
  • playwright 1.22.0
  • protego 0.2.1
  • pyasn1 0.4.8
  • pyasn1-modules 0.2.8
  • pycparser 2.21
  • pydantic 1.9.1
  • pydispatcher 2.0.5
  • pyee 8.1.0
  • pyopenssl 22.0.0
  • pyparsing 3.0.9
  • pypydispatcher 2.1.2
  • queuelib 1.6.2
  • redis 4.3.3
  • requests 2.28.0
  • requests-file 1.5.1
  • rq 1.10.1
  • scrapy 2.6.1
  • scrapy-playwright 0.0.17
  • scrapy-splash 0.8.0
  • service-identity 21.1.0
  • six 1.16.0
  • tldextract 3.3.0
  • twisted 22.4.0
  • twisted-iocpsupport 1.0.2
  • typing-extensions 4.2.0
  • urllib3 1.26.9
  • w3lib 1.22.0
  • websockets 10.1
  • wrapt 1.14.1
  • zope.interface 5.4.0
scraping/pyproject.toml pypi
  • black ^21.12b0 develop
  • flake8 ^3.9.2 develop
  • isort ^5.10.1 develop
  • mypy ^0.910 develop
  • pytest ^5.2 develop
  • scrapyd-client ^1.2.0 develop
  • Scrapy ^2.6.1
  • core *
  • message_queue *
  • python ^3.9
  • scrapy-playwright ^0.0.17
  • scrapy-splash ^0.8.0
workers/poetry.lock pypi
  • atomicwrites 1.4.0 develop
  • attrs 21.4.0 develop
  • black 21.12b0 develop
  • flake8 3.9.2 develop
  • isort 5.10.1 develop
  • mccabe 0.6.1 develop
  • more-itertools 8.12.0 develop
  • mypy 0.910 develop
  • mypy-extensions 0.4.3 develop
  • pathspec 0.9.0 develop
  • platformdirs 2.5.0 develop
  • pluggy 0.13.1 develop
  • py 1.11.0 develop
  • pycodestyle 2.7.0 develop
  • pyflakes 2.3.1 develop
  • pytest 5.4.3 develop
  • toml 0.10.2 develop
  • tomli 1.2.3 develop
  • types-redis 4.1.15 develop
  • wcwidth 0.2.5 develop
  • beautifulsoup4 4.10.0
  • certifi 2021.10.8
  • charset-normalizer 2.0.11
  • click 8.0.3
  • colorama 0.4.4
  • core 0.1.6
  • database 0.1.6
  • deprecated 1.2.13
  • extract 0.1.6
  • extruct 0.13.0
  • greenlet 1.1.2
  • html-text 0.5.2
  • html5lib 1.1
  • idna 3.3
  • isodate 0.6.1
  • jstyleson 0.0.2
  • lxml 4.7.1
  • message-queue 0.1.6
  • mf2py 1.1.2
  • packaging 21.3
  • psycopg2 2.9.3
  • pydantic 1.9.0
  • pyparsing 3.0.7
  • pyrdfa3 3.5.3
  • rdflib 6.1.1
  • rdflib-jsonld 0.6.2
  • redis 4.1.3
  • requests 2.27.1
  • rq 1.10.1
  • six 1.16.0
  • soupsieve 2.3.1
  • sqlalchemy 1.4.31
  • typing-extensions 4.0.1
  • urllib3 1.26.8
  • w3lib 1.22.0
  • webencodings 0.5.1
  • wrapt 1.13.3
workers/pyproject.toml pypi
  • black ^21.12b0 develop
  • flake8 ^3.9.2 develop
  • isort ^5.10.1 develop
  • mypy ^0.910 develop
  • pytest ^5.2 develop
  • types-redis ^4.1.10 develop
  • core *
  • database *
  • extract *
  • message_queue *
  • python ^3.10
  • redis ^4.1.1
  • rq ^1.10.1
.github/workflows/build-and-push-images.yaml actions
  • actions/checkout v2 composite
  • docker/build-push-action v2.8.0 composite
  • docker/login-action v1.12.0 composite
  • docker/metadata-action v3.6.2 composite
.github/workflows/lint.yaml actions
  • actions/checkout v2 composite
  • actions/setup-python v2 composite
.github/workflows/test.yaml actions
  • actions/checkout v2 composite
  • actions/setup-python v2 composite
  • snok/install-poetry v1 composite
monitoring/Dockerfile docker
  • python 3.10.2 build
scraping/Dockerfile docker
  • easypi/scrapyd 1.3.0-2.6.1 build
start-job/Dockerfile docker
  • python 3.10.2 build
workers/Dockerfile docker
  • python 3.10.2 build
monitoring/poetry.lock pypi
  • atomicwrites 1.4.1 develop
  • black 22.8.0 develop
  • flake8 5.0.4 develop
  • isort 5.10.1 develop
  • mccabe 0.7.0 develop
  • more-itertools 8.14.0 develop
  • mypy 0.971 develop
  • mypy-extensions 0.4.3 develop
  • pathspec 0.9.0 develop
  • platformdirs 2.5.2 develop
  • pluggy 0.13.1 develop
  • py 1.11.0 develop
  • pycodestyle 2.9.1 develop
  • pyflakes 2.5.0 develop
  • pytest 5.4.3 develop
  • tomli 1.2.3 develop
  • wcwidth 0.2.5 develop
  • altair 4.2.0
  • attrs 22.1.0
  • blinker 1.5
  • cachetools 5.2.0
  • certifi 2022.6.15
  • charset-normalizer 2.1.1
  • click 8.1.3
  • colorama 0.4.5
  • commonmark 0.9.1
  • core 0.2.0
  • database 0.2.0
  • decorator 5.1.1
  • entrypoints 0.4
  • gitdb 4.0.9
  • gitpython 3.1.27
  • greenlet 1.1.3
  • idna 3.3
  • importlib-metadata 4.12.0
  • jinja2 3.1.2
  • jsonschema 4.14.0
  • markupsafe 2.1.1
  • numpy 1.23.2
  • packaging 21.3
  • pandas 1.4.3
  • pillow 9.2.0
  • plotly 5.10.0
  • protobuf 3.20.1
  • psycopg2-binary 2.9.3
  • pyarrow 9.0.0
  • pydantic 1.9.2
  • pydeck 0.8.0b1
  • pygments 2.13.0
  • pympler 1.0.1
  • pyparsing 3.0.9
  • pyrsistent 0.18.1
  • python-dateutil 2.8.2
  • pytz 2022.2.1
  • pytz-deprecation-shim 0.1.0.post0
  • requests 2.28.1
  • rich 12.5.1
  • semver 2.13.0
  • six 1.16.0
  • smmap 5.0.0
  • sqlalchemy 1.4.40
  • streamlit 1.12.2
  • tenacity 8.0.1
  • toml 0.10.2
  • toolz 0.12.0
  • tornado 6.2
  • typing-extensions 4.3.0
  • tzdata 2022.2
  • tzlocal 4.2
  • urllib3 1.26.12
  • validators 0.20.0
  • watchdog 2.1.9
  • zipp 3.8.1
monitoring/pyproject.toml pypi
  • black ^22.8.0 develop
  • flake8 ^5.0.4 develop
  • isort ^5.10.1 develop
  • mypy ^0.971 develop
  • pytest ^5.2 develop
  • core *
  • database *
  • numpy ^1.23.1
  • pandas ^1.4.3
  • plotly ^5.9.0
  • python ^3.10
  • streamlit ^1.11.0