SPRUCE

Spruce helps estimate the environmental impact of your cloud usage. By leveraging open source models and data, it enriches usage reports generated by cloud providers and allows you to build reports and visualisations. Having the greenops and finops data in the same place makes it easier to expose your costs and impacts side by side.

Spruce uses Apache Spark to read and write the usage reports (typically in Parquet format) in a scalable way and, thanks to its modular approach, splits the enrichment of the data into configurable stages.

A typical sequence of stages would be: - estimation of embedded emissions from resources used - estimation of energy used - application of PUE and other overheads - application of carbon intensity factors

Please note that this is currently a prototype which handles only CUR reports from AWS. Not all AWS services are covered.

One of the benefits of using Apache Spark is that you can use EMR on AWS to enrich the CURs at scale without having to export or expose any of your data.

Prerequisites

You will need to have CUR reports as inputs. Those are generated via DataExports and stored on S3 as Parquet files.

Local install

With Apache Maven, Java and Apache Spark installed locally and added to the $PATH.

mvn clean package spark-submit --class com.digitalpebble.spruce.SparkJob --driver-memory 8g ./target/spruce-1.0.jar ./curs ./output

Docker

Build the Docker image with docker build -t digitalpebble/spruce:1.0 .

The command below processes the data locally by mounting the directories containing the CURs and output as volumes: docker run -it -v ./curs:/curs -v ./output:/output digitalpebble/spruce:1.0 \ /opt/spark/bin/spark-submit \ --class com.digitalpebble.spruce.SparkJob \ --driver-memory 4g \ --master 'local[*]' \ /usr/local/lib/spruce-1.0.jar \ /curs /output/enriched

Explore the output

Using DuckDB locally or Athena on AWS:

```sql create table enriched_curs as select * from 'output//.parquet';

select lineitemproductcode, productservicecode, round(sum(operationalemissionsco2eqg),2) as co2usageg, round(sum(energyusagekwh),2) as energyusagekwh from enrichedcurs where operationalemissionsco2eqg > 0.01 group by lineitemproductcode, productservicecode order by co2usage_g desc; ```

should give an output similar to

| lineitemproductcode | productservicecode | co2usageg | energyusagekwh | |------------------------|---------------------|-------------|------------------| | AmazonS3 | AWSDataTransfer | 659.2 | 3.31 | | AmazonRDS | AWSDataTransfer | 361.59 | 1.09 | | AmazonEC2 | AWSDataTransfer | 162.59 | 1.43 | | AmazonECR | AWSDataTransfer | 88.75 | 0.8 | | AmazonVPC | AWSDataTransfer | 40.55 | 0.38 | | AWSELB | AWSDataTransfer | 6.3 | 0.06 |

To measure the proportion of the costs for which emissions where calculated

sql select round(covered * 100 / "total costs", 2) as percentage_costs_covered from ( select sum(line_item_unblended_cost) as "total costs", sum(line_item_unblended_cost) filter (where operational_emissions_co2eq_g is not null) as covered from enriched_curs where line_item_line_item_type like '%Usage' );

License

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0