Visions
Visions: An Open-Source Library for Semantic Data - Published in JOSS (2020)
Cost-Effective Big Data Orchestration Using Dagster: A Multi-Platform Approach
Cost-Effective Big Data Orchestration Using Dagster: A Multi-Platform Approach - Published in JOSS (2026)
https://github.com/moj-analytical-services/splink
Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends
h2o
H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.
com.linkedin.isolation-forest
A distributed Spark/Scala implementation of the isolation forest algorithm for unsupervised outlier detection, featuring support for scalable training and ONNX export for easy cross-platform inference.
https://github.com/awslabs/deequ
Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.
fugue
A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.
https://github.com/broadinstitute/gatk
Official code repository for GATK versions 4 and up
workbench
Workbench: An easy to use Python API for creating and deploying AWS SageMaker Models
https://github.com/commoncrawl/cc-pyspark
Process Common Crawl data with Python and Spark
https://github.com/commoncrawl/cc-index-table
Index Common Crawl archives in tabular format
https://github.com/bytedance/cloudshuffleservice
Cloud Shuffle Service(CSS) is a general purpose remote shuffle solution for compute engines, including Spark/Flink/MapReduce.
https://github.com/azavea/geotrellis-collections-api-research
A research project to investigate using GeoTrellis as a REST service
https://github.com/ccao-data/service-spark-iasworld
Service for extracting tables from the CCAO system-of-record and uploading them to the Data Department's data warehouse
https://github.com/aveek-saha/cricket-score-predictor
A Big data application to predict the outcome of a T20 cricket match.
https://github.com/dadananjesha/spark-streaming
Spark Streaming KPI Processing is a real-time data processing application built using Apache Spark Streaming
https://github.com/dadananjesha/credit-card-fraud-detection
Credit Card Fraud Detection is a state-of-the-art real-time streaming analytics solution designed to detect fraudulent credit card transactions instantly.
https://github.com/rumbledb/rumble
⛈️ RumbleDB 2.0.0 "Lemon Ironwood" 🌳 for Apache Spark | Run queries on your large-scale, messy JSON-like data (JSON, text, CSV, Parquet, ROOT, AVRO, SVM...) | No install required (just a jar to download) | Declarative Machine Learning and more
https://github.com/bigbio/pgatk-io
High performance io library for proteogenomics
spark-dynamic-executor-time-prediction
Neural Network Models for Predicting Execution Time with Dynamic Executor Allocation in Apache Spark.
https://github.com/dadananjesha/redshift-etl-project
The project covers the complete data pipeline—from importing data from an RDS source to HDFS using Sqoop, processing data with Spark, to executing analytical queries on an AWS Redshift cluster.
https://github.com/johnsnowlabs/johnsnowlabs
Gateway into the John Snow Labs Ecosystem
https://github.com/awslabs/data-on-eks
DoEKS is a tool to build, deploy and scale Data Platforms on Amazon EKS
acid
Generate the AutoCorrelation Integral Drill (ACID) test set with scripts and workflows for robust validation of autocorrelation algorithms. 🌐📊
https://github.com/ai-team-uoa/geotriples
Publishing Big Geospatial data as Linked Open Geospatial Data
https://github.com/arturbomtempo-learning/projects-for-interdisciplinary-work-2
Projects created to better learn about the technologies used in the Interdisciplinary Work 2 course of the Computer Science program.
https://github.com/dadananjesha/azuredataengine
AzureDataEngine is a robust, scalable batch processing data architecture built on the Azure platform. It efficiently extracts, transforms, and loads massive datasets for machine learning applications, leveraging Azure Blob Storage, PostgreSQL, Databricks, and Key Vault to ensure reliability and maintainability.
https://github.com/SETL-Framework/setl
A simple Spark-powered ETL framework that just works 🍺
analysis-pipelines
Enables data scientists to compose pipelines of analysis which consist of data manipulation, exploratory analysis & reporting, as well as modeling steps. Data scientists can use tools of their choice through an R interface, and compose interoperable pipelines between R, Spark, and Python.
pysparklyr
Extension to {sparklyr} that allows you to interact with Spark & Databricks Connect
graphster
spark-based library that helps construct and query knowledge graphs from unstructured and structured data