Data

Tools

Indexes

Applications

Experiments

Open Source Science

@uwdata/mosaic-core

An extensible framework for linking databases and interactive views.

https://github.com/uwdata/mosaic

Science Score: 65.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ✓
    DOI references
    Found 1 DOI reference(s) in README
  • ○
    Academic publication links
  • ○
    Committers with academic emails
  • ✓
    Institutional organization owner
    Organization uwdata has institutional domain (idl.cs.washington.edu)
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (13.8%) to scientific vocabulary

Keywords

data duckdb interaction scalability visualization

Keywords from Contributors

marimo widgets embedding asp constraints draco recomendation mesh interpretability dag
Last synced: 4 months ago · JSON representation ·

Repository

An extensible framework for linking databases and interactive views.

Basic Info
  • Host: GitHub
  • Owner: uwdata
  • License: other
  • Language: TypeScript
  • Default Branch: main
  • Homepage: https://idl.uw.edu/mosaic/
  • Size: 21 MB
Statistics
  • Stars: 1,113
  • Watchers: 18
  • Forks: 81
  • Open Issues: 69
  • Releases: 22
Topics
data duckdb interaction scalability visualization
Created about 3 years ago · Last pushed 4 months ago
Metadata Files
Readme License Citation Codeowners

README.md

Mosaic: An Extensible Framework for Linking Databases and Interactive Views

  • 📈 Explore massive datasets
    Visualize, select, and filter datasets with millions or billions of records.
  • 🚀 Flexible deployment
    Build data-driven web apps, or interact with data directly in Jupyter notebooks.
  • 🛠️ Interoperable & extensible
    Create new components that seamlessly integrate across selections and datasets.
  • 🦆 Powered by DuckDB
    Mosaic pushes computation to DuckDB, both server-side and in your browser via WebAssembly.

Mosaic is an extensible architecture for linking data visualizations, tables, input widgets, and other data-driven components, leveraging a backing database for scalable processing of both static and interactive views. With Mosaic, you can visualize and explore millions and even billions of data points at interactive rates.

The key idea is to have interface components "publish" their data needs as declarative queries that can be managed, optimized, and cross-filtered by a coordinator that proxies access to DuckDB.

Learn more about Mosaic at the documentation site, or read the Mosaic research paper.

If referencing Mosaic, please use the following citation:

bibtex @article{heer2024mosaic, title={Mosaic: An Architecture for Scalable \& Interoperable Data Views}, author={Heer, Jeffrey and Moritz, Dominik}, journal={IEEE Transactions on Visualization and Computer Graphics}, year={2024}, volume={30}, number={1}, pages={436-446}, doi={10.1109/TVCG.2023.3327189} }

Repository Structure

This repository contains a set of related packages, spanning the core Mosaic architecture, DuckDB servers, the vgplot visualization library, and example projects.

Mosaic Architecture (packages/mosaic)

  • mosaic-core: The core Mosaic components. A central coordinator, parameters and selections for linking scalar values or query predicates (respectively) across Mosaic clients, and filter groups with materialized views of pre-aggregated data. The Mosaic coordinator can send queries either over the network to a backing server (socket and rest clients) or to a client-side DuckDB-WASM instance (wasm client).
  • mosaic-sql: An API for convenient construction and analysis of SQL queries. Query objects then coerce to SQL query strings.

vgplot Libraries (packages/vgplot)

vgplot is a visualization grammaer API that demonstrates the features and performance of the Mosaic architecture. It includes a layered grammar of interactive graphics alongside table and input widget components. For convenience, the vgplot package re-exports much of the mosaic-core, mosaic-sql, mosaic-plot, and mosaic-inputs packages. For most vgplot applications, it is sufficient to either import @uwdata/vgplot alone or in conjunction with @uwdata/mosaic-spec.

  • vgplot: A visualization grammar API for building interactive Mosaic-powered visualizations and dashboards. This package provides convenient, composable methods that combine multiple Mosaic packages (core, inputs, plot, etc.) in an integrated API. This API re-exports much of the mosaic-core, mosaic-sql, mosaic-plot, and mosaic-inputs packages, enabling use in a stand-alone fashion.
  • mosaic-inputs: Standalone data-driven components such as input menus, text search boxes, and sortable, load-on-scroll data tables.
  • mosaic-plot: An interactive grammar of graphics implemented on top of Observable Plot. Marks (plot layers) serve as individual Mosaic clients. These marks can push data processing (binning, hex binning, regression) and optimizations (such as M4 for line/area charts) down to the database. This package also provides interactors for linked selection, filtering, and highlighting using Mosaic Params and Selections.
  • mosaic-spec: Declarative specification of Mosaic-powered applications as JSON or YAML files. This package provides a parser and code generation framework for reading specifications in a JSON format and generating live Mosaic visualizations and dashboards using the vgplot API.
  • widget: A Jupyter widget for Mosaic. Given a declarative specification, will generate web-based visualizations while leveraging DuckDB in the Jupyter kernel. Create interactive Mosaic plots over Pandas and Polars data frames or DuckDB connections.

DuckDB Servers (packages/server)

  • duckdb-server: A Python-based server that runs a local DuckDB instance and support queries over Web Sockets or HTTP, returning data in either Apache Arrow or JSON format.
  • duckdb-server-rust: A Rust-based server similar to duckdb-server (Python) and mosaic-duckdb (Node.js) with additional support for HTTP/2. We are still evaluating what server component works best. DuckDB support for Rust is often delayed compared to Python.
  • mosaic-duckdb: A Promise-based Node.js API to DuckDB, along with a data server that supports transfer of Apache Arrow and JSON data over either Web Sockets or HTTP. Due to quality and maintenance issues involving the Node.js DuckDB client and Arrow extension, we recommend using the Python-based duckdb-server package instead. However, we retain this package for both backwards compatibility and internal testing use.

Examples (packages/examples)

  • vanilla-example: A basic example demonstrating how to deploy Mosaic in a website.
  • svelte-example: An example application that demonstrates howto create Mosaic components within web frameworks such as Svelte and React.
  • vega-example: A proof-of-concept example integrating Vega-Lite with Mosaic for data management and cross-view linking.

Build and Usage Instructions

To build and develop Mosaic locally:

  • Clone https://github.com/uwdata/mosaic.
  • Run npm i to install dependencies.
  • Run npm test to run the test suite.
  • Run npm run build to build client-side bundles.
  • Run uv build --all-packages to build the Python packages.

To run local interactive examples:

  • Run npm run dev to launch a local web server and view examples. By default, the examples use DuckDB-WASM in the browser. For greater performance, launch and connect to a local DuckDB server as described below.

To launch a local DuckDB server:

  • Install uv, if not already present.
  • Run npm run server to launch the duckdb-server. This runs the server in development mode, so the server will restart if you change its code.

To use Mosaic with DuckDB Python in Jupyter Notebooks:

To use Mosaic with DuckDB-WASM in Observable Notebooks:

To use Mosaic and DuckDB in Observable Framework:

Owner

  • Name: UW Interactive Data Lab
  • Login: uwdata
  • Kind: organization
  • Location: Seattle, WA

Visualization + Analysis

Citation (CITATION.cff) 

citation-file-format: 1.2.0
creator:
  - family-names: Heer
    given-names: Jeffrey
  - family-names: Moritz
    given-names: Dominik
title: "Mosaic: An Architecture for Scalable & Interoperable Data Views"
url: https://github.com/uwdata/mosaic
license: BSD 3-Clause
preferred-citation:
  type: article
  authors:
    - family-names: Heer
      given-names: Jeffrey
    - family-names: Moritz
      given-names: Dominik
  title: "Mosaic: An Architecture for Scalable & Interoperable Data Views"
  journal: IEEE Transactions on Visualization and Computer Graphics
  year: 2024
  doi: 10.1109/TVCG.2023.3327189

Committers

Last synced: 6 months ago

All Time
  • Total Commits: 903
  • Total Committers: 21
  • Avg Commits per committer: 43.0
  • Development Distribution Score (DDS): 0.3
Past Year
  • Commits: 295
  • Committers: 15
  • Avg Commits per committer: 19.667
  • Development Distribution Score (DDS): 0.492
Top Committers
Name Email Commits
jheer j****r@g****m 632
Dominik Moritz d****z@g****m 134
dependabot[bot] 4****] 94
Oh-Hyun Kwon oh@k****o 12
andyrooger 4****r 5
Francisco Rivera f****1@g****m 4
Torsten Sprenger m****l@s****e 3
weifan_certik w****u@c****m 2
Péter Ferenc Gyarmati d****y@g****m 2
Ming Fang m****g@m****m 2
Ron Pechuk 3****k 2
Trevor Manz t****z@g****m 2
Alex Kesling a****x@k****o 1
Alexander Kolov me@a****m 1
Derek Perkins d****k@n****o 1
Geoffrey Donaldson g****d@g****m 1
Kevin Zheng 1****7 1
Michael Keller m****r 1
Mike Bostock m****k@g****m 1
Rick Smit r****t@g****m 1
eknowles 8****s 1
Committer Domains (Top 20 + Academic)
nozzle.io: 1 alexkolov.com: 1 kesling.co: 1 mac.com: 1 certik.com: 1 spren9er.de: 1 kwon.io: 1

Issues and Pull Requests

Last synced: 4 months ago

All Time
  • Total issues: 148
  • Total pull requests: 564
  • Average time to close issues: about 1 month
  • Average time to close pull requests: 5 days
  • Total issue authors: 56
  • Total pull request authors: 30
  • Average comments per issue: 1.79
  • Average comments per pull request: 0.59
  • Merged pull requests: 360
  • Bot issues: 0
  • Bot pull requests: 226
Past Year
  • Issues: 56
  • Pull requests: 382
  • Average time to close issues: 5 days
  • Average time to close pull requests: 3 days
  • Issue authors: 35
  • Pull request authors: 18
  • Average comments per issue: 2.18
  • Average comments per pull request: 0.55
  • Merged pull requests: 253
  • Bot issues: 0
  • Bot pull requests: 164
View more stats
Top Authors
Issue Authors
  • domoritz (33)
  • derekperkins (14)
  • frtennis1 (10)
  • jheer (10)
  • declann (7)
  • mhkeller (5)
  • maelp (5)
  • spren9er (4)
  • Unemyr (3)
  • RyanChapman2x (3)
  • andyrooger (3)
  • bbollen23 (2)
  • harrylojames (2)
  • jaanli (2)
  • manzt (2)
Pull Request Authors
  • dependabot[bot] (264)
  • jheer (158)
  • domoritz (127)
  • kwonoh (25)
  • derekperkins (16)
  • rpechuk (12)
  • frtennis1 (8)
  • peter-gy (7)
  • spren9er (5)
  • andyrooger (4)
  • kzhang31415 (3)
  • declann (3)
  • manzt (3)
  • JessamineQ (2)
  • jerryc214 (2)
Top Labels
Issue Labels
projects (18) enhancement (9) bug (6) documentation (4) question (3) good first issue (1) needs_details (1) python (1)
Pull Request Labels
dependencies (264) javascript (156) rust (82) github_actions (12) python (10) go (4) projects (2) on hold (1)

Packages

  • Total packages: 12
  • Total downloads:
    • cargo 3,603 total
    • npm 167,619 last-month
    • pypi 1,038 last-month
  • Total dependent packages: 10
    (may contain duplicates)
  • Total dependent repositories: 10
    (may contain duplicates)
  • Total versions: 258
  • Total maintainers: 4
proxy.golang.org: github.com/uwdata/mosaic/packages/server/duckdb-server-go
  • Versions: 0
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Stargazers count: 2.3%
Forks count: 3.2%
Average: 4.1%
Dependent packages count: 5.3%
Dependent repos count: 5.6%
Last synced: 4 months ago
npmjs.org: @uwdata/mosaic-sql

SQL query construction and analysis.

  • License: BSD-3-Clause
  • Latest release: 0.18.0
    published 6 months ago
  • Versions: 26
  • Dependent Packages: 7
  • Dependent Repositories: 2
  • Downloads: 115,439 Last month
Rankings
Dependent packages count: 2.8%
Stargazers count: 3.3%
Downloads: 3.5%
Average: 4.7%
Forks count: 6.1%
Dependent repos count: 7.6%
Maintainers (2)
jheer luke-s-snyder
Last synced: 5 months ago
proxy.golang.org: github.com/uwdata/mosaic
  • Versions: 31
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent packages count: 5.7%
Average: 5.9%
Dependent repos count: 6.0%
Last synced: 4 months ago
npmjs.org: @uwdata/mosaic-core

Scalable and extensible linked data views.

  • License: BSD-3-Clause
  • Latest release: 0.18.0
    published 6 months ago
  • Versions: 29
  • Dependent Packages: 2
  • Dependent Repositories: 2
  • Downloads: 12,286 Last month
Rankings
Stargazers count: 3.3%
Downloads: 5.4%
Forks count: 6.1%
Average: 6.3%
Dependent repos count: 7.6%
Dependent packages count: 8.8%
Maintainers (2)
jheer luke-s-snyder
Last synced: 4 months ago
npmjs.org: @uwdata/mosaic-inputs

Mosaic input components.

  • License: BSD-3-Clause
  • Latest release: 0.18.0
    published 6 months ago
  • Versions: 29
  • Dependent Packages: 1
  • Dependent Repositories: 2
  • Downloads: 11,316 Last month
Rankings
Stargazers count: 3.3%
Downloads: 5.2%
Forks count: 6.1%
Dependent repos count: 7.6%
Average: 8.6%
Dependent packages count: 21.0%
Maintainers (2)
jheer luke-s-snyder
Last synced: 5 months ago
npmjs.org: @uwdata/vgplot

An API for interactive Mosaic-powered visualizations and dashboards.

  • License: BSD-3-Clause
  • Latest release: 0.18.0
    published 6 months ago
  • Versions: 30
  • Dependent Packages: 0
  • Dependent Repositories: 3
  • Downloads: 9,806 Last month
Rankings
Stargazers count: 3.3%
Downloads: 5.3%
Forks count: 6.1%
Dependent repos count: 6.4%
Average: 14.6%
Dependent packages count: 52.0%
Maintainers (2)
jheer luke-s-snyder
Last synced: 5 months ago
pypi.org: mosaic-widget

A Jupyter widget for Mosaic

  • Versions: 25
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 599 Last month
Rankings
Dependent packages count: 6.6%
Average: 19.1%
Downloads: 19.9%
Dependent repos count: 30.6%
Maintainers (2)
Dominik.Moritz jheer
Last synced: 4 months ago
npmjs.org: @uwdata/mosaic-duckdb

A Promise-based DuckDB API and Node.js data server.

  • License: BSD-3-Clause
  • Latest release: 0.18.0
    published 6 months ago
  • Versions: 26
  • Dependent Packages: 0
  • Dependent Repositories: 1
  • Downloads: 499 Last month
Rankings
Stargazers count: 3.3%
Forks count: 6.1%
Dependent repos count: 10.3%
Average: 19.4%
Downloads: 25.5%
Dependent packages count: 52.0%
Maintainers (2)
jheer luke-s-snyder
Last synced: 5 months ago
pypi.org: duckdb-server

A DuckDB server for Mosaic

  • Versions: 18
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 439 Last month
Rankings
Dependent packages count: 9.8%
Average: 37.4%
Dependent repos count: 65.0%
Maintainers (2)
Dominik.Moritz jheer
Last synced: 4 months ago
npmjs.org: @uwdata/mosaic-spec

Declarative specification of Mosaic-powered applications.

  • License: BSD-3-Clause
  • Latest release: 0.18.0
    published 6 months ago
  • Versions: 20
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 8,331 Last month
Rankings
Dependent repos count: 33.9%
Average: 41.2%
Dependent packages count: 48.4%
Maintainers (2)
jheer luke-s-snyder
Last synced: 5 months ago
npmjs.org: @uwdata/mosaic-plot

A Mosaic-powered plotting framework based on Observable Plot.

  • License: BSD-3-Clause
  • Latest release: 0.18.0
    published 6 months ago
  • Versions: 20
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 9,942 Last month
Rankings
Dependent repos count: 33.9%
Average: 41.2%
Dependent packages count: 48.4%
Maintainers (2)
jheer luke-s-snyder
Last synced: 5 months ago
crates.io: duckdb-server

DuckDB Server for Mosaic.

  • Versions: 4
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 3,603 Total
Rankings
Dependent repos count: 26.5%
Dependent packages count: 35.1%
Average: 52.6%
Downloads: 96.1%
Maintainers (1)
domoritz
Last synced: 5 months ago

Dependencies

.github/workflows/deploy.yml actions
  • actions/checkout v3 composite
  • actions/configure-pages v3 composite
  • actions/deploy-pages v2 composite
  • actions/setup-node v3 composite
  • actions/upload-pages-artifact v2 composite
.github/workflows/test.yml actions
  • actions/checkout v3 composite
  • actions/setup-node v3 composite
  • actions/setup-python v4 composite
package-lock.json npm
  • 939 dependencies
package.json npm
  • esbuild ^0.19.3 development
  • eslint ^8.49.0 development
  • lerna ^7.3.0 development
  • mocha ^10.2.0 development
  • nodemon ^3.0.1 development
  • rimraf ^5.0.1 development
  • timezone-mock ^1.3.6 development
  • typescript ^5.2.2 development
  • vite ^4.4.9 development
  • vitepress 1.0.0-rc.20 development
  • yaml ^2.3.2 development
packages/core/package.json npm
  • @duckdb/duckdb-wasm ^1.27.0
  • @uwdata/mosaic-sql ^0.3.4
  • apache-arrow ^11.0.0
packages/duckdb/package.json npm
  • duckdb ~0.8.1
  • ws ^8.14.2
packages/inputs/package.json npm
  • @uwdata/mosaic-core ^0.3.4
  • @uwdata/mosaic-sql ^0.3.4
  • isoformat ^0.2.1
packages/sql/package.json npm
packages/vega-example/package.json npm
  • vite ^4.4.8 development
  • @uwdata/mosaic-core ^0.3.4
  • @uwdata/mosaic-sql ^0.3.4
  • @uwdata/vgplot ^0.3.4
  • vega ^5.25.0
  • vega-embed ^6.22.2
packages/vgplot/package.json npm
  • @observablehq/plot ^0.6.11
  • @uwdata/mosaic-core ^0.3.4
  • @uwdata/mosaic-inputs ^0.3.4
  • @uwdata/mosaic-sql ^0.3.4
  • d3 ^7.8.5
  • isoformat ^0.2.1
  • topojson-client ^3.1.0
packages/widget/package.json npm
  • anywidget ^0.6.3 development
  • @uwdata/vgplot ^0.3.4
  • apache-arrow ^11.0.0
packages/widget/pyproject.toml pypi
  • anywidget >=0.6.3
  • duckdb ==0.8.1
  • pyarrow *
packages/duckdb-server/pyproject.toml pypi
  • diskcache *
  • duckdb ==0.8.1
  • pandas *
  • pyarrow *
  • socketify *
  • ujson *