Data

Tools

Indexes

Applications

Experiments

Open Source Science

vega-plus

Make Vega charts of large datasets

https://github.com/vega/vega-plus

Science Score: 59.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ✓
    DOI references
    Found 1 DOI reference(s) in README
  • ✓
    Academic publication links
    Links to: arxiv.org
  • ✓
    Committers with academic emails
    5 of 11 committers (45.5%) from academic institutions
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (13.5%) to scientific vocabulary
Last synced: 6 months ago · JSON representation

Repository

Make Vega charts of large datasets

Basic Info
Statistics
  • Stars: 44
  • Watchers: 6
  • Forks: 2
  • Open Issues: 4
  • Releases: 0
Created over 5 years ago · Last pushed about 3 years ago
Metadata Files
Readme License Citation

README.md

VegaPlus

A demo of how to run Vega by offloading computational-intensive operations to a separate database management system. We have written a demo paper about the research behind. Please cite us if you use Vega Plus in a publication.

bib @article{yang2022vegaplus, Author = {Junran Yang and Hyekang Kevin Joo and Sai S. Yerramreddy and Siyao Li and Dominik Moritz and Leilani Battle}, Title = {Demonstration of VegaPlus: Optimizing Declarative Visualization Languages}, Year = {2022}, Eprint = {arXiv:2201.06742}, Doi = {10.1145/3514221.3520168}, }

Demos

  • 3M flights in the browser with DuckDB-WASM: https://vega.github.io/vega-plus/demo-duckdb
  • 3M flights in the browser with SQL.JS: https://vega.github.io/vega-plus/demo-sqlite

Workspace Packages

We used yarn workspaces to structure this repository.

vega-plus-core

The primary VegaPlus library component to be used in your application. Install with yarn add vega-plus-core.

demo

An interactive web demo of VegaPlus components with examples of updating data and chart variables.

server

The middleware server for using the VegaPlus with a DBMS backend (we now support PostgreSQL and DuckDB). Install wih yarn add vega-plus-server.

transform-db

The customized Vega transform used by VegaPlus that sends queries to, and receives results from a DBMS. Install wih yarn add vega-transform-db.

Developers

Pre-Requisite

  1. Install and start PostgreSQL.
  2. Create a PostgreSQL database named vega_plus, e.g., createdb vega_plus. You don't need to do anything if you want to use DuckDB.

Installation

  1. Run git clone https://github.com/vega/vega-plus.git.
  2. Run yarn --frozen-lockfile and yarn build to install VegaPlus library dependencies.
  3. For using prepopulated database in demo, look at additional notes.

Running Middleware Server

  1. Run yarn build:server to install VegaPlus server dependencies.
  2. Run yarn start:server_pg to start the application server with postgres, for DuckDB run yarn start:server_duck
  3. All the config information for the databases and server (including user, password and ports to be used) is currently stored in the ./packages/server/duck_db.js/./packages/server/postgres_db.js files and can be customized.

Running the Web Demo

  1. Make sure you have the middleware server running.
  2. In another terminal window, run cd /path/to/dev/repos/vega-plus.
  3. Run yarn build:app to build dependencies for the demo/application UI.
  4. Run yarn start:app to start the web server.
  5. Open a browser tab to localhost:1234.
  6. Upload the cars dataset from ./sample_data/data/cars.json to the data input or click on the Upload Demo Data button.
  7. After uploading a dataset to database, upload a cars vega spec from ./sample_data/data/specs/specs/ to the specs inputor click on the Show me a Demo Spec button and see the visualization.

Running Unit Tests

  1. Again make sure you have the middleware server running.
  2. In another terminal window, run cd /path/to/dev/repos/vega-plus.
  3. The Unit Tests assume a prepopulated database, either do so by uploading data using the web demo or use the provided database (look at additional notes).
  4. For running the unit tests:
    1. For PostgreSQL, yarn test transform_pg
    2. For DuckDB, yarn test transform_duckdb

Additional Notes

  1. Prepopulated Database, We have provided prepopulated databases and scripts for PostgreSQL and DuckDB in ./packages/server/database
    1. For PostgreSQL, use a command like psql dbname < infile. For example, psql postgresql://postgres@localhost/scalable_vega < ./packages/server/database/scalable_vega.sql
    2. For DuckDB you can run yarn start:duckdb-sample which will populate a duckdb database file with 5 tables. You can customize the name of the db file being used by making changes in ./packages/server/server.js and ./packages/server/database/duckdb_insertion.js
  2. If you face Cannot find name 'expect' type error while running tests
    1. Run yarn add @types/jest -D

Owner

  • Name: Vega
  • Login: vega
  • Kind: organization

Data Visualization Languages & Tools

GitHub Events

Total
  • Watch event: 5
Last Year
  • Watch event: 5

Committers

Last synced: almost 3 years ago

All Time
  • Total Commits: 406
  • Total Committers: 11
  • Avg Commits per committer: 36.909
  • Development Distribution Score (DDS): 0.66
Top Committers
Name Email Commits
Sai Yerramreddy s****7@g****m 138
Andrew Fichman a****n@g****m 110
Junran j****g@t****u 49
Dominik Moritz d****z@g****m 38
Sai Yerramreddy s****r@u****u 30
Leilani Battle l****t@.****u 18
JunranY 3****Y@u****m 13
JunranY j****n@c****u 5
Junran Yang j****n@c****u 3
Kevin Hyekang Joo j****j@g****m 1
Leilani Battle l****t@g****m 1
Committer Domains (Top 20 + Academic)
cs.washigton.edu: 1 cs.washington.edu: 1 .mit.edu: 1 umd.edu: 1 terpmail.umd.edu: 1

Issues and Pull Requests

Last synced: 7 months ago

All Time
  • Total issues: 69
  • Total pull requests: 57
  • Average time to close issues: about 1 month
  • Average time to close pull requests: 1 day
  • Total issue authors: 4
  • Total pull request authors: 3
  • Average comments per issue: 0.51
  • Average comments per pull request: 0.39
  • Merged pull requests: 52
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • domoritz (27)
  • SaiArrow (8)
  • leibatt (6)
  • joos2010kj (1)
Pull Request Authors
  • JunranY (17)
  • SaiArrow (12)
  • domoritz (5)
Top Labels
Issue Labels
enhancement (4) bug (3) documentation (1)
Pull Request Labels

Packages

  • Total packages: 1
  • Total downloads:
    • npm 7 last-month
  • Total dependent packages: 0
  • Total dependent repositories: 1
  • Total versions: 4
  • Total maintainers: 2
npmjs.org: vega-plus

Vega-plus extends the Vega dataflow to a client-server architecture to utilize the scalability advantage of DBMSs, we automatically translate Vega transform operators to SQL queries. To offload intensive calculations to the DBMS, combine vega-plus with on

  • License: BSD-3-Clause
  • Latest release: 0.0.4
    published almost 4 years ago
  • Versions: 4
  • Dependent Packages: 0
  • Dependent Repositories: 1
  • Downloads: 7 Last month
Rankings
Stargazers count: 6.9%
Forks count: 9.8%
Dependent repos count: 10.3%
Average: 23.7%
Downloads: 39.5%
Dependent packages count: 51.9%
Maintainers (2)
jryang saisyr
Last synced: 7 months ago

Dependencies

package.json npm
  • @types/jest ^27.4.1 development
  • express ^4.17.3 development
  • gh-pages ^3.2.3 development
  • jest ^27.5.1 development
  • node-fetch ^2.6.7 development
  • parcel-bundler ^1.12.3 development
  • ts-jest ^27.1.3 development
  • tsc-watch ^4.6.2 development
  • typescript 4.6.2 development
  • webpack ^5.70.0 development
  • @hpcc-js/wasm ^1.13.0
  • @observablehq/graphviz 0.2.1
  • @types/node ^17.0.21
  • body-parser ^1.19.2
  • fs ^0.0.1-security
  • vega ^5.22.0
  • vega-encode ^4.9.0
  • vega-transform-db ^1.0.10
  • vega-transforms ^4.10.0
  • web-worker ^1.2.0
.github/workflows/test.yml actions
  • actions/checkout v2 composite
  • actions/setup-node v2 composite
  • postgres * docker