uptasticsearch

An Elasticsearch client tailored to data science workflows.

https://github.com/uptake/uptasticsearch

Science Score: 36.0%

This score indicates how likely this project is to be science-related based on various indicators:

○
CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
✓
.zenodo.json file
Found .zenodo.json file
○
DOI references
○
Academic publication links
✓
Committers with academic emails
1 of 32 committers (3.1%) from academic institutions
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (16.0%) to scientific vocabulary

Keywords

data-engineering data-science document-database elasticsearch etl nosql python r

Keywords from Contributors

projection interactive distribution interpretability parquet arrow data-profiling standardization flexible optim

Last synced: 6 months ago · JSON representation

Repository

An Elasticsearch client tailored to data science workflows.

Basic Info

Host: GitHub
Owner: uptake
License: bsd-3-clause
Language: R
Default Branch: main
Homepage: https://uptake.github.io/uptasticsearch/
Size: 894 KB

Statistics

Stars: 49
Watchers: 6
Forks: 35
Open Issues: 15
Releases: 7

Topics

data-engineering data-science document-database elasticsearch etl nosql python r

Created over 8 years ago · Last pushed 8 months ago

Metadata Files

Readme Changelog Contributing License Codeowners

uptasticsearch

$CRAN\_Status\_Badge$ $CRAN\_Download\_Badge$

Introduction

uptasticsearch tackles the issue of getting data out of Elasticsearch and into a tabular format in R. It should work for all versions of Elasticsearch from 1.0.0 onwards, but is not regularly tested against all of them. If you run into a problem, please open an issue.

How it Works
Installation
- R
Usage Examples
- Get a Batch of Documents
- Aggregation Results

How it Works

The core functionality of this package is the es_search() function. This returns a data.table containing the parsed result of any given query. Note that this includes aggs queries.

Installation

R

Releases of this package can be installed from CRAN:

r install.packages( 'uptasticsearch' , repos = "http://cran.rstudio.com" )

or from conda-forge

shell conda install -c conda-forge r-uptasticsearch

To use the development version of the package, which has the newest changes, you can install directly from GitHub

r remotes::install_github( "uptake/uptasticsearch" , subdir = "r-pkg" )

Usage Examples

The examples presented here pertain to a fictional Elasticsearch index holding some information on a movie theater business.

Example 1: Get a Batch of Documents

The most common use case for this package will be the case where you have an Elasticsearch query and want to get a data frame representation of many resulting documents.

In the example below, we use uptasticsearch to look for all survey results in which customers said their satisfaction was "low" or "very low" and mentioned food in their comments.

```r library(uptasticsearch)

Build your query in an R string

qbody <- '{ "query": { "filtered": { "filter": { "bool": { "must": [ { "exists": { "field": "customercomments" } }, { "terms": { "overallsatisfaction": ["very low", "low"] } } ] } } }, "query": { "matchphrase": { "customercomments": "food" } } } }'

Execute the query, parse into a data.table

commentDT <- essearch( eshost = 'http://mydb.mycompany.com:9200' , esindex = "surveyresults" , querybody = qbody , scroll = "1m" , ncores = 4 ) ```

Example 2: Aggregation Results

Elasticsearch ships with a rich set of aggregations for creating summarized views of your data. uptasticsearch has built-in support for these aggregations.

In the example below, we use uptasticsearch to create daily timeseries of summary statistics like total revenue and average payment amount.

```r library(uptasticsearch)

Build your query in an R string

qbody <- '{ "query": { "filtered": { "filter": { "bool": { "must": [ { "exists": { "field": "pmtamount" } } ] } } } }, "aggs": { "timestamp": { "datehistogram": { "field": "timestamp", "interval": "day" }, "aggs": { "revenue": { "extendedstats": { "field": "pmtamount" } } } } }, "size": 0 }'

Execute the query, parse result into a data.table

revenueDT <- essearch( eshost = 'http://mydb.mycompany.com:9200' , esindex = "transactions" , size = 1000 , querybody = qbody , n_cores = 1 ) ```

In the example above, we used the date_histogram and extended_stats aggregations. es_search() has built-in support for many other aggregations and combinations of aggregations, with more on the way. Please see the table below for the current status of the package. Note that names of the form "agg1 - agg2" refer to the ability to handled aggregations nested inside other aggregations.

|Agg type | support? | |:------------------------------------------|:--------:| |"cardinality" |YES | |"date_histogram" |YES | |datehistogram - cardinality |YES | |datehistogram - extendedstats |YES | |datehistogram - histogram |YES | |datehistogram - percentiles |YES | |datehistogram - significantterms |YES | |datehistogram - stats |YES | |datehistogram - terms |YES | |"extended_stats" |YES | |"histogram" |YES | |"percentiles" |YES | |"significant terms" |YES | |"stats" |YES | |"terms" |YES | |terms - cardinality |YES | |terms - datehistogram |YES | |terms - datehistogram - cardinality |YES | |terms - datehistogram - extendedstats |YES | |terms - datehistogram - histogram |YES | |terms - datehistogram - percentiles |YES | |terms - datehistogram - significantterms |YES | |terms - datehistogram - stats |YES | |terms - datehistogram - terms |YES | |terms - extendedstats |YES | |terms - histogram |YES | |terms - percentiles |YES | |terms - significant_terms |YES | |terms - stats |YES | |terms - terms |YES |

Owner

Name: Uptake
Login: uptake
Kind: organization
Location: Chicago, IL

Website: https://www.uptake.com
Repositories: 11
Profile: https://github.com/uptake

Uptake

GitHub Events

Total

Create event: 5
Release event: 1
Issues event: 49
Watch event: 2
Delete event: 4
Issue comment event: 59
Push event: 30
Pull request review event: 42
Pull request review comment event: 36
Pull request event: 55
Fork event: 1

Last Year

Create event: 5
Release event: 1
Issues event: 49
Watch event: 2
Delete event: 4
Issue comment event: 59
Push event: 30
Pull request review event: 42
Pull request review comment event: 36
Pull request event: 55
Fork event: 1

Committers

Last synced: 9 months ago

All Time

Total Commits: 249
Total Committers: 32
Avg Commits per committer: 7.781
Development Distribution Score (DDS): 0.406

Past Year

Commits: 25
Committers: 2
Avg Commits per committer: 12.5
Development Distribution Score (DDS): 0.04

Top Committers

Name	Email	Commits
James Lamb	j**0@g**m	148
Michael Frasco	m**6@g**m	16
Austin Dickey	a**y@u**m	14
James Lamb	j**b@u**m	9
Weiwen Gu	w**u@u**m	9
csyhuang	c**g@u**u	8
Daniel Maynard	D**3@g**m	6
drkarthi	k**h@g**m	3
Chris Black	c**k@g**m	3
Nick Paras	n**p@g**m	3
William Dearden	w**n@u**m	2
Nick Paras	n**s@u**m	2
Ankur Srivastava	a**a@c**m	2
Eric	e**7@g**m	2
Matthew Cohen	m**t@m**z	2
Weiwen Gu	g**w@g**m	2
yashgoswami-infy	5****y	2
Bernard Beckerman	b**n@g**m	2
Kyle Szela	k**4@g**m	1
Mohneet	m**t@g**m	1
Stephanie	s**r@g**m	1
varadpoddar	v**3@g**m	1
Jim Rennie	j**e@g**m	1
Eric	e**c@E**l	1
Timothy Chang	t**g@v**m	1
James McElveen	j**2@g**m	1
Eric Hall	e**l@u**m	1
Austin Dickey	a**y@g**m	1
Yuan (Terry) Tang	t****n	1
İrfan Şener	2****r	1
and 2 more...

Committer Domains (Top 20 + Academic)

uptake.com: 6 valencehealth.com: 1 mattbsg.xyz: 1 ch1data08.uptake.com: 1 uchicago.edu: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time

Total issues: 82
Total pull requests: 92
Average time to close issues: over 3 years
Average time to close pull requests: 9 days
Total issue authors: 8
Total pull request authors: 14
Average comments per issue: 1.34
Average comments per pull request: 1.53
Merged pull requests: 79
Bot issues: 0
Bot pull requests: 1

Past Year

Issues: 6
Pull requests: 42
Average time to close issues: 17 days
Average time to close pull requests: 2 days
Issue authors: 2
Pull request authors: 2
Average comments per issue: 0.83
Average comments per pull request: 0.52
Merged pull requests: 36
Bot issues: 0
Bot pull requests: 1

View more stats

Top Authors

Issue Authors

jameslamb (74)
sebastiz (2)
bstaszkiewicz (1)
jayqi (1)
ntdef (1)
hadley (1)
fracanz (1)
skirmer (1)

Pull Request Authors

jameslamb (74)
chrsblck (3)
yashgoswami-infy (2)
MattBSG (2)
varadpoddar (1)
geraudster (1)
darktrooper3 (1)
paulokenza (1)
irfansener (1)
luisdelahoz (1)
dependabot[bot] (1)
ghost (1)
lakshay-angrish (1)
bernardbeckerman (1)

Top Labels

Issue Labels

help wanted (55) enhancement (45) python (44) R (31) good first issue (20) docs (16) ci (15) Hacktoberfest (9) question (5) bug (3) maintenance (2) awaiting-response (1)

Pull Request Labels

maintenance (29) docs (12) ci (7) python (2) enhancement (1) awaiting-response (1)

Packages

Total packages: 2
Total downloads:
- cran 194 last-month
Total docker downloads: 42,767

Total dependent packages: 0
(may contain duplicates)
Total dependent repositories: 0
(may contain duplicates)
Total versions: 8
Total maintainers: 1

cran.r-project.org: uptasticsearch

Get Data Frame Representations of 'Elasticsearch' Results

Homepage: https://github.com/uptake/uptasticsearch
Documentation: http://cran.r-project.org/web/packages/uptasticsearch/uptasticsearch.pdf
License: BSD_3_clause + file LICENSE
Latest release: 1.0.0
published 12 months ago

Versions: 7
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 194 Last month
Docker Downloads: 42,767

Rankings

Forks count: 2.0%

Stargazers count: 7.1%

Average: 23.6%

Dependent packages count: 29.8%

Dependent repos count: 35.5%

Downloads: 43.4%

Maintainers (1)

jaylamb20@gmail.com

Last synced: 6 months ago

conda-forge.org: r-uptasticsearch

Homepage: https://github.com/uptake/uptasticsearch
License: BSD-3-Clause
Latest release: 0.4.0
published almost 6 years ago

Versions: 1
Dependent Packages: 0
Dependent Repositories: 0

Rankings

Forks count: 25.6%

Dependent repos count: 34.0%

Average: 36.9%

Stargazers count: 36.9%

Dependent packages count: 51.2%

Last synced: 6 months ago

Dependencies

r-pkg/DESCRIPTION cran

R >= 3.3.0 depends
assertthat >= 0.2.0 imports
data.table * imports
futile.logger * imports
httr * imports
jsonlite * imports
purrr * imports
stringr * imports
utils * imports
uuid * imports
covr * suggests
knitr * suggests
rmarkdown * suggests
testthat * suggests

.github/workflows/ci.yml actions

actions/checkout v2 composite
r-lib/actions/setup-pandoc v2 composite
r-lib/actions/setup-r v2 composite
re-actors/alls-green v1.2.2 composite

uptasticsearch

Science Score: 36.0%

Keywords

Keywords from Contributors

Repository

Basic Info

Statistics

Topics

Metadata Files

README.md

uptasticsearch

Introduction

Table of contents

How it Works

Installation

R

Usage Examples

Example 1: Get a Batch of Documents

Build your query in an R string

Execute the query, parse into a data.table

Example 2: Aggregation Results

Build your query in an R string

Execute the query, parse result into a data.table

Owner

GitHub Events

Total

Last Year

Committers

All Time

Past Year

Top Committers

Committer Domains (Top 20 + Academic)

Issues and Pull Requests

All Time

Past Year

Top Authors

Issue Authors

Pull Request Authors

Top Labels

Issue Labels

Pull Request Labels

Packages

cran.r-project.org: uptasticsearch

Rankings

Maintainers (1)

conda-forge.org: r-uptasticsearch

Rankings

Dependencies