Data

Tools

Indexes

Applications

Experiments

Open Source Science

Recent Releases of RAthena

RAthena - RAthena 2.6.1

Bug Fix:

  • Prevent assuming role from AWS_ROLE_ARN. This caused confusing when connecting through web identity (#177)
  • Support dbplyr::in_catalog when working with dplyr::tbl (#178)

- R
Published by DyfanJones about 3 years ago

RAthena - RAthena 2.6.0

Bug Fix:

  • Delay Python to R conversion to prevent 64 bit integer mapping to R's base 32 bit integer (#168) causing the follow bug in Data Scan info message. Thanks to @juhoautio for identifying issue. INFO: (Data scanned: -43839744 Bytes)

Feature:

  • Add clear_s3_resource parameter to RAthena_options to prevent AWS Athena output AWS S3 resource being cleared up by dbClearResult (#168). Thanks to @juhoautio for the request.
  • Support extra boto3 parameters for boto3.session.Session class and client method (#169)
  • Support endpoint_override parameter allow default endpoints for each service to be overridden accordingly (#169). Thanks to @aoyh for the request and checking the package in development.

- R
Published by DyfanJones almost 4 years ago

RAthena - RAthena 2.5.1

Bug Fix:

  • Fixed unit test helper function test_data to use size parameter explicitly.

- R
Published by DyfanJones about 4 years ago

RAthena - RAthena 2.5.0

Feature:

  • Allow all information messages to be turned off (noctua # 178).
  • Allow RAthena_options to change 1 parameter at a time without affecting other pre-configured settings
  • Return warning message for deprecated retry_quiet parameter in RAthena_options function.

- R
Published by DyfanJones about 4 years ago

RAthena - RAthena 2.4.0

Feature:

  • Add support dbplyr 2.0.0 backend API.
  • Add method to set unload on a package level to allow dplyr to benefit from AWS Athena unload methods (noctua # 174).

Bug Fix:

  • Ensure dbGetQuery, dbExecute, dbSendQuery, dbSendStatement work on older versions of R (noctua # 170). Thanks to @tyner for identifying issue.
  • Caching would fail when statement wasn't a character (noctua # 171). Thanks to @ramnathv for identifying issue.

- R
Published by DyfanJones about 4 years ago

RAthena - v-2.3.0

Feature:

```python import awswrangler as wr import getpass

bucket = getpass.getpass() path = f"s3://{bucket}/data/"

if "awswranglertest" not in wr.catalog.databases().values: wr.catalog.createdatabase("awswrangler_test")

cols = ["id", "dt", "element", "value", "mflag", "qflag", "sflag", "obstime"]

df = wr.s3.readcsv( path="s3://noaa-ghcn-pds/csv/189", names=cols, parsedates=["dt", "obs_time"]) # Read 10 files from the 1890 decade (~1GB)

wr.s3.toparquet( df=df, path=path, dataset=True, mode="overwrite", database="awswranglertest", table="noaa" );

wr.catalog.table(database="awswrangler_test", table="noaa") ```

```r library(DBI) con <- dbConnect(RAthena::athena())

Query ran using CSV output

system.time({ df = dbGetQuery(con, "SELECT * FROM awswrangler_test.noaa") })

Info: (Data scanned: 80.88 MB)

user system elapsed

57.004 8.430 160.567

RAthena::RAthenaoptions(cachesize = 1)

Query ran using UNLOAD Parquet output

system.time({ df = dbGetQuery(con, "SELECT * FROM awswrangler_test.noaa", unload = T) })

Info: (Data scanned: 80.88 MB)

user system elapsed

21.622 2.350 39.232

Query ran using cache

system.time({ df = dbGetQuery(con, "SELECT * FROM awswrangler_test.noaa", unload = T) })

Info: (Data scanned: 80.88 MB)

user system elapsed

13.738 1.886 11.029

```

- R
Published by DyfanJones over 4 years ago

RAthena - v-2.2.0

Bug Fix:

  • sqltranslateenv correctly translates R functions quantile and median to AWS Athena equivalents (noctua # 153). Thanks to @ellmanj for spotting issue.

Feature:

  • Support AWS Athena timestamp with time zone data type.
  • Properly support data type list when converting data to AWS Athena SQL format.

```r library(data.table) library(DBI)

x = 5

dt = data.table( var1 = sample(LETTERS, size = x, T), var2 = rep(list(list("var3"= 1:3, "var4" = list("var5"= letters[1:5]))), x) )

con <- dbConnect(RAthena::athena())

> Version: 2.2.0

sqlData(con, dt)

Registered S3 method overwritten by 'jsonify':

method from

print.json jsonlite

Info: Special characters "\t" has been converted to " " to help with Athena reading file format tsv

var1 var2

1: 1 {"var3":[1,2,3],"var4":{"var5":["a","b","c","d","e"]}}

2: 2 {"var3":[1,2,3],"var4":{"var5":["a","b","c","d","e"]}}

3: 3 {"var3":[1,2,3],"var4":{"var5":["a","b","c","d","e"]}}

4: 4 {"var3":[1,2,3],"var4":{"var5":["a","b","c","d","e"]}}

5: 5 {"var3":[1,2,3],"var4":{"var5":["a","b","c","d","e"]}}

> Version: 2.1.0

sqlData(con, dt)

Info: Special characters "\t" has been converted to " " to help with Athena reading file format tsv

var1 var2

1: 1 1:3|list(var5 = c("a", "b", "c", "d", "e"))

2: 2 1:3|list(var5 = c("a", "b", "c", "d", "e"))

3: 3 1:3|list(var5 = c("a", "b", "c", "d", "e"))

4: 4 1:3|list(var5 = c("a", "b", "c", "d", "e"))

5: 5 1:3|list(var5 = c("a", "b", "c", "d", "e"))

``` v-2.2.0 now converts lists into json lines format so that AWS Athena can parse with sql array/mapping/json functions. Small down side a s3 method conflict occurs when jsonify is called to convert lists into json lines. jsonify was choose in favor to jsonlite due to the performance improvements (noctua # 156).

- R
Published by DyfanJones over 4 years ago

RAthena - RAthena-2.1.0

Bug Fix:

  • dbIsValid wrongly stated connection is valid for result class when connection class was disconnected.
  • sql_translate_env.paste broke with latest version of dbplyr. New method is compatible with dbplyr>=1.4.3 noctua # 149.

Feature:

  • sql_translate_env: add support for stringr/lubridate style functions, similar to Postgres backend.
  • dbConnect add timezone parameter so that time zone between R and AWS Athena is consistent noctua # 149.

- R
Published by DyfanJones over 4 years ago

RAthena - RAthena-v2.0.1

This is a hot fix patch to fix keyboard interrupt not raising errors correctly.

- R
Published by DyfanJones almost 5 years ago

RAthena - RAthena-v2.0.0

- R
Published by DyfanJones almost 5 years ago

RAthena - RAthena-v1.12.0

- R
Published by DyfanJones about 5 years ago

RAthena - RAthena-v1.11.1

- R
Published by DyfanJones over 5 years ago

RAthena - RAthena-v1.10.0

RAthena now supports keyboard interrupt. This means AWS Athena queries will be stopped when R has been interrupted.

- R
Published by DyfanJones over 5 years ago