Recent Releases of noctua
noctua - noctua 2.6.2
Feature:
- Add catalog support (#194)
- fix
dbExistsTableto catch update AWS error message. - add support to
dbplyr 2.3.3.9000+
Bug Fix:
- Can now write/append an empty data frame (#200, @mgirlich).
Internals:
- Remove AWS calls to AWS Glue
- Remove reader soft dependency
- R
Published by DyfanJones over 2 years ago
noctua - noctua 2.6.1
Bug Fix:
- Prevent assuming role from
AWS_ROLE_ARN. This caused confusing when connecting through web identity (RAthena # 177) - Support
dbplyr::in_catalogwhen working withdplyr::tbl(RAthena # 178)
- R
Published by DyfanJones about 3 years ago
noctua - noctua 2.6.0
Feature:
- Add clears3resource parameter to RAthena_options to prevent AWS Athena output AWS S3 resource being cleared up by
dbClearResult(RAthena # 168). Thanks to @juhoautio for the request. - Support extra
pawsparameters (RAthena # 169) - Support
endpoint_overrideparameter allow default endpoints for each service to be overridden accordingly (RAthena # 169). Thanks to @aoyh for the request and checking the package in development.
- R
Published by DyfanJones almost 4 years ago
noctua - noctua 2.5.1
Bug Fix:
- Fixed unit test helper function
test_datato usesizeparameter explicitly.
- R
Published by DyfanJones about 4 years ago
noctua - noctua 2.5.0
Feature:
- Allow all information messages to be turned off (#178)
- Allow
noctua_optionsto change 1 parameter at a time without affecting other pre-configured settings - Return warning message for deprecated
retry_quietparameter innoctua_optionsfunction.
- R
Published by DyfanJones about 4 years ago
noctua - noctua 2.4.0
Feature:
- Add support
dbplyr2.0.0 backend API. - Add method to set unload on a package level to allow
dplyrto benefit fromAWS Athena unloadmethods (#174).
Bug Fix:
- Ensure
dbGetQuery,dbExecute,dbSendQuery,dbSendStatementwork on older versions ofR(#170). Thanks to @tyner for identifying issue. - Caching would fail when statement wasn't a character (#171). Thanks to @ramnathv for identifying issue.
- R
Published by DyfanJones about 4 years ago
noctua - v-2.3.0
Feature:
- Add support to
AWS Athena UNLOAD(#160). This is to take advantage of read/write speedparquethas to offer.
```python import awswrangler as wr
import getpass bucket = getpass.getpass() path = f"s3://{bucket}/data/"
if "awswranglertest" not in wr.catalog.databases().values: wr.catalog.createdatabase("awswrangler_test")
cols = ["id", "dt", "element", "value", "mflag", "qflag", "sflag", "obstime"]
df = wr.s3.readcsv( path="s3://noaa-ghcn-pds/csv/189", names=cols, parsedates=["dt", "obs_time"]) # Read 10 files from the 1890 decade (~1GB)
wr.s3.toparquet( df=df, path=path, dataset=True, mode="overwrite", database="awswranglertest", table="noaa" );
wr.catalog.table(database="awswrangler_test", table="noaa") ```
```r library(DBI)
con <- dbConnect(noctua::athena())
Query ran using CSV output
system.time({ df = dbGetQuery(con, "SELECT * FROM awswrangler_test.noaa") })
Info: (Data scanned: 80.88 MB)
user system elapsed
57.004 8.430 160.567
noctua::noctuaoptions(cachesize = 1)
Query ran using UNLOAD Parquet output
system.time({ df = dbGetQuery(con, "SELECT * FROM awswrangler_test.noaa", unload = T) })
Info: (Data scanned: 80.88 MB)
user system elapsed
21.622 2.350 39.232
Query ran using cache
system.time({ df = dbGetQuery(con, "SELECT * FROM awswrangler_test.noaa", unload = T) })
Info: (Data scanned: 80.88 MB)
user system elapsed
13.738 1.886 11.029
```
- R
Published by DyfanJones over 4 years ago
noctua - v-2.2.0
Bug Fix:
- sqltranslateenv correctly translates R functions quantile and median to AWS Athena equivalents (#153). Thanks to @ellmanj for spotting issue.
Feature:
- Support AWS Athena timestamp with time zone data type.
- Properly support data type list when converting data to AWS Athena SQL format.
```r library(data.table) library(DBI)
x = 5
dt = data.table( var1 = sample(LETTERS, size = x, T), var2 = rep(list(list("var3"= 1:3, "var4" = list("var5"= letters[1:5]))), x) )
con <- dbConnect(noctua::athena())
> Version: 2.2.0
sqlData(con, dt)
Registered S3 method overwritten by 'jsonify':
method from
print.json jsonlite
Info: Special characters "\t" has been converted to " " to help with Athena reading file format tsv
var1 var2
1: 1 {"var3":[1,2,3],"var4":{"var5":["a","b","c","d","e"]}}
2: 2 {"var3":[1,2,3],"var4":{"var5":["a","b","c","d","e"]}}
3: 3 {"var3":[1,2,3],"var4":{"var5":["a","b","c","d","e"]}}
4: 4 {"var3":[1,2,3],"var4":{"var5":["a","b","c","d","e"]}}
5: 5 {"var3":[1,2,3],"var4":{"var5":["a","b","c","d","e"]}}
> Version: 2.1.0
sqlData(con, dt)
Info: Special characters "\t" has been converted to " " to help with Athena reading file format tsv
var1 var2
1: 1 1:3|list(var5 = c("a", "b", "c", "d", "e"))
2: 2 1:3|list(var5 = c("a", "b", "c", "d", "e"))
3: 3 1:3|list(var5 = c("a", "b", "c", "d", "e"))
4: 4 1:3|list(var5 = c("a", "b", "c", "d", "e"))
5: 5 1:3|list(var5 = c("a", "b", "c", "d", "e"))
``` v-2.2.0 now converts lists into json lines format so that AWS Athena can parse with sql array/mapping/json functions. Small down side a s3 method conflict occurs when jsonify is called to convert lists into json lines. jsonify was choose in favor to jsonlite due to the performance improvements (#156).
- R
Published by DyfanJones over 4 years ago
noctua - v-2.1.0
Bug Fix:
dbIsValidwrongly stated connection is valid for result class when connection class was disconnected.sql_translate_env.pastebroke with latest version ofdbplyr. New method is compatible withdbplyr>=1.4.3(#149).
Feature:
sql_translate_env: add support forstringr/lubridatestyle functions, similar to Postgres backend.write_binnow doesn't chunk writeBin if R version is greater than 4.0.0 https://github.com/HenrikBengtsson/Wishlist-for-R/issues/97 (#149)dbConnectaddtimezoneparameter so that time zone betweenRandAWS Athenais consistent.
- R
Published by DyfanJones over 4 years ago
noctua - noctua-v2.0.1
This is a hot fix patch to fix keyboard interrupt not raising errors correctly.
- R
Published by DyfanJones almost 5 years ago