Recent Releases of datatools
datatools - v1.3.4
Added json2jsonl. It will render a JSON array document as JSON lines. A -as-dataset option is included. If
an top level attribute name is provided and matches the object it will render the result as a dataset load
componatible object. Of the top level attribute name is not found then that object is skipped with a message
written to standard error.
Full Changelog: https://github.com/caltechlibrary/datatools/compare/v1.3.3...v1.3.4
- Go
Published by rsdoiel 8 months ago
datatools - Improved CRLF (EOL) handling for Windows
Added CRLF option on CSV write to better support Windows, see issue #24. This is a minor release bump. The prior behavior of CSV writes was to default to LF regardless of operating system. Now by default the use of CRLF is set to return for Windows and false for everyone else. It can be explicitly set using the -crlf command line option for programs that output CSV content.
Full Changelog: https://github.com/caltechlibrary/datatools/compare/v1.2.12...v1.3.0
- Go
Published by rsdoiel 11 months ago
datatools - fixed issue #22
fix issue #22, codemeta2cff will now correctly identifying DOI without https://doi.org prefix.
Full Changelog: https://github.com/caltechlibrary/datatools/compare/v1.2.11...v1.2.12
- Go
Published by rsdoiel about 1 year ago
datatools - Bug fix issue #21
This release addresses issue #21 changing the CopyrightYear (copyrightYear in JSON) to an int.
- Go
Published by rsdoiel about 1 year ago
datatools - Minor bug fix
The reldate cli was exiting with a panic after error message. The error message is now displayed followed by a call of os.Exit(1) called to avoid the panic.
Full Changelog: https://github.com/caltechlibrary/datatools/compare/v1.2.9...v1.2.10
- Go
Published by rsdoiel over 1 year ago
datatools - Introducing jsonobjects2csv
This release features a new tool, jsonobjects2csv, which will convert JSON expressing a list of objects into a CSV file. If the objects contain complex attributes (lists and objects) those get converted into YAML (might change representation in the future).
- Go
Published by rsdoiel almost 2 years ago
datatools - Maintenance release, option hanlding and docs
Normalized handling of showHelp, showLicense and showVersion. Fixed man pages.
Full Changelog: https://github.com/caltechlibrary/datatools/compare/v1.2.6...v1.2.7
- Go
Published by rsdoiel almost 2 years ago
datatools - Maintenance release
Fixed a regression in jsonjoin which had the wrong man page again and was as failing tests. Updated to go1.22 and updated the dependencies. Brought build process up to date with Caltech Library Digital Library Development Group practices.
Full Changelog: https://github.com/caltechlibrary/datatools/compare/v1.2.5...v1.2.6
- Go
Published by rsdoiel almost 2 years ago
datatools - fix jsonjoin behavior
The documentation and behavior of jsonjoin was confusing and inconsistent. This release fixes both some ambiguous code and documents the expected behavior for join JSON objects. Also corrects problem where the man page for jsonjoin returns the manual page for json2toml.
- Go
Published by rsdoiel over 2 years ago
datatools - bug fix in JSON output
This release addresses issue #16 and normalizes JSON output to not encode HTML entities in Unicode code points notation.
- Go
Published by rsdoiel over 2 years ago
datatools - changes in codemeta2cff, see issue #15
This release features a change related to issue #15 for codemeta2cff command line tool. It has also be compiled with go 1.20.5.
Zip install files have been updated to reflect availability of installer.sh and for Linux/macOS (where uname is available) now reflect that preferred description of the CPU and OS.
Linux (64 bit): Intel style CPU use Linux-x8664, ARM uses Linux-aarch64 Raspberry Pi OS uses Linx-armv7l Windows uses Windows-x8664 (Windows running on Intel) and Windows-arm64 (for Windows on ARM) macOS uses macOS-arm64 (for the newer M1, M2 Apple CPU) and macOS-x86_64 for the older generation of Intel based Macs
- Go
Published by rsdoiel over 2 years ago
datatools - Added missing csv2tab
Add missing csv2tab to convert a file from comma separated values to tab separated values
- Go
Published by rsdoiel almost 3 years ago
datatools - Bug fix, issue #14
Fixed bug in configuration handling for setting the delimiter used by sql2csv.
- Go
Published by rsdoiel almost 3 years ago
datatools - Code cleanup and simplification
This release drops the requirement for cli package to be available in favor of our idiomatic use of flag package. Also simplifies the string cli implementation, e.g. options always come before the verb. For all command line programs you can use the input/output options with "-" as the name to specify explicitly to use standard input or standard output. This was implemented in some of the command line programs but not is implemented across the board. Drop parameters like generate-markdown, generate-manpage in favor of Markdown documents and Pandoc. "examples" is no longer a support option as the man pages supply examples directly as does the "-help" option.
This release include man pages in the release zip file. These can be copied into your MANPATH appropriately.
- Go
Published by rsdoiel almost 3 years ago
datatools - SQL Query to CSV tool addition
This minor release adds sql2csv command line tool let you easily run a SQL query and render the results CSV encoded. Support SQLite3 and MySQL 8 databases. May support Postgres (as yet untested). The cli can be configured to use another character as column delimited (e.g. tab) via command line options.
- Go
Published by rsdoiel almost 3 years ago
datatools - codemeta support improvements
Added missing codemeta maintainer support to codemeta2cff as well as the example Pandoc templates codemeta-md.tmpl and codemeta-cff.tmpl. The maintainer attribute of the codemeta object is a list of people or organizations similar to author and contributor.
- Go
Published by rsdoiel about 3 years ago
datatools - Fixed typo in generated CITATION.cff
"keywords" was missing the "s" in generation of CITATION.cff by codemeta2cff. See Issue #10.
- Go
Published by rsdoiel over 3 years ago
datatools - Updated support for CITATION.cff 1.2
This release is for updates to citation2ff tool bring the GitHub CITATION.cff file generation up to the 1.2 standard described at --
https://github.com/citation-file-format/citation-file-format/blob/main/schema-guide.md
- Go
Published by rsdoiel over 3 years ago
datatools - Updated codemeta support
This release features a change in the codemeta package. The change can be breaking, i.e. the affiliation value now expects a PersonOrOrganization struct rather than a string. Fixed a bug in the Makefile for installing from source on macOS. Now using a "mv" instead of "cp" to install to solve a security issue when copying the binary.
- Go
Published by rsdoiel over 3 years ago
datatools - Released as a snap
This release incorporates the snap/snapcraft.yaml file for releasing these tools as a snap. This release has been compiled with Go 1.18.1.
Full Changelog: https://github.com/caltechlibrary/datatools/compare/v1.0.4...1.0.5
- Go
Published by rsdoiel over 3 years ago
datatools - Experiment with snaps
This release is just an experiment with generating amd64 snaps for use with Linux. Snap support may or not make it to a stable release.
Full Changelog: https://github.com/caltechlibrary/datatools/compare/v1.0.4...1.0.5-dev-2
- Go
Published by rsdoiel almost 4 years ago
datatools - Added sample feature in jsonrange
This pre-release adds an experimental feature to jsonrange that allows you to take a "sample" of a list of objects you are iterating through with JSON range. The sample option works like limit but does a shuffle of the full array before applying limit. e.g. if I have an array of [1,3,4,5,6,7,8,9,10,11] and use the command echo '[1,2,3,4,5,6,7,8,9,10,11]' | jsonrange -sample=3 I will get a random sample of three elements from the array.
- Go
Published by rsdoiel over 4 years ago
datatools - Stable release
Added codemeta2cff cli. Improving codemeta structured data support.
- Go
Published by rsdoiel over 4 years ago
datatools - bug fixes, codemeta2cff
This is a bug fix release. The initial pass at codemeta2cff didn't write the version string.
- Go
Published by rsdoiel over 4 years ago
datatools - Added codemeta2cff
Add a new command line tool, codemeta2cff which will convert the codemeta.json file to a CITATION.cff file. This tool is a proof of concept and will likely change in upcoming releases.
- Go
Published by rsdoiel over 4 years ago
datatools - Bug fix in reldate
The command line tool reldate confused Monday and Month. This has been corrected.
- Go
Published by rsdoiel over 4 years ago
datatools - go1.16 support
Updated to support compiling with go1.16. Updated templates, version and copyright year. Added experimental M1 compiled binaries.
- Go
Published by rsdoiel almost 5 years ago
datatools - Stability Release
Minor additions to support compilation with go1.16. This library of command line tools has been stable since 2019, time to make this version 1.
- Go
Published by rsdoiel almost 5 years ago
datatools - Maintenance release, compiled with go v1.12, four new commands
This is primarily a maintenance release. It is compiled using Go v1.12.4. It includes four new commands toml2json, yaml2json, json2toml, json2yaml adding support for working with TOML and YAML content.
- Go
Published by rsdoiel over 6 years ago
datatools - Updates for conference talk
Minor bug fixes. This release is packaged so that the demos at OR 2018 in Bozenman, MT have a release number on the cli.
- Go
Published by rsdoiel over 7 years ago
datatools - csv tools normalization, bug fixes
Fixed issues #1, #2 in this release. Normalized CSV utilities options support lazy quotes and trimming leading space for CSV inputs.
Updated docs and added quoting example in how-to/csvcols/index.md.
- Go
Published by rsdoiel about 8 years ago
datatools - csvrows - bug fixes and added random row support
Fixed some range expression bugs, added tests, now supporting ranges expressions like 11-13 as well as colon notation of 11:13. Range expressions can be compound expressions e.g. 1,2,8-10 or 1,2,8:10.
Fixed output bug on csvrows. If no rows expressions provided and input is read from standard in then all rows will be written back out (validated through go's csv encoder/decoder). Show header option now respected correctly in csvrows.
- Go
Published by rsdoiel about 8 years ago
datatools - Quoting bug fix in jsoncols
Fixed a quoting bug in jsoncols, updated test_cmds.bash to test for bug.
- Go
Published by rsdoiel about 8 years ago
datatools - Bug fixes, normalization of options and io, documentation changes
datatools now uses a common vocabulary for options and input/output/error handling. This changed all the tools and in some cases changed the names of options. This will brake some scripts where the option names changed, were removed or behavior standardized.
The new string command conforms with the changes while retaining its "action" oriented syntax. A number of string specific commands have been removed along with the experimental vcard2json command. The documentation has been substantially re-organized.
- Go
Published by rsdoiel about 8 years ago
datatools - string command
The experimental string functions (e.g. toupper, tolower, joinstring, splitstring) have been consolidated in a single command called string. Starting migrating to new Cli object in cli v0.0.5. Most cmds still using old Config. Added testcmds.bash for testing the _string command and will add more tests for others as I migrate to new cli code.
- Go
Published by rsdoiel about 8 years ago
datatools - string functions
Added several new string related command line tools - joinstring, hasprefix, hassuffix, trimprefix, trimsuffix, toupper, tolower and totitle. Bug fixes, cleaned up docs. timefmt had two options renamed -input became -input-format and -output became -output-format to better align with the usage of -input and -output in the rest of the tools.
- Go
Published by rsdoiel about 8 years ago
datatools - csvcleaner added
This release adds another tool, csvcleaner, which does some CSV normalization based for column count, leading/trailing spaces, comments and delimiter based on options selected.
- Go
Published by rsdoiel about 8 years ago
datatools - splitstrings cli
This release adds a cli called splitstring which supports splitting a string into a JSON array or a new line delimited list.
- Go
Published by rsdoiel about 8 years ago
datatools - Bug fixes, jsoncols
When v0.0.14 was pushed out jsoncols defaulted to pushing out CSV formatted output, this broke the jsoncols as JSON filter use case. CSV output is now an option and jsoncols reverts to outputing JSON formatted results.
- Go
Published by rsdoiel about 8 years ago
datatools - Incremental Improvemnts
Bugs fixes, documentation fixes, jsoncols now uses encoding/csv for output.
- Go
Published by rsdoiel over 8 years ago
datatools - jsonjoin improvements
Add support to explicitly create an empty root JSON object to join against or specify a root JSON object read from standard input or a specified file. Improved documentation describing default behavior as branching merge with options for flat merges via -update and -overwrite.
- Go
Published by rsdoiel over 8 years ago
datatools - Harmonizing csv commands
All the csv commands now support setting a character delimiter. This allows us to use the csv commands with other delimiters (e.g. colon or pipe delimited files). Unless noted the delimiter affects both input and output.
Here's an example of using csvfind one a pipe delimited file called file.txt looking for "Fred" in the second column.
shell
csvfind -i file.txt -d '|' -col 2 "Fred"
- Go
Published by rsdoiel over 8 years ago
datatools - Combined release of shelltools and datatools
This release marks the merging of shelltools and datatools projects. Moving forward improvements, bug fixes will happen in the datatools project.
- Go
Published by rsdoiel over 8 years ago
datatools - New parameters and installation process
The primary feature of this release is individual Zip files for compiled binaries based on platform. Additional two commands, jsoncols and jsonrange, now have a "-permissive" option which suppress errors codes and messages (useful in Bash scripting where an empty string is sufficient).
- Go
Published by rsdoiel over 8 years ago
datatools - Column counting from 1 instead of zero
I've updated the command line options for csvcols, csvfind and csvjoin to reference column number starting with column one rather than column zero. This makes the behavior match how cut works (e.g. -f 1 for first field) as well as how most people think of counting columns in spreadsheets.
- Go
Published by rsdoiel over 8 years ago
datatools - new cli, bug fixes and cli option normalization
Changes:
- csvcols has an option change, -f is now -col
Additions:
- csvfind will help you find things in a CSV file by searching a single colum
- search options include exact match, contains and Levenshtein edit distance
- csvjoin will let you join two CSV files on a column
- includes support for approximate matching via Lenenshtein edit distance on join column
Documentation has been re-organized. Beginning to add a "How To" section for website. Lots of small BUG fixes.
datatools Go package is beginning to combine common functionality used between the CLI. dotpath continues to be debugged.
- Go
Published by rsdoiel over 8 years ago
datatools - Added csv2json cli
Added a new command line tool named csv2json. This will convert a CSV file into a series of JSON blobs output as either a JSON array or one JSON blob per line. Use case is turning a CSV file into a set of JSON blobs that could be easily stored in dataset.
- Go
Published by rsdoiel over 8 years ago
datatools - Minor tweaks, bug fixes and improvements
- Adding leading and trailing new line for CSV to Markdown table conversion (csv2mdtable).
- Improved docs.
- Linted and formatted Bash scripts.
- Improved Makefile
- Go
Published by rsdoiel over 8 years ago
datatools - jsonrange refinement
In this release updates were focused on jsonrange. It now returns keys by default, handles input/output like jsoncols and can returns lengths or last key index, and values depending on how you set options.
- Go
Published by rsdoiel almost 9 years ago
datatools - dotpath replacing jid
The primary purpose of the release is the swapping out of jid for a local submodule named dotpath. This removes a dependency for an external package and keeps the code minimal (dot path in only about 150 lines). It does mean that interactive mode of jsoncols has been removed. If you need this then use jid which is nice for this type of exploration. There are sure to be bugs introduced by this switch... YMMV
- Go
Published by rsdoiel almost 9 years ago
datatools - Proof of concept data tools for CSV, JSON and XLSX files
This is a set of tools for working with JSON, CSV and Excel Workbook content. It includes the following tools - csvcols - filters or display CSV row outpout - csvjoin - joins two CSV files by match designated column values in each file - jsoncols - filters JSON dot paths into columns in CSV format (by default) - jsonrange - iterates over JSON object (e.g. an array identified by a dot path) rendering one element per line - xlsx2json - converts an Excel Workbook sheet into a JSON 2d array - xlsx2csv - converts an Excel Workbook sheet into CSV - csv2mdtable - converts CSV into a Github Flavored Markdown table - csv2xlsx - convert CSV into a designated Excel Workbook sheet
- Go
Published by rsdoiel almost 9 years ago