Recent Releases of irdmtools
irdmtools - Minor improvement add code to use a ".env" file.
This releasee adds support for ".env" files to the Config struct and funcs.
Full Changelog: https://github.com/caltechlibrary/irdmtools/compare/v0.0.89...v0.0.90
- Go
Published by rsdoiel over 1 year ago
irdmtools - Fix issue where author only has a last name in CrossRef data
This release addresses Issue #80 where data from CrossRef didn't have a given name so author was identified as an organization. Check was revised to only check Family name to identify as person.
- Go
Published by rsdoiel over 1 year ago
irdmtools - Fix CrossRef published date handling
CrossRef seems to be improving the date handling. The "published" property in "work" is often populated, I've updated how publication date is derived in the simplified records to take advantage of that before falling back to the old date sequence preferences.
- Go
Published by rsdoiel over 1 year ago
irdmtools - Improved exit codes for doi2rdm
The exit codes for doi2rdm now differentiate between a problem in formatting and something not being available. If the DOI is not found the exit code will be ENOEXIT (2), if their is a problem cause by formats of the documents being process you'll get an exit code of ENOEXEC (8).
- Go
Published by rsdoiel over 1 year ago
irdmtools - Rename mapped field name
The creators field value for creator type has been remapped from "organization" to "organizational" to match the RDM naming convention.
- Go
Published by rsdoiel almost 2 years ago
irdmtools - Bug fix for doi2rdm
This fixes a bug in how CrossRef authors were mapped for organizational authorship. See issue #78.
Full Changelog: https://github.com/caltechlibrary/irdmtools/compare/v0.0.84...v0.0.85
- Go
Published by rsdoiel almost 2 years ago
irdmtools - Versioned installers included
This is a minor release fixing the installer.sh and installer.ps1 scripts to support installing a specific version. This is needed to decouple the irdmtools releases from the harvester releases.
- Go
Published by rsdoiel almost 2 years ago
irdmtools - added rdmds2citation, improved doi2rdm
This release includes journal title normalization via ISSN prefix in the options.yaml provided to doi2rdm. In addition a new tool, rdmds2citation, will take a dataset collection of RDM objects and map them to a citation object collections. This was included for future feeds work as we continue migrations from EPrints to RDM.
- Go
Published by rsdoiel about 2 years ago
irdmtools - Performance improvements
Improve Performance in getRecordFromPg().
- Go
Published by rsdoiel about 2 years ago
irdmtools - doi2rdm datacite resource mapping change
This release switches from using the citeproc resource type in the DataCite dois data object to using .data.attributes.types.resourceTypeGeneral instead. Also added a "preprint"/"Preprint" resource mapping to our options.yaml default file.
- Go
Published by rsdoiel about 2 years ago
irdmtools - Bug fixes for doi2rdm
This release include bug fixes for for doi2rdm and the DataCite support. Subjects are de-duplicated based on the subject string and if .data.creators[*].nameType is not set I use the fallback of checking for family and given names to set .creators[*].person_or_org.type to "personal" in the RDM JSON document.
- Go
Published by rsdoiel about 2 years ago
irdmtools - Big fix: persson_or_org type mapping
This release fixes a person or org type mapping problem for creators and contributors in doi2rdm for DataCite records.
- Go
Published by rsdoiel about 2 years ago
irdmtools - DataCite integration into doi2rdm
This release include DataCite integration for doi2rdm. This allows retrieving arXiv records from DataCite and rendering a minimal RDM JSON document suitable for importing into RDM. Also includes some minor bug fixes.
- Go
Published by rsdoiel about 2 years ago
irdmtools - Bug fix: ep2ds2citation
Fixed bug where clpid and orcid were not mapping correctly for citations.editor info.
- Go
Published by rsdoiel about 2 years ago
irdmtools - refined ep3ds2citation release
v0.0.75 features an improved implementation of ep3ds2citation. The citation record structure now includes a map of information about the EPrint record's primary document. This includes URL and file related metadata. With this update a "citation" record can provide enough information to create a simple "landing page" if needed or serve as a basis for collection analysis.
- Go
Published by rsdoiel about 2 years ago
irdmtools - refining ep3ds2citations
This release fixes some data alignment in generation a citations dataset collection using ep3ds2citations also normalizes the behavior or cite.ID (reflects ds collection prefix), cite.Collection (reflects the ds collection name) and cite.CollectionID the raw ID in the source dataset collection.
- Go
Published by rsdoiel about 2 years ago
irdmtools - refining ep3ds2citations
This release contains a refinement of ep3ds2citations based on using in feeds.library.caltech.edu repository, newtooling branch.
Full Changelog: https://github.com/caltechlibrary/irdmtools/compare/v0.0.70...v0.0.73
- Go
Published by rsdoiel over 2 years ago
irdmtools - experimental ep3ds2citations
This release includes an experimental tool to aggregate our dataset collections with EPrint structured records into a citations dataset collection.
- Go
Published by rsdoiel over 2 years ago
irdmtools - doi2rdm changes
The tool doi2rdm now expects the options to come from a YAML file provided on the command line. Included in the support options is mapping the publisher name via DOI prefix or ISSN.
To see the YAML file use the -show-yaml option. You can then modify it to better suit your organization.
Example: Save the YAML, edit it with "nano" then use the options.yaml to retrieve the DOI from CrossRef and see how the JSON looks for importing into RDM.
doi2rdm -show-yaml >options.yaml
nano options.yaml
doi2rdm options.yaml https://doi.org/10.1163/15733823-00211P01
Full Changelog: https://github.com/caltechlibrary/irdmtools/compare/v0.0.68...v0.0.70
Full Changelog: https://github.com/caltechlibrary/irdmtools/compare/v0.067...v0.0.70
- Go
Published by rsdoiel over 2 years ago
irdmtools - Bug fix, issue #74
For rdm2eprint the resulting "official_url" field in the JSON output should always be populated.
Full Changelog: https://github.com/caltechlibrary/irdmtools/compare/v0.067...v0.0.68
- Go
Published by rsdoiel over 2 years ago
irdmtools - Maintenance release
This release focuses on updating the dependencies in the built as well as a problem in building the item lists in ep3util where ghost elements would show up at {} in the JSON output. This release compile with go1.21.6.
Full Changelog: https://github.com/caltechlibrary/irdmtools/compare/v0.0.66...v0.067
- Go
Published by rsdoiel over 2 years ago
irdmtools - fix, issue 71
This fixes the mapping of the official url, primary and related objects in rdm2eprint.
Full Changelog: https://github.com/caltechlibrary/irdmtools/compare/v0.0.65...v0.0.66
- Go
Published by rsdoiel over 2 years ago
irdmtools - fix: issue #69
This is a minor change to solve issue #69.
Full Changelog: https://github.com/caltechlibrary/irdmtools/compare/v0.0.64...v0.0.65
- Go
Published by rsdoiel over 2 years ago
irdmtools - rdmutil feature addition
This release is includes a new feature included in rdmutil cli. There is now an action called getrecordversions which will return an RDM record's versions as an array (oldest to newest). It is based on querying the rdmrecordsmetadataversion table and returns the rows json, created, updated and versionid columns.
Full Changelog: https://github.com/caltechlibrary/irdmtools/compare/v0.0.63...v0.0.64
- Go
Published by rsdoiel over 2 years ago
irdmtools - Merged preprint patch
Merged Tom's preprint patch.
What's Changed
- Add preprint by @tmorrell in https://github.com/caltechlibrary/irdmtools/pull/70
Full Changelog: https://github.com/caltechlibrary/irdmtools/compare/v0.0.62...v0.0.63
- Go
Published by rsdoiel over 2 years ago
irdmtools - Added stale id list action to rdmutil
If you've configured rdmutil to access RDM's PostgreSQL database it can produce a list of stale versions of publicly available records using the "getallstale_ids" action.
- Go
Published by rsdoiel over 2 years ago
irdmtools - Refinement for rdmutil get_all_ids
Issues #96 from the feeds repository identified a problem where getallids returned not just all published RDM records but all versions of all published rdmrecords. I've added a restriction to check if the id included is the latest version of the record. The getall_ids options in rdmutil is meant to return a list of the latest version of all published records' rdm ids.
Full Changelog: https://github.com/caltechlibrary/irdmtools/compare/v0.0.60...v0.0.61
- Go
Published by rsdoiel over 2 years ago
irdmtools - rdm2eprint crowswalks primary_object and related_objects
This release addresses issue #66 enabling rdm2eprint to crosswalk RDM's Record.Files.Entries and Record.Files.DefaultPeview elements to primary object and related objects in the EPrint structure.
Full Changelog: https://github.com/caltechlibrary/irdmtools/compare/v0.0.59...v0.0.60
- Go
Published by rsdoiel over 2 years ago
irdmtools - fixes issue #67
RDM DOI from PIDS are now mapped correctly to EP3 struct.
Full Changelog: https://github.com/caltechlibrary/irdmtools/compare/v0.0.58...v0.0.59
- Go
Published by rsdoiel over 2 years ago
irdmtools - fix: issue #65 resource mapping
This release is primarily to include an updated resource mapping identified in issue #65. There are some other minor things that have been fixed with the feeds demo development.
- Go
Published by rsdoiel over 2 years ago
irdmtools - improved ep3util harvesting
This release includes some bug fixed but primarily focuses on improvements to ep3util which can now harvest EPrints directly from the MySQL database for the repository if the environment variables of EPRINTDBHOST, EPRINTDBUSER and EPRINTDBPASSWORD are set.
The feeds demo folder has been renamed feeds-demo (for clarity). The included Bash script make_dataset.bash now can run specific updates by dataset collection. Also successfully updates the thesis example collection.
Full Changelog: https://github.com/caltechlibrary/irdmtools/compare/v0.0.55...v0.0.56
- Go
Published by rsdoiel over 2 years ago
irdmtools - interim progress and bug fixes
This release features updated Go package requirements and the beginning of a demo for building a feeds.library.caltech.edu like static site using irdmtools, dataset, datatools and Pandoc. It also includes some bug fixes. The is just an interim release as I work through some of the critical bugs as I build out the feeds demo.
- Go
Published by rsdoiel over 2 years ago
irdmtools - improved mapping
These release fixes some of the issues discussed in issue #55. Fixed editor handling, corrects series handling, fixed imprint handling, adds other numbering system handling to rdm2eprint.
- Go
Published by rsdoiel over 2 years ago
irdmtools - new tools, rdm2eprint, ep3util
This release is focused on adding two new tools to back port our RDM content to our feeds system. ep3util mirrors rdmutil in irdmtools. It uses the same command structure but is limited to read only functionality. rdm2eprint will read a RDM record via RDM's JSON API and return an EPrint compatible structure for metadata (i.e. excludes support for .documents and .documents.files). It can be sued to process lists of RDM ids or individual rdm ids. It can support rendering as EPrint XML for comparison and debugging purposes. It includes a harvest option which retrieve records from RDM, convert to an EPrint JSON sturcture then save them in a dataset collection.
Also fixed a JSON encoding/decoding bug by using custom JSON encoders/decoders across irdmtools.
- Go
Published by rsdoiel over 2 years ago
irdmtools - Upgrade crossrefapi dependency
This release features crossrefapi v1.0.6 which fixes some oddness regarding HTML entities getting encoded as Unicode code points.
- Go
Published by rsdoiel over 2 years ago
irdmtools - improved ROR retrieve in doi2rdm
These release trims the https://ror.org/ prefix from the ROR retrieve in doi2rdm for funders (issue #50). The func lookupROR now takes a trimPrefix parameter. It also includes an additional Python script geteprintdocuments.py which will fetch the documents for an eprint id from a remote EPrints server via scp and place them in a simplified directory called "problem_files".
- Go
Published by rsdoiel over 2 years ago
irdmtools - improvements to doi2rdm
This release features updates to doi2rdm. It includes changes indicated by issue #40, #48 and #50.
Full Changelog: https://github.com/caltechlibrary/irdmtools/compare/v0.0.49...v0.0.50
- Go
Published by rsdoiel almost 3 years ago
irdmtools - fixes and eprintrest tool added
This release includes a new web service called eprintrest which replicates the EPRints 3.3 REST API for the dataset EPrint for those eprints with a status of "archive". It is minimal but very fast. Requires access to the MySQL eprints database as well as the archives directory for the repository you want to access locally. See the man page for eprintrest in this release for details.
Includes fix for issue #53, problem with duplicate subjects in eprint2rdm.
- Go
Published by rsdoiel almost 3 years ago
irdmtools - Fixed blocking migration issues
This release contains the fixes related to eprint2rdm needed for the migration project in the caltechauthors repository. They related to metadata crosswalks from our customized legacy EPrints repository.
- Go
Published by rsdoiel almost 3 years ago
irdmtools - bug fixes, metdata fixes
This release features two bug fixes in eprint2rdm discovered in our migration project. Rights are now assigned appropriately (had title and description swapped) and EPrint's eprint.date_type with an empty string or NULL is now treated as a "published" date when eprint.date is not NULL.
- Go
Published by rsdoiel almost 3 years ago
irdmtools - mid-migration release
This release includes the changes made based on migrating the majority records from EPrints to RDM. mkeprintredirects.py now uses Postgres database to generate a CSV file of eprintid, rdmid and record status which is then processed into a NginX redirects-map.conf file output.
Full Changelog: https://github.com/caltechlibrary/irdmtools/compare/v0.0.45...v0.0.46
- Go
Published by rsdoiel almost 3 years ago
irdmtools - Fix: related_identifier, DOI handling
This release relates to issues that came to light in caltechauthors repository, issue 71.
Full Changelog: https://github.com/caltechlibrary/irdmtools/compare/v0.0.44...v0.0.45
- Go
Published by rsdoiel almost 3 years ago
irdmtools - Fix: related identifier and identifier list handling
Fixed related identifier and identifier mappings in eprint2rdm.go. Corrected crossref.go's call from AddRelatedIdentifier to AddIdentifier from simplified.go.
Full Changelog: https://github.com/caltechlibrary/irdmtools/compare/v0.0.43...v0.0.44
- Go
Published by rsdoiel almost 3 years ago
irdmtools - fix: improved validation of people elements in contributors and creators
fix: improved validation of people elements in contributors and creators
Full Changelog: https://github.com/caltechlibrary/irdmtools/compare/v0.0.42...v0.0.43
- Go
Published by rsdoiel almost 3 years ago
irdmtools - fix: eprint2rdm custom fields
eprint2rdm was not processing the caltech:group custom field. Fixes empty id string problem.
- Go
Published by rsdoiel almost 3 years ago
irdmtools - Fix duplicate keywords and subjects bug
eprint2rdm was producing duplicate subjects. Fixed and made sure that EPrint keyword and subjects mapped uniquely to .metadata.subjects.
Full Changelog: https://github.com/caltechlibrary/irdmtools/compare/v0.0.40...v0.0.41
- Go
Published by rsdoiel almost 3 years ago
irdmtools - Critical bugs fixed
This release closes the remaining critical bugs needed to be resolved before we migrate CaltechAUTHORS from EPrints to RDM. Fixes include correct mapping of keywords, subjects, errata and notes as well as improved handling of suggests -> internal_notes for review.
Full Changelog: https://github.com/caltechlibrary/irdmtools/compare/v0.0.39...v0.0.40
- Go
Published by rsdoiel almost 3 years ago
irdmtools - bug fixes, improvements in crosswalks
This release contains bug and improvements in crosswalks.
- Go
Published by rsdoiel almost 3 years ago
irdmtools - Fixes to invenio_api.go, eprints_to_rdm.py
This is a minor update, included fixes in invenioapi.go for handling errors better on GET, POST, PUT, PATCH, and DELETE operations. Tom's improvements to eprintsto_rdm.py.
- Go
Published by rsdoiel almost 3 years ago
irdmtools - eprints_to_rdm.py and eprint2rdm improvements
Improved log handling in eprintstordm.py and fixes to eprint2rdm
- Go
Published by rsdoiel almost 3 years ago
irdmtools - debugging: campus only files and file descriptions
This release includes improvements in handling campus only files and adding file descriptions as additional descriptions in eprintstordm.py. Removed the "uploadcampusonlyfile" method from rdmutil class. It is handled in the main processing loop for record versions. The S3 bucket is hard coded for campus only files. This will need to be changed if other institutions use eprintstordm.py.
- Go
Published by rsdoiel almost 3 years ago
irdmtools - Fixes and improvements in rdmutil, irdm/irdmtools.py
This is a debugging release. There are fixes and improvements to rdmutil, irdm/irdmtools.py, eprintstordm.py.
- Go
Published by rsdoiel almost 3 years ago
irdmtools - feature: check_doi improvements
This release contains improvements in handling duplicate doi issues on migrating materials from our old EPrints repository to RDM.
- Go
Published by rsdoiel almost 3 years ago
irdmtools - debugging release
debugging eprintstordm.py, updated eprint2rdm to use environment for EPRINT_HOST. Minor bug fixes in irdmtools/fixup.py
- Go
Published by rsdoiel almost 3 years ago
irdmtools - Debugging release
This release is to test debugging eprintstordm.py migration workflow. Changes included so far are to eprintstordm.py, irdm/irdmtools.py, irdm/fixup.py, eprint2rdm.go, rdmutil.go and invenio_api.go.
- Go
Published by rsdoiel almost 3 years ago
irdmtools - rdmutil basic functionality working
rdmutil can now create draft, add files, send them to the community and review requests. Some of the command actions of rdmutil have been reorganized as a result.
- Go
Published by rsdoiel almost 3 years ago
irdmtools - debugging release
This is an "in progress" release of irdmtools. rdmutil can successfully create new records, create drafts of existing records, upload files to drafts, set files to be enabled/disabled. The submission for review (send to community), submit draft and publish draft do not work in our test deployment of RDM.
- Go
Published by rsdoiel almost 3 years ago
irdmtools - Added support to work with draft to rdmutil
This release includes many updates to rdmutil to provide support to create draft records, add files to them and publish them. This release is for testing purposes.
- Go
Published by rsdoiel almost 3 years ago
irdmtools - Added get_files, get_file, retrieve_file to rdmutil
Added getfiles, getfile, retrieve_file to rdmutil. Now requires v0.0.17 simplified package. Bug fix for issue #43. Prep work for adding file upload support for rdmutil.
- Go
Published by rsdoiel almost 3 years ago
irdmtools - Bug fixes, issue #39 and #41
This release includes bug fixes for issues #39 and #41.
- Go
Published by rsdoiel almost 3 years ago
irdmtools - bug fixes, issue #38
Implemented a different approach to handle multiple ISSN (issue #38 repoened). Improved subjects, keywords and series handling. Improved "imprint:imprint" handling.
- Go
Published by rsdoiel almost 3 years ago
irdmtools -
Remove extraneous ISSN occurrences in eprint2rdm and crossref import to simplified model. ISSN will only show in the "journal:journal" element of the Custom Fields in RDM's schema.
- Go
Published by rsdoiel almost 3 years ago
irdmtools - Bug fixes, issue #37 and #38.
This is a bug fix release with updates to the CrossRef and EPrint to RDM imports reflecting changes needed by issues #37 and #38.
- Go
Published by rsdoiel almost 3 years ago
irdmtools - bug fixes, issue #33, #34.
This release fixes some regressions that from v0.0.20 and v0.0.21 release.
- Go
Published by rsdoiel about 3 years ago
irdmtools - Bug fixes, issue #30
Changed the behavior of what is mapped as "rights.description.en" from CrossRef. I am now mapping in the "content-version" value if it is not empty, otherwise I map the description value across.
- Go
Published by rsdoiel about 3 years ago
irdmtools - Bug fixes, issue #31
Bug fixes, issue #31, a nil pointer reference when comparing affiliations for creators.
- Go
Published by rsdoiel about 3 years ago
irdmtools - Bug fix, issue #29
Fixed incorrect published online date handling.
- Go
Published by rsdoiel about 3 years ago
irdmtools - Bug, issue #28
This release addresses issue #28, and removes some DEBUG statements from output.
- Go
Published by rsdoiel about 3 years ago
irdmtools - Bug fixes, issue #13 through #27
This release fixes a few bugs found in doi2rdm but mainly fixes the mappings for eprint2rdm. Depends on updates in simplified@v0.0.16.
- Go
Published by rsdoiel about 3 years ago
irdmtools - Bug fixes, several issues related to doi2rdm
Fixed issue #10, issue #22, #issue #23, issue #24, issue #25 which impacted doi2rdm.
- Go
Published by rsdoiel about 3 years ago
irdmtools - Bug fixes, issue #20
Now depends on simplified v0.0.14 for fixes to solve issue #20.
- Go
Published by rsdoiel about 3 years ago
irdmtools - Bug fixes, issue #19
Updated dependencies to latest simplified and crossrefapi packages. Improved mapping base on issue #19.
- Go
Published by rsdoiel about 3 years ago
irdmtools - Issue #18 fix, updated to v1.0.4 crossrefapi package
This release fixes bug #18 and updates the crossrefapi package dependency to v1.0.4.
- Go
Published by rsdoiel about 3 years ago
irdmtools - Documentation improvements
This release include man page distribution with the release zip files.
- Go
Published by rsdoiel about 3 years ago
irdmtools - Updates to people2vocabulary
These release includes some bug fixes and code improvements but the primary use facing change is that the option -csv defaults to true and a new option -clrules which defaults to true will enable Caltech Library specific rules.
- Go
Published by rsdoiel about 3 years ago
irdmtools - Improved identifier filtering in people2vocabulary
people2vocabulary will convert the feeds' people.csv and filter non RDM identifiers when rendering the YAML vocabulary. The identifiers supported now are "clpid", "orcid", "isni".
- Go
Published by rsdoiel about 3 years ago
irdmtools - Added csv import to people2vocabulary
This release will map a CSV file (e.g. feeds.library.caltech.edu/people/people.csv) to the RDM people vocabulary file format.
- Go
Published by rsdoiel about 3 years ago
irdmtools - Bug fix people2vocabulary options, installer.sh fixes
Fixed a minor bug in missing the -version and -license options in people2vocabulary. Fixed a bad merge of installer.sh.
- Go
Published by rsdoiel about 3 years ago
irdmtools - Added people2vocabulary
Now requires v0.0.12 of simplified and includes a cli for converting a JSON array of Person into a people vocabulary file suitable for import into RDM.
- Go
Published by rsdoiel about 3 years ago
irdmtools - Improved RDM 11 model support
This release includes bug fixes and improved RDM 11 model support. The doi2rdm cli is implemented at the proof of concept level (e.g. you get a basic record from CrossRef and rendered as a simplified Record). Requires simplified v0.0.11 or better.
- Go
Published by rsdoiel about 3 years ago
irdmtools - improved logging, updated man pages, bug fixes
This release is some minor bug fixes, improved logging and updated man pages. Additional testing and development is needed on doi2rdm for the CrossRef crosswalk. Add ".cl_annotations" to track custom Caltech Library fields exported from EPrints. This will eventually be removed as the Invenio RDM data model improves (e.g. Invenio RDM 12 should include more of the journal fields we need).
The current release requires the latest versions of crossrefapi, eprinttools, and simplified Caltech Library Go packages.
- Go
Published by rsdoiel about 3 years ago
irdmtools - improvements to eprint2rdm, new doi2rdm, bug fixes
This release sees additional configuration available for eprint2rdm (e.g. resource type maps, contributor type maps). The addition of a proof of concept doi2rdm which can convert a CrossRef works record to a simplified record model. Bug fixes and an examples harvester Bash script using eprint2rdm, dataset v2.
- Go
Published by rsdoiel about 3 years ago
irdmtools - all-ids added to eprint2rdm
This release adds a feature to eprint2rdm of fetching all the eprint ids from an EPrints repository using the EPrint REST API. Documentation improvements show an example of a simple EPrints harvester using Bash, dataset and eprint2rdm from irdmtools.
- Go
Published by rsdoiel about 3 years ago
irdmtools - Proof of concept
This is a proof of concept release for irdmtools. It features two command line programs. rdmutil is for working with an Invenio RDM repository (e.g. get a metadata record, get record ids, harvest into a dataset v2 collection) and eprint2rdm which talks to an EPrint repository REST API, retrieves an EPrint record and returns a JSON structure in a simplified record format almost ready for import into Invenio RDM via it's REST API.
The purpose of this release is to establish a proof of concept for a general purpose institutional repository and data management tool that can work with multiple institutional repository systems, facilitate repository migration between systems.
- Go
Published by rsdoiel about 3 years ago