What's Changed

Update GitHub actions by @robkooper in https://github.com/clowder-framework/clowder/pull/461
Fix/escape space name description by @lmarini in https://github.com/clowder-framework/clowder/pull/459
Fix NoSuchElementException for anonymous users in spaces listing and … by @robkooper in https://github.com/clowder-framework/clowder/pull/460
Release/v1.23.0 by @lmarini in https://github.com/clowder-framework/clowder/pull/462

Full Changelog: https://github.com/clowder-framework/clowder/compare/v1.22.1...v1.23.0

- JavaScript
Published by lmarini 7 months ago

v1.22.1 - 2023-11-10

Fixed

Fixed hearthbeat bug where if the owner was not set in extractors_info.json clowder would throw and error and the extractor would not get registered.

- JavaScript
Published by lmarini over 2 years ago

1.22.0 - 2023-09-12

Added

Users can be marked as ReadOnly #405
Delete button to delete section #347
When parameter on API endpoints to enable pagination #266
Extractors can now specify an extractor_key and an owner (email address) when sending a registration or heartbeat to Clowder that will restrict use of that extractor to them.
Added a dropdown menu to space listing to select all spaces, your spaces and also the spaces you have access to. #374
Support for SMTP_FROM in docker-compose yml file. #417
Keycloak provider with secure social #419
Documentation on how to do easy testing of pull requests
citation.cff
New GET sections endpoint to file API and fix missing section routes in javascriptRoutes #410
Google's model viewer within viewer_three.js previewer

Fixed

Updated lastModifiesDate when updating file or metadata to a dataset, added lastModified to UI 386
Disabled button after creating new dataset #311
Changed default to 'Viewer' while inviting users to new spaces #375
Complex extracted JSON metadata objects using arrays are now being indexed properly for search.
Fixed positioning problems related to how the 3D models appear on the screen
Search results are checked to verify nothing has been put in trash before display #377
Previewer source URL in the documentation to point to the Clowder GitHub repo. #395

- JavaScript
Published by lmarini over 2 years ago

1.21.0 - 2022-08-23

Important: This update requires a MongoDB update schema due to a bug in the original migration of showing summary statistics at the space level. Make sure to start the application with -DMONGOUPDATE=1. You can also run the fixCounts.js script prior to upgrading to minimize the downtime.

Added

api.Files jsonfile, adds two fields "downloads" and "views" #228
Dataset and file scala.html pages incl schema.org jsonld metadata for (google)datasetsearch #335
MiniUser and LicenseData now have to_jsonld methods to return string part of #335 metadata
LicenseData has urlViaAttributes used by it's to_jsonld to guess url when empty, for #335
MRI previewer for NIFTY (.nii) files.
Dataset page usually defaults to Files tab, but if no files will now show Metadata first
HEIC (.heic) and HEIF (.heif) mimetypes to support new Apple iPhone image file format.
In the docker container the folder /home/clowder/data is now whitelisted by default for uploading by reference. This can be changed using the environment variable CLOWDER_SOURCEPATH.
The current CLA for developers of clowder.

Fixed

Send email to all admins in a single email when a user submits 'Request access' for a space
Send email to all admins and request user in a single email when any admin accepts/rejects 'Request access' for a space #330
script/code to count space in files was not correct #366
github actions would fail for docker builds due to secrets not existing
Fix to remove dataset from a space #349

Changed

Utils.baseURL now on RequestHeader instead of Request[Any]
MongoDB Service log error:'Not all dataset IDs found for Dataset|Folder bulk get request', now incl all the IDs notFound

- JavaScript
Published by robkooper over 3 years ago

clowder - v1.20.3

1.20.3 - 2022-06-10

Fixed

docker images are build using newest java version #361

Changed

docker builds images for amd64 and arm64 #322

- JavaScript
Published by lmarini over 3 years ago

clowder - 1.20.2

1.20.2 - 2022-04-30

Fixed

swagger lint action
When downloading a file with a ' in the name it would save the file as blob
Fix for a rare race condition with masonry where tiles could end up overlapping in space page.
Fixes bug where same extractor shows up multiple times and all Clowder instances index db on reindex #327

Changed

Changed Enabled By SuperAdmin to read Enabled by Server Admin #344

- JavaScript
Published by robkooper almost 4 years ago

clowder - 1.20.1

1.20.1 - 2022-04-04

Fixed

Fix to incorrect sorting for default cases. #248
Masonry package was not loaded from right location sometimes.
Fixes bug where "Select a Collection" Parent Collection dropdown on Collection page was not working due to missing parameter #343

Added

Documentation: Installing Clowder on Apple Silicon M1
Documentation: Customizing clowder's deployment, simplified duplicate instructinos.
Documentation: Added "How to contribute documentation" page
Documentation: New Sphinx plugins for dropdowns and menus.

- JavaScript
Published by lmarini almost 4 years ago

clowder - v1.20.0

Fixed

Conf file and code had incosistent spelling of BagIt. Now all have capital B and I.
When event stream is disabled don't show for logged in user #280
three.js is no longer associated with application/octet-stream, now with models #305

Changed

Download of dataset/collection now has optional parameter bagit (default false) to download items in bagit format.

- JavaScript
Published by robkooper about 4 years ago

clowder - v1.19.5

Fixed

Removed JMSAppender and SocketServer from log4j.
Cleaned up getting started documentation.

- JavaScript
Published by robkooper about 4 years ago

clowder - v1.19.4

1.19.4 - 2021-11-11

Fixed

Error with adding dataset to collection. #306

- JavaScript
Published by robkooper over 4 years ago

clowder - v1.19.3

1.19.3 - 2021-11-11

See fix-counts.js for a script that can be run before this update (as well as the update for 1.19.0) to pre populate the migration. This will speed up the update and will not impact the running instance.

Fixed

If a space has a lot of datasets, rendering the space page is very slow. Files in a space is now cached.
Set permissions for folders to be 777 this fixes clowder-helm#5

- JavaScript
Published by lmarini over 4 years ago

clowder - v1.19.2

1.19.2 - 2021-10-20

Fixed

Error with library dependencies broke search capabilities, rolled back to known working versions

- JavaScript
Published by robkooper over 4 years ago

clowder - v1.19.1

1.19.1 - 2021-10-19

Added

Support the DefaultAWSCredentialsProviderChain for passing in credentials to the S3ByteStorageService.

Fixed

Cleaning up after a failed upload should no longer decrement the file + byte counts.
Fix the broken preview after file deletion within a folder. #277
Fix public spaces not displaying correctly if not logged in.

Changed

Now building mongo-init and monitor docker containers with python 3.8
Upgraded extractor parameters jsonform to version 2.2.5.

Removed

Check image is now part of ncsa/checks

- JavaScript
Published by lmarini over 4 years ago

clowder - v1.19.0

1.19.0 - 2021-10-05

Important: This update requires a MongoDB update schema due to the new ability of showing summary statistics at the space level. Make sure to start the application with -DMONGOUPDATE=1.

Fixed

Adding dataset to space. Space list on dataset page would be empty - fixed error when no spaces would load. #274
Typos "success" when returning status from API and "occurred" when logging to console.
If a dataset had multiple folders the layout would be wrong.
Collections created using api route are now indexed upon creation. #257

Added

Mime type for geojson
"when" parameter in a few GET API endpoints to enable pagination #266
Show space statistics (bytes, users. etc) #119
"id" in GET metadata.jsonld endpoints #278
'POST /api/files/uploadToDataset' now allows folder_id for uploading file to folder. #232

- JavaScript
Published by lmarini over 4 years ago

clowder - v1.18.1

1.18.1 - 2021-08-16

This release fixes a critical issue where invalid zip files could result in the files not being uploaded correctly. To check to see if you are affected, please use the following query:

db.uploads.find({"status": "CREATED", "contentType": "application/x-zip-compressed"}, {"author.fullName": 1, "author.email": 1, "filename": 1, "uploadDate": 1, "length": 1})

If any files are returned, you should check to see if these files affected and are missing from clowder.

Fixed

When zip file is uploaded, it will parse the file to check if it is a valid zip file, this couuld result in files not stored in final storage space #264
Updated swagger documentation
Return 404 not found when calling file/dataset/space api endpoints with an invalid ID #251
Line breaks in welcome message breaks swagger build #187

Changed

Added more information when writing files to make sure files are written correctly
Made cilogon group check debug message instead of error message

- JavaScript
Published by robkooper over 4 years ago

clowder - v1.18.0

1.18.0 - 2021-07-08

Added

Added folder and folder id to API call GET /api/datasets/:id/files. #34
Ability to queue archive / unarchive for full datasets.
API status endpoint GET /api/status will now show what storage type is used and for superadmins will show more information about the backend storage.
GET /api/files/bulkRemove now returns status of files deleted, not found, no permission, or errors.

Fixed

When uploading a file, any extractors marked disabled at the space level would be ignored. #246
RabbitMQ will not use connection if it does not exist.
Previews returns 404 if preview is not found GET /api/previews/:id.
Added index for comments, will speed up index creation.
If using S3 storage in docker, it was not reflected correctly in the docker-compose file.
Docker image for mongo-init now based on python:3.7-slim to reduce size.

- JavaScript
Published by lmarini over 4 years ago

clowder - v1.17.0

1.17.0 - 2021-04-29

Fixed

Close channel after submitting events to RabbitMQMessageService.

Added

Endpoint /api/datasets/createfrombag to ingest datasets in BagIt format. Includes basic dataset metadata, files, folders and technical metadata. Downloading datasets now includes extra Datacite and Clowder metadata.
Endpoint /api/files/bulkRemove to delete multiple files in one call. #12
Log an event each time that a user archives or unarchives a file.
Persist name of message bus response queue, preventing status messages from getting lost after a reboot.

Changed

Updated Sphinx dependencies due to security and changes in required packages.

- JavaScript
Published by lmarini almost 5 years ago

clowder - v1.16.0

1.16.0 - 2021-03-31

Fixed

Remove the RabbitMQ plugin from the docker version of clowder

Added

Added a sort and order parameter to /api/search endpoint that supports date and numeric field sorting. If only order is specified, created date is used. String fields are not currently supported.
Added a new /api/deleteindex admin endpoint that will queue an action to delete an Elasticsearch index (usually prior to a reindex).
JMeter testing suite.

Changed

Consolidated field names sent by the EventSinkService to maximize reuse.
Add status column to files report to indicate if files are ARCHIVED, etc.
Reworked auto-archival configuration options to make their meanings more clear.

- JavaScript
Published by lmarini almost 5 years ago

clowder - v1.15.1

1.15.1 - 2021-03-12

Fixed

Several views were throwing errors trying to access a None value in EventSinkService when a user was not logged in. Replaced get() with getOrElse().
Consolidated field names sent by the EventSinkService to maximize reuse.
Changed EventSinkService logging to debug to minimize chatter.
Don't automatically create eventsink queue and bind it to eventsink exchange. Let clients do that so that we don't have a queue for the eventsink filling up if there are no consumers.

- JavaScript
Published by lmarini almost 5 years ago

clowder - v1.15.0

1.15.0 - 2021-03-03

Added

CSV/JSON previewer using Vega.
Previewer for FBX files.
created search option for filtering by upload/creation date of resource.
EventSinkService to track user activity. All events are published to the message queue. Multiple consumers are available in event-sink-consumers.

Fixed

Clowder will no longer offer a Download button for a file until it has been PROCESSED.
When space created through api the creator was not added to space as admin #179.

Changed

/api/me will now return some of the same information as response headers. Can be used by other services to single sign on when running on same host.
RabbitMQPlugin has been split into ExtractorRoutingService and MessageService to isolate the rabbitmq code from the extraction code.

Removed

the toolserver is no longer build as part of clowder since it is no longer maintained. We are working on a newer version that will be included in future versions of clowder.

- JavaScript
Published by lmarini almost 5 years ago

clowder - v1.14.1

1.14.1 - 2021-02-02

Google will no longer work as login provider, we are working on this issue #157.
If non local accounts are used the count can be wrong. Use the fixcounts script to fix this.

Fixed

Error logging in with Orcid due to changed URL. #91
Fixed error in url for Twitter login.
Users count was not correct if using anything else but local accounts. #136
Files were not properly reindexed when the Move button was used to move a file into or out of a folder in a dataset.
When adding a file to a dataset by URL, prioritize the URL content-type header over the file content type established by looking at the file name extension. #139
Wrap words across lines to stay within interface elements. #160

- JavaScript
Published by lmarini about 5 years ago

clowder - v1.14.0

1.14.0 - 2021-01-07

Added

Added a new /api/reports/metrics/extractors report for summarizing extractor usage by user. Database administrators can use scripts/updates/UpdateUserId.js to assign user IDs to older extraction event records based on resource ownership in order to improve the accuracy of the report for older data.

Changed

api/reports/storage/spaces endpoint now accepts a space parameter for ID rather than requiring a space filter.
Datasets and collections in the trash are no longer indexed for discovery in search services.

- JavaScript
Published by lmarini about 5 years ago

clowder - v1.13.0

1.13.0 - 2020-12-02

Added

Ability to submit multiple selected files within a dataset to an extractor.
Support for Amplitude clickstream tracking. See Admin -> Customize to configure Amplitude apikey.
UpdateUserId.js to scripts/updates. This code adds userid to each document in extractions collection in mongodb. userid is taken from author id in uploads.files if exists, else it taken from author id in datasets collection.

Fixed

An extractor with file matching set to */* (all file types) would incorrectly send out dataset events.
Space Editors can now delete tags on files, datasets and sections.
GeospatialViewer previewer no longer shows if file does not contain geospatial layers.

- JavaScript
Published by lmarini about 5 years ago

clowder - v1.12.2

1.12.2 - 2020-11-19

Changed

/api/reindex admin endpoint no longer deletes and swaps a temporary index, but reindexes in-place.

- JavaScript
Published by lmarini over 5 years ago

clowder - v1.12.1

1.12.1 - 2020-11-05

Fixed

Error uploading to spaces that did not have extractors enabled/disabled (personel spaces).
If extractor does not have any parameters, there would be an error message in the console of the browser.
If the extractor did not have a user_id it would create an error and not record the event.

Changed

Docker Images are now pushed to github container registry

- JavaScript
Published by lmarini over 5 years ago

clowder - v1.12.0

1.12.0 - 2020-10-19

Warning: - This update modifies the MongoDB schema. Make sure to start the application with -DMONGOUPDATE=1. - This update modifies information stored in Elasticsearch used for text based searching. Make sure to initiate a reindex of Elasticsearch from the Admin menu or by POST /api/reindex.

Added

Global extractors page now shows more information, including submission metrics, logs (using Graylog), job history and extractors maturity. Extractors can be grouped using labels. User can filter list of extractors by labels, space, trigger and metadata key.
Users have more refined options to set extractors triggers at the space level. They can now follow global settings, disable and enable triggers.
Ability to set chunksize when downloading files. Set defult to 1MB from 8KB. This will result in faster downloads and less CPU usage at the cost of slightly more memory use.
Support for parsing of Date and Numeric data in new metadata fields. New search operators <, >, <=, >= have been added to search API now that they can be compared properly.
Track user_id with every extraction event. #94
Added a new storage report at GET api/reports/storage/spaces/:id for auditing user storage usage on a space basis.
The file and dataset metrics reports also have support for since and until ISO8601 date parameters.
Added viewer_hop a 3D models previewer for *.ply and *.nxz files. Added mimetype.nxz=model/nxz and mimetype.NXZ=model/nxz as new mimetypes in conf/mimetypes.conf

Fixed

Ignore the update field when posting to /api/extractors. #89
Search results were hardcoded to be in batches of 2.
Fixed permissions checks on search results for search interfaces that would cause misleading counts. #60

- JavaScript
Published by lmarini over 5 years ago

clowder - v1.11.2

1.11.2 - 2020-10-13

Fixed

Clowder healthcheck was not correct, resulting in docker-compose never thinking it was healthy. This could also result in traefik not setting up the routes.

- JavaScript
Published by lmarini over 5 years ago

clowder - v1.11.1

1.11.1 - 2020-09-29

Added

Added healtz endpoint that is cheap and quick to return, useful for kubernetes live/ready checks.

Fixed

Fixed health check script when using custom path prefix.
Proxy will no correctly handle paths that end with a / at the end.
Submitting an extraction will always return a 500 error, see #84
Added MongoDB index for folders.files.

Changed

Updated update-clowder script to work with migration to github. Has the ability now to push a message to MSTEAMS as well as influxdb.

- JavaScript
Published by lmarini over 5 years ago

clowder - v1.11.0

1.11.0 - 2020-08-31

Added

Downloaded datasets now include DataCite v4 XML files in the output /metadata folder.
Script to clean extractors' tmp files scripts/clean-extractors-tmpfiles/.
Script for RabbitMQ error queue cleanup scripts/rmq-error-shovel/.
Ability to use HTML formatting in the welcome message on the home page. #51
Expose a read-only list of extractors to all users.

Changed

Improved test script scripts/tester/tester.sh to report successes once a day.

Fixed

Escape colon characters on search values for search box and advanced search to allow those values in a search.
Typesafe now only offers https access. #49
If uploading files by url > 2147483647 it would fail. #54

- JavaScript
Published by lmarini over 5 years ago

clowder - v1.10.1

Fixed

Queue threads (e.g. Elasticsearch indexer) will no longer crash permanently if the queue connection to Mongo is lost temporarily.
Docker images would not build correctly on GitHub.
If monitor HTTP server would crash, it would not restart correctly.
Don't call server side twice when rendering list of files on dataset page. #7
Fixed Sphinx build errors and switched to using pipenv. Now building docs on readthedocs.

Added

GitHub artifacts can be uploaded using SCP to remote server.

- JavaScript
Published by lmarini over 5 years ago

clowder - v2.0-alpha.1

Remove all unused and legacy code
- XML RDF export
- Rabbitmq monitoring using manager API
- Geostreams API
- Deprecated endpoints
Every plugin should become a Guice Trait + Implementation
For services that are not implemented, provide an no-op implementation
Elasticsearch upgrade from 2.3.5 to 7.5

https://opensource.ncsa.illinois.edu/confluence/display/CATS/2.0

- JavaScript
Published by lmarini over 5 years ago

clowder - v1.10.0

Added

Ability to mark multiple files in a dataset and perform bulk operations (download, tag, delete) on them at once.

Fixed

Return thumbnail as part of the file information. #8
Datasets layout on space page would sometimes have overlapping tiles.

Changed

mongo-init script with users would return with exit code -1 if user exists, now returns exit code 0.

- JavaScript
Published by lmarini over 5 years ago

clowder - v1.9.0

Warning: This update modifies information stored in Elasticsearch used for text based searching. To take advantage of these changes a reindex of Elasticsearch is required. A reindex can be started by an admin from the Admin menu.

Added

Ability to delete extractor, both from API and GUI. CATS-1044
Add tags endpoint now returns the added tags. CATS-1053
Ability to search by creator name and email address for all resources.
List Spaces/Datasets/Collections created by each user on their User Profile page. CATS-1056
Allow user to easily flip through the files in a dataset. CATS-1058
Ability to filter files and folders in a dataset when sorting is enabled.
Visualize existing relations between datasets on the dataset page. This can be extended other resource types. CATS-1000
S3ByteStorageService verifies bucket existence on startup and creates it if it does not exist. CATS-1057
Can now switch storage provider in Docker compose, for example S3 storage. See env.example for configuration options.
Script to test extractions through the API.

Fixed

When adding tags to a section of an image, show the new tag without having to refresh the page. CATS-1053

Changed

Removed buttons to remove datasets from spaces and collections from certain pages. Moved Remove button for subcollections to right side of page to be consistent with other pages. CATS-1055
Upgraded swagger documentation to openapi v3.0.1.

- JavaScript
Published by lmarini over 5 years ago

clowder - v1.8.4

Warning: This update modifies how information is stored in Elasticsearch for text based searching. To take advantage of these changes a reindex of Elasticsearch is required. This can be started by an admin either from GUI or through the API.

Fixed

Fixed a bug related to improper indexing of files in nested subfolders, which could also affect searching by parent dataset.

- JavaScript
Published by lmarini almost 6 years ago

clowder - v1.8.3

Warning: This update modifies how information is stored in Elasticsearch for text based searching. To take advantage of these changes a reindex of Elasticsearch is required. This can be started by an admin either from GUI or through the API.

Changed

Elasticsearch indexer will now store new metadata fields as strings to avoid unexpected behavior on date fields.
When reindexing use a temporary index to reindex while the current one is in use then swap.

Fixed

Ability to delete tags from sections and files on the file page.
User-owned resources will now appear in search results regardless of space permissions.
Updating space ownership for datasets and collections will correctly reindex those resources for searches.
Missing index in statistics which would slow down system when inserting download/views.

Added

GitHub Actions to compile and test the code base, create documentation and docker images.
Code of Conduct as MD file (will be displayed by GitHub).
Templates for Bug, Feature and Pull Request on GitHub.

- JavaScript
Published by lmarini almost 6 years ago

clowder - v1.8.2

Fixed

Use the passed-in length within S3ByteStorageService.save.

- JavaScript
Published by lmarini almost 6 years ago

clowder - v1.8.1

Removed

Removed unused RDF libraries. This was probably used by the rdf/xml functionality that was removed a while back but the dependencies were never removed.
Removed Jena validation of JSON-LD metadata. It was creating a blank graph and clients couldn't upload metadata when Clowder runs in a location that doesn't not have access to https://clowderframework.org/contexts/metadata.jsonld.

Added

Scripts to migrate files on disk AWS S3. CATS-1034

Changed

Include collection prefix in path when saving to S3.
Include length of file in FileService when saving the bytes to any backend service. This helps optimize S3 implementation.
Upgraded sbt from 0.13.0 to 0.13.6 to fix build failures. CATS-1038

Fixed

Calling api/Files.removeFile should no longer decrement related counters twice. CATS-929

- JavaScript
Published by lmarini about 6 years ago

clowder - v1.8.0

Ability for users to provide run time parameters when manually submitting extractions.
Improvements to the search web interface and web API endpoints.
A new tree folder like view starting at spaces and going down to files (available in the “Tree View” tab on the home page).
Minor improvements to the pagination and extraction pages.
Improvements to the Docker build; support for MongoDB 3.6.
Internal optimizations of MongoDB queries.
Bug fixes.
For the full list please see the changelog.

- JavaScript
Published by lmarini over 6 years ago

clowder - v1.7.4

Fixed

Extractors that don't specify EXTRACT as categories don't show up in the manual submission page. CATS-1023

- JavaScript
Published by lmarini over 6 years ago

Recent Releases of clowder

clowder - v1.23.0

What's Changed

clowder - v1.22.1

v1.22.1 - 2023-11-10

Fixed

clowder - 1.22.0

1.22.0 - 2023-09-12

Added

Fixed

clowder - 1.21.0

1.21.0 - 2022-08-23

Added

Fixed

Changed

clowder - v1.20.3

1.20.3 - 2022-06-10

Fixed

Changed

clowder - 1.20.2

1.20.2 - 2022-04-30

Fixed

Changed

clowder - 1.20.1

1.20.1 - 2022-04-04

Fixed

Added

clowder - v1.20.0

Fixed

Changed

clowder - v1.19.5

Fixed

clowder - v1.19.4

1.19.4 - 2021-11-11

Fixed

clowder - v1.19.3

1.19.3 - 2021-11-11

Fixed

clowder - v1.19.2

1.19.2 - 2021-10-20

Fixed

clowder - v1.19.1

1.19.1 - 2021-10-19

Added

Fixed

Changed

Removed

clowder - v1.19.0

1.19.0 - 2021-10-05

Fixed

Added

clowder - v1.18.1

1.18.1 - 2021-08-16

Fixed

Changed

clowder - v1.18.0

1.18.0 - 2021-07-08

Added

Fixed

clowder - v1.17.0

1.17.0 - 2021-04-29

Fixed

Added

Changed

clowder - v1.16.0

1.16.0 - 2021-03-31

Fixed

Added

Changed

clowder - v1.15.1

1.15.1 - 2021-03-12

Fixed

clowder - v1.15.0

1.15.0 - 2021-03-03

Added

Fixed

Changed

Removed

clowder - v1.14.1

1.14.1 - 2021-02-02