Data

Tools

Indexes

Applications

Experiments

Open Source Science

Recent Releases of Efficiently Learning Relative Similarity Embeddings with Crowdsourcing

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v1.0.3

Scientific Software - Peer-reviewed - Python
Published by stsievert over 2 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v1.0.2

Scientific Software - Peer-reviewed - Python
Published by stsievert over 2 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v1.0.1

Scientific Software - Peer-reviewed - Python
Published by stsievert over 2 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v1.0.0

Salmon is ready for a v1.0 release! Some recent highlights, some of which are result from feedback from experimentalists:

Enhancements:

  • Allows offline embedding to work with only pip install (#139)
  • Documentation improved (better install docs, offline embedding, ...) (#139)

Improvements:

  • Adds licenses to docs (#139)
  • Future-proofs by locking deps in Docker container + linking to docs on github (#139)
  • Allows customization on frontend (custom divs on query page, passed through config) (#139)
  • Allows RoundRobin to be user-specific (#129)
  • ...

Scientific Software - Peer-reviewed - Python
Published by stsievert over 2 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v1.0.0rc11

Scientific Software - Peer-reviewed - Python
Published by stsievert almost 3 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v1.0.0rc10

Scientific Software - Peer-reviewed - Python
Published by stsievert almost 3 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v1.0.0rc9

Scientific Software - Peer-reviewed - Python
Published by stsievert almost 3 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v1.0.0rc8

Scientific Software - Peer-reviewed - Python
Published by stsievert about 3 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v1.0.0rc7

Scientific Software - Peer-reviewed - Python
Published by stsievert about 3 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v1.0.0rc6

Scientific Software - Peer-reviewed - Python
Published by stsievert about 3 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v1.0.0rc5

Scientific Software - Peer-reviewed - Python
Published by stsievert over 3 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v1.0.0rc4

Scientific Software - Peer-reviewed - Python
Published by stsievert over 3 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v1.0.0rc3

Scientific Software - Peer-reviewed - Python
Published by stsievert over 3 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v1.0.0rc2

update PyPI package

Scientific Software - Peer-reviewed - Python
Published by stsievert over 3 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v1.0.0rc1

Scientific Software - Peer-reviewed - Python
Published by stsievert over 3 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.10.0rc9

Scientific Software - Peer-reviewed - Python
Published by stsievert over 3 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.10.0rc8

Scientific Software - Peer-reviewed - Python
Published by stsievert over 3 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.10.0rc7

Scientific Software - Peer-reviewed - Python
Published by stsievert over 3 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.10.0rc6

Scientific Software - Peer-reviewed - Python
Published by stsievert over 3 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.10.0rc5

Scientific Software - Peer-reviewed - Python
Published by stsievert over 3 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.10.0rc4

Scientific Software - Peer-reviewed - Python
Published by stsievert over 3 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.10.0rc0

Scientific Software - Peer-reviewed - Python
Published by stsievert over 3 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.9.1rc8

Scientific Software - Peer-reviewed - Python
Published by stsievert over 3 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.9.1rc7

Scientific Software - Peer-reviewed - Python
Published by stsievert almost 4 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.9.1rc6

Scientific Software - Peer-reviewed - Python
Published by stsievert almost 4 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.9.1rc4

Scientific Software - Peer-reviewed - Python
Published by stsievert almost 4 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.9.1rc3

Scientific Software - Peer-reviewed - Python
Published by stsievert almost 4 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.9.1rc2

Scientific Software - Peer-reviewed - Python
Published by stsievert almost 4 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.9.1rc1

Scientific Software - Peer-reviewed - Python
Published by stsievert almost 4 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.9.0

Breaking changes * Key n has been removed from downloaded config at /config (so is a valid config for re-upload) (#125). * Renames alg_ident to sampler in CSV at /responses (#125)

Improvements * Allows customizing title in html: title (#125) * Only change docs on release (lesson learned in https://github.com/stsievert/salmon/issues/124#issuecomment-980813015)

Maintenance * Documentation on experiment initialization and configuration is much better at https://docs.stsievert.com/salmon/init.html (#125). * Defaults are specified at https://docs.stsievert.com/salmon/generated/salmon.triplets.manager.Config.html * Documentation on samplers improved (#125) * Rule-of-thumb developed on selecting sampling algorithms (both in an FAQ in #124 and at https://docs.stsievert.com/salmon/samplers.html) * File structure (init.yaml) better documented. * Makes it clear that ARR is recommended in the default config, and that the default config (while simple) is not likely of interest. * Renames "algorithms" to "sampler" on the documentation (#124). * CI testing fixed (#125) * Allows customizing score for ARR

Scientific Software - Peer-reviewed - Python
Published by stsievert about 4 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.9.0rc4

Scientific Software - Peer-reviewed - Python
Published by stsievert about 4 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.9.0rc3

Scientific Software - Peer-reviewed - Python
Published by stsievert about 4 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.9.0rc2

Scientific Software - Peer-reviewed - Python
Published by stsievert about 4 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.9.0rc1

A pre-release to resolve https://github.com/stsievert/salmon/issues/124#issuecomment-980799884.

Scientific Software - Peer-reviewed - Python
Published by stsievert about 4 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.8.1rc5

v0.8.1rc5

Scientific Software - Peer-reviewed - Python
Published by stsievert about 4 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.8.1rc4

Scientific Software - Peer-reviewed - Python
Published by stsievert about 4 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.8.1rc3

v0.8.1rc3

Scientific Software - Peer-reviewed - Python
Published by stsievert about 4 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.8.1rc2

v0.8.1rc2

Scientific Software - Peer-reviewed - Python
Published by stsievert about 4 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.8.1rc1

Scientific Software - Peer-reviewed - Python
Published by stsievert about 4 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.8.0

Backward incompatible changes:

  • Format of init.yaml file changed (#122). See details below.

Improvements

In addition, some documentation has been updated:

  • Detailing running Salmon offline; example given with scoring queries with custom embedding (6baaff80fb0c6d1a28a709cab6c7d5137748271b)
  • FAQ about specifying custom participant ID (ab76df80cbc229db48b9ea85bfc629c1ed683391)
  • Using HTTPS with Salmon (issue #119)
  • About setting/changing username/password after/before launch (issue #115)
  • Allow users to use arrow keys to submit answers (implemented in #122 as part of the HTML configuration; tracked in issue #121).

Details on YAML change

yaml instructions: Click buttons debrief: Goodbye! max_queries: 4 ...

Now:

yaml html: instructions: Click buttons debrief: Goodbye! max_queries: 4 ...

Salmon will raise an informative error if an old configuration file is uploaded.

Scientific Software - Peer-reviewed - Python
Published by stsievert about 4 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.7.0

Changes since v0.6.0 (https://github.com/stsievert/salmon/compare/v0.6.0...v0.7.0):

  • Viz: better showing response time on dashboard (show histograms of response {rate, gap}) (#110)
  • API: allow setting n_search in Adaptive. (#110)
  • API: set default mu=0.05 for the CKL. (#110)
  • DOC: better show number of queries needed (FAQ + adaptive benchmarks) (#110)
  • API: implement validation sampling (#112, #118)
  • API: allow customization of n_top for ARR (#113).
  • ENH: add HTML id tag to target div on query_page.html (useful for custom javascript/css) (#116).
  • API: implement synchronous round robin, SRR (#118)
  • API: allow a custom participant ID and allow specifying the number of samplers per user (#118)
  • API: allow an initial embedding for offline fitting (useful if downloaded from dashboard) (#118).

Backwards incompatible * Renamed RandomSampling to Random. (#118) * (internal) renames Runner to Sampler and salmon.backend.runner to salmon.backend.sampler. (#118)

Contributors: @stsievert, @crcox

For more detail, see #110, #112, #113, #118. More tasks have been completed; general maintenance tasks and documentation improvements are hidden.

Scientific Software - Peer-reviewed - Python
Published by stsievert over 4 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.7.0rc4

Scientific Software - Peer-reviewed - Python
Published by stsievert over 4 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.7.0rc3

Scientific Software - Peer-reviewed - Python
Published by stsievert over 4 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.7.0rc2

Scientific Software - Peer-reviewed - Python
Published by stsievert over 4 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.7.0rc1

Scientific Software - Peer-reviewed - Python
Published by stsievert over 4 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.6.1rc9

Scientific Software - Peer-reviewed - Python
Published by stsievert over 4 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.6.1rc8

Scientific Software - Peer-reviewed - Python
Published by stsievert over 4 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.6.1rc7

Scientific Software - Peer-reviewed - Python
Published by stsievert over 4 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.6.1rc6

Scientific Software - Peer-reviewed - Python
Published by stsievert over 4 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.6.1rc5

Scientific Software - Peer-reviewed - Python
Published by stsievert over 4 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.6.1rc4

Scientific Software - Peer-reviewed - Python
Published by stsievert over 4 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.6.1rc3

Scientific Software - Peer-reviewed - Python
Published by stsievert over 4 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.6.0

Breaking changes * API: Endpoint /init_exp has been renamed to /init (#106). * API: defaults for salmon.triplets.offline.OfflineEmbedding changed. (#100) * API: class RR for "round robin" has been renamed to ARR for "active round robin." (#109) * API: salmon.triplets.algs has been renamed to salmon.triplets.samplers. (#109)

Improvements since v0.5.2 The main improvement is better showing how to enable good adaptive performance with Salmon (#100). This showed clear benefits from using adaptive algorithms – fewer responses will be required with the sampler ARR. Some changes around this include the following:

  • ENH/MAINT: better setting of max_epochs for adaptive runners. This increases the response rate at which adaptive and random sampling perform the same (2 responses/sec → 10 responses/sec) (#100).
  • MAINT: change defaults on adaptive runners (mostly, change default optimizer from SGD to Adadelta)
  • MAINT: keep number of queries for RR to be 1 * n (don't post the database if not required).

These changes enabled showing the improvements:

  • DOC: show how response rate affects Salmon embedding performance for adaptive algs, and provide a reference configuration (#100).

Additionally, some other smaller features have been implemented:

  • DOC: firm up salmon.triplets.offline.OfflineEmbedding (make new properties, better document, etc).
  • API: allow users to provide authorization credentials (#106)
  • API/DOC: improve attributes/documentation for salmon.triplets.offline.OfflineEmbedding in salmon/triplets/offline.py (#100)

Maintenance * DOC: provide methods for troubleshooting launching process (#104) * DOC/MAINT: allow the database to be properly reset (#107) * DOC: clean documentation. Add more classes, better descriptions, reference papers, etc (#100) * DOC: docs only changed on releases (#108).

Scientific Software - Peer-reviewed - Python
Published by stsievert almost 5 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.5.2

This is a small bug-fix release.

  • MAINT, TST: It fixes a typo on ZIP file upload w/o numeric filenames (#101)

This release is based off a single PR, #101. Other features in that PR include:

  • MAINT: Cleans examples/datasets (remove unused code, better organize) (#101)
  • MAINT/TST: Adds random state back in – but only for initialization (and test it too) (#101)
  • TST: fix zip file tests so they run with pytest (→ remove xfail) (#101)
  • TST: test uploads of each example in examples/ (#101)

Scientific Software - Peer-reviewed - Python
Published by stsievert almost 5 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.5.1

This is primarily a bug fix release:

  • BUG: remove bug where same random state used to ask queries (#96)

Other improvements:

  • MAINT: don't return repeated queries from round robin sampling
  • MAINT: Files are ordered if numeric filenames are given in targets.zip (#95).
  • MAINT: Offline embedding improved (#95; adaptive learning rate method used).
  • DOC: Better show adaptive example

Scientific Software - Peer-reviewed - Python
Published by stsievert almost 5 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.5.0

What's new in this release? * ENH: Adaptive sampling now works as expected for triplets (#92) * API: an endpoint to get the embedding is implemented (#93). * MAINT: initialization (#91, #90). * DOC: better documentation around offline embeddings (#92).

What's new since the v0.4.0 release? These are the most important changes in the minor releases.

  • API: implements OGD and sample weights for offline embeddings
  • MAINT: remove skorch for 40× faster & lower memory usage (mostly #87, #90 is slightly related)
  • DOC improvements (#71, #57)
  • MAINT: Backend performance (#66, #74) and stability (#71, #78) improvements.
  • VIZ: show more information on dashboards, including embeddings (#67) and algorithm timings (#71).

Scientific Software - Peer-reviewed - Python
Published by stsievert almost 5 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.4.5

This release limits memory usage, to help prevent Dask workers and the database from becoming overloaded.

Scientific Software - Peer-reviewed - Python
Published by stsievert about 5 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.4.4

This release makes the following improvements:

  • API: implements OGD and sample weights for offline embeddings
  • DOC: warns experimentalists to use HTTP and new AMI
  • MAINT: Remove skorch. This means Salmon embeddings are much quicker, about 40× faster than before with Skorch. (no data preprocessing is required, only a torch.from_numpy call).
  • MAINT: Train for longer after each response received (200 epochs, not 1).

All of these were implemented in #87.

Scientific Software - Peer-reviewed - Python
Published by stsievert about 5 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.4.3

Scientific Software - Peer-reviewed - Python
Published by stsievert about 5 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.4.2

Scientific Software - Peer-reviewed - Python
Published by stsievert about 5 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.4.1

  • DOC: improvements in #57, showing adaptive gain (#58), showing dashboard (#71)
  • VIZ: showing embeddings on dashboard (#67), algorithm timings on dashboard (#71, also API), adaptive algorithm creation (#57).
  • MAINT: Backend improvements: running searches/updates in parallel (#66, #74)
  • MAINT: adaptive algorithm stability (#71, #78)
  • API: Experimentalist requests: skip button in (#59)
  • API: allowing targets: int to create N targets
  • MAINT: implement uncertainty sampling (another method different to information gain)

Scientific Software - Peer-reviewed - Python
Published by stsievert over 5 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.4.0

Notable features since v0.3.0:

  • Implementation, integration and documentation of adaptive algorithms (#42, #44)
  • Integrate Dask for adaptive algorithms (#47)
  • Documentation showing adaptive algorithms work as expected (#57, #49).

Other changes since v0.3.0 (some of which are in minor releases):

  • Allow download of logs and show server side metric on the dashboard (#36).
  • Support text targets via CSV file upload (#46).
  • Allow users to download responses as CSV files (#51).
  • Add button for user to skip query (#53, #59).
  • Backend stability fixes (#39).
  • Frontend stability (#45).
  • Dashboard stability fixes (#41).
  • Documentation improvements (#41, #58, #57, #55).

Scientific Software - Peer-reviewed - Python
Published by stsievert over 5 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.3.6

Shows "skip button."

Scientific Software - Peer-reviewed - Python
Published by stsievert over 5 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.3.5

New features:

  • #51, MAINT: download responses as CSV files

Scientific Software - Peer-reviewed - Python
Published by stsievert over 5 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.3.4

Support CSV upload.

Scientific Software - Peer-reviewed - Python
Published by stsievert over 5 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.3.3

Maintenance fix for when ZIP target file uploaded and targets key not specified in YAML file.

Scientific Software - Peer-reviewed - Python
Published by stsievert over 5 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.3.2

Dashboard fixes in #41.

Scientific Software - Peer-reviewed - Python
Published by stsievert over 5 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.3.1

Some deploy fixes for AWS in #39.

Scientific Software - Peer-reviewed - Python
Published by stsievert over 5 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.3.0

A variety of deploy fixes. Notable features include the following:

  • Allowing upload/download of experiment and persist through reboot
  • Beta implementation of adaptive algorithms
  • Interactive dashboard graphs
  • various EC2 deployment fixes (transparent huge memory pages off, memory swapping, etc)

Scientific Software - Peer-reviewed - Python
Published by stsievert over 5 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.2.6

Deploy to experimentalists. Changes include the following:

  • a port change (8000 => 8421)
  • no RedirectResponse on file upload (so experiments aren't overwritten if refreshed)
  • AMI change

Scientific Software - Peer-reviewed - Python
Published by stsievert over 5 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing -

Scientific Software - Peer-reviewed - Python
Published by stsievert over 5 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.2.4

Document how performant v0.2.3 is under load testing.

Scientific Software - Peer-reviewed - Python
Published by stsievert over 5 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.2.3

Primarily a usability release with some experimentalist feedback

  • MAINT: better show errors (on file upload, on plotting)
  • DOC: add to experimentalist docs
  • ENH: allow answering by key presses
  • TST: test errors propogated, no repeats in targets

Scientific Software - Peer-reviewed - Python
Published by stsievert over 5 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.2.2

Bump for #13 to show errors

Scientific Software - Peer-reviewed - Python
Published by stsievert over 5 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.2.1

A minor fix; the AMI files might have a bug on reload.

Scientific Software - Peer-reviewed - Python
Published by stsievert almost 6 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.2.0

Implements basics for experimentalists. It incorporates some of their feedback, and implements a dashboard.

Hopefully this release is stable.

Pull requests:

  • #9: BUG: participant ID and recover responses
  • #11: ENH: implement features requested by experimentalists

Scientific Software - Peer-reviewed - Python
Published by stsievert almost 6 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.1.2

Scientific Software - Peer-reviewed - Python
Published by stsievert about 6 years ago

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing - v0.1.0

This is an initial release. This release has a (very) basic implementation of the triplets interface.

This API is still alpha, and may change without warning.

Scientific Software - Peer-reviewed - Python
Published by stsievert about 6 years ago