mephisto - v1.5.4

Overview

A housekeeping release devoted to paying down urrent tech debt, e.g.: - port UI code to Typescript - upgrade Python version to 3.10 - making UI fully mobile-compatible for standard task generators - update existing examples with InHouse provider - add missing doc pages for providers - remove deprecated packages

What's Changed

Updated UI validation and fixed error handling in remote procedures by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1259
Make FormComposer andVideoAnnotator render well on mobile by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1260
Removed deprecated mephisto-task and mephisto-worker-addons libs by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1261
Port existing Task examples to use In-House provider by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1262
Added doc pages for In-House and Mock providers by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1263
Auto-submit partial Task results right before Unit expiration by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1264
Ported core packages and examples to Typecript by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1265
Fixed auto-redirect upon Task completion in In-House provider by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1266

Full Changelog: https://github.com/facebookresearch/Mephisto/compare/v1.5.3...v1.5.4

- Python
Published by meta-paul over 1 year ago

mephisto - v1.5.3

Overview

This release offers clarity and ease of use for Model-in-the-Loop tasks. The pattern of communicating with the server over the course of the task (called "remote procedures") was added to FormComposer, and an example called "Interactive Generation of a Look-Alike Image" was provided. Other examples of remote procedure tasks were brought up-to-date as well.

FormComposer gainded a few useful additions: - dynamic fieldsets (to save multiple instances of the same response fieldset) - additional flexibility for coding up UI triggers - buttons as an element of a form fieldset

This is useful when: - Worker should react to continuously generated content - Worker should guide your algorithm to achieve a goal

What's Changed

Renamed React component files to jsx; fixed VideoAnnotator bug by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1256
Added example task 'Interactive Image Generation' by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1257
Updated remote procedure examples; Refactored remote procedure blueprint by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1258

Full Changelog: https://github.com/facebookresearch/Mephisto/compare/v1.5.2...v1.5.3

- Python
Published by meta-paul over 1 year ago

mephisto - v1.5.2

Overview

Mephisto now offers a UI for managing qualifications and qualification assigns. We've nested it under TaskReview app since it already can run a local web server. Some new capabilities were also added:

Now workers can be granted qualifications directly (without context of a specific Task Unit)
Qualifications now support not only name, but also description
Multiple qualifications can now be assigned to a single Task Unit
Screenshots and all details are described in the docs

Note that this code change includes a database migration. It should port all your existing qualifications data automatically. Though just in case we'd recommend backing up the qualifications part of your database (using mephisto db export --qualification-only command as described here)

Some other changes include bug fixes and reorganization of local Mephisto packages for more transparent naming and versioning.

What's Changed

Qualifications management UI by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1254
Added local builds of Mephisto packages by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1255

Full Changelog: https://github.com/facebookresearch/Mephisto/compare/v1.5.1...v1.5.2

- Python
Published by meta-paul over 1 year ago

mephisto - v1.5.1

Overview

Patch with small fixes to VideoAnnotator and docs update.

VideoAnnotator

Display validation error for overlapping video segments on the same track
Add support for multiple video formats (not just mp4)
Fix a few outdated unittests for generators

For docs

Add "Known Issues" page in the docs
Add FormComposer screenshots

What's Changed

v1.5.1 by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1253

Full Changelog: https://github.com/facebookresearch/Mephisto/compare/v1.5.0...v1.5.1

- Python
Published by meta-paul over 1 year ago

mephisto - v1.5.0

Overview

This release introduces a couple of key features, and a number of smaller improvements.

Video Annotator

VideoAnnotator is a new Task Generator for convenient and functional video annotation.

Key features:

Worker can indicate multiple video segments along multiple tracks, and annotate each one
For each segment, you can specify (via JSON config) desired annotation input (as a list of standard form fields)
VideoAnnotator JSON config works similar to FormComposer (including pre-signed S3 URLs, and support for custom JS triggers)
Workers inputs are downloadable in JSON
When viewing a completed unit, worker inputs are also available in WebVTT format

InHouse provider

The new InHouse provider enables using Mephisto with your own "in-house" crowd - and basically crowdsourcing your task to people you know. (This used to be "mock" provider last year, but now we're using "mock" only for testing).

The initial implementation is quite simple: a worker navigates to your Task page, types in their worker nickname, and proceeds to the next available Task unit (if any). Afterwards you can review their results and take notes or grant qualifications, just like you do that with the 3rd party workers.

Future roadmap for InHouse provider would include: proper worker authorization; allowing workers to view researcher notes; restricting worker acces based on their qualification.

Feature improvements

TaskReview app

Visualized selection-based response stats in Task Review
Added ability to view completed task units, without assessing them

DataPorter feature

Now you can port only granted worker qualifications (without all other data), to easily build pool of known workers collaboratively

FormComposer

Enabled CSS insertions in FormComposer

Grafana

Updated Grafana dockerized deployment and docs
Added access to Grafana dashboard from TaskReview app

General housekeeping

Updated and added missing architecture diagrams in Mephisto docs
Refactored increasingly nested CLI commands, and updated their Help strings
Add example of GoldUnit and Worker Onboarding
upgraded NodeJS to v20

Included PRs

Bump eslint-plugin-flowtype from 4.7.0 to 8.0.3 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1157
Bump @storybook/addon-actions from 6.5.16 to 8.0.10 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1158
Bump peaceiris/actions-gh-pages from 3 to 4 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1163
Bump @babel/plugin-proposal-class-properties from 7.16.7 to 7.18.6 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1161
Bump pre-commit/action from 2.0.0 to 3.0.1 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1164
Bump pytest from 5.4.3 to 8.2.0 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1166
Additional unittests for Data Porter feature by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1178
Added porting of worker qualifications only for Data Porter by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1180
Visualization of selection-based response stats in Task Review by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1193
Enable CSS insertions in FormComposer by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1194
Bump mdutils from 1.4.0 to 1.6.0 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1165
Bump requests from 2.31.0 to 2.32.0 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1172
Bump tornado from 6.3.3 to 6.4.1 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1179
Bump urllib3 from 1.26.17 to 1.26.19 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1187
Bump ws from 7.5.9 to 7.5.10 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1188
Bump ws from 5.2.3 to 5.2.4 in /mephisto/abstractions/architects/router/node by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1189
Bump torch from 1.13.1 to 2.3.1 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1181
Bump tabulate from 0.8.10 to 0.9.0 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1171
Bump pdoc from 14.4.0 to 14.5.1 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1192
Bump braces from 3.0.2 to 3.0.3 in /packages/annotation-toolkit by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1184
Bump cypress-io/github-action from 3.1.0 to 6.7.1 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1186
Bump pillow from 10.2.0 to 10.3.0 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1138
Bump @storybook/node-logger from 6.5.16 to 8.1.10 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1191
Experimental functions for resuming a task run by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/910
Update architecture diagrams by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1206
Update Grafana dockerized deployment and docs, refactor CLI commands by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1207
Fixed Docker testing matrix, removed unneeded image versions by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1208
Add example of GoldUnit usage by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1211
Add examples of worker onboarding and screening by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1212
Update Mephisto manual installation instructions by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1213
Added access to Grafana dashboard from TaskReview app by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1214
Create mephisto task addons by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1218
Update help for mephisto CLI command by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1219
Updated NodeJS to v22 by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1221
Accommodate lengthy task instructions in Form Composer by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1222
Add in house provider by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1227
Include token values in downloaded data for FormComposer tasks by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1230
Add Video Annotator feature by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1247

Full Changelog: https://github.com/facebookresearch/Mephisto/compare/v1.4.0...v1.5.0

- Python
Published by meta-paul over 1 year ago

mephisto - v1.4.0

Overview

In this release we added Data Porter feature that allows to move full/partial data dumps between different isntances of Mephisto. This enables things like:

backups/restores of Mephisto data
consolidating task data across different Mephisto instances
sharing task data with your colleagues

We also introduced a simple (forward-only) workflow for database migrations, enabled custom code snippets in Form Composer, and updated numerous library versions.

Last but not least, mephisto.ai documentation got a major update as well.

Included PRs

Bump pyarrow from 9.0.0 to 14.0.1 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1083
Bump @cypress/request and cypress in /examples/formcomposerdemo/webapp by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1106
Bump @cypress/request and cypress in /packages/react-form-composer by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1105
Bump ip from 2.0.0 to 2.0.1 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1103
Bump aiohttp from 3.8.6 to 3.9.2 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1101
Bump follow-redirects from 1.15.2 to 1.15.4 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1094
Bump pillow from 10.0.1 to 10.2.0 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1099
Fixing agent desync on RemoteProcedure tasks by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/1063
Mock architect actions recording by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/1064
Script to write out qualifications by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/1069
Enabled custom code insertions for FormComposer by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1108
Display qualifications granted to a worker in TaskReview app by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1127
Update mephisto.ai documentation by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1128
Data porter feature by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1153
Bump webpack-dev-middleware from 5.3.3 to 5.3.4 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1130
Bump webpack-dev-middleware from 5.3.3 to 5.3.4 in /mephisto/review_app/client by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1129
Bump xmltodict from 0.12.0 to 0.13.0 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1126
Bump @babel/core from 7.17.7 to 7.24.0 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1118
Bump follow-redirects from 1.15.2 to 1.15.6 in /examples/staticreacttask/webapp by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1116
Bump actions/setup-python from 2 to 5 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1113
Bump actions/setup-node from 2 to 4 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1111
Bump dorny/paths-filter from 2 to 3 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1109
Bump react-shadow from 19.1.0 to 20.4.0 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1121
Bump follow-redirects from 1.15.5 to 1.15.6 in /mephisto/review_app/client by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1114
Bump follow-redirects from 1.15.5 to 1.15.6 in /packages/mephisto-task-multipart by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1112
Bump react-player from 2.9.0 to 2.16.0 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1148
Bump @storybook/addon-links from 6.5.16 to 8.0.10 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1151
Bump aiohttp from 3.9.2 to 3.9.4 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1146
Bump idna from 3.4 to 3.7 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1143
Bump ejs from 3.1.9 to 3.1.10 in /mephisto/review_app/client by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1150
Bump follow-redirects from 1.15.4 to 1.15.6 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1154
Bump @storybook/react from 6.5.16 to 8.0.10 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1152
Bump ejs from 3.1.8 to 3.1.10 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1149
Bump mypy from 0.971 to 1.10.0 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1147
Bump tar from 6.1.13 to 6.2.1 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1142
Added unittests for Data Porter command options by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1156
Bump codecov/codecov-action from 3 to 4 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1115
Bump express from 4.18.2 to 4.19.2 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1135
Bump pyyaml from 5.4.1 to 6.0.1 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1125
Bump express from 4.18.3 to 4.19.2 in /mephisto/review_app/client by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1136
Bump pdoc from 10.0.4 to 14.4.0 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/1123

Full Changelog: https://github.com/facebookresearch/Mephisto/compare/v1.3.0...v1.4.0

- Python
Published by meta-paul almost 2 years ago

mephisto - v1.3.0

Overview

This release introduces FormComposer task generator. It aims to simplify creating React front-end for form-based Tasks. Main highlights:

Task front-end can be defined via JSON configuration (no need for custom code)
Multiple form versions are supported within a single Task
Multi-version forms can be generated within the same Task, based on variables' values
Expiring (presigned) URLs are supported for S3 files (for data safety)
Task Review app support includes preview of uploaded media files

Included PRs

Mephisto v1.3 by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1102

Full Changelog: https://github.com/facebookresearch/Mephisto/compare/v1.2.1...v1.3.0

- Python
Published by meta-paul about 2 years ago

mephisto - v1.2.1

Overview

This release rounded out TaskReview app functionality. Now you can conveniently give feedback and bonuses to individual workers while performing review of their submissions. This feature works with all currently supported providers.

Included PRs

V1.2.1 dev by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1100

Full Changelog: https://github.com/facebookresearch/Mephisto/compare/v1.2.0...v1.2.1

- Python
Published by meta-paul about 2 years ago

mephisto - Mephisto 1.2.0

Overview

This release is focused on adding a new provider (Prolific) and better data review experience. It has a larger diff than usual, and we will return to a more frequent release cadence afterwards.

See the list of changes when upgrading from v1.1.0 below. Note that only essential dependency updates are listed.

Big Changes

Added integration with Prolific provider, as an alternative to Mturk for data collection. Expect some minor additions in the subsequent releases, but the core functionality is already there. #1008
Added UI-driven review of collected task results, a major upgrade over the previously used command-line script. Support for bonuses and feedback for workers to be added in the subsequent releases. When jobs have had no activity for a set duration, they'll be shut down automatically (suspecting an issue). #1058
Added dockerization support for running Mephisto. #1008

Included PRs

Feature + UX Improvements

Getting complete MTurk pay for individual units by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/446
Adding gold units and functionality by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/923
including hydra_configs in package (for 1.1.1) by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/942
Adding missing hook registration for MTurk Sandbox wrap_crowd_source by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/987
Reorganizing pyproject.toml to use Extras over Groups by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/1002
Added AWS credentials pre-commit hook by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1018
Update heroku_architect.py by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/1025
ec2 architect uninterruptable shutdown by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/1023
Permissive socket upload by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/1026
Adding auth implementation of prolific client by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/1028
Add Prolific provider (#1008) by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1012
Improve logger in Python code by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1045
Laying down basic strokes of standard get_data response by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/1065
Prolific workspace management by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/1067
Update heroku_architect.py by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/1062 1076
Create a reusable app for task review by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1058
Prevent oversampling of study submissions by any single worker by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1080

Bugfixes

Fixed Static task to not explicitly check for objData.file1 by @gzhihongwei in https://github.com/facebookresearch/Mephisto/pull/966
Refactoring QA from unit assign to use in onboarding by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/960
dropped the self from cantcancelexpirations by @arjunan-k in https://github.com/facebookresearch/Mephisto/pull/971
Fixing missing events in wrapcrowdsource.js by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/988
InaccessiblePrometheusServer Exception by @mojtaba-komeili in https://github.com/facebookresearch/Mephisto/pull/1016
Updated error message to manually delete EC2 server by @gzhihongwei in https://github.com/facebookresearch/Mephisto/pull/1007
quick fix: change order of checks in operator by @PReithofer in https://github.com/facebookresearch/Mephisto/pull/1039
Small fixes from local branch by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/1054
fix race condition in generate_units by @PReithofer in https://github.com/facebookresearch/Mephisto/pull/1056

Docs + Examples

Corrected documentation inconsistencies and typos by @gzhihongwei in https://github.com/facebookresearch/Mephisto/pull/965
Adding citation note to readme by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/989
Fix typo in operations/README.md by @eltociear in https://github.com/facebookresearch/Mephisto/pull/992
Add additional python versions to test by @pringshia in https://github.com/facebookresearch/Mephisto/pull/925
fix rendering of assignment url for mock tasks by @edwardguo61 in https://github.com/facebookresearch/Mephisto/pull/1005
New testing option for ec2 (reuse existing server) by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/1036
Small typo fix for running the statictestscript by @benmatselby in https://github.com/facebookresearch/Mephisto/pull/1019
Add unittests for Prolific integration by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1046
running black across codebase by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/1043
Add unittests for Prolific integration (2) by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1047
Updated Prolific integrations docs by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1049
Clean up TODO comments by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1050
Update copyright headers across the codebase by @meta-paul in https://github.com/facebookresearch/Mephisto/pull/1075

Dependencies

Update to React 18 by @pringshia in https://github.com/facebookresearch/Mephisto/pull/905
Upgrading codecov version by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/1027

New Contributors

@gzhihongwei made their first contribution in https://github.com/facebookresearch/Mephisto/pull/965
@arjunan-k made their first contribution in https://github.com/facebookresearch/Mephisto/pull/971
@eltociear made their first contribution in https://github.com/facebookresearch/Mephisto/pull/992
@edwardguo61 made their first contribution in https://github.com/facebookresearch/Mephisto/pull/1005
@benmatselby made their first contribution in https://github.com/facebookresearch/Mephisto/pull/1019
@meta-paul made their first contribution in https://github.com/facebookresearch/Mephisto/pull/1018
@PReithofer made their first contribution in https://github.com/facebookresearch/Mephisto/pull/1039

Full Changelog: https://github.com/facebookresearch/Mephisto/compare/v1.1.0...v1.2.0

- Python
Published by meta-paul about 2 years ago

mephisto - Mephisto 1.1.0

Overview

This release contains a number of Stability and Quality of Life improvements from v1.0.0 and v1.0.2. The big focus has been on collection success rate, documentation, and various bugfixes.

For those updating from 1.0, see the 1.0.2 changes as well for a complete changelog.

Breaking changes

Mephisto no longer supports Python 3.7. This is as of #898, wherein to get the ParlAI peer imports working we needed to bump to ParlAI 1.7.0, which no longer supports Python 3.7.

Big Changes

When launching on MTurk, Mephisto will request that you run the mephisto scripts mturk cleanup script inline before launching. This exercise prevents having long-running tasks that haven't been cleaned up properly. #918
When jobs have had no activity for a set duration, they'll be shut down automatically (suspecting an issue). #919

Big Fixes

Lots of documentation, thanks @Etesam913!
Significant fixes to proper execution and state tracking in Mephisto, which should improve stability across all job types (#875, #878, #902, #922)
Major fix for Heroku tasks using the useMephistoLiveTask hook, which was causing all messages to be double sent in both directions. (#921)

Included PRs

Feature + UX Improvements

Enhance logging messages with rich by @Etesam913 in https://github.com/facebookresearch/Mephisto/pull/869
Added ability to customize html-static-task-example submit button by @Etesam913 in https://github.com/facebookresearch/Mephisto/pull/879
Adds ability to clear onboarding for a worker by @Etesam913 in https://github.com/facebookresearch/Mephisto/pull/886
Adding more general function with better name by @mojtaba-komeili in https://github.com/facebookresearch/Mephisto/pull/893
Making MTurk cleanup run daily per requester by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/918
12 hour inactivity shutdown by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/919

Bugfixes

[Bugfix] Fixing double update bug on heroku by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/921
Fixing Worker Pool race condition for assignments by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/875
ParlAI Assignment Fixes pt 2 by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/878
Resolving static return status bug by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/902
Make self.metadata backward-compatible by @chiehminwei in https://github.com/facebookresearch/Mephisto/pull/861
Finding OSTYPE if not exists by @mojtaba-komeili in https://github.com/facebookresearch/Mephisto/pull/892
fix breaks in createvalidationfunction for screening units by @macota in https://github.com/facebookresearch/Mephisto/pull/864
Fix mephisto web cli command by @Etesam913 in https://github.com/facebookresearch/Mephisto/pull/865
Removing threadsafety as we already lock connections behind threads by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/897
Extending some 1.1 changes by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/922

Testing & DevX

Add path filter by @Etesam913 in https://github.com/facebookresearch/Mephisto/pull/868
Add cypress test to simple static task by @Etesam913 in https://github.com/facebookresearch/Mephisto/pull/881
fix #874: Upgrade pip and create maindatadirectory in Dockerfile by @mjkanji in https://github.com/facebookresearch/Mephisto/pull/917
pre-commit install by @mojtaba-komeili in https://github.com/facebookresearch/Mephisto/pull/894
Some fixes made to run a live demo of Mephisto functionality by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/909

Docs + Examples

Adds cypress section to frontend docs by @Etesam913 in https://github.com/facebookresearch/Mephisto/pull/860
Adds tips/feedback section to docs by @Etesam913 in https://github.com/facebookresearch/Mephisto/pull/838
Add tests to tips example by @Etesam913 in https://github.com/facebookresearch/Mephisto/pull/833
Add searchbar to docs by @Etesam913 in https://github.com/facebookresearch/Mephisto/pull/858
Update searchbar colors by @Etesam913 in https://github.com/facebookresearch/Mephisto/pull/863
Auto Generate Blueprint Documentation by @Etesam913 in https://github.com/facebookresearch/Mephisto/pull/870
Auto Generate Architect Documentation by @Etesam913 in https://github.com/facebookresearch/Mephisto/pull/871
Auto Generate Requester Documentation by @Etesam913 in https://github.com/facebookresearch/Mephisto/pull/872
Auto Generate Provider Documentation by @Etesam913 in https://github.com/facebookresearch/Mephisto/pull/873
Update deploy docs github action to auto-generate docs by @Etesam913 in https://github.com/facebookresearch/Mephisto/pull/877
Add Screening Example by @Etesam913 in https://github.com/facebookresearch/Mephisto/pull/883
Creates a readme for simplestatictask by @Etesam913 in https://github.com/facebookresearch/Mephisto/pull/882
Improve onboarding example by @Etesam913 in https://github.com/facebookresearch/Mephisto/pull/887
Add info on link_task_source to docs by @Etesam913 in https://github.com/facebookresearch/Mephisto/pull/885
Update documentation to reflect that makequalificationdict takes 3 arguments. by @xksteven in https://github.com/facebookresearch/Mephisto/pull/908

Dependencies

Bump terser from 5.12.1 to 5.14.2 in /docs/web by @dependabot in https://github.com/facebookresearch/Mephisto/pull/853
Update @blueprintjs/core version by @pringshia in https://github.com/facebookresearch/Mephisto/pull/876
Bump oauthlib from 3.2.0 to 3.2.1 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/901
Bump protobuf from 3.19.4 to 3.19.5 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/903
Bump joblib from 1.1.0 to 1.2.0 by @dependabot in https://github.com/facebookresearch/Mephisto/pull/904
Upgrading to poetry 1.2, resolving dependency issues by @JackUrb in https://github.com/facebookresearch/Mephisto/pull/898
Upgrade yarn 3.2.1->3.2.4 by @pringshia in https://github.com/facebookresearch/Mephisto/pull/920

New Contributors

@chiehminwei made their first contribution in https://github.com/facebookresearch/Mephisto/pull/861
@mjkanji made their first contribution in https://github.com/facebookresearch/Mephisto/pull/917

Full Changelog: https://github.com/facebookresearch/Mephisto/compare/v1.0.3...v1.1.0

- Python
Published by JackUrb over 3 years ago

mephisto - v1.0.3: Last version, but ParlAI-stable

Overview

v1.0.2 shipped with a bug that broke ParlAI-chat related tasks. This release only includes the fix for that (#859), but is otherwise equivalent to v1.0.2.

- Python
Published by JackUrb over 3 years ago

mephisto - v1.0.2: Key Bugfixes and QoL Upgrades

Overview

Some issues still remaining from v1.0.0 (and even sooner) have been ironed out, so this release aims to fix those as well as include some new QoL upgrades.

Highlights

A status mismatch issue related to collection slowdowns on long-running tasks has been squashed (#830)
Adding testing for our example files (#795, #800)
Upgrading completely to Poetry as our package manager (#779)
Core work for new worker addons package, to be included in coming release notes (#791, #785)

Additional Changes

MTurk improvements and fixes:
- #810: Functionality to specify the lifetime of a task
- #780: Fix to HIT tags that were being improperly submitted as individual characters
- #767: Fix to storing and restoring AWS credentials
Bugfix for deploying to the HerokuArchitect via git (#824)
Locking down the version of Node that our router uses to solve issues deploying on the EC2Architect (#762)
Cleanup to the Operator.shutdown method to allow disconnected onboardings to be cleared (#769)
Developer QoL:
- #814: Consolidating the AgentState classes and introducing a metadata attribute
- #770: Making Agent and OnboardingAgent pull from the same base class for code deduplication
Bugfixes to the RemoteProcedureBlueprint:
- #763: Fixing an issue that prevented reloading from restoring the state
- #798, #782: Cleaning up on the RemoteProcedure examples
- #784: Adding a new RemoteProcedure example using a text model-in-the-loop

- Python
Published by JackUrb over 3 years ago

mephisto - 1.0.1 Release: minor 1.0.0 Bugfixes

Overview

Following the 1.0.0 release, a few bugs were surfaced. The largest involved an update of mephisto-task already published, and this release syncs the underlying Mephisto library to refer to the same version.

Full Changelog

#728 Fixes an issue preventing live tasks from working over https
#749 Adds back some missing mephisto-task package versions that were preventing correct builds at runtime
#756 Fixes a small bug in the HTMLStaticBlueprint
#745 Clarifying correct typing for data passed via handleSubmit in our StaticBlueprint tasks
#754 fixes docker imaging issue if mephisto metrics install has already been run locally
#747 Set some defaults for max_num_concurrent_tasks to encourage people to use this option more consistently

- Python
Published by JackUrb almost 4 years ago

mephisto - Mephisto 1.0.0 Release

This PR captures all of the work the team has done for Mephisto up to now, setting our first stable API, prioritizing user-friendly features, and providing a massively updated documentation site. Read more about this release on the blog.

- Python
Published by JackUrb about 4 years ago

mephisto - EC2 Architect, Documentation, and Various Bugfixes

Overall this update doesn't change any existing semantics for Mephisto, but includes some new functionality (including launches on EC2 instead of Heroku!) and some small bugfixes. In total, this release contains the following: - EC2 architect and followup fixes (#603, #606, #609, #618, #622) - Documentation overhaul (#611, #612, #613) - Early implementation for using Gold labelling worker quality control (#585) - Bugfixes for existing blueprints (#588, #594, #637) - Various lint and dependency updates

- Python
Published by JackUrb about 4 years ago

mephisto - Hydra and ParlAI fixes, `BlueprintMixin`s.

Minor version update

This release primarily serves to help with the transition from hydra 1.0 to 1.1, though also includes some additional fixes, especially for ParlAI tasks. This release also includes BlueprintMixins, which serve as an easier way to inject first-class features into blueprints, and to provide shared functionality across otherwise disjoint tasks. Key changes: - Blueprint Mixin refactor, and the ScreenTaskRequired mixin (#582, #573, #566) - Compensation HIT launch script (#586) - PyPI Release prep (#548) - ParlAI Onboarding conversion fix, and other bugfixes (#573, #554, #555) - JS Package reorg (#565, #574, #570) - Hydra migration fixes (#587) - Other bugfixes (#535, #547, #560)

- Python
Published by JackUrb over 4 years ago

mephisto - Hydra 1.1 upgrade

Notices

The upgrade to Hydra 1.1 (#494) means that previous scripts may not function properly. Remediation steps for this are included in #529.
The old method of directly initializing Mephisto data model elements (such as Units and Agents) is being changed to <Class>.get(db, db_id) rather than a direct constructor. Both are supported for now, but the old version will be deprecated in 1.0.

This version release primarily focuses around our upgrade to hydra 1.1 (#494).
Hydra 1.1 means we can now specify profiles for common launch patterns between jobs (#496).
Deprecating direct data model initialization to move off of metaclasses in 1.0 (#437).
Stability improvements - better state tracking for MTurk units (#527)
UX improvements - easier local development of static react tasks (#509), centralizing review utilities for scripts (#526), better messaging for unexpected behaviors (#528, #530)

- Python
Published by JackUrb over 4 years ago

mephisto - Dependency Updates, Bugfixes

This release is fairly minor, including fixes to Botocore and a number of other underlying dependencies.

Also included are #486 and #489, which aim to improve detection of disconnects with the HerokuArchitect and clarify messaging about heroku setup failures.

- Python
Published by JackUrb over 4 years ago

mephisto - Speedup Improvements + More

Summary

This release focuses on Mephisto speedups, including fixes to reduce the runtime of many common operations. Most of these are further reduced by using the (BETA) MephistoSingletonDB. In combination, these reduce Mephisto's cpu usage by 90% or more for the same number of active tasks, allowing for increased concurrency.

It also includes a number of bugfixes and stability improvements since January

Included changes

max_num_concurrent_units option for all Mephisto tasks #405
Hook to register callbacks after the completion of a task #339
Speedups for interactions requiring the MTurkProvider #435
Speedups for overall Mephisto operations by no longer loading AgentState for every Agent, preventing unnecessary data reads. #341
Significant speedups using the MephistoSingletonDB #432 #441
Other speed-related fixes #440
Flask option for Heroku router #398
Various improvements to HerokuArchitect #403 #413 #411 #418
MTurk related bugfixes #382 #385 #445
Other bugfixes #400 #421 #410 #439 #443 #456

- Python
Published by JackUrb almost 5 years ago

mephisto - Deprecating ParlAI-MTurk

This version removes any references to the old ParlAI-MTurk implementation, as it is soon to be removed entirely from the ParlAI repo.

- Python
Published by JackUrb about 5 years ago

mephisto - Small bug patch on v0.3.1

Contains all of the good changes of v0.3.1 but with one less bad change - shutdowns won't run into the race condition of list iterators changing sizes anymore.

- Python
Published by JackUrb about 5 years ago

mephisto - v0.3.1 - Stability and Shutdown improvements

Version 0.3.1 brings a few stability improvements: - Improved checking to ensure that only one unit is linked to an MTurk HIT at a time - Properly returning live tasks to the pool if a worker returns while in the WAITING status - Correct final statuses on static task units - Cleaner and more consistent shutdown behavior, as well as a force_shutdown implementation to use in tests and automated scripts.

It also has a few improvements with ParlAI tasks.

- Python
Published by JackUrb about 5 years ago

mephisto - First release post path migration

This Mephisto release contains no core functionality improvements over v0.2.2, but has now removed the backwards-compatible import locations across the codebase. It also includes the creation of a more complete README structure for the project.

This signifies the completion of the plan set out in #285.