https://github.com/deepset-ai/farm - v0.8.0 DPR Improvements

DPR Improvements

DPR - improve loading of datasets #733 @voidful DPR - enable saving and loading of other model types, e.g., RoBERTa models #765 @Timoeller @julian-risch DPR - fix conversion of BiAdaptiveModel #753 @bogdankostic

torch 1.8.1 and transformers 4.6.1

Bump transformers version to 4.6.1 #787 @Timoeller @julian-risch Bump torch version to 1.8.1 #767 @Timoeller @julian-risch

Multi-task Learning

Implement Multi-task Learning and added example #778 @johann-petrak

List of Evaluation Metrics

Allow list of metrics and add tests and pythondoc #777 @johann-petrak

Misc

Reduce number of logging messages by Processor about returning problematic ids #772 @johann-petrak Add farm.__version__ tag #761 @johann-petrak Add value of docstride, maxseqlen, maxquerylength in error message #784 @ftesser Convert QACandidates with empty or whitespace answers to noanswers on doc level #756 @julian-risch

String comparison: Should replace "is" with "==": #774 @johann-petrak Fix reference before assignment in DataSilo #738 @bogdankostic Changing QA_input format in tutorial #735 @julian-risch Fix TextPairClassificationProcessor example by adding metric #780 @julian-risch

- Python
Published by julian-risch almost 5 years ago

https://github.com/deepset-ai/farm - DPR patch release

A patch release focusing on bug fixes for Dense Passage Retrieval DPR Fix saving and loading of DPR models and Processors in #746 Fix DPR tokenization statisticss in #738 Fix cosine similarity in DPR training #741

Misc Fix tuple input for TextPairClassification inference #723

- Python
Published by Timoeller about 5 years ago

https://github.com/deepset-ai/farm - v0.7.0 QA Model Confidence

QA Confidence Scores

In response to several requests from the community, we now provide more meaningful confidence scores for the predictions of extractive QA models. #690 #705 @julian-risch @timoeller @lalitpagaria To this end, predicted answers got a new attribute called confidence, which is in the range [0,1] and can be calibrated with the probability that a prediction is an exact match. The intuition behind the scores is the following: After calibration, if the average confidence of 100 predictions is 70%, then on average 70 of the predictions will be correct. The implementation of the calibration uses a technique called temperature scaling. The calibration can be executed on a dev set by running the eval() method in the Evaluator class and setting the parameter calibrateconfscores to true. This parameter is false by default as it is still an experimental feature and we continue working on it. The score attribute of predicted answers and their ranking remain unchanged so that the default behavior is unchanged. An example shows how to calibrate and use the confidence scores.

Misc

Refactor Text pair handling, that also add Text pair regression #713 @timoeller Refactor Textsimilarity processor #711 @timoeller Refactor Regression and inference processors #702 @timoeller Fix NER probabilities #700 @brandenchan Calculate squad evaluation metrics overall and separately for text answers and no answers #698 @julian-risch Re-enable testdprmodules also for windows #697 @ftesser Use Path instead of String in ONNXAdaptiveModel #694 @skiran252 Big thanks to all contributors!

- Python
Published by julian-risch over 5 years ago

https://github.com/deepset-ai/farm - Patch Release - Bugfix QA Inference

This is just a small patch to change the return types of offsets in our QAInferencer, see #693

It is needed to fix RestAPI related issues where int64 cannot decoded within JSONs.

- Python
Published by Timoeller over 5 years ago

https://github.com/deepset-ai/farm - Patch Release - Bugfix QA Processing

Patch release

This is just a quick patch release to bugfix some input validation for Question Answering [closed] Fix/missing truncation bug #679

Additional feature for QA

Still, another interesting feature slipped in: We can now filter QA predictions to not contain duplicate answers. [closed] Added filter_range parameter that allows to filter answers with similar start/end indices #680

Additional test

[part: tokenizer][task: QA] Add integration test for QA processing #683

Misc

[closed] Remove "qas" inference input wherever possible #681 [closed] Added parameter names to convertfromtransformers call in questionansweringcrossvalidation.py #672

- Python
Published by Timoeller over 5 years ago

https://github.com/deepset-ai/farm -

Simplification of Preprocessing

We wanted to make preprocessing for all our tasks (e.g. QA, DPR, NER, classification) more understandable for FARM users, so that it is easier to adjust to specific use cases or extend the functionality to new tasks.

To achieve this we followed two design choices: 1. Avoid deeply nested calls 2. Keep all high-level descriptions in a single place

Question Answering Preprocessing

We especially focussed on making QA processing more sequential and divided the code into meaningful snippets #649

The code snippets are (see related method): - convert the input into FARM specific QA format - tokenize the questions and texts - split texts into passages to fit the sequence length constraint of Language Models - [optionally] convert labels (disabled during inference) - convert question, text, labels and additional information to PyTorch tensors

Breaking changes

Switching to FastTokenizers (based on Huggingface tokenizer project written in Rust) as default Tokenizer. We changed the use_fast=True parameter in the Tokenizer.load() method. Support for slow, python-based Tokenizers will be implemented for all tasks in the next release.
The Processor.datasetfromdicts method by default returns an additional parameter problematic_sample_ids that keeps track of which input sample caused problems during preprocessing: dataset, tensor_names, problematic_sample_ids = processor.dataset_from_dicts(dicts=dicts)

Update to transformers version 4.1.1 and torch version 1.7.0

Transformers comes with many new features, including model versioning, that we do not want to miss out on. #665 Model versions can now be specified like: model = Inferencer.load( model_name_or_path="deepset/roberta-base-squad2", revision="v2.0", task_type="question_answering", )

DPR enhancements

MultiGPU support #619
Added tests #643
Bugfixes and smaller enhancements #629 #655 #663

Misc

Cleaner logging and error handling #639
Benchmark automation via CML #646
Disable DPR tests on Windows, since they do not work with PyTorch 1.6.1 #637
Option to disable MLflow logger #650
Fix to Earlystopping and custom head #617
Adding probability of masking a token parameter for LM task #630

Big thanks to all contributors! @ftesser @pashok3d @Timoeller @tanaysoni @brandenchan @bogdankostic @kolk @tholor

- Python
Published by Timoeller over 5 years ago

https://github.com/deepset-ai/farm - v0.5.0

Add Dense Passage Retriever (DPR) incl. Training & Inference (#513, #601, #606)

Happy to introduce a completely new task type to FARM: Text similarity with two separate transformer encoders

Why? We observe a big shift in Information Retrieval from sparse methods (BM25 etc.) towards dense methods that encode queries and docs as vectors and use vector similarity to retrieve the most similar docs for a certain query. This is not only helpful for document search but also for open-domain Question Answering. Dense methods outperform sparse methods already in many domains and are especially powerful if the matching between query and passage cannot happen via "keywords" but rather relies on semantics / synonyms / context.

What? One of the most promising methods at the moment is "Dense Passage Retrieval" from Karphukin et al. (https://arxiv.org/abs/2004.04906). In a nutshell, DPR uses one transformer to encode the query and a second transformer to encode the passage. The two encoders project the different texts into the same vector space and are trained jointly on a similarity measure using in-batch-negatives.

How? We introduce a new class BiAdaptiveModel that has two language models plus a prediction head. In the case of DPR, this will be one question encoder model and one passage encoder model.
See the new example script dpr_encoder.py for training / fine-tuning a DPR model. We also have a tight integration in Haystack, where you can use it as a Retriever for open-domain Question Answering.

Refactor conversion from / to Transformers #576

We simplified conversion between FARM <-> Transformers. You can now run:

```python

Transformers -> FARM

model = Converter.convertfromtransformers("deepset/roberta-base-squad2", device="cpu")

FARM -> Transformers

transformermodels = Converter.converttotransformers(youradaptive_model) ``` Note: In case your FARM AdaptiveModel has multiple prediction heads (e.g. 1x NER, 1x Text Classification), the conversion will return a list with two transformer models (both with one head respectively).

Upgrade to Transformers 3.3.1 #579

Transformers 3.3.1 comes with a few new interesting features, incl. support for Retrieval-Augmented Generation (RAG) which can be used to generate answers rather than extracting answers. In contrast to GPT-3, the generation is conditioned on a set of retrieved documents, and is, therefore, more suitable for most QA applications in the industry that rely on a domain corpus.
Thanks to @lalitpagaria, we'll support RAG also in Haystack soon (see https://github.com/deepset-ai/haystack/pull/484)

Details

Question Answering

Improve Speed: Vectorize Question Answering Prediction Head #603
Fix removal of yes no answers #540
Fix QA bug that rejected spans at beginning of passage #564
Added warning about that Natural Questions Inference. #565
Remove loss index from QA PH #589

Other

Catch empty datasets in Inferencer #605
Add option to set evaluation batch size #607
Infer model type from config #600
Fix random behavior when loading ELECTRA models #599
Fix import for Python3.6 #581
Fixed conversion of BertForMaskedLM to transformers #555
Load correct config for DistilBert model #562
Add passages per second calculation to benchmarks #560
Fix batching in ONNX forward pass #559
Add ONNX conversion & Inference #557

Big thanks to all contributors! @ftesser @lalitpagaria @himanshurawlani @Timoeller @tanaysoni @brandenchan @bogdankostic @kolk @tholor

- Python
Published by tholor over 5 years ago

https://github.com/deepset-ai/farm - v0.4.9

Minor patch: Relax PyTorch version requirements

Installing FARM in environments where torch's GPU version was already installed via pip (e.g. torch 1.6.0+cu101), caused version trouble. This is especially annoying in Google Colab environments. Change: Allow all torch 1.6.x versions incl 1.6.0+cu101 etc

Further changes: - Nested cross validation by @PhilipMay #508

- Python
Published by tholor over 5 years ago

https://github.com/deepset-ai/farm - 0.4.8

Minor release

Experimental Support for fast Rust Tokenizers (#482)

While preprocessing is usually not the bottleneck in our pipelines, there's still significant time spent on it (~ 20 % for QA inference). We saw substantial speed-ups with HuggingFace's "FastTokenizers" that are based on rust. We are therefore introducing a basic "experimental" implementation with this release. We are planning to stabilizing it and having a smoother fit into the FARM processor.

Usage: python tokenizer = Tokenizer.load(pretrained_model_name_or_path=""bert-base-german-cased"", do_lower_case=False, use_fast=True)

Upgrade to transformers 3.1.0 (#464)

The latest transformers release has quite interesting new features - one of them being basic support of a DPR model class (Dense Passage Retriever). This will simplify our dense passage retriever integration in Haystack and the upcoming DPR training which we plan to have in FARM.

Details

Question Answering

Add asserts on docstride and maxseq_len to prevent issues with sliding window #538
fix Natural Question inference processing #521

Other

Fix logging of error msg for FastTokenizer + QA #541
Fix truncation warnings in tokenizer #528
Evaluate model on best model when doing early stopping #524
Bump transformers version to 3.1.0 #515
Add warmup run to component benchmark #504
Add optional s3 auth via params #511
Add option to use fast HF tokenizer. #482
CodeBERT support for embeddings #488
Store test eval result in variable #506
Fix typo f1 micro vs. macro #505

Big thanks to all contributors! @PhilipMay @lambdaofgod @Timoeller @tanaysoni @brandenchan @bogdankostic @kolk @tholor

- Python
Published by tholor over 5 years ago

https://github.com/deepset-ai/farm - 0.4.7

Main changes

Support for MiniLM Model (#464)

Interesting model from Microsoft that is up to 2.7x faster than BERT, while showing similar or better performance on many tasks (Paper). We found it particularly interesting for QA and also published a fine-tuned model on SQuAD 2.0: deepset/minilm-uncased-squad2

Benchmarks per component (#491)

Measuring the speed of individual components in the pipeline while respecting CUDA's async behaviour. We were especially interested in analyzing how much time we spend for QA in preprocessing, language model, and prediction head. Turns out it's on average about 20% : 50% : 30%. Interestingly, there's a high variance in the prediction head depending on the relevance of the question. We will use that information to further optimize performance in the prediction head. We'll share more detailed benchmarks soon.

Support for PyTorch 1.6 (#502)

We now support 1.6 and 1.5.1

Details

Question Answering

Pass max_answers param to processor #503
Deprecate QA input dicts with [context, qas] as keys #472
Squad processor verbose feature #470
Propagate QA ground truth in Inferencer #469
Ensure QAInferencer always has tasktype "questionanswering" #460

Other

Download models from (private) S3 #500
fix initializedataloaders in datasilo #476
Remove torch version wildcard in requirements #489
Make num processes parameter consistent across inferencer and data silo #480
Remove restapischema argument in inferencefromdicts() #474
farm.datahandler.utils: Add encoding to open write in splitfile method #466
Fix and document Inferencer usage and pool handling #429
Remove assertions or replace with logging error #468
Remove baskets without features in createdataset. #471
fix bugs with regression label standardization #456

Big thanks to all contributors! @PhilipMay @Timoeller @tanaysoni @brandenchan @bogdankostic @kolk @rohanag @lingsond @ftesser

- Python
Published by tholor almost 6 years ago

https://github.com/deepset-ai/farm - 0.4.6

Main changes

Upgrading to Pytorch 1.5.1 and transformers 3.0.2
Important bug fix for language model training from scratch
Bug fixes and big refactorings for Question Answering, incl. a specialized QAInferencer with dedicated In- and Output objects to simplify usage and code completion: ``` from farm.infer import QAInferencer from farm.data_handler.inputs import QAInput, Question

nlp = QAInferencer.load( "deepset/roberta-base-squad2", tasktype="questionanswering", batchsize=16, numprocesses=0)

input = QAInput( doc_text="My name is Lucas and I live on Mars.", questions=Question(text="Who lives on Mars?", uid="your-id"))

res = nlp.inferencefromobjects([input], return_json=False)[0]

High level attributes for your query

print(res.question) print(res.context) print(res.noanswergap)

...

Attributes for individual predictions (= answers)

pred = res.prediction[0] print(pred.answer) print(pred.answertype) print(pred.answersupport) print(pred.offsetanswerstart) print(pred.offsetanswerend)

...

```

Details

Question Answering

Add meta attribute to QACandidate for Haystack #455
Fix start and end offset checks in QA #450
Fix offset_end character for QA #449
Dedicated Input Objects for QA #445
Question Answering improvements: cleaner code, more typed objects, better compatibility between SQuAD and Natural Questions #411, #438, #419

Other

Upgrade pytorch and python versions #447
Upgrade transformers version #448
Fix randomisation of train file for training from scratch #427
Fix loading of saved models with class weights #431
Remove raising exception errors in processor #451
Fix bug in benchmark tests with if statement #430
Remove hardcoded seeds from trainer #424
Conditional numtrainingsteps setting #437
Add badge with link to doc page #432
Fix for CI problem - closing multiprocessing.pool again #403

:manfarmer: :womanfarmer: Thanks to all contributors for making FARMer's life better! @PhilipMay, @tstadel, @brandenchan, @tanaysoni, @Timoeller, @tholor, @bogdankostic

- Python
Published by tholor almost 6 years ago

https://github.com/deepset-ai/farm - 0.4.5

Minor release including an important bug fix for Question Answering

Important Bug Fix QA

Fixing a bug that was introduced in 0.4.4 (#416 and #417) that resulted in returning only a single answer per document in certain situations. This caused particular trouble for open-domain QA settings like in haystack.

Speed optimization training from scratch

Adding multiple optimizations and bug fixes to improve training from scratch, incl.: - Enable usage of DistributedDataParallel - Enable Automatix Mixed Precision Training - Fix bugs in StreamingDataSilo - Fix bugs in Checkpointing (important for training via spot / on-demand instances)

This helped to boost training time in our benchmark from 616 hours down to 160 hours See #305 for details

Other changes:

Add model optimization to inference loading #415
Proper attribute assignment in QA with yes / no answer #414

- Python
Published by tanaysoni almost 6 years ago

https://github.com/deepset-ai/farm - 0.4.4

ELECTRA Model

We welcome a new language model to the FARM family that we found to be a really powerful alternative to the existing ones. ELECTRA is trained using a small generator network that replaces tokens with plausible alternatives and a discriminative model that predicts which learns to detect these replaced tokens (see the paper for details: https://arxiv.org/abs/2003.10555). This makes pretraining more efficient and improves down-stream performance for quite many tasks.

You can load it as usual via

LanguageModel.load("google/electra-base-discriminator") See HF's model hub for more model variants

Natural Questions Style QA

With QA being our favorite and focussed down-stream task, we are happy to support an additional style of QA in FARM ( #334). In contrast to the popular SQuAD-based models, these NQ models support binary answers, i.e. questions like "Is Berlin the capital of Germany?" can be answered with "Yes" and an additional span that the model used as a "supporting fact" to give this answer.

The implementation leverages the option of prediction heads in FARM by having one QuestionAnsweringHead that predicts a span (like in SQuAD) and one TextClassificationHead that predicts what type of answer the model should give (current options: span, yes, no, is_impossible).

Example:

``` QAinput = [ { "qas": ["Is Berlin the capital of Germany?"], "context": "Berlin (/bɜːrˈlɪn/) is the capital and largest city of Germany by both area and population." } ] model = Inferencer.load(modelnameorpath="../models/roberta-base-squad2-nq", batchsize=batchsize, gpu=True) result = model.inferencefromdicts(dicts=QAinput, returnjson=False) print(f"Answer: {result[0].prediction[0].answer}")

Answer: yes ```

See this new example script for more details on training and inference.

Note: This release includes the initial version for NQ, but we are already working on some further simplifications and improvements in #411 .

New speed benchmarking

With inference speed being crucial for many deployments - especially for QA, we introduce a new benchmarking tool in #321. This allows us to easily compare the performance of different frameworks (e.g. ONNX vs. pytorch), parameters (e.g. batch size) and code optimizations of different FARM versions. See the readme for usage details and this spreadsheet for current results.

A few more changes ...

Modeling - Add support for Camembert-like models #396 - Speed up in BERTLMHead by doing argmax on logits on GPU #377 - Fix bug in BERT-style pretraining #369 - Remove additional XLM-R tokens #360 - ELECTRA: use gelu for pooled output of ELECTRA model #364

Data handling - Option to specify text col name in TextClassificationProcessor and RegressionProcessor #387 - Document magic data loading in TextClassificationProcessor PR #383 - multilabel support for datasilo.calculateclass_weights #389 - Implement Prediction Objects for Question Answering #405 - Removing lambda function from AdaptiveModel so the class can be pickable #345 - Add target device optimisations for ONNX export #354

Examples / Docs - Add script to reproduce results from COVID-QA paper #412 - Update tutorials #348 - Docstring Format fix #382

Other - Adjust code to squad inferencing #367 - Convert pydantic objects to regular classes #410 - Rename top n recall to top n accuracy #409 - Add test for embedding extraction #394 - Quick fix CI problems with OOM and unclosed worker pool #406 - Update transformers version to 2.11 #407 - Managing pytorch pip find-links directive #393 - Zero based Epoch Display in Training Progress Bar #398 - Add stalebot #400 - Update pytorch to 1.5.0 #392 - Question answering accuracy test #357 - Add init.py files for farm.conversion module #365 - Make onnx imports optional #363 - Make ONNXRuntime dependency optional #347

:manfarmer: :womanfarmer: Thanks to all contributors for making FARMer's life better! @PhilipMay , @stefan-it, @ftesser , @tstadel, @renaud, @skirdey, @brandenchan, @tanaysoni, @Timoeller, @tholor, @bogdankostic

- Python
Published by tanaysoni almost 6 years ago

https://github.com/deepset-ai/farm - 0.4.3

:1234: Changed Multiprocessing in Inferencer

The Inferencer has now a fixed pool of processes instead of creating a new one for every inference call. This accelerates the processing a bit and solves some problems when using it in combination with Frameworks like gunicorn/FastAPI etc (#329)

Old: ... inferencer.inference_from_dicts(dicts, num_processes=8)

New: Inferencer(dicts, num_processes=8) ...

:fast_forward: Streaming Inferencer

You can now also use the Inferencer in a "streaming mode". This is especially useful in production scenarios where the Inferencer is part of a bigger pipeline (e.g. consuming documents from elasticsearch) and you want to get predictions as soon as they are available (#315)

Input: Generator yielding dicts with your text Output: Generator yielding your predictions

dicts = sample_dicts_generator() # it can be a list of dicts or a generator object results = inferencer.inference_from_dicts(dicts, streaming=True, multiprocessing_chunksize=20) for prediction in results: # results is a generator object that yields predictions print(prediction)

:olderwoman: :olderman: "Classic" baseline models for benchmarking + S3E Pooling

While Transformers are conquering many of the current NLP tasks, there are still quite some tasks (e.g. some document classification) where they are a complete overkill. Benchmarking Transformers with "classic" uncontextualized embedding models is a common, good practice and is now possible without switching frameworks. We added basic support for loading in embeddings models like GloVe, Word2vec and FastText and using them as a "LanguageModels" in FARM (#285)

See the example script

We also added a new pooling method to get sentence or document embeddings from these models that can act as a strong baseline for transformer-based approaches (e.g Sentence-BERT). The method is called S3E and was recently introduced by Wang et al in "Efficient Sentence Embedding via Semantic Subspace Analysis" (#286)

See the example script

A few more changes ...

Modeling - Cross-validation for Question-Answering #335 - Add option to use maxseqlen tokens for LM Adaptation/Training-from-scratch instead of real sentences #314 - Add english glove models #339 - Implicitly connect heads with processor + check for connection #337

Evaluation & Inference - Registration of custom evaluation reports #331 - Standalone Evaluation with pretrained models #330 - tqdm progress bar in inferencer #338 - Group NER preds by sample #327 - Fix Processor configs when loading Inferencer #318

Other - Fix the IOB2 to simple tags check #324 - Update config when saving model to include changes of parameters #323 - Fix Issues with NER format Conversion #322 - Fix error message in loading of Tokenizer #317 - Less verbosity, Fix which Samples and Baskets being Thrown Away #313

:manfarmer: :womanfarmer: Thanks to all contributors for making FARMer's life better! @brandenchan, @tanaysoni, @Timoeller, @tholor, @bogdankostic, @gsarti

- Python
Published by tanaysoni about 6 years ago

https://github.com/deepset-ai/farm - 0.4.2

:fast_forward: Scalable preprocessing: StreamingDataSilo

Allows you to load data lazily from disk and preprocess a batch on-the-fly when needed during training.

stream_data_silo = StreamingDataSilo(processor=processor, batch_size=batch_size)

=> Allows large datasets that don't fit in memory (e.g. for training from scratch) => Training directly starts. No initial time for preprocessing needed.

:rocket: Better Inference: Speed, scalability, standardization

ONNX support:

Microsoft recently added optimizations to the ONNX-runtime and reported substantial speed-ups compared to PyTorch. Since these improvements can be particularly useful for inference-heavy tasks such as QA, we added a way to export your AdaptiveModel to the ONNX format and load it into the Inferencer:

model = AdaptiveModel (...) model.convert_to_onnx(Path("./onnx_model")) inferencer = Inferencer.load(model_name_or_path=Path("./onnx_model")) => See example
=> Speed improvements depend on device and batch size. On a Tesla V100 we measured improvements between 30% - 260% for doing end-to-end QA inference on a large document and we still see more potential for optimizations.

| Batch Size | PyTorch | ONNX | ONNX V100 optimizations | Speedup | |------------|---------|------|-----------|---------| | 1 | 27.5 | 12.8 | 10.6 | 2.59 | | 2 | 17.5 | 11.5 | 9.1 | 1.92 | | 4 | 12.5 | 10.7 | 8.3 | 1.50 | | 8 | 10.6 | 10.2 | 8.2 | 1.29 | | 16 | 10.5 | 10.1 | 7.8 | 1.38 | | 32 | 10.1 | 9.8 | 7.8 | 1.29 | | 64 | 9.9 | 9.8 | 7.8 | 1.26 | | 128 | 9.9 | 9.8 | 7.7 | 1.28 | | 256 | 10.0 | 9.8 | 7.9 | 1.26 |

Embedding extraction:

Extracting embeddings from a model at inference time is now more similar to other inference modes.

Old model = Inferencer.load(lang_model, task_type="embeddings", gpu=use_gpu, batch_size=batch_size) result = model.extract_vectors(dicts=basic_texts, extraction_strategy="cls_token", extraction_layer=-1) New model = Inferencer.load(lang_model, task_type="embeddings", gpu=use_gpu, batch_size=batch_size, extraction_strategy="cls_token", extraction_layer=-1) result = model.inference_from_dicts(dicts=basic_texts, max_processes=1)

=> The preprocessing can now also utilize multiprocessing => It's easier to reuse other methods like Inference.inference_from_file()

:leftrightarrow: New tasks: TextPairClassification & Passage ranking

Added supprt for text pair classification and ranking. Both can be especially helpful in semantic search settings where you want to (re-)rank search results and will be incorporated in our haystack framework soon. Examples: - MSMARCO passage ranking - ASNQ text pair classification

A few more changes ...

Faster & simpler Inference

Make extract_vectors more compatible to other inference types #292
Add test for onnx qa inference. Fix bug in loading PHs for ONNX. #297
Add ONNX Inference for Question Answering #288
Improve inferencer for better multiprocessing with QA / haystack #278
Scalable Qa aggregation #268
Allow for multiple queries in QA inference when using rest_api format #246
Decouple n_best in QA predictions #269
Correct keyword argument for maxprocesses when used by calcchunksize() #255
Add document id to QA inference #265
Refactor no answer handling in Question Answering #258

Streaming Data Silo / Training from scratch

StreamingDataSilo for loading & preprocessing batches lazily during training #239
Fix dict chunking in StreamingDataSilo for LMFinetuning #284
Add example for training with AWS SageMaker #283
Fix deletion of old training checkpoints #282
Fix epoch number for saving a training checkpoint #281
Fix Train Step calculations for Checkpointing #279
Implement len() for StreamingDataSilo #274
Refactor StreamingDataSilo to support multiple train epochs #266
Fix serialization for saving train checkpoints #271

Modeling

Add support for text pair classification (ASNQ) and ranking (MSMarco) #237
Add conversion of lm_finetuned to HF transformers #290
Added next_sentence_head in examples/lm_finetuning.py. #273
Quickfix loading pred head #256
Maked use of 'language' **kwargs if present in LanguageModel.load. #262
Add the option to define the language model class manually #264
Fix XLMR Bug When Calculating Start of Second Sequence #240

Examples / Tutorials / Experiments

Add data handling for GermEval14, add checks for correct data files #259
Fix separator in CoNLL_de experiment config #254
Use correct German conll03 data + conversion #248
Bugfix parameter loading through experiment configs #252
Add early stopping to experiment #253
Fix Tutorial: Add missing param in initialize_optimizer #245

Other

Add Azure test pipeline #270
Fix progress bar in datasilo #267
Turn off prints and logging during testing #260
Pin Werkzeug version in requirements.txt #250
Add ConnectionError handling for MLFlow logger #236
Clearer message when DataSilo calculates Sequence Lengths #293
Add metric to textpairclassification example #294
Add preprocessed CORD-19 dataset #295

:manfarmer: :womanfarmer: Thanks to all contributors for making FARMer's life better! @brandenchan, @tanaysoni, @Timoeller, @tholor, @bogdankostic, @andra-pumnea, @PhilipMay, @ftesser, @guggio

- Python
Published by tanaysoni about 6 years ago

https://github.com/deepset-ai/farm - 0.4.1

:manfarmer: :arrowscounterclockwise: :hugs: Full compatibility with Transformers' models

Open-source is more than just public code. It's a mindset of sharing, being transparent and collaborating across organizations. It's about building on the shoulders of other projects and advancing together the state of technology. That's why we built on the top of the great Transformers library by huggingface and are excited to release today an even deeper compatibility that simplifies the exchange & comparison of models.

1. Convert models from/to transformers model = AdaptiveModel.convert_from_transformers("deepset/bert-base-cased-squad2", device="cpu", task_type="question_answering") transformer_model = model.convert_to_transformers()

2. Load models from their new model hub: LanguageModel.load("TurkuNLP/bert-base-finnish-cased-v1") Inferencer.load("deepset/bert-base-cased-squad2", task_type="question_answering") ...

:rocket: Better & Faster Training

Thanks to @BramVanroy and @johann-petrak we got some really hot new features here: - Automatic Mixed Precision (AMP) Training: Speed up your training by ~ 35%! Model params are usually stored with FP32 precision. Some model layers don't need that precision and can be reduced to FP16, which speeds up training and reduces memory footprint. AMP is a smart way of figuring out, for which params we can reduce precision without sacrificing performance (Read more). Test it by installing apex and setting "use_amp" to "O1" in one of the FARM example scripts.

More flexible Optimizers & Schedulers: Choose whatever optimizer you like from PyTorch, apex or Transformers. Take your preferred learning rate schedule from Transformers or PyTorch (Read more)
Cross-validation: Get more reliable eval metrics on small datasets (see example)
Early Stopping: With early stopping, the run stops once a chosen metric is not improving any further and you take the best model up to this point. This helps prevent overfitting on small datasets and reduces training time if your model doesn't improve any further (see example).

:fast_forward: Caching & Checkpointing

Save time if you run similar pipelines (e.g. only experimenting with model params): Store your preprocessed dataset & load it next time from cache: data_silo = DataSilo(processor=processor, batch_size=batch_size, caching=True) Start & stop training by saving checkpoints of the trainer: trainer = Trainer.create_or_load_checkpoint( ... checkpoint_on_sigterm=True, checkpoint_every=200, checkpoint_root_dir=Path(“/opt/ml/checkpoints/training”), resume_from_checkpoint=“latest”) The checkpoints include the state of everything that matters (model, optimizer, lr_schedule ...) to resume training. This is particularly useful, if your training crashes (e.g. because you are using spot cloud instances).

:cloud: Integration with AWS SageMaker & Training from scratch

We are currently working a lot on simplifying large scale training and deployment. As a first step, we are adding support for training on AWS SageMaker. The interesting part here is the option to use Spot Instances and save about 70% of costs compared to regular instances. This is particularly relevant for training models from scratch, which we introduce in a basic version in this release and will improve over the next weeks. See this tutorial to get started with using SageMaker for training on down-stream tasks.

:computer: Windows support

FARM now also runs on Windows. This implies one breaking change: We now use pathlib and therefore expect all directory paths to be of type Path instead of str #172

A few more changes ...

Modelling

[enhancement] ALBERT support #169
[enhancement] DistilBERT support #187
[enhancement] XLM-Roberta support #181
[enhancement] Automatically infer layer dims of prediction head #195
[bug] Implement nextsentpred flag #198

QA

[enhancement] Encoding of QA IDs #171
[enhancement] Remove repeat QA preds from overlapping passages #186
[enhancement] More options to control predictions of Question Answering Head #183
[bug] Fix QA example #203

Training

[enhancement] Use AMP instead of naive fp16. More optimizers. More LR Schedules. #133
[bug] Fix for use AMP instead of naive fp16 (#133) #180
[enhancement] Add early stopping and custom metrics #165
[enhancement] Add checkpointing for training #188
[enhancement] Add train loss to tqdm. add desc for data preproc. log only 2 samples #175
[enhancement] Allow custom functions to aggregate loss of prediction heads #220

Eval

[bug] Fixed micro f1 score #179
[enhancement] Rename classification_report to report #173

Data Handling

[enhancement] Add caching of datasets in DataSilo #177
[enhancement] Add option to limit number of processes in datasilo #174
[enhancement] Add maxmultiprocessingchunksize as a param for DataSilo #168
[enhancement] Issue59 - Add cross-validation for small datasets #167
[enhancement] Add max_samples argument to TextClassificationProcessor #204
[bug] Fix bug with added tokens #197

Other

[other] Disable multiprocessing in lm_finetuning tests to reduce memory footprint #176
[bug] Fix device arg in examples #184
[other] Add error message to train/dev split fn #190
[enhancement] Add more seeds #192

:manfarmer: :womanfarmer: Thanks to all contributors for making FARMer's life better! @brandenchan, @tanaysoni, @Timoeller, @tholor, @maknotavailable, @johann-petrak, @BramVanroy

- Python
Published by tanaysoni over 6 years ago

https://github.com/deepset-ai/farm - 0.3.2

:paintbrush: Fundamental Re-design of Question Answering

We believe QA is one of the most exciting tasks for transfer learning. However, the complexity of the task lets pipelines easily become messy, complicated and slow. This is unacceptable for production settings and creates a high barrier for developers to modify or improve them.

We put substantial effort in re-designing QA in FARM with two goals in mind: making it the simplest & fastest pipeline out there. Results: - :bulb: Simplicity: The pipeline is cleaner, more modular and easier to extend. - :rocket: Speed: Preprocessing of SQuAD 2.0 got down to 42s on a AWS p3.8xlarge (vs. ~ 20min in transformers and early versions of FARM). This will not only speed up training cycles and reduce GPU costs, but has also a big impact at inference time, where most time is actually spend on preprocessing.

See this blog post for more details and to learn about the key steps in a QA pipeline.

:briefcase: Support of proxy servers

Good news for our corporate users: Many of you approached us that the automated downloads of datasets / models caused problem in environments with proxy servers. You can now pass the proxy details to Processor and LanguageModel in the format used by the requests library

Example:
``` proxies = {"https": "http://user:pass@10.10.10.10:8000"}

languagemodel = LanguageModel.load(pretrainedmodelnameorpath = "bert-base-cased", language = "english", proxies=proxies ) ... processor = BertStyleLMProcessor(datadir="data/lmfinetunenips", tokenizer=tokenizer, maxseqlen=128, maxdocs=25, nextsent_pred=True, proxies = proxies, ) ```

Modelling

[enhancement] QA redesign #151
[enhancement] Add backwards compatibility for loading prediction head #159
[enhancement] Raise an Exception when an invalid path is supplied for loading a saved model #137
[bug] fix context in QA formatted preds #163
[bug] Fix loading custom vocab in transformers style for LM finetuning #155

Data Handling

[enhancement] Allow to load dataset from dicts in DataSilo #127
[enhancement] Option to supply proxy server #136
[bug] Fix tokenizer for multiple whitespaces #156

Inference

[enhancement] Change context in QA formatted preds to not split words #138

Other

[enhancement] Add test for output format of QA Inferencer #149
[bug] Fix classification report for multilabel #150
[bug] Fix inference in docclassificationcola example #147

Thanks to all contributors for making FARMer's life better! @johann-petrak, @brandenchan, @tanaysoni, @Timoeller, @tholor, @cregouby

- Python
Published by tanaysoni over 6 years ago

https://github.com/deepset-ai/farm - 0.3.1

Improved Question Answering

Aggregation over multiple passages

When asking questions on long documents, the underlying Language Model needs to cut the document in multiple passages and answer the question on each of them. The output needs to be aggregated.

Improved QA Inferencer

The QA Inferencer - projects model predictions back to character space - can be used in the FARM demos UI - writes predictions in SQuAD style format, so you can compare the model accuracy with other frameworks

Modelling

[closed] Refactor squad qa #131
[enhancement][part: model] Fix passing kwargs to LM loading (e.g. proxy) #132

- Python
Published by tanaysoni over 6 years ago

https://github.com/deepset-ai/farm - 0.3.0

Major Changes

Adding Roberta & XLNet

Welcome RoBERTa and XLNet on the FARM :tada:! We did some intense refactoring in FARM to make it easier to add more language models. However, we will only add models where we see some decent advantages. One of the next models to follow will very likely be ALBERT ...

For now, we support Roberta/XLNet on (Multilabel) Textclassification, Text Regression and NER. QA will follow soon.

:warning: Breaking Change - Loading of Language models has changed: Bert.load("bert-base-cased") -> LanguageModel.load("bert-base-cased")

Migrating to tokenizers from the transformers repo.

Pros: - It's quite easy to add a tokenizer for any of the models implemented in transformers. - We rather support the development there than building something in parallel - The additional metadata during tokenization (offsets, startofword) is still created via tokenizewithmetadata - We can use encode_plus to add model specific special tokens (CLS, SEP ...)

Cons: - We had to deprecate our attribute "neversplitchars" that allowed to adjust the BasicTokenizer of BERT. - Custom vocab is now realized by increasing vocab_size instead of replacing unused tokens

:warning: Breaking Change - Loading of tokenizers has changed: BertTokenizer.from_pretrained("bert-base-cased") -> Tokenizer.load("bert-base-cased")

:warning: Breaking Change - neversplitchars: is no longer supported as an argument for the Tokenizer

Modelling:

[enhancement] Add Roberta, XLNet and redesign Tokenizer #125
[bug] fix loading of old tokenizer style #129

Data Handling:

[bug] Fix name of squad labels in experiment config #121
[bug] change arg in squadprocessor from labels to label_list #123

Inference:

[enhancement] Add option to disable multiprocessing in Inferencer(#117) #128
[bug] Fix logging verbosity in Inferencer (#117) #122

Other

[enhancement] Tutorial update #116
[enhancement] Update docs for api/ui docker #118

- Python
Published by tanaysoni over 6 years ago

https://github.com/deepset-ai/farm - 0.2.2

Major Changes

Parallelization of Data Preprocessing :rocket:

Data preprocessing via the Processor is now fast while maintaining a low memory footprint. Before, the parallelization via multiprocessing was causing serious memory issues on larger data sets (e.g. for Language Model fine-tuning). Now, we are running a small chunk through the whole processor (-> Samples -> Featurization -> Dataset ...). The multiprocessing is handled by the DataSilo now which simplifies implementation.

With this new approach we can still easily inspect & debug all important transformations for a chunk, but only keep the resulting dataset in memory once a process has finished with a chunk.

Multilabel classification

We support now also multilabel classification. Prepare your data by simply setting multilabel=true in the TextClassificationProcessor and use the new MultiLabelTextClassificationHead for your model. => See an example here

Concept of Tasks

To further simplify multi-task learning we added the concept of "tasks". With this you can now use one TextClassificationProcessor to preprocess data for multiple tasks (e.g. using two columns in your CSV for classification). Example: 1. Add the tasks to the Processor: ``` processor = TextClassificationProcessor(...)

news_categories = ["Sports", "Tech", "Politics", "Business", "Society"]
publisher = ["cnn", "nytimes","wsj"]

processor.add_task(name="category", label_list=news_categories, metric="acc", label_column_name="category_label")
processor.add_task(name="publisher", label_list=publisher, metric="acc", label_column_name="publisher_label")

```

Link the data to right PredictionHead by supplying the task name at initialization: category_head = MultiLabelTextClassificationHead(layer_dims=[768,5)], task_name="action_type") publisher_head = MultiLabelTextClassificationHead(layer_dims=[768, 3], task_name="parts")

Update to transformers 2.0

We are happy to see how huggingface's repository is growing and how they made another major step with the new 2.0 release. Since their collection of language models is awesome, we will continue building upon their language models and tokenizers. However, we will keep following a different philosophy for all other components (dataprocessing, training, inference, deployment ...) to improve usability, allow multitask learning and simplify usage in the industry.

Modelling:

['enhancement] Add Multilabel Classification (#89)
- ['enhancement] Add PredictionHead for Regression task (#50)
[enhancement] Introduce concept of "tasks" to support of multitask training using multiple heads of the same type (e.g. for multiple text classification tasks) (#75)
[enhancement] Update dependency to transformers 2.0 (#106)
[bug] TypeError: classificationreport() got an unexpected keyword argument 'targetnames' #93
[bug] Fix issue with class weights (#82)

Data Handling:

[enhancement] Chunkwise multiprocessing to reduce memory footprint in preprocessing large datasets (#88)
[bug] Threading Error upon building Data Silo #90
[bug] Multiprocessing causes data preprocessing to crash #110 (https://github.com/deepset-ai/FARM/issues/102)
[bug] Multiprocessing Error with PyTorch Version 1.2.0 #97
[bug] Windows fixes (#109)

Inference:

[enhancement] excessive uncalled-for warnings when using the inferencer #104
[enhancement] Get probability distribution over all classes in Inference mode (#102)
[enhancement] Add InferenceProcessor (#72)
[bug] Fix classifcation report bug with binary doc classification

Other:

[enhancement] Add more tests (#108)
[enhancement] do logging within run_experiment() (#37)
[enhancement] Improved logging (#82, #87 #105)
[bug] fix custom vocab for bert-base-cased (#108)

Thanks to all contributors: @tripl3a, @busyxin, @AhmedIdr, @jinnerbichler, @Timoeller, @tanaysoni, @brandenchan , @tholor

👩‍🌾 Happy FARMing!

- Python
Published by tanaysoni over 6 years ago

https://github.com/deepset-ai/farm - Add multiprocessing to Datahandler, unify Processors

Besides fixing various smaller bugs, we focussed in this release on two major changes:

1. Speeding things up :rocket: :
- By adding multiprocessing to the data preprocessing, we reduced the execution time for many tasks from hours to minutes. Since the functionality is mostly hidden in the parent class, the user doesn't have to implement anything on his own. However, this required changing the interface of the processor slightly. _dict_to_samples and _sample_to_features must now be classmethods and all objects accessed by them must be class attributes. - Multi-GPU support is now also available for the "building blocks mode"

2. Making the processor more user friendly :blush: : - Instead of having one individual processor per dataset, we have implemented a more generic TextClassificationProcessor that you can instantiate easily for various predefined tasks (GNAD, GermEval ...) or your own dataset in CSV/TSV format processor = TextClassificationProcessor(tokenizer=tokenizer, max_seq_len=128, data_dir="../data/germeval18", columns=["text", "label", "unused"], label_list=["OTHER", "OFFENSE"], metrics=["f1_macro"] ) Thanks for contributing @brandenchan @tanaysoni @tholor @Timoeller @tripl3a @Seb0 @waldemarhahn !

Modeling: - [bug] Accuracy metric in LM finetuning always zero #30 - [enhancement] Multi-GPU only enabled in experiment mode #57 - [bug] Wrong number of total steps for linear warmup schedule #46

Data Handling: - [enhancement] Unify redundant Processor; add new NERProcessor and TextClassificationProcessor - [enhancement] Add parallel dataprocessing #45 - [bug] dev_size param in run-by-config is being ignored #49 - [bug] output_dir parameter in run by config is being ignored #39 - [bug] Error when running by config with a list of batch sizes #38

Documentation: - [bug] LM finetuning example missing data #47 - [bug] Colab Notebook referenced in readme does not work #27

Other: - [enhancement] Proposition: improve dependency management with pipenv #35

- Python
Published by tanaysoni almost 7 years ago

https://github.com/deepset-ai/farm - Initial Release

First release of FARM package

Contributor list: @brandenchan @tanaysoni @tholor @Timoeller @tripl3a

- Python
Published by Timoeller almost 7 years ago

Recent Releases of https://github.com/deepset-ai/farm

https://github.com/deepset-ai/farm - v0.8.0 DPR Improvements

DPR Improvements

torch 1.8.1 and transformers 4.6.1

Multi-task Learning

List of Evaluation Metrics

Misc

https://github.com/deepset-ai/farm - DPR patch release

https://github.com/deepset-ai/farm - v0.7.0 QA Model Confidence

QA Confidence Scores

Misc

https://github.com/deepset-ai/farm - Patch Release - Bugfix QA Inference

https://github.com/deepset-ai/farm - Patch Release - Bugfix QA Processing

Patch release

Additional feature for QA

Additional test

Misc

https://github.com/deepset-ai/farm -

Simplification of Preprocessing

Question Answering Preprocessing

Breaking changes

Update to transformers version 4.1.1 and torch version 1.7.0

DPR enhancements

Misc

https://github.com/deepset-ai/farm - v0.5.0

Add Dense Passage Retriever (DPR) incl. Training & Inference (#513, #601, #606)

Refactor conversion from / to Transformers #576

Transformers -> FARM

FARM -> Transformers

Upgrade to Transformers 3.3.1 #579

Details

Question Answering

Other

https://github.com/deepset-ai/farm - v0.4.9

Minor patch: Relax PyTorch version requirements

https://github.com/deepset-ai/farm - 0.4.8

Minor release

Experimental Support for fast Rust Tokenizers (#482)

Upgrade to transformers 3.1.0 (#464)

Details

Question Answering

Other

https://github.com/deepset-ai/farm - 0.4.7

Main changes

Support for MiniLM Model (#464)

Benchmarks per component (#491)

Support for PyTorch 1.6 (#502)

Details

Question Answering

Other

https://github.com/deepset-ai/farm - 0.4.6

Main changes

High level attributes for your query

...

Attributes for individual predictions (= answers)

...

Details

Question Answering

Other

https://github.com/deepset-ai/farm - 0.4.5

Important Bug Fix QA

Speed optimization training from scratch

Other changes:

https://github.com/deepset-ai/farm - 0.4.4

ELECTRA Model

Natural Questions Style QA

New speed benchmarking

A few more changes ...

https://github.com/deepset-ai/farm - 0.4.3

:1234: Changed Multiprocessing in Inferencer

:fast_forward: Streaming Inferencer

:olderwoman: :olderman: "Classic" baseline models for benchmarking + S3E Pooling

A few more changes ...

https://github.com/deepset-ai/farm - 0.4.2

:fast_forward: Scalable preprocessing: StreamingDataSilo

:rocket: Better Inference: Speed, scalability, standardization

ONNX support:

Embedding extraction:

:leftrightarrow: New tasks: TextPairClassification & Passage ranking

A few more changes ...