rtg - v0.7.1: updated REST API; fix bfloat16 issue; dockers w/ both arm64, amd64

• Bfloat16 is not default and used only when it is available
• /translate REST API is updated
  • accepts params lp_alpha, beam_size, num_hyp
  • produces n-best hyps, scores, time taken per API call
  • Web APP: relative paths to static files are correctly updated when base prefix is used
  • Docker files are updated. multi-arch build with linux/arm64 linux/amd64 have been released to docker hub

- Jupyter Notebook
Published by thammegowda over 3 years ago

rtg -

• Improvements:
  • Autocast / mixed precision: bfloat16 instead of float16. Now we can train larger models on larger batches using 16bit float ops without loss becoming infinity!
    
  • WARNING: we need pytorch 1.10 or newer. Please upgrade!
  • validation BLEU scores are computed without teacher forcing i.e., similar to inference. BLEU is more realistic estimate of test time bleu
  • WARNING: validations can be slower. Dont use too big validation set
  • schedule:
  • inverse_sqrt support scaler multiplier term, similar to noam
  • inverse_root schedule added, generalization of inverse_sqrt
• fixes
  • rtg.prep CLI arguments works now
  • optimizer state loading now works while resuming training
  • parent model will be recreated if missing even after _PREPARED flag exists

- Jupyter Notebook
Published by thammegowda almost 4 years ago

rtg - v0.6.1

• rtg.fork accepts multiple to_dir; thus supports cloning multiple times at once
• Bug fix: early stopping on distributed parallel training
• rtg.tool.augment to support data augmentations
• Add attention visualization in rtg.serve; powered by plotly
• rtg.pipeline and rtg.fork: uses relative symlinks instead of absolute paths
• rtg.decode shows decoding speed (segs, srctoks, hyptoks)
• batch_size is auto adjusted based on number of workers and gradient_accum (huh! finally)
• batch_size normalizer in distributed training setting (fix! faster convergence now)
• support for byte encoding added

- Jupyter Notebook
Published by thammegowda about 4 years ago

rtg - v0.6.0

• Redesign of registry; using decorators to register all modules

• optim block is split into optimizer schedule and criterion; as a result,

  • this version is not backward compatible with prior versions Refer to migration guide
  • Migration instruction: https://isi-nlp.github.io/rtg/v0.6.0/#migrate-to-0_6
  • NoamOpt replaced with ScheduledOptimizer which takes scheduler and optimizer objects which are independently configurable from conf.yml

• Transformer sequence classification model: tfmcls, supports initialization from pretrained NMT (picks encoder layers, source embeddings, and source vocabs from NMT experiment)

- Jupyter Notebook
Published by thammegowda over 4 years ago

rtg - v0.5.2 - minor bug fixes

• Fix rtg.decode bug fix (partial migration to new API)
• test case added for decode api so we can catch such errors in future

- Jupyter Notebook
Published by thammegowda over 4 years ago

rtg - 0.5.1 - tfmcls, source preproc and target post proc,

• Add rtg-params command that shows trainable parameters in model (layer wise as well as total)
• rtg.serve supports flexible transformations on source (pre processing) and target (post processing)
• Travis build configured to auto run tests
• sequence classification is now supported via tfmcls model
  

- Jupyter Notebook
Published by thammegowda over 4 years ago

rtg - 0.5.0 - DDP , NLcodec + NLDb, scaling to large datasets

• DDP: multinode training see scripts/slurm-multinode-launch.sh
• FP16 and mixed-precision (upgrade from APEX to torch's built in AMP)
• NLCodec & NLDb integration for scaling to large datasets using pyspark backend
• Web UI rtg-serve
• Cache ensemble state for rtg-decode
• Docker images for 500-eng model
• Parent-child transfer: Shrink parent model vocab and embeddings to child datasets
• Fix packaging of flask app: now templates and static files are also included in PyPI package

- Jupyter Notebook
Published by thammegowda almost 5 years ago

rtg - 0.4.2

• Fix issue with decboscut complaining that tensors are not on contigous storage
• REST API
• Docker for deployment

- Jupyter Notebook
Published by thammegowda over 5 years ago

rtg - Fix description type and platform type issues in setup.py

- Jupyter Notebook
Published by thammegowda over 5 years ago

rtg - First release to PyPi

- Jupyter Notebook
Published by thammegowda over 5 years ago