foundation-model-benchmarking-tool - Llama3.1-8b for p5en

What's Changed

• Update pyproject.toml by @dheerajoruganty in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/290
• Create config-llama3-1-8b-p5en-djl.yml by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/291
• Update config-llama3-1-8b-p5en-djl.yml by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/292

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v2.1.3...v2.1.4

- Jupyter Notebook
Published by aarora79 about 1 year ago

foundation-model-benchmarking-tool - SGLang, s3 fix

What's Changed

• Update README.md with new python version by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/284
• Custom predictor: Taking a Ollama Custom Server for TinyLlama 1.1b for example by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/285
• Customize config files doc update by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/286
• SGLang integration by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/287

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v2.1.2...v2.1.3

- Jupyter Notebook
Published by aarora79 over 1 year ago

foundation-model-benchmarking-tool - Updated deepseek prompt, bug fixes

What's Changed

• pricing fallback bug fix by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/280
• bug fix in the template.yml file by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/282

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v2.1.1...v2.1.2

- Jupyter Notebook
Published by aarora79 over 1 year ago

foundation-model-benchmarking-tool - Deepseek, uv, Qwen2.5, ConvFinQA, chores

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v2.1.0...v2.1.1

- Jupyter Notebook
Published by aarora79 over 1 year ago

foundation-model-benchmarking-tool - Deepseek, uv, Qwen2.5, chores

What's Changed

• qwen generic config file + prompt template by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/266
• Qwen + llama3.3 tests for accuracy & readme + doc updates by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/269
• Adding sonnet3.5 & llama3.3 as judges by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/272
• Ollama end to end support by @dheerajoruganty in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/271
• Ollama fix + deepseek prompt & config file by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/273
• prompt template naming fix for deepseek by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/274
• chore: improve no latency metrics data available messaging by @tonyksong in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/275
• VLLM GPU support by @dheerajoruganty in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/277
• Deepseek and uv changes by @aarora79 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/279

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v2.0.26...v2.1.0

- Jupyter Notebook
Published by aarora79 over 1 year ago

foundation-model-benchmarking-tool - Bug fixes, new config file options

What's Changed

• Fix for the hf token content by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/265
• parameterizing tp depree, instance type and batch size by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/264. See https://aws-samples.github.io/foundation-model-benchmarking-tool/cli.html.

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v2.0.25...v2.0.26

- Jupyter Notebook
Published by aarora79 over 1 year ago

foundation-model-benchmarking-tool - Graviton 4, SageMaker fixes

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v2.0.23...v2.0.24

- Jupyter Notebook
Published by aarora79 over 1 year ago

foundation-model-benchmarking-tool - SageMaker BYOE metrics fix

What's Changed

• Summarization Prompt Template by @dheerajoruganty in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/255
• HF dataset loading bug fixes + new config file by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/256
• llama3.2 1b g6e.2xl config file for smaller prompts by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/257

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v2.0.22...v2.0.23

- Jupyter Notebook
Published by aarora79 over 1 year ago

foundation-model-benchmarking-tool - Amazon Nova models, multi-modal

What's Changed

• NOVA models config file + bedrock converseAPI predictor by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/253
• Update template.yml by @antara678 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/254

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v2.0.21...v2.0.22

- Jupyter Notebook
Published by aarora79 over 1 year ago

foundation-model-benchmarking-tool - EC2 pricing through API, misc. config file changes

What's Changed

• copy s3 content update + llama3-8b triton g5.48xl config by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/244
• HF token bug fix + docs update for copy s3 content command by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/245
• EC2 instance utilization metrics by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/248
• Fallback pricing for EC2 by @dheerajoruganty in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/249
• Update EC2 Metrics by @dheerajoruganty in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/250
• Update Fallback Pricing YAML Path by @dheerajoruganty in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/252
• chore(ec2): update byoe yml and docs by @tonyksong in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/251

New Contributors

• @tonyksong made their first contribution in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/251

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v2.0.20...v2.0.21

- Jupyter Notebook
Published by aarora79 over 1 year ago

foundation-model-benchmarking-tool - embeddings models on SageMaker

What's Changed

• Update config-ec2-llama3-1-70b-inf2-48xl-deploy-ec2-djl.yml by @aarora79 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/229
• Update config-ec2-llama3-1-70b-inf2-48xl-deploy-ec2-djl.yml by @aarora79 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/230
• fix for triton ep names by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/231
• Add Initial support for bge-base-en-v1-5 embedding model and Llama 3.2 11b-Vision-Instruct on FMBench by @dheerajoruganty in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/227

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v2.0.16...v2.0.17

- Jupyter Notebook
Published by aarora79 over 1 year ago

foundation-model-benchmarking-tool - torch version 2.4

What's Changed

• Update pyproject.toml by @antara678 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/226

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v2.0.15...v2.0.16

- Jupyter Notebook
Published by aarora79 over 1 year ago

foundation-model-benchmarking-tool - Ollama support

What's Changed

• Add BYO Ollama Support by @dheerajoruganty in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/223
• Change Llama 3 8b and 70b Model IDs by @dheerajoruganty in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/225

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v2.0.14...v2.0.15

- Jupyter Notebook
Published by aarora79 over 1 year ago

foundation-model-benchmarking-tool - FMBench orchestrator

What's Changed

• Configuration files for llama3.1 70b on large prompt payloads + longbench dataset by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/216
• adding support for llama3 summarization prompt by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/217
• changing file name for llama3 summarization prompt by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/218
• Config files for llama3.1 8b instruct on g6e instances by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/219
• All config files for llama3.1 8b on g6e instances using DJL by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/220
• make config file naming convention consistent for llama3.1 8b/70b on g6e by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/221
• Config files for all llama3.2 models - tested by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/222

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v2.0.13...v2.0.14

- Jupyter Notebook
Published by aarora79 over 1 year ago

foundation-model-benchmarking-tool - pricing.yml updates

What's Changed

• Update pricing.yml by @aarora79 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/210
• Rename config-llama3-8b-g6e.4xl-tp-2-mc-max-djl-ec2.yml to config-lla… by @aarora79 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/212
• add mixtral config file for AWQ version - g6e.48xl by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/214
• pricing update + retry logic added to bedrock predictor by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/215

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v2.0.11...v2.0.13

- Jupyter Notebook
Published by aarora79 over 1 year ago

foundation-model-benchmarking-tool - Llama3 with Triton+DJL on Neuron

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v2.0.10...v2.0.11

- Jupyter Notebook
Published by aarora79 over 1 year ago

foundation-model-benchmarking-tool - Llama3 on g6e

What's Changed

• Add support and pricing for g6e instances by @dheerajoruganty in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/207
• Config file for llama3 8b on inf2 using triton with DJL by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/205
• Add config files for g6e instances by @dheerajoruganty in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/208
• Add concurrency=3 for g6e instance configs by @dheerajoruganty in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/209

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v2.0.9...v2.0.10

- Jupyter Notebook
Published by aarora79 over 1 year ago

foundation-model-benchmarking-tool - Triton-DJL support, Tokenizer from HF

What's Changed

• Contains a configuration file trn and doc fix for triton on AWS chips by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/201
• Integration triton inference server with djl by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/204

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v2.0.8...v2.0.9

- Jupyter Notebook
Published by aarora79 over 1 year ago

foundation-model-benchmarking-tool - v2.0.8

What's Changed

• bug fix + updated triton vllm config file by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/200
• Add Support for Metrics While Benchmarking on Neuron and EC2 by @dheerajoruganty in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/193
• Fix to correctly parse response in message API format for SageMaker inference by @fespigares in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/165
• Tagging by @antara678 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/188
• adding platform identification for deployment by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/187
• update llama2 7b quick file by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/198
• Bug fix for evals by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/196
• Update config-ec2-llama3-8b.yml by @antara678 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/199
• Triton integration by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/194

New Contributors

• @fespigares made their first contribution in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/165

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v2.0.7...v2.0.8

- Jupyter Notebook
Published by aarora79 over 1 year ago

foundation-model-benchmarking-tool - Triton inference server

What's Changed

• Add Support for Metrics While Benchmarking on Neuron and EC2 by @dheerajoruganty in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/193
• Fix to correctly parse response in message API format for SageMaker inference by @fespigares in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/165
• Tagging by @antara678 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/188
• adding platform identification for deployment by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/187
• update llama2 7b quick file by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/198
• Bug fix for evals by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/196
• Update config-ec2-llama3-8b.yml by @antara678 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/199
• Triton integration by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/194

New Contributors

• @fespigares made their first contribution in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/165

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v2.0.6...v2.0.7

- Jupyter Notebook
Published by aarora79 over 1 year ago

foundation-model-benchmarking-tool - Multiple model copies on a single EC2 instance

What's Changed

• Fix EC2 container not running due to bad flag by @dheerajoruganty in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/185
• remove unecessary --runtime=nvidia from the docker command by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/186
• Multi model w djl by @aarora79 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/191

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v2.0.5...v2.0.6

- Jupyter Notebook
Published by aarora79 over 1 year ago

foundation-model-benchmarking-tool - Intel CPU support with vLLM

What's Changed

• add ec2 instructions to enable docker runs without sudo by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/181
• clean up analytics, create documentation for it by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/182
• prompt template key fix by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/184
• Support for Intel Instances and Configuration Updates by @dheerajoruganty in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/183

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v2.0.4...v2.0.5

- Jupyter Notebook
Published by aarora79 almost 2 years ago

foundation-model-benchmarking-tool - AMD CPU support

What's Changed

• Deployment Support for x86 AMD CPUs on EC2 by @dheerajoruganty in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/178
• amd support by @aarora79 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/179

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v2.0.3...v2.0.4

- Jupyter Notebook
Published by aarora79 almost 2 years ago

foundation-model-benchmarking-tool - EFA support for /tmp dir, Mistral on AWS Chips

What's Changed

• config file update for llama3-8b by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/175
• Added Inital Support for AMD x86 CPUs by @dheerajoruganty in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/177
• Adding support for custom tmp directory + preliminary config files for mistral on inf2/trn1 by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/176

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v2.0.2...v2.0.3

- Jupyter Notebook
Published by aarora79 almost 2 years ago

foundation-model-benchmarking-tool - Llama3.1-8b on SageMaker, DJL serving fixes

What's Changed

• Update CITATION.cff by @aarora79 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/172
• Adding youtube logo on embedded image link by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/173
• Update README.md by @aarora79 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/174

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v2.0.1...v2.0.2

- Jupyter Notebook
Published by aarora79 almost 2 years ago

foundation-model-benchmarking-tool - Model evaluations

What's Changed

• Aws inf by @antara678 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/166

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v1.0.52...v2.0.0

- Jupyter Notebook
Published by aarora79 almost 2 years ago

foundation-model-benchmarking-tool - Llama3.1 on AWS Chips

What's Changed

• Support for inf2 instance by @antara678 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/158
• Support for llama3.1 on Inf2 + Other neuron deployment updates by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/159
• updating the manifext.txt file by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/160
• updated neuron deploy code by @antara678 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/161
• Update neuron_deploy.py with logger statements by @antara678 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/162
• support for llama3.1 on neuron + usemessagesapi metadata addition by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/164
• Add files via upload by @antara678 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/163

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v1.0.51...v1.0.52

- Jupyter Notebook
Published by aarora79 almost 2 years ago

foundation-model-benchmarking-tool - FMBench website, Llama3.1

What's Changed

• Update config-ec2-llama3-8b-inf2-48xl.yml by @antara678 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/148
• config file for llama 2 70B by @antara678 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/134
• fmbench website by @antara678 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/151
• Gh website-updated logo path by @antara678 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/152
• adding support for llama3.1 on Amazon Bedrock by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/156

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v1.0.50...v1.0.51

- Jupyter Notebook
Published by aarora79 almost 2 years ago

foundation-model-benchmarking-tool - FMBench support for RAG + Model evaluations

This version contains [Work in Progress] code for evaluating both:

1. Majority Vote: Using a Panel of LLM Evaluators to check for RAG eval on whether a given candidate model output is correct or incorrect.
2. Average Pooling: Using a Panel of LLM Evaluators to evaluate candidate model responses using 'user-defined' subjective evaluation criteria.

- Jupyter Notebook
Published by madhurprash almost 2 years ago

foundation-model-benchmarking-tool - Llama3-8b on EC2 inf2.48xlarge

What's Changed

• Update README.md by @antara678 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/143
• Ec2 inf2 by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/145
• Update manifest.txt by @antara678 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/146
• Add Material for Mkdocs Config by @dheerajoruganty in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/147

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v1.0.49...v1.0.50

- Jupyter Notebook
Published by aarora79 almost 2 years ago

foundation-model-benchmarking-tool - Per token latency metrics - TTFT, TPOT

What's Changed

• rebaseline by @aarora79 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/140
• Add ttft tpot ttlt metrics by @aarora79 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/141
• updated manifest by @aarora79 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/142

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v1.0.48...v1.0.49

- Jupyter Notebook
Published by aarora79 almost 2 years ago

foundation-model-benchmarking-tool - Phi-3, bug fixes

What's Changed

• update create combination with test count limit by @haozhx23 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/124
• min/max iterations per combination handling by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/133
• handling endpoint metrics by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/136
• add write multiple to s3 by @haozhx23 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/123
• tokenizer file handling fix by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/137
• Lllama3 70B on ec2 by @antara678 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/139
• adding phi config file by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/138

New Contributors

• @haozhx23 made their first contribution in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/124

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v1.0.47...v1.0.48

- Jupyter Notebook
Published by aarora79 almost 2 years ago

foundation-model-benchmarking-tool - FMBench in a Docker container

What's Changed

• Containerize FMBench by @dheerajoruganty in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/127
• handling deployment return values from various scripts by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/128
• cloudformation template generalized for gov cloud by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/129
• Update ekscluster-creationsteps.md by @antara678 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/132

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v1.0.46...v1.0.47

- Jupyter Notebook
Published by aarora79 almost 2 years ago

foundation-model-benchmarking-tool - Native EC2, EKS support. FMbench now in GovCloud.

What's Changed

• adding EC2 djl rest prediction support for llama3 on fmbench by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/113
• Python Script to Plot Sagemaker Endpoint Utilization Metrics , Heatmap to visualize Cost for Various RPMs by @dheerajoruganty in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/108
• Do eks by @bainskb in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/110
• gemma support added to FMBench by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/116
• added the changes to get get do-eks to work up untill the metrics not… by @bainskb in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/121
• Ec2 deployment by @antara678 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/125
• EKS integration with FMBench by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/126

New Contributors

• @bainskb made their first contribution in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/110

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v1.0.45...v1.0.46

- Jupyter Notebook
Published by aarora79 almost 2 years ago

foundation-model-benchmarking-tool - More analytics

What's Changed

• Update config-bedrock-llama3.yml by @antara678 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/109

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v1.0.44...v1.0.45

- Jupyter Notebook
Published by aarora79 almost 2 years ago

foundation-model-benchmarking-tool - Llama3-8b on g6, SageMaker Endpoint metrics

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v1.0.43...v1.0.44

- Jupyter Notebook
Published by aarora79 almost 2 years ago

foundation-model-benchmarking-tool - v1.0.42: llama3-8b all config, misc. fixes

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v1.0.41...v1.0.42

- Jupyter Notebook
Published by aarora79 about 2 years ago

foundation-model-benchmarking-tool - Easy to run in the Amazon EC2

What's Changed

• readme update by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/99

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v1.0.39...v1.0.40

- Jupyter Notebook
Published by aarora79 about 2 years ago

foundation-model-benchmarking-tool - FMBench internal website

What's Changed

• Add renderfmbenchwebsite.py and Update README by @dheerajoruganty in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/97

New Contributors

• @dheerajoruganty made their first contribution in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/97

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v1.0.38...v1.0.39

- Jupyter Notebook
Published by aarora79 about 2 years ago

foundation-model-benchmarking-tool - FMBench on EC2 minus S3 dependency

What's Changed

• Issue 85 - Remove S3 depencies by @RajeshRamchander in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/89

New Contributors

• @RajeshRamchander made their first contribution in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/89

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v1.0.37...v1.0.38

- Jupyter Notebook
Published by aarora79 about 2 years ago

foundation-model-benchmarking-tool - Llama3-8b instance specific config files

What's Changed

• html report generation with quarto, llama3-8b on trn1 by @aarora79 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/90
• config file reorg by @aarora79 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/91
• fix broken symlinks by @aarora79 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/92
• Grammar nits by @jimburtoft in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/93
• Config files for Llama3 for different instance types by @aarora79 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/96

New Contributors

• @jimburtoft made their first contribution in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/93

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v1.0.36...v1.0.37

- Jupyter Notebook
Published by aarora79 about 2 years ago

foundation-model-benchmarking-tool - Concurrency bug fix, instance count > 1 support

What's Changed

• Fix bug at high concurrency level, support for instance count > 1 by @aarora79 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/87
• bump up version for concurrency fix and instance count > 1 support by @aarora79 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/88

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v1.0.35...v1.0.36

- Jupyter Notebook
Published by aarora79 about 2 years ago

foundation-model-benchmarking-tool - Open-Orca dataset and more support for BYO dataset

What's Changed

• OpenOrca and byo dataset updates by @aarora79 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/86

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v1.0.34...v1.0.35

- Jupyter Notebook
Published by aarora79 about 2 years ago

foundation-model-benchmarking-tool - Bug fixes

- Jupyter Notebook
Published by aarora79 about 2 years ago

foundation-model-benchmarking-tool - Add missing package

- Jupyter Notebook
Published by aarora79 about 2 years ago

foundation-model-benchmarking-tool - Report enhancements, Llama3 on Inf2

1. Report enhancements: New business summary chart, config file embedded in the report, version numbering and others.

2. Additional config files: Meta Llama3 on Inf2, Mistral instruct with lmi-dist on p4d and p5 instances.

- Jupyter Notebook
Published by aarora79 about 2 years ago

foundation-model-benchmarking-tool - Llama3 benchmarking now available

What's Changed

• Support for Llama3 added to FMBench by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/60

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v1.0.30...v1.0.31

- Jupyter Notebook
Published by aarora79 about 2 years ago

foundation-model-benchmarking-tool - Bedrock support

What's Changed

• support for US-west2 by @antara678 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/36
• s3 bucket timeout template.yml by @antara678 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/37
• Updating djl serving script, toml files by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/38
• updated docker container support for region by @antara678 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/40
• adding region to stack-role cft by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/41
• changes to djl script, adding region to buckets and the p5 config file by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/43
• price per token added by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/44
• To add support for bring your own sagemaker pre existing endpoint by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/48
• testing p5, byo ep and price per token bug fix by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/49
• Updated llama chat model inf parameters and 70b djl config file by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/52
• Bedrock functionality, code cleanup, new config files by @aarora79 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/57
• config file updates by @aarora79 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/59

New Contributors

• @antara678 made their first contribution in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/36

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/compare/v1.0.24...v1.0.30

- Jupyter Notebook
Published by aarora79 about 2 years ago

foundation-model-benchmarking-tool - v1.0.24

New in this release

1. Support for HuggingFace datasets as well as bring your own datasets, more here.

2. Support for external endpoints. No longer limited to Amazon SageMaker endpoints, more here.

3. Added config files for Mistral-7B-Instruct, Mistral-7B-Instruct-v0.2-AWQ, huggingface-tc-distilbert-base-uncased (from SageMaker JumpStart), meta-textgenerationneuron-llama-2-70b-f (on AWS Inferentia2).

- Jupyter Notebook
Published by aarora79 about 2 years ago

foundation-model-benchmarking-tool - First release: v1.0.20

What's Changed

• S3 integration with FMBT by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/8
• S3 updates cleaned up by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/9
• Fmbt python version by @aarora79 in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/13
• Cost estimation functionality addition by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/15
• Adding support for default tokenizer if tokenizer is not already provided by @madhurprash in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/18

New Contributors

• @madhurprash made their first contribution in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/8
• @aarora79 made their first contribution in https://github.com/aws-samples/foundation-model-benchmarking-tool/pull/13

Full Changelog: https://github.com/aws-samples/foundation-model-benchmarking-tool/commits/v1.0.20

- Jupyter Notebook
Published by aarora79 about 2 years ago