promptfoo - 0.118.3

What's Changed

New Features

• Add AWS Bedrock support for OpenAI GPT OSS models by @jonathancaevans in https://github.com/promptfoo/promptfoo/pull/5444
• Add Amazon Bedrock API key authentication support by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5468
• Ability to filter evals view by severity by @typpo in https://github.com/promptfoo/promptfoo/pull/5443
• Check cloud permissions for target before running red team by @sklein12 in https://github.com/promptfoo/promptfoo/pull/5400
• Make vars and context available for request transform by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/5461

Bug Fixes

• Add Vertex AI responseSchema file loading support by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5414
• Close menus when mouse leaves by @typpo in https://github.com/promptfoo/promptfoo/pull/5456
• Default sharing to false by @sklein12 in https://github.com/promptfoo/promptfoo/pull/5473
• Handle empty function arguments in OpenAI Responses API tool callbacks by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5454
• Improve Windows Python detection and add sys.executable support by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5467
• Prioritize tool calls over content in openrouter provider by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5417
• Support commandLineOptions.envPath in config files by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5415
• Support setting HELICONEAPIKEY for Cloud Gateway by @javdl in https://github.com/promptfoo/promptfoo/pull/5465
• Token tracking by @sklein12 in https://github.com/promptfoo/promptfoo/pull/5239

Chores

• Add "results" menu, link to red team reports view by @typpo in https://github.com/promptfoo/promptfoo/pull/5459
• Bump version 0.118.3 by @sklein12 in https://github.com/promptfoo/promptfoo/pull/5474
• Include provider response metadata on test case transform by @will-holley in https://github.com/promptfoo/promptfoo/pull/5316
• Refactor Crescendo maxTurns property by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/4528
• Remove accidental server directory by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5471
• Replace direct process.env calls with environment helpers by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5472
• Reorganize misplaced test files from src/ to test/ directory by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5470

Docs

• Fix enterprise email by @typpo in https://github.com/promptfoo/promptfoo/pull/5463

Dependencies

• Bump openai from 5.18.1 to 5.19.1 by @dependabot[bot] in https://github.com/promptfoo/promptfoo/pull/5466

Tests

• Add Tusk test runner workflow for src Jest unit tests by @use-tusk[bot] in https://github.com/promptfoo/promptfoo/pull/5469

New Contributors

• @javdl made their first contribution in https://github.com/promptfoo/promptfoo/pull/5465
• @jonathancaevans made their first contribution in https://github.com/promptfoo/promptfoo/pull/5444

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.118.2...0.118.3

- TypeScript
Published by sklein12 6 months ago

promptfoo - 0.118.2

What's Changed

New Features

• feat(providers): Add support for Meta Llama API provider by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5432
• feat(providers): Support TLS certs in http provider by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5452
• feat(providers): add support for xAI Grok Code Fast models by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5425

Bug Fixes

• fix(sharing): Share when it's enabled via the Config or the CLI command by @sklein12 in https://github.com/promptfoo/promptfoo/pull/5404
• fix: Update util.ts to reflect correct Anthropic Haiku 3.5 pricing by @capttrousers in https://github.com/promptfoo/promptfoo/pull/5436
• fix(grader): reduce grader false positives by @typpo in https://github.com/promptfoo/promptfoo/pull/5431

Chores

• chore: drop Node.js 18 support by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5428
• chore(http): improve PFX debug logging + tests by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5445
• chore(webui): Show footer on custom metrics dialog by @will-holley in https://github.com/promptfoo/promptfoo/pull/5424
• chore: silence dotenv commercial logging messages by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5453
• chore: remove example by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5420
• test: CoverBot: Added tests for analytics tracking and red team reporting components (src/app) by @use-tusk[bot] in https://github.com/promptfoo/promptfoo/pull/5441
• test: optimize Python Unicode test suite for CI reliability by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5449

Docs

• docs(site): add more guardrails assertion doc by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/5434
• docs(site): add multi-lingual RAG evaluation guidance by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5447
• docs(site): optimize OG image generation performance by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5451
• docs(site): update blog post by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5422

Dependencies

• chore: bump the github-actions group with 3 updates by @dependabot[bot] in https://github.com/promptfoo/promptfoo/pull/5440
• chore: update dependencies (non-breaking) by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5448
• chore: update dependencies to latest minor/patch versions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5433

New Contributors

• @capttrousers made their first contribution in https://github.com/promptfoo/promptfoo/pull/5436

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.118.1...0.118.2

- TypeScript
Published by faizanminhas 6 months ago

promptfoo - 0.118.1

What's Changed

New Features

• feat(redteam): Add AI auto-fill for HTTP target configuration in redteam target setup ui by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5391
• feat(redteam): Handle uploaded signatureAuth in target setup ui by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5405

Bug Fixes

• fix(providers): Handle Qwen tool call responses in openrouter provider by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5416

Chores

• chore(site): integrate pylon chat into site by @typpo in https://github.com/promptfoo/promptfoo/pull/5407

Docs

• docs(site): avoid logging full image/base64; use boolean presence only by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5408

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.118.0...0.118.1

- TypeScript
Published by faizanminhas 6 months ago

promptfoo - 0.118.0

What's Changed

New Features - feat(providers): add support for database-stored certificates in HTTP provider for promptfoo cloud by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5401

Bug Fixes - fix: stop progress bar to show a clearer share error message by @sklein12 in https://github.com/promptfoo/promptfoo/pull/5399

Chores - chore(internals)!: send provider-transformed output directly to test context transforms by @will-holley in https://github.com/promptfoo/promptfoo/pull/5376 Breaking: contextTransform now receives the provider transform directly. - chore(providers): sanitize sensitive credentials in HTTP provider debug logs by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5387 - chore: warn when tests and red-team configuration are both present during generation by @typpo in https://github.com/promptfoo/promptfoo/pull/5398 - chore(release): bump version to 0.118.0 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5402

Docs - docs(site): update model-graded metrics by @ladyofcode in https://github.com/promptfoo/promptfoo/pull/5285 - docs(site): remove references to "parallel" introduced by #5376 by @will-holley in https://github.com/promptfoo/promptfoo/pull/5403

Tests - test: add tests for CoverBot store management and red-team reporting components (src/app) by @use-tusk[bot] in https://github.com/promptfoo/promptfoo/pull/5372

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.117.11...0.118.0

- TypeScript
Published by mldangelo 6 months ago

promptfoo - 0.117.11

What's Changed

New Features

• feat: MCP Agent example to red team with tool call results by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5379
• feat: medical offlabel use by @typpo in https://github.com/promptfoo/promptfoo/pull/5342
• feat: modelaudit ability to remove recent paths by @typpo in https://github.com/promptfoo/promptfoo/pull/5330
• feat(redteam): add -t/--target option to redteam generate command by @will-holley in https://github.com/promptfoo/promptfoo/pull/5338

Bug Fixes

• fix: Address design nits in redteam setup UI by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5264
• fix: allow custom ApiProvider instances in defaultTest configuration by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5381
• fix: mcp eval example by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5390
• fix: Prioritize tool calls over thinking for openrouter reasoning models by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5395
• fix: use model role for gemini ai studio models by @typpo in https://github.com/promptfoo/promptfoo/pull/5386

Chores

• chore: Adjust padding in plugins page by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5396
• chore: bump version 0.117.11 by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5397
• chore(CI): enable and refactor Docker build for caching by @josephyi in https://github.com/promptfoo/promptfoo/pull/5374
• chore: remove promptfoo/package-lock.json by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5380
• chore: visual formatting for modelaudit flat list by @typpo in https://github.com/promptfoo/promptfoo/pull/5331
• refactor(webui): Clicking "show more" on eval results metric pills renders dialog by @will-holley in https://github.com/promptfoo/promptfoo/pull/5337

Docs

• docs: expose sidebar on pages that aren't in the sidebar by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5377
• docs: model audit ci/cd by @typpo in https://github.com/promptfoo/promptfoo/pull/5335
• docs: remove orphaned star animation gif by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5383
• docs(site): fix context transform examples to use context.vars.prompt by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5393
• docs: update site user count to 150,000+ across site constants and pages by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5394

Dependencies

• chore: bump @aws-sdk/client-bedrock-runtime from 3.873.0 to 3.876.0 by @dependabot[bot] in https://github.com/promptfoo/promptfoo/pull/5392
• chore: bump openai from 5.15.0 to 5.16.0 by @dependabot[bot] in https://github.com/promptfoo/promptfoo/pull/5388

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.117.10...0.117.11

- TypeScript
Published by faizanminhas 6 months ago

promptfoo - 0.117.10

What's Changed

New Features

• feat: improve HuggingFace dataset fetching performance and reliability by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5346
• feat: add Google AI Studio default providers by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5361
• feat: share model audit scans to cloud by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5336
• feat: add google vertex credentials in config by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5179

Bug Fixes

• fix(CI): refactor docker build by @josephyi in https://github.com/promptfoo/promptfoo/pull/5353
• fix: safe raw HTTP templating via Nunjucks raw-wrap + CRLF normalization by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/5358
• fix: improve JSON export error handling for large datasets by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5344
• fix(internals): defaultTest.provider doesn't override by @will-holley in https://github.com/promptfoo/promptfoo/pull/5348
• fix: replace raw-request editor with auto-growing textarea to prevent layout overflow by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/5369

Chores

• chore: better error messages for browser by @typpo in https://github.com/promptfoo/promptfoo/pull/5226
• chore: improve strategy presets by @typpo in https://github.com/promptfoo/promptfoo/pull/5357
• chore: set onboarding defaults to gpt 5 by @typpo in https://github.com/promptfoo/promptfoo/pull/5360
• chore: update dependencies to latest minor versions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5363
• chore: log posthog errors to debug by @typpo in https://github.com/promptfoo/promptfoo/pull/5359
• chore: sync dependencies by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5367
• test: clean up skipped tests and add FunctionCallbackHandler coverage by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5366
• chore: bump version 0.117.10 by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/5373

Docs

• docs: add critical git workflow guidelines to CLAUDE.md by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5362
• docs: add SARIF output format documentation for ModelAudit by @typpo in https://github.com/promptfoo/promptfoo/pull/5364

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.117.9...0.117.10

- TypeScript
Published by MrFlounder 6 months ago

promptfoo - 0.117.8

What's Changed

• feat(redteam): make unblock call optional for multi-turn strategies by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/5292
• fix: add lru-cache dependency by @typpo in https://github.com/promptfoo/promptfoo/pull/5309
• chore: many plugins and strategies selected warning by @typpo in https://github.com/promptfoo/promptfoo/pull/5306
• chore: add max max concurrency to generate by @typpo in https://github.com/promptfoo/promptfoo/pull/5305
• chore: bump version 0.117.8 by @typpo in https://github.com/promptfoo/promptfoo/pull/5311
• ci: add depcheck by @typpo in https://github.com/promptfoo/promptfoo/pull/5310

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.117.7...0.117.8

- TypeScript
Published by typpo 6 months ago

promptfoo - 0.117.7

What's Changed

New Features

• feat: Add unverifiable claims red team plugin by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5190
• feat(site): add hero image for red teaming tools blog post by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5291
• feat(webui): Demarcate redteam results by @will-holley in https://github.com/promptfoo/promptfoo/pull/5255

Bug Fixes

• fix(webui/reports): Don't exclude failure cases from stats by @will-holley in https://github.com/promptfoo/promptfoo/pull/5298
• fix: lower sharing chunk size by @sklein12 in https://github.com/promptfoo/promptfoo/pull/5270
• fix(internals): Gracefully handle object responses during target purpose discovery by @will-holley in https://github.com/promptfoo/promptfoo/pull/5236
• fix(site): fix YAML front matter parsing error in jailbreaking blog post by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5287
• fix(webui): Improved handling of long loglines by @will-holley in https://github.com/promptfoo/promptfoo/pull/5227

Chores

• chore(webui): Rename "Redteam" to "Red Team" in evals datagrid by @will-holley in https://github.com/promptfoo/promptfoo/pull/5288
• chore: bump version 0.117.7 by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5299
• test: CoverBot: Added test coverage for History page component (src/app) by @use-tusk[bot] in https://github.com/promptfoo/promptfoo/pull/5289

Docs

• docs(site): add AI Safety vs AI Security blog post with interactive quiz by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5268
• docs(site): add blog post about prompt injection vs jailbreaking differences by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5282
• docs(site): document transform and contextTransform for model-graded assertions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5258
• docs(site): improve context assertion documentation by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5249
• docs: add open source ai red teaming tools post by @ladyofcode in https://github.com/promptfoo/promptfoo/pull/5259
• docs: add red team github action info by @typpo in https://github.com/promptfoo/promptfoo/pull/5294

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.117.6...0.117.7

- TypeScript
Published by faizanminhas 6 months ago

promptfoo - 0.117.6

What's Changed

New Features

• feat: Add Agent provider types in red team setup by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5244
• feat: add update check for modelaudit package by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5278
• feat: add update notification banner to web UI by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5279
• feat: edit and replay requests in details dialog by @typpo in https://github.com/promptfoo/promptfoo/pull/5242
• feat: Surface run options and probes on red team review page by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5272

Bug Fixes

• fix: composite indices and query optimization by @typpo in https://github.com/promptfoo/promptfoo/pull/5275
• fix: exclude errors from report by @typpo in https://github.com/promptfoo/promptfoo/pull/5271
• fix: Fix json-output example by @sklein12 in https://github.com/promptfoo/promptfoo/pull/5213
• fix: handle json schema for openrouter provider by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5284
• fix: handle thinking tokens for openrouter by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5263
• fix: OpenAI Responses API function callbacks and Azure implementation by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5176
• fix(provider): Remove maxTokens for gpt-5 calls by @AISimplyExplained in https://github.com/promptfoo/promptfoo/pull/5224
• fix(providers): Validate that OpenAI response reasoning outputs have summary items by @will-holley in https://github.com/promptfoo/promptfoo/pull/5235
• fix(site): suppress noisy font loading warnings in OG image plugin by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5254
• fix: throw error instead of failing when trace data is unavailable by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5192
• perf(webui): Reduces eval results load-time when filters are applied via search param by @will-holley in https://github.com/promptfoo/promptfoo/pull/5234

Chores

• chore: add bias to foundation plugins list by @typpo in https://github.com/promptfoo/promptfoo/pull/5280
• chore: Add .serena to .gitignore by @sklein12 in https://github.com/promptfoo/promptfoo/pull/5225
• chore: bump version 0.117.6 by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5273
• chore: fix model id name by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/5232
• chore: improve generated constants handling to prevent accidental commits by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5148
• chore: remove file by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/5229
• chore: show final prompt in table view for attacks that mutate prompts by @typpo in https://github.com/promptfoo/promptfoo/pull/5269
• chore: simplify eval progress bar by @typpo in https://github.com/promptfoo/promptfoo/pull/5238
• chore: update dark mode styles, formatting, etc by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5251
• chore(webui): Don't show loading animations while streaming eval results by @will-holley in https://github.com/promptfoo/promptfoo/pull/5201
• chore(webui/eval results): Sticky header sticks to the top of the viewport by @will-holley in https://github.com/promptfoo/promptfoo/pull/5208
• test: CoverBot: Added tests for red team reporting components (src/app) by @use-tusk[bot] in https://github.com/promptfoo/promptfoo/pull/5228

Docs

• docs: Add AWS Bedrock Guardrails image testing documentation by @Mantej-Singh in https://github.com/promptfoo/promptfoo/pull/5253
• docs: add july release notes by @ladyofcode in https://github.com/promptfoo/promptfoo/pull/5133
• docs: hide events banner by @typpo in https://github.com/promptfoo/promptfoo/pull/5217
• docs: separate malicious code plugin documentation by @typpo in https://github.com/promptfoo/promptfoo/pull/5222
• docs(site): add cross-links between multimodal strategy documentation by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5241
• docs(site): add missing meta descriptions and optimize existing ones for SEO by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5247
• docs(site): enhance OG image generation with full metadata support by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5246
• docs(site): remove unused markdown-page.md by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5245

Dependencies

• chore: bump @anthropic-ai/sdk from 0.58.0 to 0.59.0 by @dependabot[bot] in https://github.com/promptfoo/promptfoo/pull/5218
• chore: bump @anthropic-ai/sdk from 0.59.0 to 0.60.0 by @dependabot[bot] in https://github.com/promptfoo/promptfoo/pull/5257
• chore: bump @aws-sdk/client-bedrock-runtime from 3.862.0 to 3.863.0 by @dependabot[bot] in https://github.com/promptfoo/promptfoo/pull/5211
• chore: bump @aws-sdk/client-bedrock-runtime from 3.863.0 to 3.864.0 by @dependabot[bot] in https://github.com/promptfoo/promptfoo/pull/5221
• chore: bump openai from 5.12.0 to 5.12.1 by @dependabot[bot] in https://github.com/promptfoo/promptfoo/pull/5210
• chore: bump openai from 5.12.1 to 5.12.2 by @dependabot[bot] in https://github.com/promptfoo/promptfoo/pull/5219
• chore: bump pypdf from 5.7.0 to 6.0.0 in /examples/rag-full in the pip group across 1 directory by @dependabot[bot] in https://github.com/promptfoo/promptfoo/pull/5252
• chore: bump the npmandyarn group with 2 updates by @dependabot[bot] in https://github.com/promptfoo/promptfoo/pull/5276

New Contributors

• @Mantej-Singh made their first contribution in https://github.com/promptfoo/promptfoo/pull/5253

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.117.5...0.117.6

- TypeScript
Published by faizanminhas 6 months ago

promptfoo - 0.117.5

What's Changed

New Features

• feat: add GPT-5 support by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5205
• feat: add collapsible header to ResultsView by @typpo in https://github.com/promptfoo/promptfoo/pull/5159
• feat: add contains-html and is-html assertions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5161
• feat: add Google Imagen image generation support by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5104
• feat: add max-score assertion for objective output selection by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5067
• feat: add selected state to provider type picker by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5152
• feat: add unified page wrapper around each red team setup step by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5136
• feat: apply plugin modifiers for crescendo by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5032
• feat: help text to nudge towards better red teams by @sklein12 in https://github.com/promptfoo/promptfoo/pull/5153
• feat: improve red team plugin selection UI with test generation by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5125
• feat: respect prompt config override in all providers by @tomquist in https://github.com/promptfoo/promptfoo/pull/5189
• feat: update red team provider selection UI by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5078
• feat(assertions): add conversational relevancy metric by @albertlieyingadrian in https://github.com/promptfoo/promptfoo/pull/2130
• feat(export): add metadata to exported evaluation files by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4886
• feat(providers): add support for Docker Model Runner provider by @wmluke in https://github.com/promptfoo/promptfoo/pull/5081
• feat(webui): add plugin and strategy filters for red team results by @will-holley in https://github.com/promptfoo/promptfoo/pull/5086

Bug Fixes

• fix: adjust padding on docs sidebar to prevent overlap by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5099
• fix: fix XML crash by @typpo in https://github.com/promptfoo/promptfoo/pull/5194
• fix: list reasoning tokens on the left side of token breakdown tooltip by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5113
• fix: map critical severity to error in ModelAudit scanner output by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5098
• fix: prevent double stateful target question in strategies page by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/4988
• fix: prevent Unicode corruption in Python providers by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5108
• fix: remove problematic caching from ModelAudit installation check by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5120
• fix: replace broken Ashby iframe with link to careers page by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5088
• fix: reset provider type correctly and handle Go providers by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5154
• fix: share debugging by @typpo in https://github.com/promptfoo/promptfoo/pull/5131
• fix(internals): force CommonJS mode for db:migrate in Node 24 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5123
• fix(openrouter): handle Gemini thinking tokens correctly by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5116
• fix(providers): correct WebP image detection in Google provider by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5171
• fix(webui): deduplicate strategy IDs by @will-holley in https://github.com/promptfoo/promptfoo/pull/5132
• fix(webui): fix custom policy validation timing issue by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5141
• fix(webui): refresh eval list when navigating back after editing eval name by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5090
• fix(webui/evals): prevent applying the same plugin/strategy multiple times by @will-holley in https://github.com/promptfoo/promptfoo/pull/5114
• fix(webui/evals): show highlights after search results by @will-holley in https://github.com/promptfoo/promptfoo/pull/5137

Chores

• chore: add link to documentation in plugin sample modal by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5193
• chore: add missing image back to home page by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5196
• chore: fix width on application details page by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5139
• chore: improve RAG metrics with detailed metadata and fix context relevance scoring by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5164
• chore: memoize context value in PostHog provider by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5089
• chore: remove accidentally committed PR description file by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5175
• chore: rename scan templates to attack profiles by @typpo in https://github.com/promptfoo/promptfoo/pull/5165
• chore: support verbosity and reasoning parameters for GPT-5 by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5207
• chore: update dependencies to latest minor and patch versions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5109
• chore: update dependencies to latest minor and patch versions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5173
• chore: update Replicate provider by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5085
• chore(providers): improve Google API key error handling and test reliability by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5147
• chore(webui): add intelligent scroll-timeline polyfill loading by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5130
• chore: bump @anthropic-ai/sdk from 0.57.0 to 0.58.0 by @dependabot[bot] in https://github.com/promptfoo/promptfoo/pull/5186
• chore: bump @aws-sdk/client-bedrock-runtime from 3.848.0 to 3.855.0 by @dependabot[bot] in https://github.com/promptfoo/promptfoo/pull/5096
• chore: bump @aws-sdk/client-bedrock-runtime from 3.855.0 to 3.856.0 by @dependabot[bot] in https://github.com/promptfoo/promptfoo/pull/5107
• chore: bump @aws-sdk/client-bedrock-runtime from 3.856.0 to 3.857.0 by @dependabot[bot] in https://github.com/promptfoo/promptfoo/pull/5126
• chore: bump @aws-sdk/client-bedrock-runtime from 3.857.0 to 3.858.0 by @dependabot[bot] in https://github.com/promptfoo/promptfoo/pull/5145
• chore: bump @aws-sdk/client-bedrock-runtime from 3.858.0 to 3.859.0 by @dependabot[bot] in https://github.com/promptfoo/promptfoo/pull/5167
• chore: bump @aws-sdk/client-bedrock-runtime from 3.859.0 to 3.861.0 by @dependabot[bot] in https://github.com/promptfoo/promptfoo/pull/5188
• chore: bump @aws-sdk/client-bedrock-runtime from 3.861.0 to 3.862.0 by @dependabot[bot] in https://github.com/promptfoo/promptfoo/pull/5198
• chore: bump @azure/identity from 4.10.2 to 4.11.0 by @dependabot[bot] in https://github.com/promptfoo/promptfoo/pull/5180
• chore: bump @azure/identity from 4.11.0 to 4.11.1 by @dependabot[bot] in https://github.com/promptfoo/promptfoo/pull/5185
• chore: bump openai from 5.10.2 to 5.11.0 by @dependabot[bot] in https://github.com/promptfoo/promptfoo/pull/5127
• chore: bump openai from 5.11.0 to 5.12.0 by @dependabot[bot] in https://github.com/promptfoo/promptfoo/pull/5187
• chore: bump version to 0.117.5 by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5206
• chore(webui/evals): filter by categorical plugins by @will-holley in https://github.com/promptfoo/promptfoo/pull/5118

Documentation

• docs: add bert-score example by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5091
• docs: add dynamic OG image generation for social media previews by @typpo in https://github.com/promptfoo/promptfoo/pull/5157
• docs: add red teaming best practices by @typpo in https://github.com/promptfoo/promptfoo/pull/5155
• docs: clarify contains-any/contains-all CSV format by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5150
• docs: fix company name by @sjhsieh in https://github.com/promptfoo/promptfoo/pull/5143
• docs: fix images by @typpo in https://github.com/promptfoo/promptfoo/pull/5197
• docs: fix multi-turn strategy documentation by @typpo in https://github.com/promptfoo/promptfoo/pull/5156
• docs: guide for evaluating LangGraph agents with Promptfoo by @Ayush7614 in https://github.com/promptfoo/promptfoo/pull/4926
• docs: include font for meta image by @typpo in https://github.com/promptfoo/promptfoo/pull/5158
• docs: make MCP image taller by @typpo in https://github.com/promptfoo/promptfoo/pull/5199
• docs: update Ollama documentation with latest models and defaultTest guidance by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5084
• docs(site): add comprehensive command line options documentation by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5135
• docs(site): add Lily Liu to team page by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5177
• docs(site): add Series A post by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5097
• docs(site): rename will.jpg to will.jpeg for consistency by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5178

Performance Improvements

• perf: make database migrations non-blocking and fix error handling by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5105

Refactorings

• style: extract helper function for deduplicating strategy IDs by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5138

Tests

• test: add tests for fix width on application details page by @use-tusk[bot] in https://github.com/promptfoo/promptfoo/pull/5140
• test: add tests for red team compliance reporting utilities in src/app by @use-tusk[bot] in https://github.com/promptfoo/promptfoo/pull/5170
• test: fix flaky Python Unicode tests by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5128
• test: fix modelGradedClosedQa test segmentation fault on macOS/Node 24 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5163
• test: increase test coverage for unified page wrapper around each red team setup step by @use-tusk[bot] in https://github.com/promptfoo/promptfoo/pull/5142

New Contributors

• @wmluke made their first contribution in https://github.com/promptfoo/promptfoo/pull/5081
• @sjhsieh made their first contribution in https://github.com/promptfoo/promptfoo/pull/5143

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.117.4...0.117.5

- TypeScript
Published by faizanminhas 7 months ago

promptfoo - 0.117.4

What's Changed

New Features

• feat: add support for OpenAI deep research models by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4661
• feat: add mcp server by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/4595
• feat: add support for connecting to existing Chrome browser sessions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5069

Bug Fixes

• fix(providers): Preserve text formatting when no images present for Google provider by @trojanowski in https://github.com/promptfoo/promptfoo/pull/5058
• fix: progress bars incrementing beyond their maximum values by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5049
• fix: resolve Jest test failures and open handles by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5052
• fix: skip validation for defaultTest to allow partial test case properties by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4732
• fix: Nested config field for custom target json by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5076
• fix(simba): fix simba host by @sklein12 in https://github.com/promptfoo/promptfoo/pull/5092
• fix: handle multi-line prompts in parseGeneratedPrompts for testGenerationInstructions by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/5093

Chores

• chore: refactor token tracking utilities, track all tokens by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4897
• chore: add new fields to eval_ran telemetry by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4638
• chore(redteam): improve redteam plugin error messaging by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4330
• test: CoverBot: Added tests for model audit and prompt management UI components (src/app) by @use-tusk[bot] in https://github.com/promptfoo/promptfoo/pull/5087
• chore: bump version 0.117.4 by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/5094

Docs

• docs(site): add AI red teaming for first-timers blog post by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5017
• docs(blog): defcon and blackhat info by @vsauter in https://github.com/promptfoo/promptfoo/pull/5050
• docs: clarifiy derivedMetrics documentation by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5068
• docs: update defcon posting by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5070
• docs: update defcon posting by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5071
• docs: switch to likert preview image by @typpo in https://github.com/promptfoo/promptfoo/pull/5083

New Contributors

• @trojanowski made their first contribution in https://github.com/promptfoo/promptfoo/pull/5058

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.117.3...0.117.4

- TypeScript
Published by MrFlounder 7 months ago

promptfoo - 0.117.3

What's Changed

New Features

• feat(eval-creator): add YAML file upload support for test cases by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5054

Bug Fixes

• fix(webui): Eval results pass rate chart rendering incorrect percentages by @will-holley in https://github.com/promptfoo/promptfoo/pull/5048
• fix: improve x.ai provider error handling for 502 errors by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5051
• fix(webui): Eval results histogram improvements by @will-holley in https://github.com/promptfoo/promptfoo/pull/5059
• fix: Infinite re-render on redteam review page by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5061
• fix: sessionid(s) in extension hooks by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/5053
• fix(google): handle multiple candidates in gemini response by @typpo in https://github.com/promptfoo/promptfoo/pull/5020
• fix: Bias Plugins should send config in remote generation by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5064

Chores

• chore(redteam): regenerate sessionId for each iteration in single-turn strategies by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4835
• chore: Change mcp log from error to debug by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/5060
• chore: Improve telemetry by @will-holley in https://github.com/promptfoo/promptfoo/pull/5062
• chore: Add simba command by @sklein12 in https://github.com/promptfoo/promptfoo/pull/5063
• chore(webui): improve redteam setup UI with progressive disclosure for advanced options by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5028
• refactor: remove redundant dotenv from Vite app by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4983
• chore: bump version 0.117.3 by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/5066
• test: CoverBot: Added tests for eval-creator components and feature flag hook (src/app) by @use-tusk[bot] in https://github.com/promptfoo/promptfoo/pull/5013

Docs

• docs(blog): grok-4 political bias post by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4953
• docs: fix cli command and remove gratuitous hover by @typpo in https://github.com/promptfoo/promptfoo/pull/5056
• docs: update user count from 100,000 to 125,000 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5046
• docs: updates to political bias post by @typpo in https://github.com/promptfoo/promptfoo/pull/5057
• docs: improve crewai eval example by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5035
• docs: update GitHub Actions to v4 across documentation and examples by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5008
• docs: add style check guidance to CLAUDE.md by @mldangelo in https://github.com/promptfoo/promptfoo/pull/5065

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.117.2...0.117.3

- TypeScript
Published by MrFlounder 7 months ago

promptfoo - 0.117.2

What's Changed

New Features

• feat(webui): First-class support for zooming eval results table by @will-holley in #4966
• feat: Grading and test generation improvements for BFLA, BOLA and RBAC by @sklein12 in #4982
• feat: New Sample Target by @sklein12 in #4979
• feat(webui): Apply metrics filter when clicking on a metric pill rendered in eval results cell by @will-holley in #4991
• feat: HTTP Target test button improvements by @faizanminhas in #5007
• feat: Add metadata filtering to eval results by @will-holley in #5014

Bug Fixes

• fix: add goal related rubric when grade crescendo turns to increase grading accuracy by @MrFlounder in #4980
• fix: update HTTP config generator endpoint to use v1 API by @mldangelo in #4989
• fix(redteam): find plugin assertion in strategy providers by @MrFlounder in #4981
• fix(site): dark mode style on redteam setup ui by @mldangelo in #5000
• fix: View logs button on redteam report by @sklein12 in #5009
• fix: undo unintended changes to http config editor by @faizanminhas in #5012
• fix: Autofocus on Redteam configuration description field by @sklein12 in #5019
• fix: remove filter icon by @sklein12 in #5021
• fix: Ollama token usage by @SamPatt in #5022
• fix(test): improve share test isolation to prevent CI timeouts by @mldangelo in #5038

Chores

• chore: revert eval view ui improvements by @mldangelo in #4969
• chore(webui): Improvements to pagination "go to" functionality by @will-holley in #4976
• chore(webui): Eval results sticky header improvements by @will-holley in #4978
• chore: update custom strategy prompt by @MrFlounder in #4994
• chore(cli): add support for 'help' argument to display command help by @mldangelo in #4823
• chore(examples): remove redteam-agent example by @mldangelo in #5001
• chore(providers): add GEMINIAPIKEY environment variable support by @mldangelo in #5004
• chore(webui): Migrate from JS to CSS for eval results scroll effects by @will-holley in #4995
• chore(webui): Eval result pagination UX improvements by @will-holley in #4993
• chore: Sort imports and turn on rule against unused imports by @faizanminhas in #5010
• chore: Make default target stateful by @faizanminhas in #4992
• chore: add medical plugins collection by @MrFlounder in #5006
• chore: Improve grading accuracy with Goal-Aware Grading for iterative/iterative tree by @MrFlounder in #4996
• chore: Add additionalRubric and storedGraderResult to GOAT and Custom providers by @MrFlounder in #5015
• chore: prevent testGenerationInstructions from being serialized if not present by @faizanminhas in #5029
• chore: Add lint rule to ensure key in jsx by @faizanminhas in #5034
• chore(webui): Eval Results UI Tweaks by @will-holley in #5023
• chore: skip goal extraction for datasets by @MrFlounder in #5036
• chore(providers): add GitHub Models provider by @mldangelo in #4998
• chore: bump version 0.117.2 by @MrFlounder in #5045
• ci: increase build job timeout from 4 to 5 minutes by @mldangelo in #5043
• test: refactor share.test.ts to prevent flaky timeouts by @mldangelo in #5037
• test: remove share.test.ts file by @mldangelo in #5044

Docs

• docs(providers): update OpenAI Assistants example by @aloisklink in #4987
• docs(redteam): improve custom strategy documentation by @mldangelo in #4990
• docs(blog): correct author attribution in DeepSeek censorship post by @mldangelo in #5002
• docs(openai): remove gpt-4.5-preview references after API deprecation by @mldangelo in #5005
• docs: remove label from featured blog post by @typpo in #5011
• docs(site): vegas contact redirect by @typpo in #5033
• docs(browser): improve browser provider documentation and examples by @mldangelo in #5030
• docs(providers): remove deprecated claude-3-sonnet-20240229 model references by @mldangelo in #5018
• docs(site): add hipaa badge by @typpo in #5039
• docs(site): add documentation for using text and embedding providers with Azure by @mldangelo in #5027
• docs(blog): fix missing blog posts by removing even-number enforcement by @mldangelo in #5042

Dependencies

• chore: bump @aws-sdk/client-bedrock-runtime from 3.846.0 to 3.848.0 by @dependabot in #4985
• chore: bump the npmandyarn group with 2 updates by @dependabot in #4984
• chore: bump @anthropic-ai/sdk from 0.56.0 to 0.57.0 by @dependabot in #5016
• chore: bump openai from 5.10.1 to 5.10.2 by @dependabot in #5024
• chore: bump the npmandyarn group with 2 updates by @dependabot in #5026
• chore: bump axios from 1.10.0 to 1.11.0 in the npmandyarn group by @dependabot in #5031

New Contributors

• @SamPatt made their first contribution in #5022

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.117.1...0.117.2

- TypeScript
Published by MrFlounder 7 months ago

promptfoo - 0.117.1

What's Changed

Fixes

• fix: move inquirer dependencies to production dependencies by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4973
• fix: grading in crescendo by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/4960
• fix: composite strategy test generation by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/4971

Chores

• chore: bump version 0.117.1 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4974

Documentation

• docs: remove tags from blog card by @typpo in https://github.com/promptfoo/promptfoo/pull/4970
• docs(blog): add system cards security analysis with vulnerability testing by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4937

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.117.0...0.117.1

- TypeScript
Published by mldangelo 8 months ago

promptfoo - 0.117.0

What's Changed

Features

• feat(http): support JKS and PFX Certificates in HTTP providers by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/4865
• feat(langfuse): add Langfuse prompt label support with improved parsing by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4847
• feat(prompts): preserve function names when using glob patterns by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4927
• feat(providers): add grok-4 support by @AISimplyExplained in https://github.com/promptfoo/promptfoo/pull/4855
• feat(providers): image understanding for Google providers by @adelmuursepp in https://github.com/promptfoo/promptfoo/pull/4767
• feat(azure): add system prompt support for azure provider by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/4869
• feat(cli): xml output by @typpo in https://github.com/promptfoo/promptfoo/pull/4912

Fixes

• fix(cli): --filter-failing not working with custom providers by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4911
• fix(google-sheets): replace hardcoded range with dynamic approach by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4822
• fix(internal): fixes filtering by metric keys which contain dots by @will-holley in https://github.com/promptfoo/promptfoo/pull/4964
• fix(providers): add thinking token tracking for Google Gemini models by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4944
• fix(providers): esm provider loading by @will-holley in https://github.com/promptfoo/promptfoo/pull/4915
• fix(providers): implement callEmbeddingApi for LiteLLM embedding provider by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4952
• fix(redteam): prevent redteam run from hanging when using an mcp client by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/4924
• fix(redteam): respect PROMPTFOODISABLEREDTEAMREMOTEGENERATION for cloud users by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4839
• fix(redteam): set pluginId on eval results by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4928
• fix(redteam): test target in http provider setup with non-200 status codes by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/4932
• fix(webui): eval results table horizontal scrolling by @will-holley in https://github.com/promptfoo/promptfoo/pull/4826
• fix(webui): fix hard-coded light mode colors in model audit interface by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4907
• fix(webui): handle null table.body in DownloadMenu disabled prop by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4913
• fix(webui): resolve pagination scrolling and layout issues in ResultsTable by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4943
• fix(webui): scrolling when tbody is outside of viewport by @will-holley in https://github.com/promptfoo/promptfoo/pull/4948

Chores

• chore(knip): integrate knip for unused code detection and clean up codebase by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/4464
• chore(linting): migrate from ESLint + Prettier to Biome by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4903
• chore(assertions): additional checking on llm-rubric response by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4954
• chore(assertions): include reason in model-graded-closedqa pass reason by @typpo in https://github.com/promptfoo/promptfoo/pull/4931
• chore(build): resolve build warnings and optimize bundle size by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4895
• chore(csv): improve __metadata warning message and test coverage by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4842
• chore(providers): improve guardrails handling in Azure providers by @will-holley in https://github.com/promptfoo/promptfoo/pull/4788
• chore(redteam): add domain-specific risks section and reduce verbose descriptions by @typpo in https://github.com/promptfoo/promptfoo/pull/4879
• chore(release): bump version 0.117.0 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4963
• chore(server): check if server is already running before starting by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4896
• chore(server): log correct eval ID instead of description in WebSocket updates by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4910
• chore(telemetry): add telemetry logging when tracing is enabled by @typpo in https://github.com/promptfoo/promptfoo/pull/4925
• chore(types): typings needed for enterprise by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4955
• chore(vscode): use Biome as default formatter of TS files in vscode by @will-holley in https://github.com/promptfoo/promptfoo/pull/4920
• chore(webui): conditionally render metrics selector by @will-holley in https://github.com/promptfoo/promptfoo/pull/4936
• chore(webui): display context values in eval results by @will-holley in https://github.com/promptfoo/promptfoo/pull/4856
• chore(webui): improves eval results table spacing by @will-holley in https://github.com/promptfoo/promptfoo/pull/4965
• chore(webui): revert eval view ui improvements by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4967
• chore(webui/eval): allow filtering results by >1 metrics simultaneously (disabled by default) by @will-holley in https://github.com/promptfoo/promptfoo/pull/4870

Dependencies

• chore(deps): add overrides to fix build issues by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4957
• chore(deps): bump @aws-sdk/client-bedrock-runtime from 3.842.0 to 3.844.0 by @dependabot[bot] in https://github.com/promptfoo/promptfoo/pull/4850
• chore(deps): bump aiohttp from 3.11.11 to 3.12.14 in /examples/redteam-langchain in the pip group across 1 directory by @dependabot[bot] in https://github.com/promptfoo/promptfoo/pull/4922
• chore(deps): bump openai from 5.8.3 to 5.9.0 by @dependabot[bot] in https://github.com/promptfoo/promptfoo/pull/4863
• chore(deps): bump openai from 5.9.2 to 5.10.1 by @dependabot[bot] in https://github.com/promptfoo/promptfoo/pull/4961
• chore(deps): move knip to dev dependencies by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/4958
• chore(deps): npm audit fix by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4962
• chore(deps): test removing knip to resolve installation errors by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4956
• chore(deps): update all example dependencies to latest versions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4900
• chore(deps): update dependencies to latest minor/patch versions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4899
• chore(deps): update non-breaking dependencies by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4935
• chore(deps): update Jest to version 30 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4939

Refactors

• refactor(eval-config): modernize eval-creator state management by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4908
• refactor(webui): improve metrics ui by @will-holley in https://github.com/promptfoo/promptfoo/pull/4938
• refactor(webui/eval results): pagination improvements by @will-holley in https://github.com/promptfoo/promptfoo/pull/4914

Tests

• test(core): coverBot: added tests for core UI components and user context hooks (src/app) by @use-tusk[bot] in https://github.com/promptfoo/promptfoo/pull/4929
• test(EnterpriseBanner): add unit tests for EnterpriseBanner component by @use-tusk[bot] in https://github.com/promptfoo/promptfoo/pull/4919
• test(redteam): add unit test for src/redteam/remoteGeneration.ts by @gru-agent[bot] in https://github.com/promptfoo/promptfoo/pull/4834
• test(server): fix flaky server share tests by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4942
• test(server): fix flaky server tests by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4968
• test(server): mock database in server tests by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4959
• test(tusk): update Tusk test runner workflow - coverage script by @use-tusk[bot] in https://github.com/promptfoo/promptfoo/pull/4921

Docs

• docs(analytics): add google tag manager by @typpo in https://github.com/promptfoo/promptfoo/pull/4904
• docs(api): improves contextTransform documentation by @will-holley in https://github.com/promptfoo/promptfoo/pull/4854
• docs(assertions): add missing deterministic assertions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4891
• docs(azure): improve Azure provider documentation by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4836
• docs(blog): add blog image generation script by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4945
• docs(blog): add truncation markers to articles without them by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4934
• docs(blog): add truncation markers to blog posts by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4906
• docs(blog): mcp proxy blog by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4860
• docs(blog): revise article tags by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4949
• docs(blog): soc2 type ii and iso 27001 blog by @vsauter in https://github.com/promptfoo/promptfoo/pull/4880
• docs(comparison): pyrit comparison by @typpo in https://github.com/promptfoo/promptfoo/pull/4679
• docs(config): clarify PROMPTFOOEVALTIMEOUTMS and PROMPTFOOMAXEVALTIME_MS descriptions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4947
• docs(enterprise): adaptive guardrails enterprise by @typpo in https://github.com/promptfoo/promptfoo/pull/4951
• docs(events): blackhat landing page by @typpo in https://github.com/promptfoo/promptfoo/pull/4862
• docs(events): defcon landing page by @typpo in https://github.com/promptfoo/promptfoo/pull/4864
• docs(events): events banner by @typpo in https://github.com/promptfoo/promptfoo/pull/4867
• docs(examples): add mischievous-user strategy to redteam multi-turn examples by @will-holley in https://github.com/promptfoo/promptfoo/pull/4837
• docs(gemini): update experimental Gemini model IDs to stable versions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4894
• docs(google): add examples for gemini URL context and code execution tools by @adelmuursepp in https://github.com/promptfoo/promptfoo/pull/4923
• docs(guide): guide for evaluating CrewAI agents with Promptfoo by @Ayush7614 in https://github.com/promptfoo/promptfoo/pull/4861
• docs(images): standardize CrewAI image filenames to kebab-case by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4941
• docs(integration): add n8n integration by @typpo in https://github.com/promptfoo/promptfoo/pull/4917
• docs(litellm): fix example with modern model IDs and proper embedding config by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4885
• docs(mcp): add mcp testing guide by @typpo in https://github.com/promptfoo/promptfoo/pull/4846
• docs(mcp): add mcp to sidebar by @typpo in https://github.com/promptfoo/promptfoo/pull/4852
• docs(metrics): add similar to model graded metrics table by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4830
• docs(providers): update available databricks models by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4887
• docs(providers): update provider index with missing providers and latest 2025 model IDs by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4888
• docs(release): add monthly release notes by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4358
• docs(resources): add arsenal link by @typpo in https://github.com/promptfoo/promptfoo/pull/4878
• docs(security): add soc2 badge by @typpo in https://github.com/promptfoo/promptfoo/pull/4877
• docs(site): add OWASP top 10 tldr blog post by @ladyofcode in https://github.com/promptfoo/promptfoo/pull/4853
• docs(site): expand June 2025 release notes with detailed feature documentation by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4881
• docs(site): improve Google AI and Vertex authentication documentation by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4892
• docs(site): improve NLP metric explanations and add SEO metadata by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4890
• docs(site): update python documentation for basePath config option by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4819
• docs(ui): better mobile wrap on homepage tabs by @typpo in https://github.com/promptfoo/promptfoo/pull/4884
• docs(ui): colors by @typpo in https://github.com/promptfoo/promptfoo/pull/4875
• docs(ui): contrast fixes by @typpo in https://github.com/promptfoo/promptfoo/pull/4901
• docs(ui): fix button clickability issue on hero sections by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4905
• docs(ui): remove bouncing down arrow in mobile by @typpo in https://github.com/promptfoo/promptfoo/pull/4882
• docs(ui): remove text shadow by @typpo in https://github.com/promptfoo/promptfoo/pull/4898

New Contributors

• @Ayush7614 made their first contribution in https://github.com/promptfoo/promptfoo/pull/4861

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.116.7...0.117.0

- TypeScript
Published by mldangelo 8 months ago

promptfoo - 0.116.7

What's Changed

Fixes

• fix: Always do remote generation if logged into cloud by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4832

Chores

• chore(providers/sagemaker): Improves error handling in SageMakerCompletionProvider by @will-holley in https://github.com/promptfoo/promptfoo/pull/4808
• chore(providers/sagemaker): Improves validation of user-provided config by @will-holley in https://github.com/promptfoo/promptfoo/pull/4809
• chore: update graderExamplesString by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/4821
• chore: bump version 0.116.7 by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4833

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.116.6...0.116.7

- TypeScript
Published by sklein12 8 months ago

promptfoo - 0.116.6

What's Changed

Fix

• fix: Failing test by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4829

Chore

• chore: bump version 0.116.6 by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4831

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.116.5...0.116.6

- TypeScript
Published by sklein12 8 months ago

promptfoo - 0.116.5

Features

• feat: add support for loading defaultTest from external files (#4720) by @mldangelo
• feat: add embedding support to LiteLLM provider (#4804) by @mldangelo
• feat: add mischievous user strategy (#4107) by @mldangelo

Fixes

• fix: add glob pattern support for loading scenario files (#4761) by @mldangelo
• fix: improve model-audit installation check dark mode display (#4816) by @typpo
• fix: pass env vars to MCP server (#4827) by @sklein12

Chores

• chore: better remote grading logs (#4820) by @sklein12
• chore: bump openai from 5.8.2 to 5.8.3 (#4817) by @dependabot
• chore: bump version 0.116.5 (#4828) by @sklein12
• chore: capitalize 'Red Team' in navigation menu for consistency (#4799) by @mldangelo
• chore: remove redundant 'Done.' message from evaluation output (#4810) by @mldangelo
• chore: remove python script result data type debug log (#4807) by @will-holley
• chore: update website with MCP Proxy (#4812) by @sklein12

Docs

• docs: add Azure OpenAI vision example (#4806) by @mldangelo
• docs: add looper guide (#4814) by @typpo
• docs: add SonarQube integration (#4815) by @typpo

Tests

• test: add unit test for src/assertions/guardrails.ts (#4765) by @gru-agent
• test: add unit test for src/redteam/commands/generate.ts (#4789) by @gru-agent
• test: add unit test for src/redteam/constants/strategies.ts (#4800) by @gru-agent
• test: add unit test for src/redteam/plugins/pii.ts (#4780) by @gru-agent
• test: add unit test for src/types/providers.ts (#4766) by @gru-agent
• test: add unit test for src/validators/redteam.ts (#4803) by @gru-agent

Full Changelog: 0.116.4...0.116.5

- TypeScript
Published by sklein12 8 months ago

promptfoo - 0.116.4

What's Changed

New Features

• feat(redteam): add support for custom multi-turn strategy by @MrFlounder in #4783
• feat(redteam): expose generate function in redteam namespace by @mldangelo in #4793

Chores

• chore: bump version 0.116.4 by @MrFlounder in #4805
• chore: rename strategy name from playbook to custom by @MrFlounder in #4798
• refactor: inline MEMORYPOISONINGPLUGIN_ID constant by @mldangelo in #4794

Docs

• docs: add doc for custom strategy by @MrFlounder in #4802
• docs: modular configuration management by @typpo in #4763

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.116.3...0.116.4

- TypeScript
Published by MrFlounder 8 months ago

promptfoo - 0.116.3

What's Changed

Features

• feat(providers): add MCP provider by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/4768
• feat(providers): add new AIMLAPI provider by @D1m7asis in https://github.com/promptfoo/promptfoo/pull/4721
• feat(assertions): add contextTransform support for RAG evaluation by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4467
• feat(assertions): add finish reason as assertion option by @christophersutton in https://github.com/promptfoo/promptfoo/pull/3879
• feat(assertions): trace assertions by @typpo in https://github.com/promptfoo/promptfoo/pull/4750
• feat(tracing): add traces to JavaScript, Python asserts by @typpo in https://github.com/promptfoo/promptfoo/pull/4745

Fixes

• fix(telemetry): prevent PostHog initialization when telemetry is disabled by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4772
• fix(redteam): fix modifiers application order in PII plugins by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/4779

Chores

• chore(schema): remove duplicate 'bias' entry in config-schema.json by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4773
• chore(telemetry): add PostHog client to app by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/4726
• chore(redteam): add reason field to give clear/customized guardrails triggering reason by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/4764
• chore(providers): expose MCP plugin in UI by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4762
• chore(providers): AWS SageMaker AI provider cleanup by @will-holley in https://github.com/promptfoo/promptfoo/pull/4667
• chore(providers): update AIML integration by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4751
• chore(redteam): improve organization of redteam strategies in setup UI by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4738
• chore(telemetry): identify to PostHog whether user is also cloud user by @will-holley in https://github.com/promptfoo/promptfoo/pull/4782
• chore(deps): bump @anthropic-ai/sdk from 0.55.1 to 0.56.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/4756
• chore(deps): bump @aws-sdk/client-bedrock-runtime from 3.840.0 to 3.842.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/4747
• chore(deps): bump @azure/identity from 4.10.1 to 4.10.2 by @dependabot in https://github.com/promptfoo/promptfoo/pull/4748
• chore(deps): bump version 0.116.3 by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/4792
• chore(deps): update pbkdf2 to 3.1.3 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4777
• chore(deps): upgrade glob from v10 to v11 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4776
• chore: expose doRedteamRun in package exports by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4758

Docs

• docs: add Gemini Live API audio by @adelmuursepp in https://github.com/promptfoo/promptfoo/pull/4729
• docs: ModelAudit vs ModelScan by @typpo in https://github.com/promptfoo/promptfoo/pull/4769
• docs: multiple MCP server connections by @typpo in https://github.com/promptfoo/promptfoo/pull/4755
• docs: update ModelAudit documentation with new features and fixes by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4699

Tests

• test: add integrity check for generated-constants.ts by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4753
• test: fix flaky Google Live test and improve test speed by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4774
• test: fix mock pollution in testCaseReader by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4775
• test: isolate mocks so tests can run in any order with --randomize by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4744

New Contributors

• @christophersutton made their first contribution in https://github.com/promptfoo/promptfoo/pull/3879
• @D1m7asis made their first contribution in https://github.com/promptfoo/promptfoo/pull/4721

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.116.2...0.116.3

- TypeScript
Published by faizanminhas 8 months ago

promptfoo - 0.116.2

What's Changed

Bug Fixes

• fix: unblock postbuild for ci by @MrFlounder in #4742

Chores

• chore: bump version 0.116.2 by @MrFlounder in #4743

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.116.1...0.116.2

- TypeScript
Published by MrFlounder 8 months ago

promptfoo - 0.116.1

What's Changed

New Features

• feat(cli): support pdb tracing in 3rd party Python scripts by @will-holley in #4723

Bug Fixes

• fix: http body parsing when it comes from yaml string by @MrFlounder in #4728
• fix: remove accidentally committed redteam.yaml file by @mldangelo in #4733
• fix: fix the case when http body has not escaped charactors by @MrFlounder in #4739
• fix: update package-lock.json by @mldangelo in #4719
• fix(webui): replace window.location.href with React Router navigation by @mldangelo in #4717
• test: fix SIGSEGV caused by better-sqlite3 in test environment by @mldangelo in #4737

Chores

• chore: Add unblocking detection to GOAT strategy by @MrFlounder in #4532
• chore: add preset for guardrails eval by @MrFlounder in #4640
• chore: Improve telemetry delivery by @sklein12 in #4655
• chore: reset generated constants after build by @mldangelo in #4731
• chore: update onboarding model defaults by @typpo in #4708
• chore(webui): improve styling of EvalsDataGrid by @mldangelo in #4736
• ci(workflows): gracefully handle missing PostHog secret in forks by @ggiiaa in #4725
• test: refactor assertion tests by @mldangelo in #4718
• chore: bump version 0.116.1 by @MrFlounder in #4741

Docs

• docs: add system prompt hardening blog post by @ladyofcode in #4630
• docs(site): add guide on humanity's last exam by @mldangelo in #4694
• docs(site): clarify self-hosting workflow for eval sharing by @mldangelo in #4730
• docs(site): fix relative link in HLE benchmark guide by @mldangelo in #4711

Dependencies

• chore: bump @anthropic-ai/sdk from 0.55.0 to 0.55.1 by @dependabot in #4710
• chore: bump @aws-sdk/client-bedrock-runtime from 3.839.0 to 3.840.0 by @dependabot in #4709

New Contributors

• @ggiiaa made their first contribution in #4725

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.116.0...0.116.1

- TypeScript
Published by MrFlounder 8 months ago

promptfoo - 0.116.0

What's Changed

New Features

• feat(redteam): add financial plugins by @typpo in https://github.com/promptfoo/promptfoo/pull/4416
• feat(redteam): add bias plugins by @AISimplyExplained in https://github.com/promptfoo/promptfoo/pull/4382
• feat(providers): add Helicone AI Gateway provider by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4662

Bug Fixes

• fix(app): use client-generated session IDs when testing targets by @will-holley in https://github.com/promptfoo/promptfoo/pull/4653
• fix(matchers): track token usage for successful API calls by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4677
• fix(providers): handle content filter errors in Azure Assistant API by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4674
• fix(providers): fix SageMaker Llama inference configuration serialization by @will-holley in https://github.com/promptfoo/promptfoo/pull/4637
• fix(redteam): respect maxConcurrency from Web UI by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4605
• fix(simulated-user): pass context variables to custom providers by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4654
• fix(telemetry): add telemetry for red teams by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4641
• fix(webui): handle undefined outputs in DownloadMenu by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4693
• fix(webui): prevent pass/fail badge from disappearing when toggling highlight by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4700
• fix(webui): support derived metrics in eval configuration uploaded via Web UI by @yexingyao in https://github.com/promptfoo/promptfoo/pull/4647
• fix(webui): use backendCounts first before counting metrics on page by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/4659
• fix(sharing): fix file outputs when sharing by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4698

Chores

• chore: enable WAL mode for SQLite by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4104
• chore(providers): add thread ID function call for OpenAI and Azure assistants by @gonzab in https://github.com/promptfoo/promptfoo/pull/2263
• chore(app): improve target test error handling by @will-holley in https://github.com/promptfoo/promptfoo/pull/4652
• chore(cli): add missing CLI options to scan-model command for feature parity by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4670
• chore(providers): convert Cloudflare AI to use OpenAI-compatible endpoints by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4683
• chore(providers): log flagged output for Azure chat models by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/4636
• chore(redteam): add centralized REDTEAM_DEFAULTS and maxConcurrency support by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4656
• chore(webui): add checkbox to clear all variables by @sangwoo-joh in https://github.com/promptfoo/promptfoo/pull/666
• chore(webui): add defaultTest variables to red team setup UI by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4671
• chore(webui): remove unused components by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4695
• chore(webui): set page titles on every page by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4668
• chore(telemetry): add pass/fail/errors to eval_run event by @typpo in https://github.com/promptfoo/promptfoo/pull/4639
• chore(telemetry): improve page view deduplication by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4651

Dependencies

• chore(deps): bump @anthropic-ai/sdk from 0.54.0 to 0.55.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/4628
• chore(deps): bump openai from 5.7.0 to 5.8.1 by @dependabot in https://github.com/promptfoo/promptfoo/pull/4664
• chore(deps): bump version to 0.116.0 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4707
• chore(deps): update minor and patch dependencies by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4686

Documentation

• docs(site): add async Python note by @typpo in https://github.com/promptfoo/promptfoo/pull/4680
• docs(site): add Garak comparison by @typpo in https://github.com/promptfoo/promptfoo/pull/4660
• docs(site): update Garak post by @typpo in https://github.com/promptfoo/promptfoo/pull/4672
• docs(site): add ModelAudit HuggingFace scanner by @typpo in https://github.com/promptfoo/promptfoo/pull/4645
• docs(redteam): add missing docs to sidebar by @typpo in https://github.com/promptfoo/promptfoo/pull/4690
• docs(redteam): remove duplicate ToxicChat plugin by @typpo in https://github.com/promptfoo/promptfoo/pull/4689
• docs(redteam): update Target Purpose documentation by @will-holley in https://github.com/promptfoo/promptfoo/pull/4523
• docs(site): add FAQ section for offline environment usage by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4650
• docs(site): add HuggingFace datasets integration documentation by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4691
• docs(site): add truncation marker to Garak blog post by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4666
• docs(site): clarify self-hosting replica limitations by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4669
• docs(site): remove copy for LLM button by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4665
• docs(site): remove unnecessary configuration review text from getting started guide by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4597
• docs(site): reorganize configuration documentation structure by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4692
• docs(site): use relative URLs for internal links and fix broken references by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4688
• docs(site): correct typos in red team agent blog post by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4634

Tests

• test: add unit test for src/server/routes/providers.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/4658
• test: verify that plugins are synced between code and documentation by @typpo in https://github.com/promptfoo/promptfoo/pull/4681

New Contributors

• @yexingyao made their first contribution in https://github.com/promptfoo/promptfoo/pull/4647
• @gonzab made their first contribution in https://github.com/promptfoo/promptfoo/pull/2263

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.115.4...0.116.0

- TypeScript
Published by mldangelo 8 months ago

promptfoo - 0.115.4

What's Changed

Features: * feat: opentelemetry tracing support by @typpo in https://github.com/promptfoo/promptfoo/pull/4600

Chores: * chore: bump version 0.115.4 by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/4635 * chore: remove invariant by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/4633 * chore: update Tusk test runner workflow by @use-tusk in https://github.com/promptfoo/promptfoo/pull/4627* * docs: prevent copy button from overlapping screenshot overlay by @typpo in https://github.com/promptfoo/promptfoo/pull/4632

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.115.3...0.115.4

- TypeScript
Published by MrFlounder 8 months ago

promptfoo - 0.115.3

What's Changed

Bug Fixes

• fix(webui): handle null scores in ResultsCharts component #4610 by @mldangelo
• fix: empty vars array on eval results #4621 by @sklein12
• fix: save sessionId for multi-turn strategies #4625 by @sklein12
• fix(redteam): skip goal extraction when remote generation is disabled #4623 by @mldangelo
• fix(test): hyperbolic provider tests failing due to env variable pollution #4619 by @mldangelo
• fix(cli): remove context schema validation from extension hooks #4622 by @will-holley

Chores

• chore: PROMPTFOODISABLETEMPLATEENVVARS controls process.env access, not env: access #4620 by @mldangelo
• chore: bump version to 0.115.3 #4626 by @sklein12

Full Changelog: v0.115.2...v0.115.3

- TypeScript
Published by sklein12 8 months ago

promptfoo - 0.115.2

What's Changed

Features

• feat(cli): add assertion generation by @sidd2pir in https://github.com/promptfoo/promptfoo/pull/4559
• feat(providers): add support for hyperbolic image and audio providers by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4260

Bug Fixes

• fix(openrouter): unpack passthrough at the root level by @mathop in https://github.com/promptfoo/promptfoo/pull/4592
• fix(webui): escape HTML special characters in output reports by @typpo in https://github.com/promptfoo/promptfoo/pull/4555
• fix(webui): sort EvalsDataGrid by creation date by @typpo in https://github.com/promptfoo/promptfoo/pull/4594
• fix(cli): include cached results in grand total by @typpo in https://github.com/promptfoo/promptfoo/pull/4581
• fix(webui): improve base64 matching by @typpo in https://github.com/promptfoo/promptfoo/pull/4609
• fix(modelaudit): use modelaudit binary by @typpo in https://github.com/promptfoo/promptfoo/pull/4525
• fix(webui): make Citations font consistent with other headers by @typpo in https://github.com/promptfoo/promptfoo/pull/4598
• fix(redteam): respect maxTurns from dev doc in crescendo by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/4527
• fix(webui): prevent Welcome component from rendering while loading eval data by @will-holley in https://github.com/promptfoo/promptfoo/pull/4604
• fix(cli): prevent RangeError in progress bar variable display by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4475
• fix(server): resolve Express.js NotFoundError when serving app by @will-holley in https://github.com/promptfoo/promptfoo/pull/4601

Chores

• chore(redteam): add cross-session leak strategy exclusions by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/4516
• chore(cli): display key metrics (success, failures, pass rate) at the bottom of output by @typpo in https://github.com/promptfoo/promptfoo/pull/4580
• chore: remove unused import by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/4530
• chore(webui): show provider breakdown only for multiple providers by @typpo in https://github.com/promptfoo/promptfoo/pull/4599
• chore(redteam): update Target Purpose Discovery by @will-holley in https://github.com/promptfoo/promptfoo/pull/4480
• chore(ci): update CodeRabbit config to be less aggressive by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4586
• chore(providers): update Gemini models to include latest 2.5 Pro Preview and Flash models by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4499
• chore(providers): update tau-simulated-user docs and example by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4468
• chore(webui): use CSS to create PDF-optimized report and browser to save as PDF by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4535
• chore(app): remove discovered purpose from report view by @will-holley in https://github.com/promptfoo/promptfoo/pull/4541
• chore(cli): add cache busting for select provider API calls by @will-holley in https://github.com/promptfoo/promptfoo/pull/4508
• chore(cli): improve concurrency log statements by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4606
• chore(eval): add first-class support for beforeAll and beforeEach extension hooks mutation of context by @will-holley in https://github.com/promptfoo/promptfoo/pull/4197
• chore(providers): document support for loading system instructions from files by @typpo in https://github.com/promptfoo/promptfoo/pull/4582
• chore(providers): enhance OpenAI provider with legacy models and new parameters by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4502
• chore(redteam): add continueAfterSuccess option to multi-turn strategies by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4570
• chore(webui): improve purpose form by @will-holley in https://github.com/promptfoo/promptfoo/pull/4603
• chore(redteam): add JSON file support to intent plugin with enhanced UI by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4574
• chore(redteam): add unblock multiturn by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/4498
• chore(ci): clean up CodeRabbit configuration and minimize automated comments by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4573

Build & Dependencies

• build: update Tusk vitest reporter by @sohil-kshirsagar in https://github.com/promptfoo/promptfoo/pull/4602
• chore(deps): bump @aws-sdk/client-bedrock-runtime from 3.830.0 to 3.835.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/4614
• chore(deps): bump openai from 5.5.0 to 5.5.1 by @dependabot in https://github.com/promptfoo/promptfoo/pull/4537
• chore(deps): bump openai from 5.5.1 to 5.6.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/4596
• chore(deps): bump openai from 5.6.0 to 5.7.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/4615
• chore(deps): bump urllib3 from 1.26.19 to 2.5.0 in /examples/docker-code-generation-sandbox by @dependabot in https://github.com/promptfoo/promptfoo/pull/4556
• chore(deps): bump urllib3 from 2.3.0 to 2.5.0 in /examples/redteam-langchain by @dependabot in https://github.com/promptfoo/promptfoo/pull/4557
• chore: bump version to 0.115.2 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4617

Documentation

• docs: add audit logging documentation for enterprise features by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4482
• docs: add feedback page and update CLI link by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4591
• docs: add ISO badge by @typpo in https://github.com/promptfoo/promptfoo/pull/4534
• docs: improve contact form by @typpo in https://github.com/promptfoo/promptfoo/pull/4531
• docs: update ModelAudit documentation by @typpo in https://github.com/promptfoo/promptfoo/pull/4585
• docs: clarify no OpenAI key required for Claude redteam by @typpo in https://github.com/promptfoo/promptfoo/pull/4524
• docs: add red team Gemini documentation by @typpo in https://github.com/promptfoo/promptfoo/pull/4542
• docs: add trust center documentation by @typpo in https://github.com/promptfoo/promptfoo/pull/4539
• docs: update contact form by @typpo in https://github.com/promptfoo/promptfoo/pull/4529
• docs(blog): add authors to blog posts and update authors.yml by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4564
• docs(blog): add descriptions and keywords to blog posts by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4565
• docs(examples): add pydantic-ai example with structured output evaluation by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4575
• docs(examples): consolidate Google Vertex Tools examples by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4587
• docs(examples): consolidate Python assertion examples into unified folder by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4588
• docs(examples): consolidate translation examples by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4590
• docs(site): document new features in ModelAudit by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4593
• docs(site): document new features in modelaudit by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4593
• docs(site): fix author reference on 2025-summer-new-redteam-agent blog post by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4563
• docs(site): Update ModelAudit scanners documentation with comprehensive scanner coverage by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4562

Tests

• test: add unit test for src/commands/delete.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/4572
• test: add unit test for src/commands/modelScan.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/4526
• test: add unit test for src/commands/show.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/4571
• test: add unit test for src/providers/azure/completion.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/4510
• test: add unit test for src/providers/ollama.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/4509
• test: add unit test for src/providers/ollama.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/4512
• test: add unit test for src/providers/openai/completion.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/4511
• test: add unit test for src/python/pythonUtils.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/4486
• test: improve mock setup and teardown for --randomize by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4569

New Contributors

• @mathop made their first contribution in https://github.com/promptfoo/promptfoo/pull/4592

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.115.1...0.115.2

- TypeScript
Published by mldangelo 8 months ago

promptfoo - 0.115.1

What's Changed

Fixes

• fix: Windows Python path validation race condition by @benbuzz790 in https://github.com/promptfoo/promptfoo/pull/4485
• fix: View results as evaluation runs by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4459

Chores

• chore: refactor modifiers and apply to all plugins by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/4454
• chore(cli): update plugin severity overrides API endpoint by @will-holley in https://github.com/promptfoo/promptfoo/pull/4460
• chore(webui): fix text length reset value to use reasonable default by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4469
• chore(webui): remove unused hook files by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4470
• chore: remove unused token usage utilities by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4471
• chore: convert console.logs to logger by @typpo in https://github.com/promptfoo/promptfoo/pull/4479
• chore: improve tusk workflow by @sohil-kshirsagar in https://github.com/promptfoo/promptfoo/pull/4461
• chore: bump version to 0.115.1 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4520

Dependencies

• chore(deps): bump @aws-sdk/client-bedrock-runtime from 3.828.0 to 3.830.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/4519
• chore(deps): bump @azure/identity from 4.10.0 to 4.10.1 by @dependabot in https://github.com/promptfoo/promptfoo/pull/4477
• chore(deps): bump openai from 5.3.0 to 5.5.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/4518
• chore(deps): update zod to 3.25.63 and zod-validation-error to 3.5.0 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4463

Documentation

• docs: add log file location section to troubleshooting guide by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4473
• docs: capitalize Promptfoo by @typpo in https://github.com/promptfoo/promptfoo/pull/4515
• docs: update red-teaming agent blog post title by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4497
• docs: improve installation and getting-started pages with tabbed interface and SEO metadata by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4395
• docs: improve Python provider documentation by @typpo in https://github.com/promptfoo/promptfoo/pull/4484
• docs: add ModelAudit binary formats documentation by @typpo in https://github.com/promptfoo/promptfoo/pull/4500
• docs: update ModelAudit documentation by @typpo in https://github.com/promptfoo/promptfoo/pull/4514
• docs: add ModelAudit weighted distribution scanner documentation by @typpo in https://github.com/promptfoo/promptfoo/pull/4501
• docs: add ModelAudit ZIP feature documentation by @typpo in https://github.com/promptfoo/promptfoo/pull/4491
• docs: separate pages for prompts, test cases, and outputs by @typpo in https://github.com/promptfoo/promptfoo/pull/4505
• docs: update model reference in guide.md by @aaronccasanova in https://github.com/promptfoo/promptfoo/pull/4513
• docs(blog): add new redteam agent documentation by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4494
• docs(examples): fix custom-grader-csv README inconsistencies by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4474
• docs(site): add llms.txt mentions and documentation standards by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4481
• docs(site): add robots.txt by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4488
• docs: fix typo in blog post by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4496
• docs: update title on blog post by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4495

Testing

• test: add unit test for src/util/cloud.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/4462
• test: add unit test for src/util/convertEvalResultsToTable.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/4457

New Contributors

• @sohil-kshirsagar made their first contribution in https://github.com/promptfoo/promptfoo/pull/4461
• @aaronccasanova made their first contribution in https://github.com/promptfoo/promptfoo/pull/4513

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.115.0...0.115.1

- TypeScript
Published by mldangelo 9 months ago

promptfoo - 0.115.0

0.115.0 (2025-06-12)

✨ Features

• feat(providers): Google live audio output (#4280) by @adelmuursepp
• feat(webui): static model-scanning UI (#4368) by @typpo
• feat(tests): configuration support for test generators (#4301) by @mldangelo
• feat(cli): per-provider token-usage statistics (#4044) by @mldangelo
• feat(providers): optional token-estimation for HTTP provider (#4439) by @mldangelo
• feat(redteam): enable HTTP-token estimation by default in red-team mode (#4449) by @mldangelo
• feat(redteam): cloud-based plugin-severity overrides (#4348) by @will-holley
• feat(providers): custom-header support for Azure API (#4409) by @yurchik11
• feat(core): maximum evaluation-time limit via PROMPTFOO_MAX_EVAL_TIME_MS (#4322) by @mldangelo
• feat(redteam): Aegis red-team dataset (#4119) by @mldangelo
• feat(providers): Mistral Magistral reasoning models (#4435) by @mldangelo
• feat(core): WebSocket header support (#4456) by @typpo

🐛 Fixes

• fix(eval): gracefully handle RangeError & truncate oversized output (#4424) by @Sly1029
• fix(providers): add timeout to ProxyAgent (#4369) by @AegisAurora
• fix(config): persist Goat configuration (#4370) by @sklein12
• fix(parser): lenient JSON parsing for MathPrompt (#4361) by @typpo
• fix(redteam): standardize plugin parameter to prompt (#4425) by @mldangelo
• fix(assertions): support snake_case fields in Python assertions (#4398) by @mldangelo
• fix(redteam): handle purpose without prompts (#4445) by @typpo
• fix(webui): stream test-cases to viewer (#4440) by @mldangelo
• fix(redteam): connect MisinformationDisinformationGrader (#4452) by @mldangelo

♻️ Refactors

• refactor(redteam): consolidate constants (#4372) by @mldangelo

📚 Documentation

• docs(blog): GPT red-team post (#4363) by @typpo
• docs(blog): Claude red-team post (#4365) by @typpo
• docs(guides): clarify completion-variable for factuality (#4385) by @mldangelo
• docs(blog): fix broken image link in GPT post (#4391) by @mldangelo
• docs(blog): update Claude-4 post date (#4392) by @mldangelo
• docs(site): move discovery docs under Tools (#4408) by @typpo
• docs(guides): GPT-4.1 vs GPT-4o MMLU comparison (#4399) by @mldangelo
• docs(blog): 100 k-users milestone post (#4402) by @mldangelo
• docs(redteam): configuration precedence section (#4412) by @typpo
• docs(policies): PromptBlock format for custom policies (#4327) by @mldangelo
• docs(site): improve copy-button positioning (#4414) by @mldangelo
• docs(workflow): GH-CLI rule improvements (#4415) by @mldangelo
• docs(blog): overflow in MCP blog post (#4367) by @AISimplyExplained
• docs(redteam): remove duplicate memory-poisoning entry (#4388) by @mldangelo

🔧 Chores & Maintenance

• chore(ci): set CodeRabbit review settings (#4413) by @sklein12
• chore(core): coding-rules for error messages (#4401) by @sklein12
• chore(core): improve RangeError diagnostics (#4431) by @mldangelo
• chore(core): prefer remote-purpose generation (#4444) by @typpo
• chore(core): remove unused types & deprecated functions (#4450) by @mldangelo
• chore(cursor): local-dev guidance for coding agents (#4403) by @mldangelo
• chore(deps): bump @aws-sdk/client-bedrock-runtime → 3.826.0 (#4366) by @dependabot
• chore(deps): bump @aws-sdk/client-bedrock-runtime → 3.828.0 (#4442) by @dependabot
• chore(deps): bump brace-expansion → 1.1.12 (#4423) by @dependabot
• chore(deps): bump openai → 5.3.0 (#4407) by @dependabot
• chore(deps): bump pip group dependencies (#4379) by @dependabot
• chore(deps): minor + patch bumps across workspaces (#4377) by @mldangelo
• chore(deps): upgrade Express → 5.1.0 (#4378) by @mldangelo
• chore(docs): add README for missing examples (#4404) by @mldangelo
• chore(providers): initial o3-pro support (#4397) by @mldangelo
• chore(providers): o3-pro improvements (#4396) by @mldangelo
• chore(redteam): delimit user-inputs in purpose discovery (#4405) by @typpo
• chore(redteam): turn off discovery by default (#4393) by @sklein12
• chore(release): bump version → 0.115.0 (#4451) by @mldangelo
• chore(ui): improve EvalOutputPromptDialog styling (#4364) by @typpo
• chore(webui): remove extra OpenAI targets (#4447) by @mldangelo
• chore(webui): add token-estimation UI (#4448) by @mldangelo

🧪 Tests

• test(redteam): unique risk-category IDs (#4390) by @mldangelo
• test(pricing): add missing o3 pricing information (#4400) by @mldangelo
• test(providers): Azure embedding (#4411) & completion (#4410) by @gru-agent
• test(redteam): graders unit tests (#4433, #4455) by @gru-agent
• test(redteam): Aegis plugin unit tests (#4434) by @gru-agent
• test(redteam): memory-poisoning plugin tests (#4453) by @gru-agent

New Contributors

• @AegisAurora made their first contribution in #4369
• @Sly1029 made their first contribution in #4424

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.114.7...0.115.0

- TypeScript
Published by mldangelo 9 months ago

promptfoo - 0.114.7

What's Changed

PR #4215 introduced a bug that broke remote red teams.

• Revert "chore(redteam): add target option to generate command (#4215)" by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4359
• chore: bump version 0.114.7 by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4360

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.114.6...0.114.7

- TypeScript
Published by sklein12 9 months ago

promptfoo - 0.114.6

What's Changed

Features

• feat(redteam): add medical plugins for testing medical anchoring bias by @AISimplyExplained in https://github.com/promptfoo/promptfoo/pull/4196

Fixes

• fix(redteam): include plugin and strategy IDs in report CSV output by @typpo in https://github.com/promptfoo/promptfoo/pull/4347
• fix(webui): reset defaultTest configuration on setup page by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4345

Chores

• chore(redteam): add target option to generate command by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4215
• chore(redteam): update OpenAI model options in redteam setup by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4344
• chore(webui): update OpenAI model options with GPT-4.1 series and o4-mini models in eval-creator by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4350
• chore(deps): bump @aws-sdk/client-bedrock-runtime from 3.823.0 to 3.825.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/4355
• chore(deps): bump openai from 5.1.0 to 5.1.1 by @dependabot in https://github.com/promptfoo/promptfoo/pull/4354
• chore(deps): bump version to 0.114.6 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4357

Documentation

• docs: update getting-started example by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4346

Tests

• test: clean up teardown and setup to remove side effects from tests by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4351

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.114.5...0.114.6

- TypeScript
Published by mldangelo 9 months ago

promptfoo - 0.114.5

What's Changed

Fixes

• fix(redteam): exclude memory poisoning plugin from strategies by @will-holley in https://github.com/promptfoo/promptfoo/pull/4317
• fix(redteam): agent discovered info dark mode by @will-holley in https://github.com/promptfoo/promptfoo/pull/4312
• fix(eval): handle undefined maxConcurrency with proper fallbacks by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4314

Chores

• chore(redteam): update custom policy template and generatedPrompts parser by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/4324
• chore(redteam): add severity levels to redteam plugin objects by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4310
• chore(redteam): store original text for encoding strategies by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4248
• chore(redteam): add emoji encoding strategy by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4263
• chore(cli): terminal cleanup on Ctrl+C by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4313
• chore(providers): improve logging when inheriting from OpenAiChatCompletionProvider by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4320
• chore(tusk): fix tusk test runner workflow configuration by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4328
• chore(tusk): add Tusk test runner workflow for even more unit tests by @use-tusk in https://github.com/promptfoo/promptfoo/pull/4326
• chore(deps): bump @anthropic-ai/sdk from 0.52.0 to 0.53.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/4333
• chore(deps): bump version 0.114.5 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4332

Documentation

• docs(site): add Tabs Fakier as Founding Developer Advocate to team page by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4315

Tests

• test: add unit test for src/redteam/providers/agentic/memoryPoisoning.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/4319
• test: improve test setup and teardown for better isolation by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4331
• test(webui): add telemetry hook tests by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4329

New Contributors

• @use-tusk made their first contribution in https://github.com/promptfoo/promptfoo/pull/4326

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.114.4...0.114.5

- TypeScript
Published by mldangelo 9 months ago

promptfoo - 0.114.4

What's Changed

Fixes

• fix(webui): restore dark mode cell highlighting without breaking status pill visibility by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4300
• fix(redteam): set plugin severity by @will-holley in https://github.com/promptfoo/promptfoo/pull/4303
• fix(redteam): remove empty values from discovery result by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/4295

Chores

• chore(templating): add PROMPTFOODISABLEOBJECT_STRINGIFY environment variable for object template handling by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4297
• chore(cli): improve token usage presentation by @typpo in https://github.com/promptfoo/promptfoo/pull/4294
• chore(providers): add base URL override for Google provider by @yurchik11 in https://github.com/promptfoo/promptfoo/pull/4255
• chore(providers): add custom headers support for Google Gemini by @yurchik11 in https://github.com/promptfoo/promptfoo/pull/4308
• chore(redteam): add tool-discovery:multi-turn alias to tool-discovery by @will-holley in https://github.com/promptfoo/promptfoo/pull/4302
• chore(redteam): remove empty values from discovery result by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/4295
• chore(redteam): improve shell injection attack generation by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/4304
• chore(redteam): update goal extraction logic by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/4285
• chore(webui): add highlight count to eval view by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4249
• chore(deps): bump @aws-sdk/client-bedrock-runtime from 3.821.0 to 3.823.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/4306
• chore(deps): bump openai from 5.0.1 to 5.0.2 by @dependabot in https://github.com/promptfoo/promptfoo/pull/4292
• chore(deps): bump openai from 5.0.2 to 5.1.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/4307
• chore(deps): bump tar-fs from 2.1.2 to 2.1.3 in npmandyarn group by @dependabot in https://github.com/promptfoo/promptfoo/pull/4293
• chore(deps): bump version to 0.114.4 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4309

Documentation

• docs: update GPT-4o to GPT-4.1 references by @typpo in https://github.com/promptfoo/promptfoo/pull/4296
• docs: refresh getting started models section by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4290
• docs: standardize file references to use file:// scheme by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4291
• docs: add descriptions to example configs by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4283
• docs(examples): update model references from gpt-4o-mini to gpt-4.1-mini by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4289

Tests

• test(redteam): add unit test for discover command by @gru-agent in https://github.com/promptfoo/promptfoo/pull/4298

New Contributors

• @yurchik11 made their first contribution in https://github.com/promptfoo/promptfoo/pull/4255

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.114.3...0.114.4

- TypeScript
Published by mldangelo 9 months ago

promptfoo - 0.114.3

What's Changed

Features

• feat: Display audio file variables in result table
  #3864 by @faizanminhas
  
• feat(redteam): Update application definition flow to collect better info
  #4244 by @faizanminhas
  

Fixes

• fix: Resolve model-graded assertion providers from providerMap
  #4273 by @mldangelo
  
• fix: File content not being loaded when referenced with file:// prefix in vars
  #3793 by @adityabharadwaj198
  
• fix: Use array as type for vars
  #4281 by @sklein12
  

Tests

• test: Add unit test for src/globalConfig/accounts.ts
  #4259 by @gru-agent
  
• test: Add unit test for src/util/config/manage.ts
  #4258 by @gru-agent
  
• test: Add vitest coverage for frontend pages
  #4274 by @mldangelo
  
• test: Add unit test for renderVarsInObject formatting
  #4254 by @mldangelo
  
• test: Add unit test for src/redteam/plugins/base.ts
  #4233 by @gru-agent
  
• test: Add unit test for src/redteam/providers/crescendo/index.ts
  #4211
  #4214 by @gru-agent
  
• test: Add unit test for src/redteam/providers/crescendo/prompts.ts
  #4213 by @gru-agent
  

Documentation

• docs: Add job board
  #4264 by @typpo
  
• docs: Add custom policy to sidebar
  #4272 by @typpo
  
• docs: Add native build guidance to troubleshooting section
  #4253 by @mldangelo
  
• docs: Add anchor links to press page section headings
  #4265 by @mldangelo
  
• docs: Add JSON schema to example
  #4276 by @ladyofcode
  
• docs: Add schema header to example configs
  #4277 by @mldangelo
  
• docs: Unify formatting across site
  #4270 by @mldangelo
  

Chores

• chore: Fix open handles in readline tests preventing graceful Jest exit
  #4242 by @mldangelo
  
• chore: Add external file loading support for response_format in OpenAI API
  #4240 by @mldangelo
  
• chore: Always have unique redteam file when running live
  #4237 by @sklein12
  
• chore: Add metadata to generated redteam.yaml
  #4257 by @typpo
  
• chore: Bump openai from 4.103.0 to 5.0.1
  #4250 by @dependabot
  
• chore: Redteam → red team
  #4268 by @typpo
  
• chore: Improve dark mode highlight styling for eval cell views
  #4269 by @mldangelo
  
• chore: Update dependencies to latest minor/patch versions
  #4271 by @mldangelo
  
• chore: Clarify wording
  #4278 by @typpo
  
• chore: Format estimated probes
  #4279 by @typpo
  
• chore: Update grader for malicious code
  #4286 by @MrFlounder
  
• chore: Add back example config to red team create flow
  #4282 by @faizanminhas
  
• chore: Bump version 0.114.3
  #4287 by @sklein12
  
• chore(webui): Hide diff filter option on /eval when single column
  #4246 by @mldangelo
  
• chore(webui): Allow toggling highlight on eval outputs
  #4252 by @mldangelo
  

New Contributors

• @ladyofcode made their first contribution in #4276

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.114.2...0.114.3

- TypeScript
Published by sklein12 9 months ago

promptfoo - 0.114.2

What's Changed

Features

• feat(redteam): Off-Topic Plugin by @AISimplyExplained in https://github.com/promptfoo/promptfoo/pull/4168
• feat(redteam): Set a goal for attacks by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/4217

Fixes

• fix: fix border radius on purpose example by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/4229
• fix: resolve env variables in renderVarsInObject (issue #4143) by @rsullivan05 in https://github.com/promptfoo/promptfoo/pull/4231
• fix: Check if body is good json before sending warning by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4239

Chores

• docs(site): clarify deepseek model aliases and fix configuration examples by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4236
• chore: bump version 0.114.2 by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4241

New Contributors

• @rsullivan05 made their first contribution in https://github.com/promptfoo/promptfoo/pull/4231

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.114.1...0.114.2

- TypeScript
Published by sklein12 9 months ago

promptfoo - 0.114.1

What's Changed

Feature

• feat(redteam): Target Discovery Agent by @will-holley in https://github.com/promptfoo/promptfoo/pull/4203
• feat(providers): add OpenAI MCP (Model Context Protocol) support to Responses API by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4180

Fix

• fix: Relax private key validation by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4216
• fix: Undefined values on red team application purpose page by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4202

Chore

• chore: Add purpose to crescendo prompt by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4212
• chore: Add purpose with goat generation by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4222
• chore: Always include raw output from http provider, status code and status text by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4206
• chore: centralize readline utilities to fix Jest open handle issues by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4219
• chore: move http data to metadata by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4209
• chore(redteam): tight up some graders by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/4210
• chore(redteam): tight up some graders by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/4224
• chore: bump version 0.114.1 by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4228

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.114.0...0.114.1

- TypeScript
Published by sklein12 9 months ago

promptfoo - 0.114.0

What's Changed

Features

• feat(providers): Add xAI image provider by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4130
• feat(cli): add validate command by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4134
• feat(redteam): add camelCase strategy by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4146
• feat: add typed row interfaces for eval queries by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4186
• feat: add goal/intent extraction by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/4178

Fixes

• fix(prompts): splitting when PROMPTFOOPROMPTSEPARATOR is contained within a string with text files by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4142
• fix: isolate proxy vars in bedrock tests by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4181
• fix(docs): Fix issue with docs links not scrolling to the top by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/4195
• fix: when there’s too many intents result won’t render error by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/4175
• fix: need to send auth request to api path by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4199
• fix: Gemini MCP integration - can not parse $schema field by @adelmuursepp in https://github.com/promptfoo/promptfoo/pull/4200

Chores

• chore(redteam): add harmful plugin preset to redteam setup ui by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4132
• chore(redteam): add label strategy-less plugins in redteam setup ui by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4131
• chore(redteam): improve style of redteam purpose field in webui by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4124
• chore(providers): add xai live search support by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4123
• chore(providers): add Claude 4 support to anthropic, bedrock, and vertex providers by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4129
• chore: bump @aws-sdk/client-bedrock-runtime from 3.816.0 to 3.817.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/4164
• chore(providers): update fal provider by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4182
• chore: remove redundant test comments by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4183
• chore: add typed interface for MCP tool schemas by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4187
• chore(redteam): add ToxicChat dataset as redteam plugin by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4121
• chore(webui): add max concurrency as an option for run in browser by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/4147
• chore(app/evals): Adds Agent Discovered Information to Redteam Report by @will-holley in https://github.com/promptfoo/promptfoo/pull/4198
• chore: bump version 0.114.0 by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4201

Docs

• docs(site): minimal copy page button + sanitize text by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4156
• docs(site): scroll to top when using by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4162
• docs: fix DOM nesting warning and sort plugins array by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4174
• docs(site): document missing redteam plugins by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4169
• docs(site): restore scroll-to-top behavior on page navigation by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4176
• docs: iterative jailbreak diagram by @typpo in https://github.com/promptfoo/promptfoo/pull/4191

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.113.4...0.114.0

- TypeScript
Published by sklein12 9 months ago

promptfoo - 0.113.4

What's Changed

Features

• feat: Server-side pagination, filtering and search for eval results table by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4054
• feat: add score to pass/fail in CSV and add json download by @typpo in https://github.com/promptfoo/promptfoo/pull/4153

Fixes

• fix(webui): defaultTest shown in webui YAML editor by @adelmuursepp in https://github.com/promptfoo/promptfoo/pull/4152
• fix: Run red team from UI without email by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4158

Docs

• docs(site): reduce sidebar padding by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4154
• chore: bump version 0.113.4 by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4160

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.113.3...0.113.4

- TypeScript
Published by sklein12 9 months ago

promptfoo - 0.113.3

What's Changed

Fix

• fix: zod error when state.answer has object by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/4136
• fix: use current working directory for redteam file if loading from cloud by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4145
• fix: Throw error on un-supported command - redteam run with a cloud target but no config by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4144
• fix: bias:gender plugin generation by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/4126

Chore:

• chore: bump openai from 4.100.0 to 4.103.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/4140
• chore: bump @aws-sdk/client-bedrock-runtime from 3.812.0 to 3.816.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/4137
• chore: bump @anthropic-ai/sdk from 0.51.0 to 0.52.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/4138
• chore(telemetry): add isRedteam property to telemetry events by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4149
• build: increase build job timeout from 3 to 4 minutes by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4150
• chore: bump version 0.113.3 by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4151

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.113.2...0.113.3

- TypeScript
Published by sklein12 9 months ago

promptfoo - 0.113.2

What's Changed

Fix

• fix: intent grader crescendo by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/4113

Chore

• chore: revert telemtry changes by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4122
• chore(deps): update peer dependencies to latest versions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4125
• chore: bump version 0.113.2 by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4128

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.113.1...0.113.2

- TypeScript
Published by sklein12 9 months ago

promptfoo - 0.113.1

What's Changed

Fixes

• fix(redteam): Skip plugins when validation fails by @faizanminhas in #4101

Chores

• chore(redteam): Target discovery agent by @sklein12 in #4084
• chore(redteam): Add log by @MrFlounder in #4108
• chore(redteam): Update purpose example by @MrFlounder in #4109
• chore(deps): Update Smithy dependencies to latest version by @mldangelo in #4105
• chore(providers): Support templated URLs in HTTP by @mldangelo in #4103
• chore(redteam): Update default REDTEAM_MODEL from 'openai:chat:gpt-4o' to 'openai:chat:gpt-4.1-2025-04-14' by @mldangelo in #4100
• chore(telemetry): Add isRunningInCi flag to telemetry events by @mldangelo in #4115
• chore: Bump version 0.113.1 by @mldangelo in #4116

Docs

• docs: Add enterprise disclaimer to self-hosting by @mldangelo in #4102

Full Changelog: 0.113.0...0.113.1

- TypeScript
Published by mldangelo 9 months ago

promptfoo - 0.113.0

What's Changed

Note: We're removing backwards compatibility for sharing to servers deployed with version < 0.103.8. Please upgrade your self-hosted servers to the latest version.

Fix

• fix: target purpose not making it into redteam config by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4097

Chore

• chore: Remove deprecated sharing setups by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4082

• chore: add vision grading example by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4090

• chore: bump version to 0.113.0 by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4099

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.112.8...0.113.0

- TypeScript
Published by sklein12 9 months ago

promptfoo - 0.112.8

Features

• feat: multilingual combinations by @typpo in https://github.com/promptfoo/promptfoo/pull/4048
• feat: add copy as markdown button to doc pages by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/4039

Fixes

• fix(providers): AI21 response validation by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4052
• fix(redteam): respect cliState.webUI in multilingual progressbar by @typpo in https://github.com/promptfoo/promptfoo/pull/4047
• fix(redteam): fix test count calculation for multiple strategies by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4065
• fix(redteam): replace other-encodings with individual morse and piglatin strategies by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4064
• fix(webui): evaluateOptions removal in YAML editor by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4059
• fix(redteam): fix open handle in video test by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4069
• fix(hooks): add missing results to afterAll hook context by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4071
• fix: telemetry key by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4093

Chores

• chore: bump @anthropic-ai/sdk from 0.50.4 to 0.51.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/4030
• chore: add headers support for url remote mcp servers by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/4018
• chore(providers): Adds support for openai codex-mini-latest by @will-holley in https://github.com/promptfoo/promptfoo/pull/4041
• chore(redteam): improve multilingual strategy performance and reliability by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4055
• chore(providers): update default openai models for openai:chat alias by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4066
• chore: Update prompt suffix help text by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4058
• chore(docs): update model IDs in documentation to reflect latest naming convention by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4046
• chore(deps): update dependencies by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4073
• chore(redteam): introduce strategy collection for other-encodings by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4075
• chore(webui): display currently selected eval in eval dialogue by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/4079
• chore: Improve memory usage when sharing results by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4050
• chore(docs): Handle index.md files for copy page by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/4081
• chore: update Google Sheets fetch to use proxy helper by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4087
• chore: simplify crypto usage in sagemaker provider by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4089
• chore: bump version 0.112.8 by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4095

Documentation

• docs: add curl example for medical agent by @typpo in https://github.com/promptfoo/promptfoo/pull/4049
• docs: update CLI docs by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4063
• docs: standardize code block titles by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4067
• docs(examples): add uniform init commands to all example READMEs by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4068

Tests

• test: add unit test for src/redteam/commands/discover.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/4034
• test: add unit test for src/redteam/commands/generate.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/4036
• test: add unit test for src/providers/ai21.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/4056
• test: add unit test for src/commands/eval.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/4062
• test: add unit test for src/evaluatorHelpers.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/4037

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.112.7...0.112.8

- TypeScript
Published by sklein12 9 months ago

promptfoo - 0.112.7

What's Changed

Features

• feat(redteam): add MCP plugin by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/3989

Bug Fixes

• fix: stringify objects in matcher templates by @typpo in https://github.com/promptfoo/promptfoo/pull/3896
• fix: Azure auth headers get set to null in subclass by @johannestang in https://github.com/promptfoo/promptfoo/pull/4015
• fix: move custom policies into the correct accordion by @typpo in https://github.com/promptfoo/promptfoo/pull/4017
• fix: update return type for task extract-goat-failure by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/4021
• fix(redteam): remove duplicate Datasets section in Plugins component by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4022

Chores

• chore: adjust framework compliance column width by @typpo in https://github.com/promptfoo/promptfoo/pull/4005
• chore: bump @aws-sdk/client-bedrock-runtime from 3.808.0 to 3.810.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/4012
• chore: bump @azure/identity from 4.9.1 to 4.10.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/4013
• chore: bump version 0.112.7 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4023
• chore: exclude response from crescendo if privacy setting is enabled by @sklein12 in https://github.com/promptfoo/promptfoo/pull/4009
• chore: remove accidentally committed example prompt by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4008
• chore: update GOAT implementation by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/4011
• chore: update multilingual description by @typpo in https://github.com/promptfoo/promptfoo/pull/4016
• chore(cli): improve color of Red Team test generation table headers by @will-holley in https://github.com/promptfoo/promptfoo/pull/4004
• chore(redteam): add link to view all logs at top of report by @typpo in https://github.com/promptfoo/promptfoo/pull/4007

Documentation

• docs(blog): Agent2Agent Protocol by @AISimplyExplained in https://github.com/promptfoo/promptfoo/pull/3981
• docs(examples): add OpenAI Agents SDK example by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4006
• docs(usage): update sharing instructions with API key details by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4010

New Contributors

• @johannestang made their first contribution in https://github.com/promptfoo/promptfoo/pull/4015

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.112.6...0.112.7

- TypeScript
Published by mldangelo 10 months ago

promptfoo - 0.112.6

What's Changed

Features

• feat(redteam): add EU AI Act mappings by @typpo in https://github.com/promptfoo/promptfoo/pull/4000
• feat(redteam): add gender bias plugin by @typpo in https://github.com/promptfoo/promptfoo/pull/3886
• feat(eval): add evaluation duration display by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3996

Bug Fixes

• fix: autowrap prompts with partial nunjucks tags by @will-holley in https://github.com/promptfoo/promptfoo/pull/3999
• fix(python): resolve paths relative to promptfooconfig when not cloud config by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/4001

Chores

• chore(providers): improve Perplexity API integration by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3990
• build: add Node.js 24 support by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3941
• chore(redteam): set plugin config type by @sklein12 in https://github.com/promptfoo/promptfoo/pull/3982
• chore(providers): add EU Claude 3.7 Sonnet model to Bedrock by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3998
• chore(redteam): update iterative tree by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/3987
• chore: bump version to 0.112.6 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/4003
• chore(deps): update dependencies by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3985
• refactor: clean up providers for redteam generate by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/3954

Documentation

• docs: add basic enterprise architecture diagram by @typpo in https://github.com/promptfoo/promptfoo/pull/3988
• docs(ci): add Azure pipelines by @typpo in https://github.com/promptfoo/promptfoo/pull/3986
• docs(ci): add Bitbucket and Travis CI by @typpo in https://github.com/promptfoo/promptfoo/pull/3997
• docs(examples): add medical agent example by @typpo in https://github.com/promptfoo/promptfoo/pull/3993
• docs(blog): add truncation marker to MCP blog post by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3984

Tests

• test: add unit test for src/redteam/types.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/3983

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.112.5...0.112.6

- TypeScript
Published by mldangelo 10 months ago

promptfoo - 0.112.5

What's Changed

Fixes

• fix(config): resolve relative paths in combineConfigs by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3942
• fix(evaluator): correctly count named scores based on contributing assertions by @timothyasp in https://github.com/promptfoo/promptfoo/pull/3968
• fix(fetch): no proxy values should take priority in fetch by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/3962
• fix(providers): combine prompt config with provider config for bedrock by @aperezfals in https://github.com/promptfoo/promptfoo/pull/3970
• fix(providers): ensure correct addition for bedrock token counts by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3762
• fix(redteam): crescendo formatting by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/3952
• fix(redteam): pii grader false positives by @typpo in https://github.com/promptfoo/promptfoo/pull/3946
• fix(redteam): shell injection false positives by @typpo in https://github.com/promptfoo/promptfoo/pull/3957
• fix(redteam): add strategy pills and output details to passed tests by @typpo in https://github.com/promptfoo/promptfoo/pull/3961

Chores

• chore(cli): add global env-file option to all commands recursively by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3969
• chore(cli): add global verbose option to all commands recursively by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3950
• chore(cli): better error handling and logging for remote generation by @sklein12 in https://github.com/promptfoo/promptfoo/pull/3965
• chore(cli): better error handling for remote generation by @sklein12 in https://github.com/promptfoo/promptfoo/pull/3956
• revert: "chore: better error handling for remote generation" by @sklein12 in https://github.com/promptfoo/promptfoo/pull/3964
• chore(cli): better response parsing errors by @sklein12 in https://github.com/promptfoo/promptfoo/pull/3955
• chore(cli): revert "feat(cli): adds global --verbose option" by @sklein12 in https://github.com/promptfoo/promptfoo/pull/3945
• chore(providers): add support for Amazon Nova Premier model by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3951
• chore(redteam): improvement, include purpose in iterative attacker prompt by @sklein12 in https://github.com/promptfoo/promptfoo/pull/3948
• chore(redteam): minor changes to category descriptions and ordering by @typpo in https://github.com/promptfoo/promptfoo/pull/3960
• chore(redteam): order attack methods by decreasing ASR by @typpo in https://github.com/promptfoo/promptfoo/pull/3959
• chore(redteam): red teamer two words by @typpo in https://github.com/promptfoo/promptfoo/pull/3976
• chore(deps): bump version 0.112.5 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3980
• chore(deps): sync dependencies by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3971
• chore(deps): update dependencies by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3943
• chore(logger): replace console.error with logger.error in MCPClient by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3944
• chore(providers): add google ai studio embedding provider and improve docs by @prasanna-lakshmi18 in https://github.com/promptfoo/promptfoo/pull/3686
• chore: lint with type info by @will-holley in https://github.com/promptfoo/promptfoo/pull/3932

Docs

• docs: how to create inline assertions for package users by @billybonks in https://github.com/promptfoo/promptfoo/pull/3974
• docs: improve Docusaurus documentation instructions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3977
• docs: instructions on how to run the documentation by @billybonks in https://github.com/promptfoo/promptfoo/pull/3973
• docs: update CLAUDE.md with additional commands and project conventions by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/3972
• docs: update user count from 75,000 to 80,000 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3940
• docs(google-vertex): fix duplicate readme by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3979
• docs(openai): update structured output external schema file example by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3967

Tests

• test: add unit test for src/redteam/plugins/pii.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/3947

New Contributors

• @aperezfals made their first contribution in https://github.com/promptfoo/promptfoo/pull/3970
• @prasanna-lakshmi18 made their first contribution in https://github.com/promptfoo/promptfoo/pull/3686

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.112.4...0.112.5

- TypeScript
Published by mldangelo 10 months ago

promptfoo - 0.112.4

What's Changed

New Features

• feat(assertions): add PI scorer by @sidd2pir in https://github.com/promptfoo/promptfoo/pull/3799
• feat(redteam): add video strategy by @typpo in https://github.com/promptfoo/promptfoo/pull/3820
• feat(evals): optionally time out eval steps by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3765

Fixes

• fix: foreign key error in better-sqlite3 and adapt new transaction API by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3937
• fix(redteam): filter null values in harmful completion provider output by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3908
• fix(python): increase timeout for python path validation by @hansmaaherra in https://github.com/promptfoo/promptfoo/pull/3914
• fix(cli): read .env file prior to calling env var getters by @will-holley in https://github.com/promptfoo/promptfoo/pull/3892

Chores

• chore(cli): add global --verbose option by @will-holley in https://github.com/promptfoo/promptfoo/pull/3931
• chore(redteam): implement agentic plugin UI by @will-holley in https://github.com/promptfoo/promptfoo/pull/3880
• chore(providers): improve error message in http provider transform by @sklein12 in https://github.com/promptfoo/promptfoo/pull/3910
• chore(cloud): improve error messages on cloud requests by @sklein12 in https://github.com/promptfoo/promptfoo/pull/3934
• chore(deps): bump @anthropic-ai/sdk from 0.40.1 to 0.41.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3930
• chore(deps): bump @aws-sdk/client-bedrock-runtime from 3.799.0 to 3.803.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3898
• chore(deps): bump @aws-sdk/client-bedrock-runtime from 3.803.0 to 3.804.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3913
• chore(deps): bump openai from 4.96.2 to 4.97.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3890
• chore: bump version 0.112.4 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3939
• chore(telemetry): implement minor telemetry changes by @will-holley in https://github.com/promptfoo/promptfoo/pull/3895
• chore(telemetry): remove assertion-used event by @sklein12 in https://github.com/promptfoo/promptfoo/pull/3894
• chore(assertions): add throw error option for LLM Rubric if provider doesn't return a result or errors out by @sklein12 in https://github.com/promptfoo/promptfoo/pull/3909
• chore(cli): allow sharing urls with auth credentials by @will-holley in https://github.com/promptfoo/promptfoo/pull/3903
• revert: "chore(cli): allow sharing urls with auth credentials" by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3918

Docs

• docs(http-provider): add documentation about returning object for custom parser by @billybonks in https://github.com/promptfoo/promptfoo/pull/3897
• docs(http-provider): fix missing return statement in HTTP provider example by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3925
• docs(blog): fix scroll to top when linking into blog post by @typpo in https://github.com/promptfoo/promptfoo/pull/3889
• docs(assertions): improve PI scorer documentation by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3924
• docs(redteam): add memory poisoning plugin documentation by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3867
• docs(usage): add information about HTTP Basic Authentication by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3919
• docs(site): fix landing page content jumping on step switch by @will-holley in https://github.com/promptfoo/promptfoo/pull/3891
• docs(blog): add mcp blog by @AISimplyExplained in https://github.com/promptfoo/promptfoo/pull/3893

Refactors

• refactor: improve self hosting environment variable handling by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3920

Tests

• test: add unit test for src/models/eval.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/3904
• test: add unit test for src/python/pythonUtils.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/3915
• test: add unit test for src/redteam/constants.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/3881
• test: fix huggingface dataset tests to mock environment variables by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3936

New Contributors

• @hansmaaherra made their first contribution in https://github.com/promptfoo/promptfoo/pull/3914
• @sidd2pir made their first contribution in https://github.com/promptfoo/promptfoo/pull/3799

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.112.3...0.112.4

- TypeScript
Published by mldangelo 10 months ago

promptfoo - 0.112.3

Release Notes (v0.112.3)

New Features

• Red team: Added memory poisoning plugin (#3785) @will-holley

Bug Fixes

• CLI: Improved progress bar visualization with thread grouping (#3768) @AISimplyExplained

Docs

• Improved red team strategy documentation (#3870) @mldangelo

Chores

• Bumped version to 0.112.2 (#3872) @sklein12
• Bumped version to 0.112.3 (#3877) @sklein12
• Implemented plumbing and prompt enabling customers to use cloud attacker and unified configurations (#3852) @MrFlounder
• Optimized Meteor tests for improved performance (#3869) @mldangelo
• Optimized Nova Sonic tests for improved performance (#3868) @mldangelo
• Retrieve unified config with provider from cloud (#3865) @sklein12
• Dataset plugins now clearly marked in setup UI (#3859) @mldangelo
• Moved maybeLoadFromExternalFile to file.ts (#3851) @benbuzz790

Full Changelog

- TypeScript
Published by sklein12 10 months ago

promptfoo - 0.112.2

What's Changed

Features

• feat(providers): support Google Search grounding #3800
• feat(providers): mcp support for all models that support function calling #3832
• feat(providers): Add support for Amazon nova-sonic #3713

Fixes

• fix: allow escaping of {{ }} placeholders in prompts #3858
• fix: Trim CSV assertion values #3863
• fix(sharing): sharing to self-hosted #3839
• fix(webui): align settings icon to top right in strategy cards #2938

Enhancements / Refactors

• chore(providers): add llama4 support for bedrock #3850
• chore: make custom metrics more obviously clickable #3682
• refactor: colocate fetching evalID #3715
• chore: Respect Max text length for variable cells in results table #3862

Documentation

• docs: updates to grading documentation #3848
• docs(site): improve pricing page #3790
• docs: add false positives #3857

Dependency & Workflow Chores

• chore(workflows): update permissions in GitHub workflows #3849
• chore: bump openai from 4.96.0 to 4.96.2 #3853
• chore: bump vite from 6.2.6 to 6.2.7 #3856
• chore: bump @aws-sdk/client-bedrock-runtime from 3.798.0 to 3.799.0 #3854
• chore: bump @aws-sdk/client-bedrock-runtime from 3.797.0 to 3.798.0 #3843
• chore: bump @anthropic-ai/sdk from 0.40.0 to 0.40.1 #3842
• chore: bump formidable from 3.5.2 to 3.5.4 #3845

Full Changelog: 0.112.1...0.112.2

- TypeScript
Published by will-holley 10 months ago

promptfoo - 0.112.1

What's Changed

Chores

• chore: set telemetry key by @sklein12 in https://github.com/promptfoo/promptfoo/pull/3838
• chore: improve chunking by @sklein12 in https://github.com/promptfoo/promptfoo/pull/3846

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.112.0...0.112.1

- TypeScript
Published by sklein12 10 months ago

promptfoo - 0.112.0

What's Changed

New Features

• feat: persist search in url by @billybonks in https://github.com/promptfoo/promptfoo/pull/3717
• feat(env): allow every env variable to be overridden within the env block in a promptfoo config by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3786
• feat: METEOR score by @adityabharadwaj198 in https://github.com/promptfoo/promptfoo/pull/3776
• feat(redteam): homoglyph strategy by @typpo in https://github.com/promptfoo/promptfoo/pull/3811
• feat: enable custom response parser to optionally return provider response by @billybonks in https://github.com/promptfoo/promptfoo/pull/3824
• feat(redteam): add more encodings by @typpo in https://github.com/promptfoo/promptfoo/pull/3815
• feat(providers): add cerebras provider by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3814

Fixes

• fix: update dependencies to address npm audit issues by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3791
• fix(provider): OpenAI Realtime history issue by @AISimplyExplained in https://github.com/promptfoo/promptfoo/pull/3719
• fix: accordion positioning in plugins view by @typpo in https://github.com/promptfoo/promptfoo/pull/3807
• fix(matchers): score results correctly with trailing newlines. by @philnash in https://github.com/promptfoo/promptfoo/pull/3823
• fix: results api returns elements ordered by date by @billybonks in https://github.com/promptfoo/promptfoo/pull/3826
• fix(webui): overlapping text results pill on narrow screens by @typpo in https://github.com/promptfoo/promptfoo/pull/3831
• fix(build): add missing strategy entries for build by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3836

Chores

• chore: write static plugin severity to metadata by @typpo in https://github.com/promptfoo/promptfoo/pull/3783
• chore: respect redteam commandLineOptions from config by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/3782
• chore: update telemetry endpoint by @sklein12 in https://github.com/promptfoo/promptfoo/pull/3751
• chore: add cloud log in link by @sklein12 in https://github.com/promptfoo/promptfoo/pull/3787
• chore: bump h11 from 0.14.0 to 0.16.0 in /examples/python-provider in the pip group across 1 directory by @dependabot in https://github.com/promptfoo/promptfoo/pull/3794
• chore: bump openai from 4.95.1 to 4.96.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3792
• chore: bump h11 from 0.14.0 to 0.16.0 in /examples/redteam-langchain in the pip group across 1 directory by @dependabot in https://github.com/promptfoo/promptfoo/pull/3796
• chore: add target option to cli redteam run by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/3795
• chore: bump @aws-sdk/client-bedrock-runtime from 3.787.0 to 3.796.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3802
• refactor: remove if string check by @billybonks in https://github.com/promptfoo/promptfoo/pull/3801 (Refactor categorized as chore)
• chore(deps): update react-router-dom to v7.5.2 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3803
• chore: add info banner for community red teams by @typpo in https://github.com/promptfoo/promptfoo/pull/3809
• chore(examples): remove moderation assertions from foundation model redteam by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3804
• chore(deps): move 'natural' to peer dependency by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3813
• refactor: remove unused datasetGenerationProvider in favor of synthesizeProvider by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3818 (Refactor categorized as chore)
• chore: resolve relative provider paths from cloud configs by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/3805
• chore: bump @aws-sdk/client-bedrock-runtime from 3.796.0 to 3.797.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3829
• chore: bump @anthropic-ai/sdk from 0.39.0 to 0.40.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3828
• chore: bump version 0.112.0 by @typpo in https://github.com/promptfoo/promptfoo/pull/3844

Documentation

• docs: donotanswer example by @typpo in https://github.com/promptfoo/promptfoo/pull/3780
• docs(plugins): harmful:bias => bias name correction by @will-holley in https://github.com/promptfoo/promptfoo/pull/3731
• docs: "red team" two words by @typpo in https://github.com/promptfoo/promptfoo/pull/3798
• docs: add self-hosting caveats by @typpo in https://github.com/promptfoo/promptfoo/pull/3808
• docs: add CLAUDE.md by @typpo in https://github.com/promptfoo/promptfoo/pull/3810
• docs(vertex): put setup and config at the top by @typpo in https://github.com/promptfoo/promptfoo/pull/3830
• docs(site): add redirect from /docs to /docs/intro by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3837

Tests

• test: add unit test for src/redteam/plugins/xstest.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/3779
• test: add unit test for src/models/eval.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/3827

New Contributors

• @billybonks made their first contribution in https://github.com/promptfoo/promptfoo/pull/3717
• @philnash made their first contribution in https://github.com/promptfoo/promptfoo/pull/3823

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.111.1...0.112.0

- TypeScript
Published by typpo 10 months ago

promptfoo - 0.111.1

What's Changed

Fixes

• fix(redteam): correct the URL format in XSTest plugin by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3777

Chores

• chore(deps): bump @azure/identity from 4.9.0 to 4.9.1 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3775
• chore(release): bump version to 0.111.1 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3778
• chore(ui): capitalize "UI" in text by @typpo in https://github.com/promptfoo/promptfoo/pull/3773

Documentation

• docs(about): add Ben Shipley to team section by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3758

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.111.0...0.111.1

- TypeScript
Published by mldangelo 10 months ago

promptfoo - 0.111.0

What's Changed

New Features

• feat(grading): update OpenAI grading model to GPT-4.1 by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/3741
• feat(assertions): modify LLM Rubric rubricPrompt rendering to support arbitrary objects by @GICodeWarrior in https://github.com/promptfoo/promptfoo/pull/3746
• feat(redteam): add donotanswer plugin by @typpo in https://github.com/promptfoo/promptfoo/pull/3754
• feat(redteam): add xstest plugin by @typpo in https://github.com/promptfoo/promptfoo/pull/3771
• feat(webui): add anchor link to specific row and show on top by @irene1391 in https://github.com/promptfoo/promptfoo/pull/1582

Chores

• chore!(redteam): default to outputting generated Redteam config in same dir as input config by @will-holley in https://github.com/promptfoo/promptfoo/pull/3721
• chore(providers): add support for gemini-2.5-flash by @AISimplyExplained in https://github.com/promptfoo/promptfoo/pull/3747
• chore: use ajv with formats everywhere by @schipiga in https://github.com/promptfoo/promptfoo/pull/3716
• chore(cli): improve readline handling and tests by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3763
• chore(eval): add warning for redteam config without test cases by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3740
• chore(providers): increase max output tokens for google:gemini-2.5-pro-exp-03-25 to 2048 in Gemini example by @will-holley in https://github.com/promptfoo/promptfoo/pull/3753
• chore(redteam): add canGenerateRemote property to redteam plugins by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3761
• chore(webui): improve Eval Quick Selector (cmd+k) by @will-holley in https://github.com/promptfoo/promptfoo/pull/3742
• chore(deps): bump @azure/identity from 4.8.0 to 4.9.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3737
• chore(deps): bump openai from 4.94.0 to 4.95.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3736
• chore(deps): bump openai from 4.95.0 to 4.95.1 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3766
• chore: bump version to 0.111.0 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3772

Documentation

• docs: update homepage by @typpo in https://github.com/promptfoo/promptfoo/pull/3733
• docs(redteam): add donotanswer to sidebar and plugins list by @typpo in https://github.com/promptfoo/promptfoo/pull/3767
• docs(redteam): add isRemote to all harmful plugins by @typpo in https://github.com/promptfoo/promptfoo/pull/3769
• docs(providers): update model IDs to latest versions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3770
• docs(about): add Asmi Gulati to team section by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3760
• docs(about): add Matthew Bou to team section by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3759

Tests

• test: add unit test for src/redteam/plugins/donotanswer.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/3755

New Contributors

• @irene1391 made their first contribution in https://github.com/promptfoo/promptfoo/pull/1582

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.110.1...0.111.0

- TypeScript
Published by mldangelo 10 months ago

promptfoo - 0.110.1

What's Changed

Features

• feat(openai): add support for GPT-4.1 model by @mldangelo
  
• feat(openai): add support for o4-mini reasoning model by @mldangelo
  
• feat: Change pass rate to ASR and add export in report by @sklein12
  

Bug Fixes

• fix: Update prompt extraction to work in more scenarios without providing a prompt by @sklein12
  
• fix(providers): output json rather than string from google live provider by @abrayne
  
• fix: google is valid function call allow property_ordering field in tool schema by @abrayne
  
• fix(cli): Use correct url for sharing validation by @will-holley
  
• fix: settings positioning in strategies view by @typpo
  
• fix(cli/redteam/poison): Write docs to the output dir by @will-holley
  
• fix(evaluator): handle prompt rendering errors gracefully by @mldangelo
  
• fix: stricter test for null or undefined in transform response by @typpo
  

Chores

• chore(dependencies): update dependencies to latest versions by @mldangelo
  
• chore: rename owasp plugin presets by @typpo
  
• chore: expand frameworks section by @typpo
  
• chore(self-hosting): update self-hosting instructions by @mldangelo
  
• chore: bump openai from 4.93.0 to 4.94.0 by @dependabot
  
• chore(cli): When sharing, show auth-gate prior to re-share confirmation by @will-holley
  
• chore: email verification analytics by @sklein12
  
• chore(cli): improves robustness of hasEvalBeenShared util by @will-holley
  
• chore: easily remove plugins/strats from review page by @typpo
  
• chore: bump the npmandyarn group with 2 updates by @dependabot
  
• chore(cli): Health check API before running Redteam by @will-holley
  
• chore: make strategies configurable where applicable by @typpo
  
• chore: remove moderation assertions from foundation model redteam example by @mldangelo
  
• chore(cli): Improve description of Redteam run command by @will-holley
  
• chore: better parsing by @MrFlounder
  

Documentation

• docs: add owasp selection image by @typpo
  
• docs(sharing): add troubleshooting section for upload issues by @mldangelo
  
• docs: best-of-n documentation fixes by @typpo
  

Performance

• perf(webui): Reduce memory usage of eval results by @will-holley
  

Refactors

• refactor: update export syntax for functions by @mldangelo
  

Tests

• test: add unit test for src/providers/google/util.ts by @gru-agent
  
• test: add unit test for src/redteam/commands/poison.ts by @gru-agent
  

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.110.0...0.110.1

- TypeScript
Published by typpo 11 months ago

promptfoo - 0.110.0

What's Changed

Features

• feat(assertions): add GLEU metric by @adityabharadwaj198 in https://github.com/promptfoo/promptfoo/pull/3674
• feat(providers): add Grok-3 support by @AISimplyExplained in https://github.com/promptfoo/promptfoo/pull/3663
• feat(providers): add support for AWS Bedrock Knowledge Base by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3576

Bug Fixes

• fix(auth): remove deprecated login flow by @sklein12 in https://github.com/promptfoo/promptfoo/pull/3650
• fix(evals): implement sharing idempotence by @will-holley in https://github.com/promptfoo/promptfoo/pull/3653
• fix(huggingface): disable var expansion for huggingface datasets to prevent array field expansion by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3687
• fix(logger): resolve [Object object] empty string error by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3638
• fix(providers): address scenario where type refers to function field rather than schema type by @abrayne in https://github.com/promptfoo/promptfoo/pull/3647
• fix(providers): handle transformRequest for Raw HTTP by @AetherPrior in https://github.com/promptfoo/promptfoo/pull/3665
• fix(providers): resolve Google Vertex AI output format by @abrayne in https://github.com/promptfoo/promptfoo/pull/3660
• fix(providers): support gemini system_instruction prompt format by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3672
• fix(share): add backward compatibility for '-y' flag by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3640
• fix(share): ensure promptfoo share respects sharing config from promptfooconfig.yaml by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3668
• fix(testCaseReader): make JSON test file parsing preserve test case structure by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3651
• fix(webui): fix eval comparison mode filter by @will-holley in https://github.com/promptfoo/promptfoo/pull/3671
• fix: correct formatting issues by @typpo in https://github.com/promptfoo/promptfoo/pull/3688

Chores

• chore(webui): add X to report drawer by @typpo in https://github.com/promptfoo/promptfoo/pull/3680
• chore(share): improve error message on sharing by @sklein12 in https://github.com/promptfoo/promptfoo/pull/3654
• chore(redteam): implement reset button for strategies by @typpo in https://github.com/promptfoo/promptfoo/pull/3684
• chore(report): make eval output text expansion clearer by @typpo in https://github.com/promptfoo/promptfoo/pull/3681
• chore(report): make it clearer that plugins on the report can be clicked by @typpo in https://github.com/promptfoo/promptfoo/pull/3683
• chore(webui): change model to target in report view by @typpo in https://github.com/promptfoo/promptfoo/pull/3646
• chore(strategies): update Large preset strategies by @typpo in https://github.com/promptfoo/promptfoo/pull/3675
• chore(docker): update base images to Node.js 22 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3666
• chore(redteam): make audio strategy remote-only by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3618
• chore(redteam): remove stale check for buildDate when fetching a config from cloud by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/3658
• chore(deps): bump @aws-sdk/client-bedrock-runtime from 3.784.0 to 3.785.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3644
• chore(deps): bump @aws-sdk/client-bedrock-runtime from 3.785.0 to 3.787.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3670
• chore(deps): bump openai from 4.92.1 to 4.93.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3643
• chore(deps): bump vite from 6.2.5 to 6.2.6 in the npmandyarn group by @dependabot in https://github.com/promptfoo/promptfoo/pull/3677

Documentation

• docs(nav): add lm security db to nav by @typpo in https://github.com/promptfoo/promptfoo/pull/3690
• docs(blog): add interactive blog on invisible Unicode threats by @AISimplyExplained in https://github.com/promptfoo/promptfoo/pull/3621
• docs: improve styles on nav buttons by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3637
• docs: update user count to 75,000+ by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3662

Refactors

• refactor: change multimodal live to live by @abrayne in https://github.com/promptfoo/promptfoo/pull/3657
• refactor(util): consolidate tool loading and rendering by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3642
• test: add unit test for src/commands/auth.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/3652

New Contributors

• @AetherPrior made their first contribution in https://github.com/promptfoo/promptfoo/pull/3665
• @adityabharadwaj198 made their first contribution in https://github.com/promptfoo/promptfoo/pull/3674

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.109.1...0.110.0

- TypeScript
Published by mldangelo 11 months ago

promptfoo - 0.109.1

What's Changed

Bug Fixes

• fix(assertions): handle both string and object outputs from llm-rubric providers by @AISimplyExplained in https://github.com/promptfoo/promptfoo/pull/3624
• fix(assertions): fix google is-valid-function-call by @abrayne in https://github.com/promptfoo/promptfoo/pull/3625
• fix(eval): handle providers array with file references to multiple providers by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3617

Chores

• chore(schema): make extensions field nullable by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3611
• chore(webui): add multi-turn tool discovery to UI by @AISimplyExplained in https://github.com/promptfoo/promptfoo/pull/3622
• chore(scripts): ensure GitHub CLI is installed in preversion by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3614
• chore(deps): bump @aws-sdk/client-bedrock-runtime from 3.782.0 to 3.784.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3619
• chore(deps): bump openai from 4.91.1 to 4.92.1 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3620
• chore(deps): update dependencies to resolve vulnerabilities by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3631

Documentation

• docs(contributing): add guidance on adding a new assertion by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3610
• docs(enterprise): add enterprise documentation by @vsauter in https://github.com/promptfoo/promptfoo/pull/3596
• docs(moderation): update moderation documentation for LlamaGuard 3 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3630
• docs(providers): clarify AWS Bedrock credential resolution order by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3633
• docs(providers): improve Lambda Labs documentation by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3615

Refactoring

• refactor(share): improve formatting of cloud sharing instructions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3628
• refactor(tests): consolidate and reorganize test files by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3616

Testing

• test(providers): add unit test for src/providers/google/util.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/3626

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.109.0...0.109.1

- TypeScript
Published by mldangelo 11 months ago

promptfoo - 0.109.0

What's Changed

Features

• feat(eval): track assertion tokens in token usage by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3551
• feat(plugins): add CCA plugin with documentation and grader by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/3590
• feat(providers): add Google valid function call support by @abrayne in https://github.com/promptfoo/promptfoo/pull/3605
• feat(providers): add Lambda Labs integration by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3601
• feat(webui): add pass rate column by @will-holley in https://github.com/promptfoo/promptfoo/pull/3580

Bug Fixes

• fix(dataset): resolve issue when generating a dataset without a providers key in configuration by @will-holley in https://github.com/promptfoo/promptfoo/pull/3603
• fix(server): prevent server crash when unknown model is selected by @schipiga in https://github.com/promptfoo/promptfoo/pull/3593

Chores

• chore(api): prefix API routes with /api/v1/ by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3587
• chore(evals): remove print option from evals data grid by @will-holley in https://github.com/promptfoo/promptfoo/pull/3595
• chore(webui): update provider selector in create eval page by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3597
• chore(deps): bump vite from 6.2.4 to 6.2.5 in the npmandyarn group by @dependabot in https://github.com/promptfoo/promptfoo/pull/3594
• chore(deps): bump @aws-sdk/client-bedrock-runtime from 3.779.0 to 3.782.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3592

Documentation

• docs(plugins): add llms.txt plugin and convert config to TypeScript by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3600
• docs(plugins): remove duplicate plugins in list by @typpo in https://github.com/promptfoo/promptfoo/pull/3599
• docs(providers): add Llama 4 model details by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3598
• docs(self-hosting): clarify configuration and sharing options by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3591

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.108.0...0.109.0

- TypeScript
Published by faizanminhas 11 months ago

promptfoo - 0.108.0

What's Changed

New Features

• feat(sharing): migrate sharing to promptfoo.app by @sklein12 in https://github.com/promptfoo/promptfoo/pull/3572
• feat(providers): add Google AI Studio tool use by @abrayne in https://github.com/promptfoo/promptfoo/pull/3564
• feat(providers): add promptfoo model endpoint by @typpo in https://github.com/promptfoo/promptfoo/pull/3534
• feat(providers): implement Google Live mock stateful API by @abrayne in https://github.com/promptfoo/promptfoo/pull/3500
• feat(redteam): add multi-turn tool discovery plugin by @AISimplyExplained in https://github.com/promptfoo/promptfoo/pull/3448
• feat(dataset-generation): output generated datasets as CSV by @will-holley in https://github.com/promptfoo/promptfoo/pull/3573

Bug Fixes

• fix(providers): make AIStudio & Live handle system prompts as thoroughly as vertex by @abrayne in https://github.com/promptfoo/promptfoo/pull/3588
• fix(providers): enable Google to load tools from vars by @abrayne in https://github.com/promptfoo/promptfoo/pull/3579
• fix(csv): update CSV docs and trim whitespace for keys in CSV test files by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/3571

Chores

• chore(redteam): add OWASP red team mappings by @typpo in https://github.com/promptfoo/promptfoo/pull/3581
• chore(webui): link URLs in metadata by @typpo in https://github.com/promptfoo/promptfoo/pull/3569
• chore(webui): use datagrids for Prompts, Datasets, and History by @will-holley in https://github.com/promptfoo/promptfoo/pull/3556
• chore(build): split test and build jobs for faster CI workflow by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3586

Dependencies

• chore(deps): bump @aws-sdk/client-bedrock-runtime from 3.778.0 to 3.779.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3563
• chore(deps): bump openai from 4.90.0 to 4.91.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3562
• chore(deps): bump openai from 4.91.0 to 4.91.1 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3577
• chore(deps): update jspdf and dompurify dependencies by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3585
• chore(deps): update to vite 6 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3584
• chore: 0.108.0 by @will-holley in https://github.com/promptfoo/promptfoo/pull/3589

Documentation

• docs: add link to API reference by @typpo in https://github.com/promptfoo/promptfoo/pull/3583
• docs: add screenshot by @typpo in https://github.com/promptfoo/promptfoo/pull/3582
• docs: update docs around Google tools and rename multimodal live by @abrayne in https://github.com/promptfoo/promptfoo/pull/3578
• docs(azure): add guidance on configuring DeepSeek models by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3559

Refactor

• refactor: rename vertexUtil to util and Google provider to AIS provider by @abrayne in https://github.com/promptfoo/promptfoo/pull/3567

Tests

• test: add unit test for src/commands/generate/dataset.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/3575

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.107.7...0.108.0

- TypeScript
Published by will-holley 11 months ago

promptfoo - 0.107.7

What's Changed

Features

• feat(evals): add evals index page by @will-holley in https://github.com/promptfoo/promptfoo/pull/3554
• feat(guardrails): implement adaptive prompting guardrails by @typpo in https://github.com/promptfoo/promptfoo/pull/3536
• feat(prompts): add support for loading prompts from CSV files by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3542
• feat(providers): load arbitrary files in nested configs in python provider by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3540
• feat(redteam): add UnsafeBench plugin for testing unsafe image handling by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3422

Fixes

• fix(assertions): include reason in python score threshold message by @typpo in https://github.com/promptfoo/promptfoo/pull/3528
• fix(assertions): log all reasons in g-eval by @schipiga in https://github.com/promptfoo/promptfoo/pull/3522
• fix(datasets): add support for jsonl test cases by @will-holley in https://github.com/promptfoo/promptfoo/pull/3533
• fix(http): template strings directly in url by @typpo in https://github.com/promptfoo/promptfoo/pull/3525
• fix(providers): add logging and fix custom python provider caching by @Mikkolehtimaki in https://github.com/promptfoo/promptfoo/pull/3507
• fix(redteam): correct tool count by @typpo in https://github.com/promptfoo/promptfoo/pull/3557
• fix(webui): handle : characters better in metadata search by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3530

Chores

• chore: fix type of Prompt to use omit by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/3526
• chore: hide navbar during report PDF generation by @typpo in https://github.com/promptfoo/promptfoo/pull/3558
• chore(dependencies): update package dependencies to latest versions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3544

Documentation

• docs: add openapi reference page by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/3550
• docs: add foundation model guide by @vsauter in https://github.com/promptfoo/promptfoo/pull/3531
• docs: rename guide by @typpo in https://github.com/promptfoo/promptfoo/pull/3546
• docs: update multi modal guide by @typpo in https://github.com/promptfoo/promptfoo/pull/3547
• docs(azure-example): update assistant prompts and test cases by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3529
• docs(red-team): add metadata to foundation models guide by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3532
• docs(sagemaker): improve documentation by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3539
• docs(troubleshooting): add guidance for better-sqlite version mismatch by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3537

Refactors

• refactor: improve google types by @abrayne in https://github.com/promptfoo/promptfoo/pull/3549
• refactor: unify google apis by @abrayne in https://github.com/promptfoo/promptfoo/pull/3548

Tests

• test: add unit test for src/python/pythonUtils.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/3508

Dependencies

• chore: bump @aws-sdk/client-bedrock-runtime from 3.775.0 to 3.777.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3521
• chore: bump @aws-sdk/client-bedrock-runtime from 3.777.0 to 3.778.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3541
• chore: bump openai from 4.89.1 to 4.90.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3520
• chore: bump version 0.107.7 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3560
• chore: bump vite from 5.4.15 to 5.4.16 in the npmandyarn group by @dependabot in https://github.com/promptfoo/promptfoo/pull/3555

New Contributors

• @Mikkolehtimaki made their first contribution in https://github.com/promptfoo/promptfoo/pull/3507

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.107.6...0.107.7

- TypeScript
Published by mldangelo 11 months ago

promptfoo - 0.107.6

What's Changed

• feat(providers): add support for Amazon SageMaker by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3413
• feat: litellm provider by @typpo in https://github.com/promptfoo/promptfoo/pull/3517
• fix(providers): support token counting for every major type of bedrock model by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3506
• fix(env): add override option to dotenv.config for --env-file support by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3502
• fix: handle circular provider references by @typpo in https://github.com/promptfoo/promptfoo/pull/3511
• chore: bump openai from 4.89.0 to 4.89.1 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3509
• chore(blog): improve pagination and post grid UI by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3504
• chore: add support for apiKeyRequired in openai provider by @typpo in https://github.com/promptfoo/promptfoo/pull/3513
• chore: bump version 0.107.6 by @typpo in https://github.com/promptfoo/promptfoo/pull/3519
• docs: owasp red teaming guide by @vsauter in https://github.com/promptfoo/promptfoo/pull/3101

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.107.5...0.107.6

- TypeScript
Published by typpo 11 months ago

promptfoo - 0.107.5

What's Changed

Features

• feat(csv): add CSV metadata column support with array values by @devin-ai-integration in https://github.com/promptfoo/promptfoo/pull/2709

Bug Fixes

• fix(togetherai): ensure max_tokens is respected in configuration by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3468
• fix(providers): handle malformed response in a21 by @typpo in https://github.com/promptfoo/promptfoo/pull/3465
• fix(csv): newlines in CSVs by @typpo in https://github.com/promptfoo/promptfoo/pull/3459
• fix(providers): simulated user bugs by @typpo in https://github.com/promptfoo/promptfoo/pull/3463
• fix(assertions): replace logical OR with nullish coalescing for thresholds by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3486
• fix(redteam): filter out template variables in entity extraction by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3476
• fix(redteam): type of ALL_STRATEGIES to be as const by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/3494

Chores

• chore: add filepaths to debug output by @typpo in https://github.com/promptfoo/promptfoo/pull/3464
• chore: remove generate test cases button from UI by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/3475
• chore(content): update user statistics by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3460
• chore(deps): update dependencies to latest versions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3453
• chore(providers): add support and docs for gemini 2.5 pro to Google Chat Provider by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/3485
• chore(providers): support refusal and JSON schemas in openai responses api by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3456
• chore(providers): update openai model costs and add missing models by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3454
• chore(redteam): add a PlinyGrader to more accurately grade Pliny results by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/3478
• chore: bump @aws-sdk/client-bedrock-runtime from 3.758.0 to 3.772.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3452
• chore: bump @aws-sdk/client-bedrock-runtime from 3.772.0 to 3.774.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3482
• chore: bump @aws-sdk/client-bedrock-runtime from 3.774.0 to 3.775.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3498
• chore: bump openai from 4.88.0 to 4.89.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3451
• chore: bump version 0.107.5 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3505
• chore: bump vite from 5.4.14 to 5.4.15 in the npmandyarn group by @dependabot in https://github.com/promptfoo/promptfoo/pull/3483

Docs

• docs: ensure consistent redteam flag usage in guides by @will-holley in https://github.com/promptfoo/promptfoo/pull/3477
• docs: reduce size of profile pic by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/3484
• docs(contributing): enhance contributing guide with additional details and formatting by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3457
• docs(examples): improve instructions for running 4o vs. 4o mini example by @will-holley in https://github.com/promptfoo/promptfoo/pull/3474
• docs(multilingual): improve multilingual strategy documentation by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3487
• docs(readme): improve README formatting and add new sections by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3461
• docs(security): add security policy by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3470
• docs(site): add Faizan to team page by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/3473
• docs(site): add will to team page by @will-holley in https://github.com/promptfoo/promptfoo/pull/3472

Tests

• test: add unit test for src/app/src/pages/redteam/setup/components/strategies/utils.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/3495
• test: add unit test for src/providers/openai/util.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/3455

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.107.4...0.107.5

- TypeScript
Published by mldangelo 11 months ago

promptfoo - 0.107.4

What's Changed

Features

• feat(providers): Added support for OpenAI Responses API by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3440

Fixes

• fix(Dockerfile): Created .promptfoo directory in Dockerfile and removed initContainer by @zewelor in https://github.com/promptfoo/promptfoo/pull/3435
• fix(providers): Fixed caching behavior for Azure assistants by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3443
• fix(providers): Resolved Go provider CallApi redeclaration issue by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/3414
• fix(redteam): Added missing constants for RAG poisoning plugin by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3375

Chores

• chore(dependencies): Bumped OpenAI from 4.87.4 to 4.88.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3436
• chore(webui): Included error message in toast by @typpo in https://github.com/promptfoo/promptfoo/pull/3437
• chore(providers): Added o1-pro by @typpo in https://github.com/promptfoo/promptfoo/pull/3438
• chore(scripts): Specified repository for postversion PR creation by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3432

Documentation

• docs(blog): Added misinformation blog post by @vsauter in https://github.com/promptfoo/promptfoo/pull/3433
• docs(examples): Added redteam-azure-assistant example by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3446
• docs(redteam): Added guidance on purpose for image redteams by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3444
• docs(redteam): Created guides section under red teaming by @typpo in https://github.com/promptfoo/promptfoo/pull/3445
• docs(site): Added responsible disclosure policy by @typpo in https://github.com/promptfoo/promptfoo/pull/3434

Tests

• test: Added unit test for src/evaluatorHelpers.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/3430

New Contributors

• @zewelor made their first contribution in https://github.com/promptfoo/promptfoo/pull/3435

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.107.3...0.107.4

- TypeScript
Published by mldangelo 11 months ago

promptfoo - 0.107.3

What's Changed

Chores

• chore(providers): improve Azure Assistant integration by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3424
• chore(providers): add Google multimodal live function callbacks by @abrayne in https://github.com/promptfoo/promptfoo/pull/3421
• refactor(providers): split Azure provider into multiple files and update model pricing by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3425
• chore(deps): bump openai from 4.87.3 to 4.87.4 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3428

Docs

• docs: add multi-modal redteam example by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3416

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.107.2...0.107.3

- TypeScript
Published by mldangelo 12 months ago

promptfoo - 0.107.2

What's Changed

New Features

• feat(assertions): update factuality grading prompt to improve compatibility across many different providers by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3408
• feat(providers): add support for OpenAI Realtime API by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3383
• feat(providers): update default Anthropic providers to latest version by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3388

Bug Fixes

• fix(providers): update Bedrock output method signature by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3409
• fix(redteam): correct strategyId for jailbreak by @typpo in https://github.com/promptfoo/promptfoo/pull/3399

Chores

• chore(cli): set PROMPTFOOINSECURESSL to true by default by @typpo in https://github.com/promptfoo/promptfoo/pull/3397
• chore(webui): add success filter mode by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/3387
• chore(webui): add more copying options in EvalOutputPromptDialog by @typpo in https://github.com/promptfoo/promptfoo/pull/3379
• chore(onboarding): update presets by @typpo in https://github.com/promptfoo/promptfoo/pull/3411
• chore(auth): improve login text formatting by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3389
• chore(init): add fallback to 'main' branch for example fetching by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3417
• chore(prompts): remove unused prompts from grading.ts by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3407
• chore(redteam): update entity extraction prompt by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3405
• chore(deps): update dependencies to latest stable versions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3385

Refactor

• refactor(providers): split Anthropic provider into modular components by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3406

Documentation

• docs(blog): add data poisoning article by @vsauter in https://github.com/promptfoo/promptfoo/pull/2566
• docs(examples): update Amazon Bedrock provider documentation by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3401
• docs(guides): add documentation on testing guardrails by @typpo in https://github.com/promptfoo/promptfoo/pull/3403
• docs(guides): add more content on agent and RAG testing by @typpo in https://github.com/promptfoo/promptfoo/pull/3412
• docs(providers): update AWS Bedrock documentation with Nova details by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3395
• docs(redteam): remove duplicate plugin entry by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3393
• docs(redteam): update examples by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3394
• docs(style): introduce a cursor rule for documentation and do some cleanup by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3404

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.107.0...0.107.2

- TypeScript
Published by mldangelo 12 months ago

promptfoo - 0.107.1

What's Changed

• chore: more copying options in EvalOutputPromptDialog by @typpo in https://github.com/promptfoo/promptfoo/pull/3379
• chore: add filter mode by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/3387
• chore(providers): update default Anthropic providers to latest version by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3388
• chore(auth): improve login text formatting by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3389
• chore(deps): update dependencies to latest stable versions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3385
• chore: PROMPTFOOINSECURESSL true by default by @typpo in https://github.com/promptfoo/promptfoo/pull/3397
• chore: bump version 0.107.1 by @typpo in https://github.com/promptfoo/promptfoo/pull/3398
• docs(redteam): remove duplicate plugin entry by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3393
• docs: update redteam examples by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3394

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.107.0...0.107.1

- TypeScript
Published by typpo 12 months ago

promptfoo - 0.107.0

What's Changed

New Features

• feat(cli): Add model-scan command by @typpo in https://github.com/promptfoo/promptfoo/pull/3323
• feat(webui): Add metadata filtering in ResultsTable by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3368
• feat(providers): Add multi-modal live sequential function calls by @abrayne in https://github.com/promptfoo/promptfoo/pull/3345
• feat(server): Load dotenv file when starting server by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3321
• feat(redteam): Add audio strategy by @typpo in https://github.com/promptfoo/promptfoo/pull/3347
• feat(redteam): Add convert to image strategy by @typpo in https://github.com/promptfoo/promptfoo/pull/3342
• feat(webui): Add download failed tests dialog by @devin-ai-integration in https://github.com/promptfoo/promptfoo/pull/3327

Bug Fixes

• fix(evaluator): Merge test case metadata with provider response metadata by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3344
• fix(redteam): Include assertion in remote grading result by @typpo in https://github.com/promptfoo/promptfoo/pull/3349
• fix(providers): Fix environment variable substitution in HTTP provider headers by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3335
• fix(redteam): Update moderation flag default and adjust test case metadata by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3377
• fix(share): Correct URL display when self-hosting by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3312
• fix(webui): Fix missing plugins in report view by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3356

Chores

• chore(providers): Add Bedrock support for DeepSeek by @typpo in https://github.com/promptfoo/promptfoo/pull/3363
• chore(docs): Add Cursor AI rules for development workflow by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3326
• chore(webui): Sync custom policies UI changes from promptfoo-cloud by @devin-ai-integration in https://github.com/promptfoo/promptfoo/pull/3257
• chore(redteam): Make image jailbreak strategy runnable by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3361
• chore(redteam): Add missing audio and image descriptions by @typpo in https://github.com/promptfoo/promptfoo/pull/3372
• chore(webui): Improve keyboard shortcut order in DownloadMenu by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3330
• chore(error): Improve malformed target response error message by @typpo in https://github.com/promptfoo/promptfoo/pull/3341
• chore(prompts): Support j2 files by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3338
• chore(providers): Add missing Bedrock models by @typpo in https://github.com/promptfoo/promptfoo/pull/3362
• chore(providers): Improve support for Azure reasoning models and update documentation by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3332
• chore(providers): Integrate DeepSeek reasoning context into output by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3285
• chore(providers): Support entire ProviderResponse output by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3343
• chore(providers): Support multi-segment prompts in google:live provider by @abrayne in https://github.com/promptfoo/promptfoo/pull/3373
• chore(redteam): Add fallback to harmful grader for specific ID patterns by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3366
• chore(redteam): Add pluginId to plugin metadata by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3367
• chore(redteam): Add strategyId metadata to test cases by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3365
• chore(release): Bump version to 0.107.0 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3378
• chore(webui): Clean up YAML from download menu by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3328
• chore(webui): Improve styling of table settings modal by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3329
• chore(webui): Improve YAML editor component by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3325
• chore(webui): Sort display metrics alphabetically in eval output cells by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3364
• chore(deps): Bump @azure/identity from 4.7.0 to 4.8.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3352
• chore(deps): Bump @babel/runtime from 7.26.7 to 7.26.10 in the npmandyarn group by @dependabot in https://github.com/promptfoo/promptfoo/pull/3348
• chore(deps): Bump openai from 4.86.2 to 4.87.3 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3353
• chore(deps): Bump the npmandyarn group with 3 updates by @dependabot in https://github.com/promptfoo/promptfoo/pull/3336
• chore(deps): Run npm audit fix by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3359
• refactor(redteam): Remove harmCategory from harmful plugin vars by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3371

Documentation

• docs(blog): Add sensitive information disclosure post by @vsauter in https://github.com/promptfoo/promptfoo/pull/3350
• docs(examples): Add foundation model redteam example by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3333
• docs(scanner): Add model scanner documentation by @typpo in https://github.com/promptfoo/promptfoo/pull/3322

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.106.3...0.107.0

- TypeScript
Published by mldangelo 12 months ago

promptfoo - 0.106.3

What's Changed

New Features

• feat(redteam): Advanced redteam configurations from cloud provider by @sklein12 in https://github.com/promptfoo/promptfoo/pull/3303

Bug Fixes

• fix(webui): Setting custom target ID by @typpo in https://github.com/promptfoo/promptfoo/pull/3319
• feat(redteam): Advanced redteam configurations from cloud provider by @sklein12 in https://github.com/promptfoo/promptfoo/pull/3303
• fix(providers): amazon nova outputs

Chores

• chore: Bump version 0.106.3 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3320
• chore(providers): Add EU Nova models to Bedrock by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3318

Documentation

• docs(self-hosting): Add a note about PROMPTFOOCONFIGDIR by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3315

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.106.2...0.106.3

- TypeScript
Published by mldangelo 12 months ago

promptfoo - 0.106.2

What's Changed

• chore(providers): add claude 3.7 thinking support in bedrock by @typpo in https://github.com/promptfoo/promptfoo/pull/3313
• chore(providers): add showThinking option to anthropic and bedrock by @typpo in https://github.com/promptfoo/promptfoo/pull/3316
• chore: Update cloud provider prefix by @sklein12 in https://github.com/promptfoo/promptfoo/pull/3311

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.106.1...0.106.2

- TypeScript
Published by typpo 12 months ago

promptfoo - 0.106.1

What's Changed

Features

• feat(providers): Google Multimodal Live provider by @abrayne in #3270
• feat(providers): add support for gpt-4o-audio-preview by @mldangelo in #3302
• feat(cloud): Fetch provider from cloud by @sklein12 in #3299
• feat(moderation): add Azure Content Safety API moderation by @MrFlounder in #3292

Fixes

• fix(providers): remove duplicate CallApi in golang completion by @MrFlounder in #3275
• fix(providers): support @smithy/node-http-handler ^4.0.0 by @aloisklink in #3288
• fix(config): env vars in promptfooconfig.yaml files are strings by @mldangelo in #3273
• fix(eval): honor evaluateOptions when config file is in a different directory by @mldangelo in #3287
• fix(providers): catch Vertex finish_reason errors correctly by @kieranmilan in #3277

Chores and Maintenance

• chore: bump version 0.106.1 by @MrFlounder in #3310
• chore(build): add pnpm support by @mldangelo in #3307
• chore(config): add fallback for eval without configuration by @mldangelo in #3279
• chore(config): enhance error message formatting by @mldangelo in #3306
• chore(dep): bump @anthropic-ai/sdk from 0.38.0 to 0.39.0 by @dependabot in #3269
• chore(dep): bump openai from 4.86.1 to 4.86.2 by @dependabot in #3305
• chore(providers): enable templating of Google API credentials by @mldangelo in #3283
• chore(providers): support for xai region by @typpo in #3281
• chore(scripts): remove unused and undocumented install script by @mldangelo in #3308
• chore(webui): set proper MIME types for JavaScript files by @mldangelo in #3271

Documentation

• docs: more bedrock multimodal docs by @typpo in #3268
• docs: show remote status for plugins by @typpo in #3272
• docs: update azure moderation doc by @MrFlounder in #3309
• docs: improve JavaScript provider documentation by @mldangelo in #3301

Tests

• test: add unit test for src/globalConfig/accounts.ts by @gru-agent in #3254
• test: add unit test for src/providers/vertexUtil.ts by @gru-agent in #3278
• test: add unit test for src/util/cloud.ts by @gru-agent in #3300
• test: add unit test for src/providers/golangCompletion.ts by @gru-agent in #3276

New Contributors

• @hwaxxer made their first contribution in #3239
• @kaibadash made their first contribution in #3255

Full Changelog: 0.105.1...0.106.0

- TypeScript
Published by MrFlounder 12 months ago

promptfoo - 0.106.0

What's Changed

• feat: base64 loader for images by @typpo in https://github.com/promptfoo/promptfoo/pull/3262
• feat: allow prompt functions to return config by @hwaxxer in https://github.com/promptfoo/promptfoo/pull/3239
• fix: infinite rerender in provider editor by @sklein12 in https://github.com/promptfoo/promptfoo/pull/3242
• chore(providers): refactor OpenAI image provider to remove OpenAI Node SDK dependency by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3245
• chore(deps): update dependencies to latest versions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3247
• chore(providers): replace OpenAI moderation provider SDK with fetch by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3248
• chore: Add Foundational Model Reports links to Resources menu and footer by @AISimplyExplained in https://github.com/promptfoo/promptfoo/pull/3250
• chore: inference limit warning by @typpo in https://github.com/promptfoo/promptfoo/pull/3253
• chore: Fix an error in Google SpreadSheet(Authenticated) with a header without a value by @kaibadash in https://github.com/promptfoo/promptfoo/pull/3255
• chore: bump version 0.106.0 by @typpo in https://github.com/promptfoo/promptfoo/pull/3267
• test: add unit test for src/providers/openai/util.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/3241
• docs(press): add new podcast to press page by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3252

New Contributors

• @hwaxxer made their first contribution in https://github.com/promptfoo/promptfoo/pull/3239
• @kaibadash made their first contribution in https://github.com/promptfoo/promptfoo/pull/3255

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.105.1...0.106.0

- TypeScript
Published by typpo 12 months ago

promptfoo - 0.105.1

What's Changed

Features

• feat(providers): add support for execution of function/tool callbacks in Vertex provider (@abrayne) #3215

Chores

• chore(cli): refactor share command (@mldangelo) #3234
• chore(deps): bump @aws-sdk/client-bedrock-runtime from 3.751.0 to 3.755.0 (@dependabot) #3213
• chore(deps): bump version 0.105.1 (@mldangelo) #3244
• chore(providers): add support for GPT-4.5 OpenAI model (@mldangelo) #3240
• chore(providers): lazy load replicate provider (@typpo) #3220
• chore(providers): support inject vars in query params for raw requests for http provider (@sklein12) #3233
• chore(redteam): map RBAC-tagIds when pulling redteam configs from the cloud (@sklein12) #3229
• chore(webui): add reusable error boundary component (@mldangelo) #3224
• chore(webui): fix progress to history redirects (@mldangelo) #3217
• chore(webui): make datasets optional in history and prompts components (@mldangelo) #3235
• revert: "chore: Map RBAC-tagIds when pulling redteam configs from the cloud" (@sklein12) #3231

Documentation

• docs: update Claude vs GPT comparison (@AISimplyExplained) #3216
• docs(command-line): update documentation with new commands and options (@mldangelo) #3223
• docs(vertex): enhance and update Vertex AI documentation (@mldangelo) #3107

Tests

• test: add unit test for src/app/src/pages/history/History.tsx (@gru-agent) #3197
• test: add unit test for src/providers/vertexUtil.ts (@gru-agent) #3208
• test: add unit test for src/server/server.ts (@gru-agent) #3198
• test(history): remove obsolete History component tests (@mldangelo) #3218

New Contributors

• @abrayne made their first contribution in #3215

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.105.0...0.105.1

- TypeScript
Published by mldangelo about 1 year ago

promptfoo - 0.105.0

What's Changed

New Features

• feat(assertions): add custom assertion scoring functions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3142
• feat(providers): add Claude 3.7 by @typpo in https://github.com/promptfoo/promptfoo/pull/3200
• feat(providers): add Databricks provider by @typpo in https://github.com/promptfoo/promptfoo/pull/3124
• feat(providers): add support for multiple providers in single config file by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3156
• feat(webui): add HTTPS option for raw request in redteam setup by @sklein12 in https://github.com/promptfoo/promptfoo/pull/3149

Bug Fixes

• fix(cli): fix list command for datasets by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3163
• fix(cli): resolve issue where script.py:myFunc fails fs stat check with PROMPTFOOSTRICTFILES=true by @dhodun in https://github.com/promptfoo/promptfoo/pull/3133
• fix(env): ensure environment variables are properly merged and rendered in Nunjucks by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3134
• fix(providers): update Go toolchain version to valid syntax by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3170
• fix(providers): add JSON stringify for debug output in http provider by @typpo in https://github.com/promptfoo/promptfoo/pull/3131
• fix(providers): correct Gemini/OpenAI format conversion by @typpo in https://github.com/promptfoo/promptfoo/pull/3206
• fix(providers): handle OpenRouter empty content by @typpo in https://github.com/promptfoo/promptfoo/pull/3205
• fix(providers): properly classify API errors with ResultFailureReason.ERROR by @AISimplyExplained in https://github.com/promptfoo/promptfoo/pull/3141
• fix(providers): remove content length header in HTTP provider by @sklein12 in https://github.com/promptfoo/promptfoo/pull/3147
• fix(site): resolve mobile responsiveness issues by @typpo in https://github.com/promptfoo/promptfoo/pull/3201
• fix(webui): improve dark mode colors by @devin-ai-integration in https://github.com/promptfoo/promptfoo/pull/3187
• fix(webui): resolve share modal infinite loop by @sklein12 in https://github.com/promptfoo/promptfoo/pull/3171

Chores

• chore!(providers): remove direct provider exports in favor of loadApiProvider by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3183
• chore(build): enable SWC for ts-node for faster dev server by @faizanminhas in https://github.com/promptfoo/promptfoo/pull/3126
• chore(eval): add eval-id to --filter-failing and --filter-errors-only eval flags by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3174
• chore(logging): replace console.error with logger.error by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3175
• chore(providers): add support for Anthropic Claude 3.7 Sonnet model by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3202
• chore(providers): add support for Claude on Vertex by @typpo in https://github.com/promptfoo/promptfoo/pull/3209
• chore(providers): update Claude 3.7 Sonnet configurations by @typpo in https://github.com/promptfoo/promptfoo/pull/3199
• chore(redteam): refactor HarmBench plugin by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3176
• chore(release): bump version to 0.105.0 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3210
• chore(webui): add pagination to eval selector by @devin-ai-integration in https://github.com/promptfoo/promptfoo/pull/3189
• chore(webui): add pagination to reports index frontend by @devin-ai-integration in https://github.com/promptfoo/promptfoo/pull/3190
• chore(webui): add toggle for application vs model testing by @typpo in https://github.com/promptfoo/promptfoo/pull/3194
• chore(webui): enhance dataset dialog and table UI by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3154
• chore(webui): improve external systems section styling by @devin-ai-integration in https://github.com/promptfoo/promptfoo/pull/3195
• chore(webui): improve prompts page view by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3135
• chore(webui): modernize UI components by @typpo in https://github.com/promptfoo/promptfoo/pull/3150
• chore(webui): refactor data loading in progress view for reusability by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3136
• chore(webui): return detailed error messages from fetch by @sklein12 in https://github.com/promptfoo/promptfoo/pull/3145
• chore(webui): sync UI improvements from cloud by @devin-ai-integration in https://github.com/promptfoo/promptfoo/pull/3164
• chore(webui): update outdated onboarding models by @typpo in https://github.com/promptfoo/promptfoo/pull/3130

Dependencies

• chore(deps): bump @aws-sdk/client-bedrock-runtime from 3.744.0 to 3.749.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3121
• chore(deps): bump @aws-sdk/client-bedrock-runtime from 3.749.0 to 3.750.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3128
• chore(deps): bump @aws-sdk/client-bedrock-runtime from 3.750.0 to 3.751.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3159
• chore(deps): bump @azure/identity from 4.6.0 to 4.7.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3160
• chore(deps): bump openai from 4.85.0 to 4.85.1 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3120
• chore(deps): bump openai from 4.85.1 to 4.85.2 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3161
• chore(deps): bump openai from 4.85.2 to 4.85.3 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3173
• chore(deps): bump openai from 4.85.3 to 4.85.4 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3192
• chore(deps): update dependencies to latest versions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3193

Refactors

• refactor(env): centralize environment variable schema by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3105
• refactor(providers): extract provider registry to dedicated module by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3127
• refactor(utils): separate database utilities from general utilities by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3184
• refactor(webui): rename progress to history by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3196

Documentation

• docs(vertex): add gemini-2.0-flash-001 fixes #3167 by @ghchinoy in https://github.com/promptfoo/promptfoo/pull/3168
• docs(metrics): improve derived metrics documentation by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3157
• docs(configuration): enhance CSV documentation with custom assertion example by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3158
• docs(press): update press page with new content and resources by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3103

Tests

• test(routes): add unit test for src/server/routes/redteam.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/3181

New Contributors

• @faizanminhas made their first contribution in https://github.com/promptfoo/promptfoo/pull/3126

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.104.4...0.105.0

- TypeScript
Published by mldangelo about 1 year ago

promptfoo - 0.104.4

What's Changed

Features

• feat(redteam): add reasoning denial of service plugin by @typpo in https://github.com/promptfoo/promptfoo/pull/3109
• feat(providers): add support for tools in Vertex provider by @kieranmilan in https://github.com/promptfoo/promptfoo/pull/3077

Bug Fixes

• fix(providers): support nested directory structures in Go provider by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3118

Chores

• chore(providers): update replicate default moderation provider by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3097
• chore(redteam): update grader prompt by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/3092
• chore(testCases): improve error message clarity in testCaseReader, clean up tests by @dhodun in https://github.com/promptfoo/promptfoo/pull/3108
• chore(testCases): improve JSON field support in CSV test cases by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3102
• chore(webui): add extension hooks support to red team configuration by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3067
• chore(webui): display suggestion note by @typpo in https://github.com/promptfoo/promptfoo/pull/3116
• chore(webui): refine suggestion behavior by @typpo in https://github.com/promptfoo/promptfoo/pull/3112

Documentation

• docs(blog): add agent security blog post by @vsauter in https://github.com/promptfoo/promptfoo/pull/3072
• docs(google-sheets): improve documentation clarity by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3104

Tests

• test(providers): add unit test for src/providers/openai/image.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/3086
• test(redteam): add unit test for src/redteam/plugins/overreliance.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/3093
• test(core): add unit test for src/table.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/3084

Dependencies

• chore(deps): bump openai from 4.84.0 to 4.85.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3095
• chore(deps): bump version to 0.104.4 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3119

New Contributors

• @kieranmilan made their first contribution in https://github.com/promptfoo/promptfoo/pull/3077
• @dhodun made their first contribution in https://github.com/promptfoo/promptfoo/pull/3108

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.104.3...0.104.4

- TypeScript
Published by mldangelo about 1 year ago

promptfoo - 0.104.3

What's Changed

Bug Fixes

• fix(testCaseReader): correctly process file:// URLs for YAML files by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3082

Chores

• chore(release): bump version to 0.104.3 by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/3091

Refactor

• refactor(prompts): consolidate prompt processing logic by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3081
• refactor(utils): move utils to util by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3083

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.104.2...0.104.3

- TypeScript
Published by mldangelo about 1 year ago

promptfoo - 0.104.2

What's Changed

Fixes

• fix(providers): Anthropic API error handling for 413s by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3078
• fix(redteam): correct foundation plugin collection expansion by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3073

Chores

• chore(providers): add extra_body support for Anthropic API by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3079
• chore(webui): add pagination and show more/less controls to intent sections by @devin-ai-integration in https://github.com/promptfoo/promptfoo/pull/2955
• chore(auth): sync email between config and login commands by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3062
• chore: remove debug log by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/3071
• chore(testCases): add HuggingFace Hub token support for datasets by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3063

Dependencies

• chore(deps): bump openai from 4.83.0 to 4.84.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3075
• chore(deps): bump version to 0.104.2 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3080

Documentation

• docs: document NO_PROXY environment variable by @typpo in https://github.com/promptfoo/promptfoo/pull/3070

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.104.1...0.104.2

- TypeScript
Published by mldangelo about 1 year ago

promptfoo - 0.104.1

What's Changed

Features

• feat(test-cases): add support for loading dynamic test cases from Python and JavaScript/TypeScript files by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2993
• feat(assertions): add threshold support for llm-rubric by @mshavliuk in https://github.com/promptfoo/promptfoo/pull/2999
• feat(package): add guardrails in node package by @typpo in https://github.com/promptfoo/promptfoo/pull/3034

Bug Fixes

• fix(providers): escape single quotes in golang provider by @gregnuttall in https://github.com/promptfoo/promptfoo/pull/3025

Chores & Improvements

• chore(assertions): improve parsing of llm-rubric outputs by @elahti in https://github.com/promptfoo/promptfoo/pull/3021
• chore(assertions): make JSON parsing less strict for matchers by @mshavliuk in https://github.com/promptfoo/promptfoo/pull/3002
• chore(assertions): parse string scores in llm rubric outputs by @mshavliuk in https://github.com/promptfoo/promptfoo/pull/3037
• chore(build): resolve CodeQL invalid Go toolchain version warning by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3022
• chore(ci): remove unused nexe build workflow by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3014
• chore(config): enhance email validation with zod schema by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3011
• chore(config): handle empty config files gracefully by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3027
• chore(deps): bump @aws-sdk/client-bedrock-runtime from 3.741.0 to 3.743.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3020
• chore(deps): bump @aws-sdk/client-bedrock-runtime from 3.743.0 to 3.744.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3038
• chore(deps): bump esbuild from 0.24.2 to 0.25.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3056
• chore(deps): bump openai from 4.82.0 to 4.83.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3019
• chore(deps): bump vitest from 2.1.8 to 2.1.9 by @dependabot in https://github.com/promptfoo/promptfoo/pull/3018
• chore(deps): update dependencies by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3032
• chore(deps): update dependencies to latest versions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3024
• chore(deps): update vitest to resolve CVE issues by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3016
• chore(download): include comment in download data by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/3052
• chore(eval): add redteamFinalPrompt to download menu by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/3035
• chore(harmful): refine grader logic for specific categories by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/3054
• chore(hooks): improve handling of absolute paths in hook/code import by @vedantr in https://github.com/promptfoo/promptfoo/pull/3060
• chore(providers): add bedrock llama3.3 support by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/3031
• chore(providers): add fireworks provider by @ershang-fireworks in https://github.com/promptfoo/promptfoo/pull/3001
• chore(providers): allow Alibaba API base URL override by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3040
• chore(providers): correct golang behavior for prompts with quotes by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3026
• chore(providers): expose deleteFromCache to evict cache keys after fetch by providers by @typpo in https://github.com/promptfoo/promptfoo/pull/3009
• chore(providers): handle edge case in openai chat completion provider by @stonyz in https://github.com/promptfoo/promptfoo/pull/3033
• chore(providers): validate dynamic method call by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3023
• chore(redteam): add --no-progress-bar support for redteam generate and run by @typpo in https://github.com/promptfoo/promptfoo/pull/3043
• chore(redteam): add support for job progress in RunEvalOptions by @typpo in https://github.com/promptfoo/promptfoo/pull/3042
• chore(redteam): enhance refusal detection by @typpo in https://github.com/promptfoo/promptfoo/pull/3015
• chore(redteam): improve progress plumbing changes by @typpo in https://github.com/promptfoo/promptfoo/pull/3053
• chore(redteam): purge signature auth from redteam config if disabled by @vedantr in https://github.com/promptfoo/promptfoo/pull/2995
• chore(redteam): support progress callback in redteam run by @typpo in https://github.com/promptfoo/promptfoo/pull/3049
• chore(release): bump version 0.104.1 by @vedantr in https://github.com/promptfoo/promptfoo/pull/3061
• chore(webui): add clear search buttons to search fields by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3048
• chore(webui): color pass rates on a gradient by @typpo in https://github.com/promptfoo/promptfoo/pull/2997
• chore(webui): ensure extensions are serialized from config in getUnifiedConfig by @vedantr in https://github.com/promptfoo/promptfoo/pull/3050
• chore(webui): ensure thumbs remain active after selection by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/3059
• chore(webui): improve column selector tooltip placement by @typpo in https://github.com/promptfoo/promptfoo/pull/3005
• chore(webui): move dropdown chevron to correct position by @typpo in https://github.com/promptfoo/promptfoo/pull/3007
• chore(webui): reorganize provider configurations by @sklein12 in https://github.com/promptfoo/promptfoo/pull/3028

Refactor

• refactor(test): split test case loading from synthesis by @mldangelo in https://github.com/promptfoo/promptfoo/pull/3004

Tests

• test(unit): add test for src/redteam/sharedFrontend.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/3051

Documentation

• docs: fix PromptFoo vs. Promptfoo capitalization by @typpo in https://github.com/promptfoo/promptfoo/pull/3013
• docs: update assert function context docs and examples by @typpo in https://github.com/promptfoo/promptfoo/pull/3008

New Contributors

• @ershang-fireworks made their first contribution in https://github.com/promptfoo/promptfoo/pull/3001
• @elahti made their first contribution in https://github.com/promptfoo/promptfoo/pull/3021
• @gregnuttall made their first contribution in https://github.com/promptfoo/promptfoo/pull/3025
• @stonyz made their first contribution in https://github.com/promptfoo/promptfoo/pull/3033

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.104.0...0.104.1

- TypeScript
Published by vedantr about 1 year ago

promptfoo - 0.104.0

What's Changed

New Features

• feat(openai): Updated default grading provider to gpt-4o-2024-11-20 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2987
• feat(assertions): Added .js file support for rubricPrompt in llm-rubric assertion by @mshavliuk in https://github.com/promptfoo/promptfoo/pull/2972
• feat(redteam): Added pandamonium strategy by @sklein12 in https://github.com/promptfoo/promptfoo/pull/2920
• feat(redteam): Added retry strategy for regression testing by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2924
• feat(redteam): Added support for base64-encoded key strings in webui in addition to file paths and file upload by @vedantr in https://github.com/promptfoo/promptfoo/pull/2983

Bug Fixes

• fix(redteam): Improved Crescendo strategy on refusals by @sklein12 in https://github.com/promptfoo/promptfoo/pull/2979
• fix(redteam): Added support for target delay in redteam setup UI by @sklein12 in https://github.com/promptfoo/promptfoo/pull/2991
• fix(redteam): Stringified guardrail headers by @typpo in https://github.com/promptfoo/promptfoo/pull/2981
• fix(redteam): Fixed harmbench plugin dataset pull location by @vedantr in https://github.com/promptfoo/promptfoo/pull/2963

Chores

• chore(redteam): Improved RBAC grader by @AISimplyExplained in https://github.com/promptfoo/promptfoo/pull/2976
• chore(redteam): Improved BOLA grader by @sklein12 in https://github.com/promptfoo/promptfoo/pull/2982
• chore(site): Added HTTP endpoint config generator link by @typpo in https://github.com/promptfoo/promptfoo/pull/2957
• chore(webui): Synced test target configuration key file UI with cloud by @devin-ai-integration in https://github.com/promptfoo/promptfoo/pull/2959
• chore(docs): Changed Docusaurus default port by @vedantr in https://github.com/promptfoo/promptfoo/pull/2964
• chore(redteam): Added foundation model plugin collection by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2967
• chore(redteam): Cleaned up key validation code by @vedantr in https://github.com/promptfoo/promptfoo/pull/2992
• chore(redteam): Sorted constants by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2988
• chore(redteam): Sorted strategy list by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2989
• chore(redteam): UI - Added new strategy presents and client-side session IDs by @sklein12 in https://github.com/promptfoo/promptfoo/pull/2968
• chore(share): Added confirmation step before generating public share link by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/2921
• chore(providers): Restructured OpenAI provider into modular files by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2953
• chore(deps): Bumped @aws-sdk/client-bedrock-runtime from 3.738.0 to 3.741.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/2973
• chore(deps): Bumped version to 0.104.0 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2994
• chore(deps): Bumped vitest from 1.6.0 to 1.6.1 in /examples/jest-integration by @dependabot in https://github.com/promptfoo/promptfoo/pull/2978
• chore: Fixed build due to duplicate import and cyclic dependency by @vedantr in https://github.com/promptfoo/promptfoo/pull/2969
• chore(docusaurus): Added ability to override port via environment variable by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2986

Documentation

• docs(blog): DeepSeek tweaks by @typpo in https://github.com/promptfoo/promptfoo/pull/2970
• docs(blog): DeepSeek redteam by @vsauter in https://github.com/promptfoo/promptfoo/pull/2966
• docs(cloud): Added service accounts by @typpo in https://github.com/promptfoo/promptfoo/pull/2984
• docs(guide): Added guide for doing evals with harmbench by @vedantr in https://github.com/promptfoo/promptfoo/pull/2943
• docs(press): Added dedicated press page by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2990
• docs(python): Updated Python provider docs to add guardrails usage example by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/2962

Tests

• test: Added unit test for src/assertions/utils.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2974
• test: Added unit test for src/redteam/plugins/rbac.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2977

New Contributors

• @mshavliuk made their first contribution in https://github.com/promptfoo/promptfoo/pull/2972

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.103.19...0.104.0

- TypeScript
Published by mldangelo about 1 year ago

promptfoo - 0.103.19

What's Changed

New Features

• feat(redteam): Add a plugin to run redteams against the HarmBench dataset by @vedantr in https://github.com/promptfoo/promptfoo/pull/2896
• feat(redteam): add hex strategy by @typpo in https://github.com/promptfoo/promptfoo/pull/2950

Chores

• chore(providers): add o3 mini as an option to OpenAI provider by @vedantr in https://github.com/promptfoo/promptfoo/pull/2940
• chore(providers): migrate Groq to use OpenAI provider - add groq reasoning example by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2952
• chore(providers): update openai api version to support o3 models by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2942
• chore(redteam): reduce false positives in politics plugin by @AISimplyExplained in https://github.com/promptfoo/promptfoo/pull/2935
• chore(deps): bump various dependencies by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2941
• chore(docs): re-add plugin documentation to the example by @vedantr in https://github.com/promptfoo/promptfoo/pull/2939
• chore(examples): Example of a very simple barebones eval with Harmbench by @vedantr in https://github.com/promptfoo/promptfoo/pull/2873
• chore: Reduced watched files for nodemon by @sklein12 in https://github.com/promptfoo/promptfoo/pull/2949
• chore(redteam): use shared penalized phrase function in `iterativeTree by @typpo in https://github.com/promptfoo/promptfoo/pull/2946

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.103.18...0.103.19

- TypeScript
Published by typpo about 1 year ago

promptfoo - 0.103.18

What's Changed

• feat(providers): add Alibaba Model Studio provider by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2908
• fix(generation): handle cases where vars is not an array by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2916
• fix(providers): handle function expressions in transform response by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/2917
• fix(webui): improve dark mode syntax highlighting in HTTP request editor by @AISimplyExplained in https://github.com/promptfoo/promptfoo/pull/2911
• fix(webui): improve spacing between Back and Next buttons by @AISimplyExplained in https://github.com/promptfoo/promptfoo/pull/2912
• fix(webui): update Next button styling to support dark mode by @AISimplyExplained in https://github.com/promptfoo/promptfoo/pull/2898
• fix: added tsx back to dependencies by @carl09 in https://github.com/promptfoo/promptfoo/pull/2923
• fix: full rubricPrompt support for json/yaml filetypes by @typpo in https://github.com/promptfoo/promptfoo/pull/2931
• chore(grader): improve false positive detection for religion grader by @AISimplyExplained in https://github.com/promptfoo/promptfoo/pull/2909
• chore(redteam): upgrade replicate moderation api to Llama Guard 3 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2904
• chore(webui): add preset collections for redteam plugins by @sklein12 in https://github.com/promptfoo/promptfoo/pull/2853
• chore: Move callEval outside of the function so we can re-use it by @sklein12 in https://github.com/promptfoo/promptfoo/pull/2897
• chore: Save test case from EvalResult by @sklein12 in https://github.com/promptfoo/promptfoo/pull/2902
• chore: bump @aws-sdk/client-bedrock-runtime from 3.734.0 to 3.738.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/2906
• chore: bump openai from 4.80.1 to 4.81.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/2905
• chore: bump version 0.103.18 by @typpo in https://github.com/promptfoo/promptfoo/pull/2932
• chore: improvements to refusal detection by @typpo in https://github.com/promptfoo/promptfoo/pull/2903
• test: configure default globalConfig mock and logger mock by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2915
• docs(examples): update and clean up DeepSeek R1 example README by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2918

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.103.17...0.103.18

- TypeScript
Published by typpo about 1 year ago

promptfoo - 0.103.17

What's Changed

New Features

• feat(launcher): Add launcher page and Cloudflare deploy action by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2599
• feat(providers): Add JFrog ML provider by @typpo in https://github.com/promptfoo/promptfoo/pull/2872

Bug Fixes

• fix(assertions): Add valueFromScript support to contains, equals, and startsWith assertions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2890
• fix(golang-provider): Support internal package imports by preserving module structure by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2888

Chores

• chore(build): Move dependencies to devDependencies by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2876
• chore(redteam): Update grader SpecializedAdviceGrader by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/2895
• chore(redteam): Update graders: imitation, overreliance by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/2882
• chore(redteam): Update graders: politics and RBAC by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/2878
• chore(redteam): Update SQL injection and shell injection graders by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/2870
• chore(redteam): Remove RedTeamProvider response by @sklein12 in https://github.com/promptfoo/promptfoo/pull/2899
• chore(deps): Move tsx to dev dependencies by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2884
• chore(deps): Update Drizzle dependencies by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2877
• chore(build): Bump version to 0.103.17 by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/2900

Documentation

• docs: Fix broken transformVars example by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2887
• docs(providers): Fix syntax and formatting in examples by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2875

Testing

• test: Add unit test for src/providers/bedrockUtil.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2879
• test: Add unit test for src/redteam/plugins/shellInjection.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2871

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.103.16...0.103.17

- TypeScript
Published by MrFlounder about 1 year ago

promptfoo - 0.103.16

What's Changed

Features

• feat(eval): Support reasoning effort and usage tokens by @joakimriedel in https://github.com/promptfoo/promptfoo/pull/2817
• feat(providers): Add support for anthropic citations by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2854
• feat(redteam): Add RAG Full Document Exfiltration plugin by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/2820
• feat(tests): Add support for loading tests from JSONL files by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2842

Bug Fixes

• fix(openrouter): Pass through passthrough by @typpo in https://github.com/promptfoo/promptfoo/pull/2863
• fix(redteam): Run strategies on intents by @typpo in https://github.com/promptfoo/promptfoo/pull/2866
• fix(sharing): Combine sharing configuration from multiple promptfooconfigs by @carl09 in https://github.com/promptfoo/promptfoo/pull/2855

Chores

• chore(eval): Support reasoning field by @typpo in https://github.com/promptfoo/promptfoo/pull/2867
• chore(providers): Add common provider types for redteam providers by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2856
• chore(providers): Update google provider with better support for latest gemini models by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2838
• chore(redteam): Add redteam run analytics by @sklein12 in https://github.com/promptfoo/promptfoo/pull/2852
• chore(package): Bump version 0.103.16 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2869
• chore(package): Ensure correct branch name when incrementing package version by @vedantr in https://github.com/promptfoo/promptfoo/pull/2851
• chore(package): Exclude test files from npm package by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2862
• chore(package): Simplify files field in package.json by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2868
• chore(deps): Remove unused dependencies by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2861
• chore(deps): Update patch and minor dependency versions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2860
• chore(dev): Upgrade development versions of Node.js to v22 and Python to 3.13 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2340

Documentation

• docs(deepseek): Deepseek censorship article by @vsauter in https://github.com/promptfoo/promptfoo/pull/2864
• docs(simulated-user): Improve simulated user example by @typpo in https://github.com/promptfoo/promptfoo/pull/2865

Tests

• test(redteam): Add unit test for src/redteam/plugins/beavertails.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2844
• test(redteam): Add unit test for src/redteam/plugins/contracts.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2845

New Contributors

• @joakimriedel made their first contribution in https://github.com/promptfoo/promptfoo/pull/2817
• @carl09 made their first contribution in https://github.com/promptfoo/promptfoo/pull/2855

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.103.15...0.103.16

- TypeScript
Published by mldangelo about 1 year ago

promptfoo - 0.103.15

What's Changed

Chores

• chore(providers): Add Hyperbolic alias by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2826
• chore(providers): Add Perplexity alias by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2836
• chore(providers): Add Cloudera alias by @typpo in https://github.com/promptfoo/promptfoo/pull/2823
• chore(providers): Make Adaline a peer dependency by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2833
• chore(providers): Support chatgpt-4o-latest alias in OpenAI provider by @vedantr in https://github.com/promptfoo/promptfoo/pull/2841
• chore(providers): Handle empty content due to Azure content filter by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/2822
• chore(assertions): Add not-is-refusal assertion by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2840
• chore(redteam): Add stack trace to generate/run errors by @typpo in https://github.com/promptfoo/promptfoo/pull/2831
• chore(redteam): Reduce science fiction jailbreaks by @typpo in https://github.com/promptfoo/promptfoo/pull/2830
• chore(redteam): Switch from stateless to stateful by @sklein12 in https://github.com/promptfoo/promptfoo/pull/2839

Documentation

• docs: Update contributing guide and fix docs build break by @vedantr in https://github.com/promptfoo/promptfoo/pull/2849
• docs: Add terms of service by @vsauter in https://github.com/promptfoo/promptfoo/pull/2821
• docs: Clean up LocalAI title by @typpo in https://github.com/promptfoo/promptfoo/pull/2824
• docs: Minor updates to provider documentation sidebar order by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2827
• docs: Update contributing guide with helpful links and update new release documentation by @vedantr in https://github.com/promptfoo/promptfoo/pull/2843
• docs: Update documentation with new models and features by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2837

Dependencies

• chore(deps): Bump OpenAI from 4.80.0 to 4.80.1 by @dependabot in https://github.com/promptfoo/promptfoo/pull/2835
• chore(deps): Bump version to 0.103.15 by @vedantr in https://github.com/promptfoo/promptfoo/pull/2850

Tests

• test: Add unit test for src/providers/portkey.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2825
• test: Add unit test for src/redteam/plugins/asciiSmuggling.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2846

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.103.14...0.103.15

- TypeScript
Published by vedantr about 1 year ago

promptfoo - 0.103.14

What's Changed

• feat(redteam): add InsultsGrader for insult detection by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/2814
• feat: ability to export to burp by @typpo in https://github.com/promptfoo/promptfoo/pull/2807
• feat: pull and set sessionIds in the request and response body by @sklein12 in https://github.com/promptfoo/promptfoo/pull/2784
• fix(azure): handle 400 response for content filter errors by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/2812
• fix(ui): ensure that a default value of the signature data field is populated into the redteam config by @vedantr in https://github.com/promptfoo/promptfoo/pull/2788
• fix: use controlled accordion for signature auth by @typpo in https://github.com/promptfoo/promptfoo/pull/2789
• fix(docs): random grammar fix for model-graded metrics by @vedantr in https://github.com/promptfoo/promptfoo/pull/2794
• chore: bump @anthropic-ai/sdk from 0.33.1 to 0.35.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/2790
• chore: bump openai from 4.79.1 to 4.79.4 by @dependabot in https://github.com/promptfoo/promptfoo/pull/2791
• chore: improve specialized advice grader by @typpo in https://github.com/promptfoo/promptfoo/pull/2793
• chore: unsafe practices grader by @typpo in https://github.com/promptfoo/promptfoo/pull/2796
• chore(deps): update LLM provider dependencies by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2795
• chore: more harmful graders by @typpo in https://github.com/promptfoo/promptfoo/pull/2797
• chore: sort by priority strategies in report view by @typpo in https://github.com/promptfoo/promptfoo/pull/2809
• chore: add graders for drugs, illegal activities, cybercrime, radicalization by @typpo in https://github.com/promptfoo/promptfoo/pull/2810
• chore: burp docs, improvements, and ui by @typpo in https://github.com/promptfoo/promptfoo/pull/2818
• chore: bump @aws-sdk/client-bedrock-runtime from 3.731.1 to 3.734.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/2815
• chore: add keyfile upload by @typpo in https://github.com/promptfoo/promptfoo/pull/2787
• test(openai): move OpenAI provider tests to dedicated file by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2802
• test: add unit test for src/assertions/contextRelevance.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2804
• test: add unit test for src/assertions/geval.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2803
• test: add unit test for src/fetch.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2781
• test: add unit test for src/assertions/contextFaithfulness.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2798
• test: add unit test for src/assertions/answerRelevance.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2799
• test: add unit test for src/assertions/contextRecall.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2800
• test: add unit test for src/assertions/modelGradedClosedQa.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2801
• refactor(fetch): remove unnecessary debug log by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2806
• docs(config): add status page link to footer by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2811

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.103.13...0.103.14

- TypeScript
Published by typpo about 1 year ago

promptfoo - 0.103.13

What's Changed

New Features

• feat: http provider auth signature support by @typpo in https://github.com/promptfoo/promptfoo/pull/2755
• feat(redteam): Add guardrail option to redteam ui & update transform response by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/2688

Bug Fixes

• fix(fetch): correct TLS options for proxy settings by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2783

Chores

• chore: improve http signature setup by @typpo in https://github.com/promptfoo/promptfoo/pull/2779
• chore(fetch): sanitize sensitive data in debug logs by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2778
• chore(redteam): enhance logging and test count formatting by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2775

Dependencies

• chore(deps): bump vite from 5.4.11 to 5.4.12 in the npmandyarn group by @dependabot in https://github.com/promptfoo/promptfoo/pull/2777
• chore(deps): bump vite from 5.4.9 to 5.4.14 in /examples/jest-integration in the npmandyarn group across 1 directory by @dependabot in https://github.com/promptfoo/promptfoo/pull/2776

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.103.12...0.103.13

- TypeScript
Published by mldangelo about 1 year ago

promptfoo - 0.103.12

What's Changed

Bug Fixes

• fix(providers): Add support for tool_resources in OpenAI assistants by @dsappet in https://github.com/promptfoo/promptfoo/pull/2772
• fix(providers): Do not set topp, presencepenalty, or frequency_penalty by default in OpenAI providers by @typpo in https://github.com/promptfoo/promptfoo/pull/2753
• fix(providers): Handle serialization bug in defaultTest for provider overrides with self references (Groq) by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2760
• fix(webui): Add error boundary to React Markdown component by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2756
• fix(redteam): Add missing strategy tags by @typpo in https://github.com/promptfoo/promptfoo/pull/2769
• fix(redteam): Empty response is not a failure for red team by @typpo in https://github.com/promptfoo/promptfoo/pull/2754
• fix(redteam): Self-harm, graphic, sexual content, competitors false positives by @typpo in https://github.com/promptfoo/promptfoo/pull/2759

Chores

• chore(providers): Add DeepSeek provider alias by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2768
• chore(types): Remove unused 'getSessionId' field from ApiProvider by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2765
• chore(redteam): Add copyright violations grader by @typpo in https://github.com/promptfoo/promptfoo/pull/2770
• chore(redteam): Show plugin in strategy stats prompt/response examples by @typpo in https://github.com/promptfoo/promptfoo/pull/2758
• chore(redteam): Improve competitors grader by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2761
• chore(lint): Resolve trailing whitespace issues in YAML file by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2767
• chore(deps): Bump @aws-sdk/client-bedrock-runtime from 3.730.0 to 3.731.1 by @dependabot in https://github.com/promptfoo/promptfoo/pull/2750
• chore(deps): Bump openai from 4.78.1 to 4.79.1 by @dependabot in https://github.com/promptfoo/promptfoo/pull/2751

Tests

• test: Add unit test for src/providers/bam.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2748
• test: Add unit test for src/redteam/graders.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2762
• test: Add unit test for src/redteam/plugins/harmful/graders.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2763
• test: Add unit test for src/redteam/plugins/harmful/graders.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2771
• test: Add unit test for src/redteam/providers/crescendo/index.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2749
• test: Add mocks to reduce CI flakes and logs by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2774

New Contributors

• @dsappet made their first contribution in https://github.com/promptfoo/promptfoo/pull/2772

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.103.11...0.103.12

- TypeScript
Published by mldangelo about 1 year ago

promptfoo - 0.103.11

What's Changed

Fixes

• fix(webui): ensure nested variables are rendered correctly by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2736
• fix(assertions): support JavaScript files in CSV assertions file:// protocol by @devin-ai-integration in https://github.com/promptfoo/promptfoo/pull/2723
• fix(redteam): don't blow up when translation fails by @typpo in https://github.com/promptfoo/promptfoo/pull/2740

Chores

• chore: update vars type definition in Test Case to support nested objects by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2738
• chore(providers): add config.o1 flag for Azure o1 model support by @devin-ai-integration in https://github.com/promptfoo/promptfoo/pull/2710
• chore(assertions): handle OpenAI tool call with content by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2741
• chore(fetch): use undici to set global proxy dispatcher by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/2737
• chore(providers): update Groq documentation with latest models by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2733
• chore(logger): expose additional logger methods by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2731

Dependencies

• chore(deps): bump @aws-sdk/client-bedrock-runtime from 3.726.1 to 3.730.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/2727
• chore(deps): bump @azure/identity from 4.5.0 to 4.6.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/2728
• chore(deps): update Docusaurus version by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2730

Refactors

• refactor: remove dynamic import for OpenAiChatCompletionProvider by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2739
• refactor: remove async imports for third-party integrations by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2746
• refactor: remove dynamic import for fetchWithProxy by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2742
• build: create dist/ using TypeScript's "module": "Node16" setting by @aloisklink in https://github.com/promptfoo/promptfoo/pull/2686
• revert: "build: create dist/ using TypeScript's "module": "Node16" setting (#2686)" by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2747

Documentation

• docs: LangChain example by @typpo in https://github.com/promptfoo/promptfoo/pull/2735
• docs: resolve duplicate route warning on docs/providers by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2676
• docs: update app details by @typpo in https://github.com/promptfoo/promptfoo/pull/2734
• docs(faq): enhance documentation on proxies and SSL certificates by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2725

Tests

• test: add unit test for src/logger.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2732
• test: Add unit test for src/providers/openai.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2700
• test: Add unit test for src/providers/websocket.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2658
• test: Add unit test for src/redteam/strategies/crescendo.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2679
• test: Add unit test for src/redteam/strategies/gcg.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2680
• test: Add unit test for src/redteam/strategies/index.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2682
• test: Add unit test for src/util/exportToFile/index.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2666

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.103.10...0.103.11

- TypeScript
Published by mldangelo about 1 year ago

promptfoo - 0.103.10

What's Changed

New Features

• feat(moderation): Add guardrail checks and logging for moderation by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/2624
• feat(redteam): Add support for built-in guardrails by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/2654

Bug Fixes

• fix: Don't throw in HTTP provider on non-2xx by @typpo in https://github.com/promptfoo/promptfoo/pull/2689
• fix: Eval description in promptfoo list evals by @typpo in https://github.com/promptfoo/promptfoo/pull/2668
• fix: Handle HTTP errors better by @typpo in https://github.com/promptfoo/promptfoo/pull/2687
• fix: Make back/next icons consistent by @typpo in https://github.com/promptfoo/promptfoo/pull/2707
• fix: Resolve defaultTest and test providers when called via Node by @typpo in https://github.com/promptfoo/promptfoo/pull/2664
• fix: WebUI should automatically refresh with new evals by @typpo in https://github.com/promptfoo/promptfoo/pull/2672
• fix(ci): Resolve redteam integration test failure by setting author by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2667
• fix(logging): Enforce single-argument type for logger methods by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2719
• fix(providers): Lazy load @azure/identity by @typpo in https://github.com/promptfoo/promptfoo/pull/2708
• fix(redteam): Adjust divergent repetition plugin prompt formatting by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/2639
• fix(ui): Don't select a stateful/stateless setting if discrepancy exists between configured providers by @vedantr in https://github.com/promptfoo/promptfoo/pull/2650
• fix(ui): Fix stateful/stateless setting for providers by @vedantr in https://github.com/promptfoo/promptfoo/pull/2649
• fix(webui): Ensure user's selection of system statefulness is correctly persisted in config and UI by @vedantr in https://github.com/promptfoo/promptfoo/pull/2645

Chores

• chore: Add email to remote inference requests by @sklein12 in https://github.com/promptfoo/promptfoo/pull/2647
• chore: Add envar for max harmful tests per request by @typpo in https://github.com/promptfoo/promptfoo/pull/2714
• chore: Bump @aws-sdk/client-bedrock-runtime from 3.726.0 to 3.726.1 by @dependabot in https://github.com/promptfoo/promptfoo/pull/2641
• chore: Bump groq-sdk from 0.11.0 to 0.12.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/2642
• chore: Bump openai from 4.78.0 to 4.78.1 by @dependabot in https://github.com/promptfoo/promptfoo/pull/2643
• chore: Check email status by @typpo in https://github.com/promptfoo/promptfoo/pull/2651
• chore: Organize advanced configurations UI by @typpo in https://github.com/promptfoo/promptfoo/pull/2713
• chore: Standardize ellipsize function across codebase by @devin-ai-integration in https://github.com/promptfoo/promptfoo/pull/2698
• chore: Update unaligned timeout by @sklein12 in https://github.com/promptfoo/promptfoo/pull/2696
• chore(assertion): Update doc by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/2705
• chore(ci): Add shell format check to CI workflow by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2669
• chore(cli): Update show command to default to most recent eval by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2718
• chore(config): Clean up and comment unused configurations by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/2646
• chore(providers): Add error handling for request transforms in HTTP provider by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2697
• chore(providers): Add validateStatus option to HTTP provider by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2691
• chore(providers): Change default validateStatus to accept all HTTP status codes by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2712
• chore(redteam): Add more abort checkpoints for redteam runs by @vedantr in https://github.com/promptfoo/promptfoo/pull/2717
• chore(redteam): Enhance debug logging in iterative provider by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2695
• chore(redteam): Improve HTTP transform configuration placeholders by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2702
• chore(webui): Add configurable validateStatus to redteam HTTP target setup by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2706

Documentation

• docs: Add redirect for troubleshooting link by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2653
• docs: Updated plugin table and harmful page by @vsauter in https://github.com/promptfoo/promptfoo/pull/2560
• docs(links): Update Discord links to new invite by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2675
• docs(strategy-table): Enhance grouping and ordering logic by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2640

Testing

• test: Add unit test for src/assertions/guardrail.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2656
• test: Add unit test for src/providers/promptfoo.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2662
• test: Add unit test for src/providers/simulatedUser.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2670
• test: Add unit test for src/providers/webhook.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2661
• test: Add unit test for src/redteam/plugins/indirectPromptInjection.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2663
• test: Add unit test for src/redteam/strategies/bestOfN.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2677
• test: Add unit test for src/redteam/strategies/likert.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2681
• test: Add unit test for src/utils/text.ts by @gru-agent in https://github.com/promptfoo/promptfoo/pull/2701
• test: Fix flaky test by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2715
• test: Make share test more robust by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2716
• test: Support randomizing test execution order by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2556

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.103.9...0.103.10

- TypeScript
Published by mldangelo about 1 year ago

promptfoo - 0.103.9

What's Changed

Features

• feat(tests): Import tests from JS/TS by @typpo in https://github.com/promptfoo/promptfoo/pull/2635
• feat(redteam): Add GCG strategy by @typpo in https://github.com/promptfoo/promptfoo/pull/2637
• feat(redteam): Add Likert-based jailbreak strategy by @AISimplyExplained in https://github.com/promptfoo/promptfoo/pull/2614

Fixes

• fix(webui): Revert 49bdcba - restore TruncatedText for var display by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2634

Chores

• chore(redteam): Catch errors during iterative attacks and continue by @sklein12 in https://github.com/promptfoo/promptfoo/pull/2631
• chore(redteam): GCG number config by @typpo in https://github.com/promptfoo/promptfoo/pull/2638
• chore(redteam): Wrap iterative providers in try/catch by @sklein12 in https://github.com/promptfoo/promptfoo/pull/2630
• chore(webui): Don't actually truncate vars because they are scrollable by @typpo in https://github.com/promptfoo/promptfoo/pull/2636

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.103.8...0.103.9

- TypeScript
Published by mldangelo about 1 year ago

promptfoo - 0.103.8

What's Changed

• fix: Running redteam from cloud by @sklein12 in https://github.com/promptfoo/promptfoo/pull/2627
• fix: redteam strategies by @sklein12 in https://github.com/promptfoo/promptfoo/pull/2629
• chore: show # plugins and strats selected by @typpo in https://github.com/promptfoo/promptfoo/pull/2628o/pull/2626

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.103.7...0.103.8

- TypeScript
Published by typpo about 1 year ago

promptfoo - 0.103.7

What's Changed

Fixes

• fix(webui): add default background for image lightbox by @typpo in https://github.com/promptfoo/promptfoo/pull/2616
• fix(openrouter): pass through openrouter-specific options by @typpo in https://github.com/promptfoo/promptfoo/pull/2620

Chores

• chore(redteam): record iterative history in metadata by @typpo in https://github.com/promptfoo/promptfoo/pull/2625
• chore(redteam): integrate grader into goat for ASR improvement by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/2612
• chore(cli): make db migrations quieter by @typpo in https://github.com/promptfoo/promptfoo/pull/2621
• chore(providers): update Azure API version for Azure provider by @vedantr in https://github.com/promptfoo/promptfoo/pull/2611
• chore: Revert "chore(redteam): expose redteam run command and auto-share remote results" by @sklein12 in https://github.com/promptfoo/promptfoo/pull/2613
• chore(deps): bump @aws-sdk/client-bedrock-runtime from 3.723.0 to 3.726.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/2618
• chore(deps): bump groq-sdk from 0.10.0 to 0.11.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/2619
• chore(deps): bump openai from 4.77.4 to 4.78.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/2617

Docs

• docs: owasp illustrations by @typpo in https://github.com/promptfoo/promptfoo/pull/2615
• docs: plugin and strategy graphics by @typpo in https://github.com/promptfoo/promptfoo/pull/2610
• docs(site): add vedant to the about page by @vedantr in https://github.com/promptfoo/promptfoo/pull/2622
• docs(site): update grid breakpoints for better spacing of team members on about page by @vedantr in https://github.com/promptfoo/promptfoo/pull/2623

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.103.6...0.103.7

- TypeScript
Published by vedantr about 1 year ago

promptfoo - 0.103.6

What's Changed

Bug Fixes

• fix(docker): correct string concatenation for BUILD_DATE in GitHub Actions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2603
• fix(providers): convert anthropic bedrock lone system messages to user messages for compatibility with model graded metrics by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2606

Chores

• chore(examples): add image saving hook for DALL-E outputs in redteam-dalle by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2607
• chore(redteam): expose redteam run command and auto-share remote results by @sklein12 in https://github.com/promptfoo/promptfoo/pull/2609
• chore(redteam): store attack prompt instead of rendered prompt in metadata by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2602
• chore(workflows): add actionlint GitHub Action for workflow validation by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2604
• chore(ci): updated yanked dependency and ruff format by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2608

Documentation

• docs(caching): expand documentation on caching mechanisms by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2605

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.103.5...0.103.6

- TypeScript
Published by mldangelo about 1 year ago

promptfoo - 0.103.5

What's Changed

New Features

• feat(fetch): Add support for custom SSL certificates by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2591

Bug Fixes

• fix(cli): filterMode failures should omit errors by @typpo in https://github.com/promptfoo/promptfoo/pull/2590
• fix(providers): Handle bad HTTP status code by @typpo in https://github.com/promptfoo/promptfoo/pull/2589
• fix(redteam): ascii-smuggling is a plugin, not a strategy by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/2585
• fix(redteam): Use OS-agnostic temp file by @typpo in https://github.com/promptfoo/promptfoo/pull/2586

Chores

• chore(assertions): Improve Python assertion configuration passing by @vedantr in https://github.com/promptfoo/promptfoo/pull/2583
• chore(debug): Enhance logging for null/undefined template variables by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2588
• chore(providers): Allow ability to set custom default embedding provider by @vedantr in https://github.com/promptfoo/promptfoo/pull/2587
• chore(providers): Improve error handling in HTTP provider by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2593
• chore(redteam): Add grader to crescendo to increase ASR by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/2594
• chore(webui): Add plugin category on the recently used cards by @typpo in https://github.com/promptfoo/promptfoo/pull/2600
• chore(webui): Highlight selected strats just like plugins by @typpo in https://github.com/promptfoo/promptfoo/pull/2601
• chore(webui): Replace initial prompt with last redteam prompt when it exists by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/2598
• chore(webui): Response parser -> response transform by @typpo in https://github.com/promptfoo/promptfoo/pull/2584

Dependencies

• chore(deps): Update dependencies to latest versions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2597

Documentation

• docs(license): Update year and clarify licensing terms by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2596
• docs(providers): Update overview table with new entries by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2592

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.103.4...0.103.5

- TypeScript
Published by mldangelo about 1 year ago

promptfoo - 0.103.4

What's Changed

Features

• feat(cli): add --filter-errors-only parameter to eval by @typpo in https://github.com/promptfoo/promptfoo/pull/2539
• feat(providers): f5 provider placeholder by @typpo in https://github.com/promptfoo/promptfoo/pull/2563
• feat(assertions): add support for specifying function names in external assertions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2548

Bug Fixes

• fix(assertions): ensure that Python assertions can reference the config as per the example given by @vedantr in https://github.com/promptfoo/promptfoo/pull/2551

Chores

• chore(providers): add support for the WATSONXAIAUTH_TYPE env by @gprem09 in https://github.com/promptfoo/promptfoo/pull/2547
• chore(providers): add debug logs to llama provider by @typpo in https://github.com/promptfoo/promptfoo/pull/2569
• chore(redteam): add debug to cyberseceval by @typpo in https://github.com/promptfoo/promptfoo/pull/2549
• chore(redteam): add english language cyberseceval by @typpo in https://github.com/promptfoo/promptfoo/pull/2561
• chore(redteam): adjust parameters for iterativeTree strategy by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2535
• chore(redteam): improve dialog content for load example configuration by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/2574
• chore(redteam): improve grader in jailbreak:tree strategy by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/2565
• chore(redteam): improve iterative provider with test case grader by @MrFlounder in https://github.com/promptfoo/promptfoo/pull/2552
• chore(redteam): improve tree node selection. Add metadata by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2538
• chore(redteam): reduce iterative image provider refusals by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2578
• chore(tests): improve misc test setup and teardown by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2579
• chore(webui): enhance metadata expand/collapse handling by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2550
• chore(webui): Add type for provider test response by @sklein12 in https://github.com/promptfoo/promptfoo/pull/2567
• chore(assertions): minor change to python assert example and revert provider to gpt4 mini by @vedantr in https://github.com/promptfoo/promptfoo/pull/2564
• chore(deps): update dependencies to latest minor and patch versions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2533
• chore(deps): bump @aws-sdk/client-bedrock-runtime from 3.716.0 to 3.721.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/2532
• chore(deps): bump @aws-sdk/client-bedrock-runtime from 3.721.0 to 3.723.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/2554
• chore(deps): bump openai from 4.77.0 to 4.77.3 by @dependabot in https://github.com/promptfoo/promptfoo/pull/2544
• chore(deps): update lock file to resolve dependency issues by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2545
• chore(deps): update lock file to resolve yanked dependency by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2581
• chore(webui): ensure provider overrides are displayed correctly by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2546

Documentation

• docs: improve dark mode styles on security page by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2562
• docs: jailbreak blog post by @typpo in https://github.com/promptfoo/promptfoo/pull/2575
• docs: missing plugins by @typpo in https://github.com/promptfoo/promptfoo/pull/2558
• docs: updates to llm vulnerability types page by @vsauter in https://github.com/promptfoo/promptfoo/pull/2527
• docs: updating typo for g-eval pages by @LiYChristopher in https://github.com/promptfoo/promptfoo/pull/2568
• docs(blog): improve the usage instructions for jailbreak dalle post by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2576
• docs(llm-vulnerability-scanner): improve dark mode styles by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2577
• docs(styles): improve dark mode styles for index page by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2580
• docs(troubleshooting): adjust sidebar order and update example version by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2557
• docs: only show frameworks in compliance section by @typpo in https://github.com/promptfoo/promptfoo/pull/2559
• chore(docs): improve dark mode on redteam configuration by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2553
• chore(docs): sort plugins by pluginId by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2536

New Contributors

• @LiYChristopher made their first contribution in https://github.com/promptfoo/promptfoo/pull/2568

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.103.3...0.103.4

- TypeScript
Published by MrFlounder about 1 year ago

promptfoo - 0.103.3

What's Changed

• feat: cyberseceval plugin by @typpo in https://github.com/promptfoo/promptfoo/pull/2523
• feat(redteam): add system prompt override plugin by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2524
• fix(gemini): support gemini thinking model by @typpo in https://github.com/promptfoo/promptfoo/pull/2526
• fix(docs): correct broken link in blog post by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2522
• fix(docs): conditionally enable gtag only in production by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2530
• chore(vertex): ability to override api version by @typpo in https://github.com/promptfoo/promptfoo/pull/2529
• chore: add more debug info to API health check by @typpo in https://github.com/promptfoo/promptfoo/pull/2531
• chore: switch cloud run to use --config param by @typpo in https://github.com/promptfoo/promptfoo/pull/2520
• docs: update owasp top 10 page by @vsauter in https://github.com/promptfoo/promptfoo/pull/2515
• docs(blog): unbounded consumption by @vsauter in https://github.com/promptfoo/promptfoo/pull/2521
• docs: misc improvements by @typpo in https://github.com/promptfoo/promptfoo/pull/2525

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.103.2...0.103.3

- TypeScript
Published by typpo about 1 year ago

promptfoo - 0.103.2

What's Changed

• feat: run redteam from cloud config by @sklein12 in https://github.com/promptfoo/promptfoo/pull/2503
• feat: divergent repetition plugin by @typpo in https://github.com/promptfoo/promptfoo/pull/2517
• docs: guardrails ui by @typpo in https://github.com/promptfoo/promptfoo/pull/2518
• feat: granular envars for memory control by @typpo in https://github.com/promptfoo/promptfoo/pull/2509
• fix: use default when importing cjs module by @typpo in https://github.com/promptfoo/promptfoo/pull/2506
• fix(moderation): handle empty output to avoid false positives by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2508
• fix(fetch): correct retries logic to ensure at least one attempt by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2513
• docs: readme overhaul by @typpo in https://github.com/promptfoo/promptfoo/pull/2502
• chore(redteam): make numIterations configurable for iterative strategy by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2511
• chore(webui): enhance styling and responsiveness for StrategyStats component by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2485
• chore(providers): make number of retry attempts configurable for HTTP provider by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2512
• chore(providers): add configurable retry attempts for AWS Bedrock. Improve error handling by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2514
• chore(redteam): handle empty and refusal responses by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2516
• docs: divergent repetition to plugins table by @vsauter in https://github.com/promptfoo/promptfoo/pull/2519

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.103.1...0.103.2

- TypeScript
Published by typpo about 1 year ago

promptfoo - 0.103.1

What's Changed

• fix: send config purpose when running in web ui by @typpo in https://github.com/promptfoo/promptfoo/pull/2504
• fix: include sharing in generated redteam config by @typpo in https://github.com/promptfoo/promptfoo/pull/2505
• docs: g-eval docs by @typpo in https://github.com/promptfoo/promptfoo/pull/2501

See notes for 0.103.0

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.103.0...0.103.1

- TypeScript
Published by typpo about 1 year ago

promptfoo - 0.103.0

What's Changed

New Features

• feat(eval): Add sheet identifier to Google Sheets URL for saving eval results by @MePriyanshuHoo in https://github.com/promptfoo/promptfoo/pull/2348
• feat(eval): Add support for Hugging Face datasets by @typpo in https://github.com/promptfoo/promptfoo/pull/2497
• feat(redteam): Ability to set the number of test cases per plugin by @typpo in https://github.com/promptfoo/promptfoo/pull/2480
• feat(redteam): Beavertails plugin by @typpo in https://github.com/promptfoo/promptfoo/pull/2500
• feat(redteam): Best-of-n jailbreak by @typpo in https://github.com/promptfoo/promptfoo/pull/2495
• feat(redteam): Dedicated custom input section by @typpo in https://github.com/promptfoo/promptfoo/pull/2493
• feat(redteam): Harmful:cybercrime:malicious-code by @typpo in https://github.com/promptfoo/promptfoo/pull/2481
• feat(redteam): Recently used plugins by @typpo in https://github.com/promptfoo/promptfoo/pull/2488
• feat(redteam): Support intent sequences by @typpo in https://github.com/promptfoo/promptfoo/pull/2487

Bug Fixes

• fix(redteam): Ability to set custom target by @typpo in https://github.com/promptfoo/promptfoo/pull/2483
• fix(redteam): Apply delay to redteam providers by @typpo in https://github.com/promptfoo/promptfoo/pull/2498
• fix(redteam): Scroll to top when changing tabs by @typpo in https://github.com/promptfoo/promptfoo/pull/2484
• fix(redteam): State management for raw HTTP requests by @typpo in https://github.com/promptfoo/promptfoo/pull/2491

Chores

• chore(redteam): Add "View Probes" button by @typpo in https://github.com/promptfoo/promptfoo/pull/2492
• chore(redteam): Enhance metadata tracking for iterative provider by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2482
• chore(redteam): Improve scoring in iterative providers by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2486
• chore(redteam): Record stateless telemetry by @typpo in https://github.com/promptfoo/promptfoo/pull/2477
• chore(examples): Revert redteam-ollama example to previous version by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2499
• chore(deps): Bump @aws-sdk/client-bedrock-runtime from 3.714.0 to 3.716.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/2479

Documentation

• docs: Cyberseceval by @typpo in https://github.com/promptfoo/promptfoo/pull/2494
• docs: Plugins overview by @vsauter in https://github.com/promptfoo/promptfoo/pull/2448
• docs: Strategy overview by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2449
• docs(providers): Add new providers to documentation by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2496

New Contributors

• @MePriyanshuHoo made their first contribution in https://github.com/promptfoo/promptfoo/pull/2348

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.102.4...0.103.0

- TypeScript
Published by mldangelo about 1 year ago

promptfoo - 0.102.4

What's Changed

• feat: add G-Eval assertion by @schipiga in https://github.com/promptfoo/promptfoo/pull/2436
• feat: ability to set delay from webui by @typpo in https://github.com/promptfoo/promptfoo/pull/2474
• fix(provider): ensure system prompt is formatted correctly for amazon nova models by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2476
• fix: resolve circular reference issue in groq provider by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2475
• chore: placeholder for ied by @typpo in https://github.com/promptfoo/promptfoo/pull/2478
• docs(red-team): update default strategy documentation by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2473

New Contributors

• @schipiga made their first contribution in https://github.com/promptfoo/promptfoo/pull/2436

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.102.3...0.102.4

- TypeScript
Published by typpo about 1 year ago

promptfoo - 0.102.3

What's Changed

• feat: pliny plugin by @typpo in https://github.com/promptfoo/promptfoo/pull/2469
• feat: meth plugin by @typpo in https://github.com/promptfoo/promptfoo/pull/2470
• fix(redteam): resolve prompt rendering issue in goat provider by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2472
• chore(deps): update dependencies to latest versions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2442

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.102.2...0.102.3

- TypeScript
Published by typpo about 1 year ago

promptfoo - 0.102.2

What's Changed

Features

• feat(eval): Add metadata filtering to promptfoo eval by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2460
• feat(redteam): Implement basic strategy to skip strategy-less tests by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2461
• feat(redteam): Show messages for multi-turn providers by @typpo in https://github.com/promptfoo/promptfoo/pull/2454
• feat(webui): Add search bar for reports by @typpo in https://github.com/promptfoo/promptfoo/pull/2458

Bug Fixes

• fix(providers): Fix O1 model detection by @typpo in https://github.com/promptfoo/promptfoo/pull/2455
• fix(redteam): Handle invalid message from GOAT by @typpo in https://github.com/promptfoo/promptfoo/pull/2462
• fix(redteam): Handle null target model responses in GOAT and improve safeJsonStringify by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2465
• fix(redteam): Improve logging and message handling in Crescendo and GOAT providers by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2467
• fix(redteam): Properly write target response to iterative by @typpo in https://github.com/promptfoo/promptfoo/pull/2447
• fix(redteam): Skip iterative turn on refusal by @typpo in https://github.com/promptfoo/promptfoo/pull/2464

Chores

• chore(providers): Add new OpenAI O1 model versions by @typpo in https://github.com/promptfoo/promptfoo/pull/2450
• chore(ci): Handle fork PRs without secrets correctly by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2443
• chore(ci): Update Node.js 22.x matrix configuration by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2444
• chore(ci): Move workflow assets to .github/assets by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2445
• chore(deps): Bump @anthropic-ai/sdk from 0.32.1 to 0.33.1 by @dependabot in https://github.com/promptfoo/promptfoo/pull/2451
• chore(deps): Bump @aws-sdk/client-bedrock-runtime from 3.712.0 to 3.714.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/2446
• chore(deps): Bump openai from 4.76.3 to 4.77.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/2452
• chore(redteam): Update target handling for model-based strategies by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2466

Documentation

• docs: Update RAG red team details by @vsauter in https://github.com/promptfoo/promptfoo/pull/2459

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.102.1...0.102.2

- TypeScript
Published by mldangelo about 1 year ago

promptfoo - 0.102.1

What's Changed

New Features

• feat(redteam): ability to upload intents from csv by @typpo in https://github.com/promptfoo/promptfoo/pull/2424
• feat(redteam): switch to rag example by @typpo in https://github.com/promptfoo/promptfoo/pull/2432

Bug Fixes

• fix(redteam): add support for entity merging in config by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2433
• fix(redteam): combine strategy configs for chained strategies by @sklein12 in https://github.com/promptfoo/promptfoo/pull/2415
• fix(redteam): don't fall back if entity and purpose extraction fails by @sklein12 in https://github.com/promptfoo/promptfoo/pull/2428
• fix(redteam): integration test by @sklein12 in https://github.com/promptfoo/promptfoo/pull/2431
• fix(redteam): make cross-session-leak not default by @typpo in https://github.com/promptfoo/promptfoo/pull/2427
• fix(redteam): remove duplicate intent plugin by @typpo in https://github.com/promptfoo/promptfoo/pull/2426
• fix(redteam): dark mode in test targets ui by @typpo in https://github.com/promptfoo/promptfoo/pull/2425
• fix(redteam): resolve invalid DOM nesting of ul elements in Strategies component by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2421
• fix(evaluator): handle circular references during error logging by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2441

Chores

• chore(cli): address punycode deprecation warning for Node.js 22 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2440
• chore(redteam): format extraction prompts as chat messages by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2429
• chore(redteam): integration tests by @sklein12 in https://github.com/promptfoo/promptfoo/pull/2413
• chore(redteam): move plugin collections out of plugin type by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2435
• chore(redteam): raise timeout on unaligned provider to 60s by @sklein12 in https://github.com/promptfoo/promptfoo/pull/2434
• chore(redteam): update owasp mappings by @typpo in https://github.com/promptfoo/promptfoo/pull/2316
• chore(redteam): update plugin and strategy display names and descriptions by @typpo in https://github.com/promptfoo/promptfoo/pull/2387
• chore(redteam): minor styling improvements to TestTargetConfiguration by @typpo in https://github.com/promptfoo/promptfoo/pull/2430
• chore(redteam): remove horizontal scroll from redteam setup tabs by @mldangelo in https://github.com/promptfoo/promptfoo/pull/2420

Dependencies

• chore(deps): bump @aws-sdk/client-bedrock-runtime from 3.709.0 to 3.712.0 by @dependabot in https://github.com/promptfoo/promptfoo/pull/2418
• chore(deps): bump groq-sdk from 0.9.0 to 0.9.1 by @dependabot in https://github.com/promptfoo/promptfoo/pull/2416
• chore(deps): bump openai from 4.76.2 to 4.76.3 by @dependabot in https://github.com/promptfoo/promptfoo/pull/2417

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.102.0...0.102.1

- TypeScript
Published by mldangelo about 1 year ago