pyrit - v0.9.0

What's Changed

Targets * HTTPTarget Improvements that properly parse the HTTP version, automatically calculate the content-length, and make headers case insensitive. * FIX: Fixed IndexError with RealtimeTarget to handle responses properly

Datasets * Social Engineering (Persuasion and Deception) Scenarios: See datasets/orchestrators/red_teaming/persuasion_deception and datasets/orchestrators/role_play/persuasion_script.yaml * Multilingual Vulnerability dataset from "A Framework to Assess Multilingual Vulnerabilities of LLMs"

Converters * Enhancements to the AsciiSmugglerConverter by adding support for two methods for encoding hidden data (embedding directly in a Unicode character (default: 😊) and appending hidden data to visible text). * ZalgoConverter: Adds Unicode characters to text to make it appear "glitchy" * ToxicSentenceGeneratorConverter: Generate toxic sentence starters based on seed prompts * FIX: Remove JSON Instructions for TranslationConverter to address intermittent failures due to JSON parsing issues and non-consistent responses from endpoints.

Orchestrators * [BREAKING] Rename MultiTurnAttackResult to OrchestratorResult as part of a bigger refactor to tack objectives and results. * FIX: Keep Conversation ID in PromptSendingOrchestrator if it is provided * FIX: Remove Harm-Specific Prevention from CrescendoOrchestrator

Scorers * Generic Scorer with Flexible Inputs: SelfAskGeneralScorer in pyrit/score/general_scorer.py. It can be configured to use different scoring types (e.g. True/False, float) and can format the prompt using a system prompt and a format string. * Criteria-Based Scorer (used with SelfAskScaleScorer): Provides evaluation criteria that is specific to a given objective. * CompositeScorer: Combines multiple True/False Results into a single True/False Result

Dependencies * Moves jupyter and ipykernel from required into an optional [dev] dependency. If you need to use Jupyter notebooks with PyRIT, you'll need to install using methods outlined here. * Moves azure-cognitiveservices-speech from required into an optional [speech] dependency.

Other * Added custom file name support to allows for saving data (image, audio, video, etc.) to storage under a custom name. * Custom Retry Decorator: pyrit_custom_result_retry to retry a function if a certain condition is true. This augments existing retry decorators which retry functions based on exception criteria. * Optimizations and various bug fixes to .devcontainer

Full list of changes

• [FEAT] New Generic Scorer with Flexible Inputs by @jbolor21 in https://github.com/Azure/PyRIT/pull/816
• MAINT post-v0.8.2.dev0 release updates by @romanlutz in https://github.com/Azure/PyRIT/pull/861
• DOC: add LM Studio support note to the user guide by @paulinek13 in https://github.com/Azure/PyRIT/pull/863
• MAINT: Make integration tests run outside of repository and various fixes by @jsong468 in https://github.com/Azure/PyRIT/pull/862
• FEAT: Add Custom File Name Support to Data Serializer by @nina-msft in https://github.com/Azure/PyRIT/pull/868
• FEAT: Add Custom Retry Decorator: pyritcustomresult_retry by @nina-msft in https://github.com/Azure/PyRIT/pull/869
• FEAT: optimized .devcontainer by @bashirpartovi in https://github.com/Azure/PyRIT/pull/871
• DOC: Fix Up Multi Turn Target Docs & OpenAI Dalle/TTS Target Docstring by @nina-msft in https://github.com/Azure/PyRIT/pull/870
• DOC: improve accessibility of the contributor guide flowchart by @paulinek13 in https://github.com/Azure/PyRIT/pull/866
• FIX: fixed the extension directory for vscode by @bashirpartovi in https://github.com/Azure/PyRIT/pull/872
• FIX jupyter set as dev dependency by @afogel in https://github.com/Azure/PyRIT/pull/857
• MAINT enhanced initialization and caching for devcontainer by @bashirpartovi in https://github.com/Azure/PyRIT/pull/873
• FIX: fixed indexing and conda cache for devcontainer by @bashirpartovi in https://github.com/Azure/PyRIT/pull/876
• FIX: Resolve mypy pre-commit error in chatmessagenormalizer_tokenizer by @nina-msft in https://github.com/Azure/PyRIT/pull/875
• MAINT: HTTPTarget Improvements by @rlundeen2 in https://github.com/Azure/PyRIT/pull/879
• FEAT: Smuggling arbitrary data through an emoji by @KutalVolkan in https://github.com/Azure/PyRIT/pull/842
• DOC fix markdown link by @dennis-rall in https://github.com/Azure/PyRIT/pull/880
• FEAT Persuasion and Deception Scenarios by @whackswell in https://github.com/Azure/PyRIT/pull/878
• FIX: Update re.split calls to use maxsplit keyword argument by @emmanuel-ferdman in https://github.com/Azure/PyRIT/pull/885
• BREAKING FEAT: orchestrator result by @rlundeen2 in https://github.com/Azure/PyRIT/pull/886
• FEAT: Added Multilingual Vulnerability Dataset by @devesh-2002 in https://github.com/Azure/PyRIT/pull/834
• FIX keep conversation ID in PromptSendingOrchestrator if it's passed in by @romanlutz in https://github.com/Azure/PyRIT/pull/889
• FEAT Adding into Criteria based scoring by @eugeniavkim in https://github.com/Azure/PyRIT/pull/874
• FIX fixed msodbcsql dep for devcontainer by @bashirpartovi in https://github.com/Azure/PyRIT/pull/895
• MAINT: Remove Azure Speech SDK as Required Dependency by @nina-msft in https://github.com/Azure/PyRIT/pull/896
• FIX pip upgrade issue on windows by @bashirpartovi in https://github.com/Azure/PyRIT/pull/901
• FEAT: Zalgo Converter by @elisetreit in https://github.com/Azure/PyRIT/pull/883
• FEAT: Composite Scorer by @rlundeen2 in https://github.com/Azure/PyRIT/pull/898
• FIX: XPIA Notebook Env Variable Fix by @jbolor21 in https://github.com/Azure/PyRIT/pull/899
• FIX: bug where scorer_type is not set in AzureContentFilterScorer by @rlundeen2 in https://github.com/Azure/PyRIT/pull/902
• MAINT: Generic Scorer Notebook Reorganizing by @jbolor21 in https://github.com/Azure/PyRIT/pull/904
• MAINT Refactor question answer orchestrator as prompt orchestrator by @AdrGav941 in https://github.com/Azure/PyRIT/pull/894
• FEAT: Toxic Sentence Generator by @0xm00n in https://github.com/Azure/PyRIT/pull/893
• FIX Removed JSON instructions for Translation Converter by @bashirpartovi in https://github.com/Azure/PyRIT/pull/910
• FIX Removing harm specific prevention for Crescendo Orchestrator @eugeniavkim in https://github.com/Azure/PyRIT/pull/911
• FIX IndexError with RealtimeTarget @bashirpartovi in https://github.com/Azure/PyRIT/pull/914
• DOC Updates to '11. Releasing PyRIT' documentation @nina-msft

New Contributors

• @afogel made their first contribution in https://github.com/Azure/PyRIT/pull/857
• @dennis-rall made their first contribution in https://github.com/Azure/PyRIT/pull/880
• @whackswell made their first contribution in https://github.com/Azure/PyRIT/pull/878
• @emmanuel-ferdman made their first contribution in https://github.com/Azure/PyRIT/pull/885
• @devesh-2002 made their first contribution in https://github.com/Azure/PyRIT/pull/834
• @elisetreit made their first contribution in https://github.com/Azure/PyRIT/pull/883
• @0xm00n made their first contribution in https://github.com/Azure/PyRIT/pull/893

Full Changelog: https://github.com/Azure/PyRIT/compare/v0.8.1...v0.9.0

- Python
Published by nina-msft 10 months ago

pyrit - v0.8.1

What's Changed

• We have a new cookbook on Precomputing turns for orchestrators
• OpenAIChatTargets now have an argument is_json_supported to allow specifying if the response_format request header should be set. This is supported by OpenAI, but not by several other providers that otherwise follow the OpenAI API.
• There is now a Docker image for PyRIT users! Check out the steps outlined in the docker/README to try it out and feel free to provide feedback in GitHub issues or on Discord.
• The Tom-and-Jerry jailbreak template was added!
• When using AAD/Entra auth with OpenAITarget, the target auto-refreshes the auth token periodically now. This addresses a bug where the token would get stale after a period of time.
• We also addressed bugs that resulted in exceptions from triggered content filters and empty exception which should lead to a smoother experience.

Full list of changes

• MAINT post-v0.8.0 release update by @romanlutz in https://github.com/Azure/PyRIT/pull/837
• MAINT: Making JSON support configurable with OpenAIChatTargets by @rlundeen2 in https://github.com/Azure/PyRIT/pull/833
• FEAT: Add Dockerized PyRIT with Jupyter Notebook Support by @ErdemOzgen in https://github.com/Azure/PyRIT/pull/784
• FEAT: add Tom-and-Jerry jailbreak by @hagsmand in https://github.com/Azure/PyRIT/pull/838
• DOC: Adding cookbook around prepending turns by @rlundeen2 in https://github.com/Azure/PyRIT/pull/840
• FIX: Small fix in cookbook by @jsong468 in https://github.com/Azure/PyRIT/pull/849
• FIX catch content_filter with 200s instead of 500s by @romanlutz in https://github.com/Azure/PyRIT/pull/850
• FIX: Amended dockerfile and requirements.txt to unblock ADO pipelines by @jsong468 in https://github.com/Azure/PyRIT/pull/853
• FIX add zero width and insert punctuation converters to init.py file by @AnnaRevutsky in https://github.com/Azure/PyRIT/pull/848
• FIX: AAD Auth refresh bug with OpenAITargets by @rlundeen2 in https://github.com/Azure/PyRIT/pull/855
• FIX handle empty exception message in validation by @romanlutz in https://github.com/Azure/PyRIT/pull/859

New Contributors

• @ErdemOzgen made their first contribution in https://github.com/Azure/PyRIT/pull/784
• @AnnaRevutsky made their first contribution in https://github.com/Azure/PyRIT/pull/848

Full Changelog: https://github.com/Azure/PyRIT/compare/v0.8.0...v0.8.1

- Python
Published by romanlutz 11 months ago

pyrit - v0.8.0

What's Changed

Targets: - HTTPTarget now supports rate limiting - Some users encountered errors in Azure OpenAI when hitting content filter errors using error code 500. PyRIT now catches content filter responses with both error codes 400 (as before) and 500 (new) and returns a clean response record.

Datasets: - fetch_babelscape_alert_dataset had a bug causing it to be limited to a single category even when users specified both. This is now fixed! - added fetch_red_team_social_bias_dataset - added fetch_darkbench_dataset - added fetch_mlcommons_ailuminate_demo_dataset

Converters: - added UnicodeReplacementConverter - added sneaky_bits option to AsciiSmugglerConverter in the encoding_mode argument. The unicode_tags argument is now removed and replaced by more options in encoding_mode (i.e., unicode_tags, unicode_tags_control, and sneaky_bits).

Scanner: A basic version was introduced in v0.7.0 that supported only sending single-turn prompts. v0.8.0 expands on this with support for most multi-turn orchestrators (incl. adversarial chat targets and scorers) and memory. This feature is still considered experimental and may change considerably in the following versions.

Other: - support for Python 3.13 in addition to 3.10-3.12. - For single-piece responses, we now have a convenient get_value() method. - PyRIT used to print warnings that torch isn't installed (unless the corresponding extra was installed). This was caused by transformers and is now turned off as it doesn't serve any purpose. - In previous versions, PyRIT started supporting .env.local as an override to the .env file for endpoint secrets. However, when using this outside of the normal repository structure (e.g., when running PyRIT without cloning this repo) the code failed to discover .env.local in the current working directory. This is now fixed.

Full list of changes

• [DevContainer] Provide a uniform development environment by @bashirpartovi in https://github.com/Azure/PyRIT/pull/787
• FEAT: Add Rate Limit Support for HTTP Target by @nina-msft in https://github.com/Azure/PyRIT/pull/786
• DOC Updating contribution docs by @bashirpartovi in https://github.com/Azure/PyRIT/pull/788
• MAINT support python 3.13 by @AdrGav941 in https://github.com/Azure/PyRIT/pull/779
• FIX: fixed dev container permission issue by @bashirpartovi in https://github.com/Azure/PyRIT/pull/789
• FEAT: simplify extraction of converted values from responses by @paulinek13 in https://github.com/Azure/PyRIT/pull/783
• MAINT: improve organization of dataset fetch functions (refactoring) by @paulinek13 in https://github.com/Azure/PyRIT/pull/785
• FEAT: Added cross-platform compatibility and needed language support for toml and docker by @bashirpartovi in https://github.com/Azure/PyRIT/pull/797
• MAINT: Update release version to 0.7.1.dev0 by @jsong468 in https://github.com/Azure/PyRIT/pull/800
• FIX: prevent data overwrite in fetch_babelscape_alert_dataset by @paulinek13 in https://github.com/Azure/PyRIT/pull/799
• DOC contributor guide flowchart, small text updates, and add Roakey to README by @romanlutz in https://github.com/Azure/PyRIT/pull/798
• DOC: clarify OpenAITarget targets httpxclientkwargs timeout settings by @clod81 in https://github.com/Azure/PyRIT/pull/801
• FIX: Add exception on response parsing when call to Openrouter.ai by @hagsmand in https://github.com/Azure/PyRIT/pull/796
• FIX make sure conversation IDs are not sent out as UUIDs to the database by @ayeganov in https://github.com/Azure/PyRIT/pull/723
• FEAT support adversarial_chat and scoring in scanner to enable automated multi-turn-orchestrators by @romanlutz in https://github.com/Azure/PyRIT/pull/706
• FIX move misplaced test file to tests/unit/converter by @romanlutz in https://github.com/Azure/PyRIT/pull/794
• FEAT: Added Red Team Social Bias dataset by @MoolmanM in https://github.com/Azure/PyRIT/pull/714
• DOC improve API reference for auth, cli, common, chatmessagenormalizer by @romanlutz in https://github.com/Azure/PyRIT/pull/793
• FEAT: UnicodeReplacementConverter by @nina-msft in https://github.com/Azure/PyRIT/pull/803
• FIX: Updating pre-commit to fix build issues by @rlundeen2 in https://github.com/Azure/PyRIT/pull/810
• MAINT: Making test_connect more resilient by @rlundeen2 in https://github.com/Azure/PyRIT/pull/806
• [FIX] fix bad domain by @mgstate in https://github.com/Azure/PyRIT/pull/815
• [FIX] Integration test fixes: add hugging face token in notebook and fix testfetchdatasets by @jsong468 in https://github.com/Azure/PyRIT/pull/819
• FEAT: Added memory config to scanner by @bashirpartovi in https://github.com/Azure/PyRIT/pull/808
• FEAT: add DarkBench dataset by @paulinek13 in https://github.com/Azure/PyRIT/pull/821
• MAINT: improving build/test time by @bashirpartovi in https://github.com/Azure/PyRIT/pull/820
• FIX handle Azure OpenAI content_filter errors with HTTP status code 500 by @romanlutz in https://github.com/Azure/PyRIT/pull/825
• FIX turn off transformers warning by @romanlutz in https://github.com/Azure/PyRIT/pull/829
• TEST: Adding integration test for content filters by @rlundeen2 in https://github.com/Azure/PyRIT/pull/830
• MAINT: Separating integration test local .env by @rlundeen2 in https://github.com/Azure/PyRIT/pull/817
• FEAT: add MLCommons AILuminate v1.0 DEMO Prompt Set by @paulinek13 in https://github.com/Azure/PyRIT/pull/828
• FIX find .env.local in current working directory by @romanlutz in https://github.com/Azure/PyRIT/pull/832
• BREAKING FEAT: Sneaky Bits - Advanced Data Smuggling Techniques by @KutalVolkan in https://github.com/Azure/PyRIT/pull/827
• FEAT add ps-fuzz prompts by @ryanjieh in https://github.com/Azure/PyRIT/pull/823

New Contributors

• @bashirpartovi made their first contribution in https://github.com/Azure/PyRIT/pull/787
• @clod81 made their first contribution in https://github.com/Azure/PyRIT/pull/801
• @hagsmand made their first contribution in https://github.com/Azure/PyRIT/pull/796
• @MoolmanM made their first contribution in https://github.com/Azure/PyRIT/pull/714
• @mgstate made their first contribution in https://github.com/Azure/PyRIT/pull/815
• @ryanjieh made their first contribution in https://github.com/Azure/PyRIT/pull/823

Full Changelog: https://github.com/Azure/PyRIT/compare/v0.7.0...v0.8.0

- Python
Published by romanlutz 11 months ago

pyrit - v0.7.0

What's Changed

Targets: * [BREAKING] OpenAIChatTarget has become more generalized to more broadly support OpenAI-compatible models. See the blog describing the changes here! * If api_version is set to None when instantiating OpenAITarget objects, it will not be added as a query parameter to requests. * Added Google Gemini example environment variables to .env_example and added integration tests for Gemini/OpenAIChatTargets

Converters: * [New] AddImageVideoConverter: PyRIT's first video converter! it allows users to add an image to a video in at a specified position. More video converters to come! * [New] InsertPunctuationConverter: Inserts various punctuation into a prompt to test model robustness to perturbations.

Orchestrators: * [New] ManyShotJailbreakOrchestrator: Prepend a faux dialogue between a human and an AI assistant within a single prompt for the target. * [New] [BREAKING] ContextComplianceOrchestrator: Update the context to prime an objective_chat_target to answer. The context is set using instructions defined in context_description_instructions_path, along with an adversarial_chat to generate the first turns to send. * [BREAKING] RolePlayOrchestrator improvements: Refactored for greater code re-use * FlipAttackOrchestrator improvement: Allow for additional converters applied after the flip attack

Memory: * Multimodal Seed Prompts Encoding Metadata: Adding non-text seed prompts to the database will automatically have metadata populated, including format (png, wav, etc.) and things like bitrate and duration for audio and video seed prompts. * SeedPrompt Duplicates: Duplicate seed prompts within the same dataset (identical dataset_name) will no longer be uploaded to memory. * Using Configured Paths for Multimodal Seed Prompts: Multimodal SeedPrompt file paths within .yaml files no longer use relative paths that break based on where the .yaml files are accessed. Instead, configured paths (located in paths.py) are used. * [BREAKING] Removed calls to disposing memory engines in Orchestrator and Prompt Target objects and replaces it with the atexit and weakref methods of cleanup in the Memory interface to ensure cleanup on process exit. Orchestrators and targets no longer support the context manager protocol. * Added get_values() method to the SeedPromptDataset class to simplify prompt values extraction from datasets. Optional filtering to retrieve the first and/or last N values has also been implemented.

Scorers: * [New] HumanInTheLoopScorerGradio: Create scores from manual human input by running the Gradio interface in a separate process and adds the scores to the database. For now, the possible scores that users can give are "safe" and "unsafe."

Datasets: * [New] Added new fetch function for Aya Red-Teaming Dataset * [New] Added Pliny's prompts from the l1b3rt4s repo as templates * [New] Added the Babelscape ALERT dataset * Added support for filtering based on harm categories for PKU-SafeRLHF and AdvBench datasets

Misc: * Other changes include various maintenance improvements and bug fixes, addition of integration tests, website enhancements, dependency updates, and doc improvements.

Full list of changes

• FIX unblock test pipelines by skipping certain tests on Ubuntu and adding Windows additionally by @romanlutz in https://github.com/Azure/PyRIT/pull/727
• MAINT: Update release version to 0.6.1.dev0 by @nina-msft in https://github.com/Azure/PyRIT/pull/731
• MAINT: Upgrading DuckDB by @jbolor21 in https://github.com/Azure/PyRIT/pull/712
• [FEAT][MAINT][4019] Make multi-modal easier to configure in seedprompt files by @shivenchawla in https://github.com/Azure/PyRIT/pull/696
• FEAT: set favicon for the website by @paulinek13 in https://github.com/Azure/PyRIT/pull/717
• FEAT: simplify extracting prompt values by @paulinek13 in https://github.com/Azure/PyRIT/pull/718
• FEAT: add a fetch function for Aya Red-teaming Dataset by @paulinek13 in https://github.com/Azure/PyRIT/pull/713
• MAINT update Roakey image to have transparent background by @romanlutz in https://github.com/Azure/PyRIT/pull/735
• FEAT Moonshot Attack Module: Insert Punctuation Attack by @u7780339 in https://github.com/Azure/PyRIT/pull/475
• FEAT: include scoredpromptid in orchestrator_identifier of the system prompt by @NicolePell in https://github.com/Azure/PyRIT/pull/725
• FEAT: Create many shot jailbreak orchestrator by @AdrGav941 in https://github.com/Azure/PyRIT/pull/709
• MAINT pre-commit hook to remove notebook header from notebooks by @jbolor21 in https://github.com/Azure/PyRIT/pull/737
• FEAT Add Encoding Data to Multimodal Seed Prompts by @jsong468 in https://github.com/Azure/PyRIT/pull/740
• FEAT added Pliny's prompts from the l1b3rt4s repo as templates by @joaodunas in https://github.com/Azure/PyRIT/pull/710
• FEAT Adding babelscape dataset by @Jarro01X in https://github.com/Azure/PyRIT/pull/738
• FIX: Upgrading Packages by @rlundeen2 in https://github.com/Azure/PyRIT/pull/741
• FIX: Increasing pipeline timout by @rlundeen2 in https://github.com/Azure/PyRIT/pull/743
• FEAT PyRIT to not upload duplicate seed-prompts by @shivenchawla in https://github.com/Azure/PyRIT/pull/742
• MAINT: Azure SQL Integration Test Misc. Updates by @nina-msft in https://github.com/Azure/PyRIT/pull/745
• FIX Small bug fixes (renaming file, editing MANIFEST) by @jsong468 in https://github.com/Azure/PyRIT/pull/746
• [BREAKING] FEAT: OpenAI Generalization Improvements by @rlundeen2 in https://github.com/Azure/PyRIT/pull/747
• FEAT: Add example_count field to ManyShotJailbreakOrchestrator by @nina-msft in https://github.com/Azure/PyRIT/pull/748
• DOC: Blog: A More Generalized OpenAIChatTarget by @rlundeen2 in https://github.com/Azure/PyRIT/pull/751
• DOC: Updating git docs by @rlundeen2 in https://github.com/Azure/PyRIT/pull/753
• FIX: Fixing integration tests broken with OpenAIChatTarget Update by @rlundeen2 in https://github.com/Azure/PyRIT/pull/755
• FEAT Video Converter: Adding Images to Videos by @jbolor21 in https://github.com/Azure/PyRIT/pull/702
• FIX: Adding back static js by @rlundeen2 in https://github.com/Azure/PyRIT/pull/761
• [BREAKING] FEAT: RolePlayOrchestrator Improvements by @rlundeen2 in https://github.com/Azure/PyRIT/pull/758
• [BREAKING] FIX: Dispose Memory in Memory vs Class Objects by @nina-msft in https://github.com/Azure/PyRIT/pull/752
• MAINT clean up dependencies by @romanlutz in https://github.com/Azure/PyRIT/pull/757
• FEAT Adding converter support to many shot jailbreak orchestrator by @AdrGav941 in https://github.com/Azure/PyRIT/pull/760
• FIX: Default API Version for TTS Target by @jbolor21 in https://github.com/Azure/PyRIT/pull/749
• [BREAKING] FEAT: Adding Context Compliance Orchestrator by @rlundeen2 in https://github.com/Azure/PyRIT/pull/763
• DOC: Add Instructions for Tagging Breaking Changes in PR Template by @nina-msft in https://github.com/Azure/PyRIT/pull/765
• FEAT: support filtering based on harm categories for PKU-SafeRLHF dataset by @paulinek13 in https://github.com/Azure/PyRIT/pull/756
• DOC Update CCA Documentation for Clarity by @eugeniavkim in https://github.com/Azure/PyRIT/pull/773
• DOC: Update OpenAI Environment Variable Names in Documentation by @nina-msft in https://github.com/Azure/PyRIT/pull/776
• FEAT: add harm categories to AdvBench Dataset by @paulinek13 in https://github.com/Azure/PyRIT/pull/732
• FIX: Allow api_version to be set to None when instantiating OpenAITarget objects by @LeoVrana in https://github.com/Azure/PyRIT/pull/764
• MAINT standardize Hugging Face token environment variable, add integration tests for Google Gemini and Open AI by @romanlutz in https://github.com/Azure/PyRIT/pull/778
• FEAT: Gradio HiTL Scorer by @mart123p in https://github.com/Azure/PyRIT/pull/722
• DOC: clarify OpenAIChatTarget usage with Ollama by @jsdlm in https://github.com/Azure/PyRIT/pull/777
• FIX: small edits to make integration tests pass by @jsong468 in https://github.com/Azure/PyRIT/pull/780
• MAINT add notice generation to component governance by @romanlutz in https://github.com/Azure/PyRIT/pull/781
• MAINT update NOTICE file by @romanlutz in https://github.com/Azure/PyRIT/pull/782

New Contributors

• @u7780339 made their first contribution in https://github.com/Azure/PyRIT/pull/475
• @NicolePell made their first contribution in https://github.com/Azure/PyRIT/pull/725
• @joaodunas made their first contribution in https://github.com/Azure/PyRIT/pull/710
• @Jarro01X made their first contribution in https://github.com/Azure/PyRIT/pull/738
• @LeoVrana made their first contribution in https://github.com/Azure/PyRIT/pull/764

Full Changelog: https://github.com/Azure/PyRIT/compare/releases/v0.6.0...releases/v0.7.0

- Python
Published by jsong468 11 months ago

pyrit - v0.6.0

What's Changed

• Cookbooks are live, and replace our How To Guide! Cookbooks try to tackle a problem and use the components that work best, instead of our typical documentation which illustrates that many pieces of PyRITs are swappable.

Cookbooks: * Sending a Million Prompts

Targets: * OllamaChatTarget: Implement ability to forward custom parameters directly to the HTTP client * HuggingFaceChatTarget: Adds optional keywords device_map, torch_dtype and attn_implementation * [New] PlaywrightTarget: Interact with web applications using Playwright. This is particularly useful for testing interactions with web interfaces like chatbots. * [New] RealtimeTarget: Send and receive audio with the Realtime API. * [New] GroqChatTarget: Interact with Groq's OpenAI-compatible API.

Converters: * [New] ANSI Escape Code Converter: AnsiAttackConverter * [New] BinaryConverter: Convert input text into binary with configurable bits per character * PDFConverter: Updates to support templated and non-templated PDF generation & enabling text injection into existing PDFs * [New] TextToHexConverter: Convert text to hexadecimal encoded utf-8 string * Add easier querying for converter-supported input/output types

Orchestrators: * RedTeamingOrchestrator & CrescendoOrchestrator now support prepended conversations. You can set a system prompt on the objective target using this feature, or provide conversation history as context to continue execution from a specific point. * ScoringOrchestrator: Add ability to score responses using filters. * PromptSendingOrchestrator: Set Skip Criteria to specify which prompts to skip being sent to the target with this orchestrator. * [New] RolePlayingOrchestrator: Single-turn orchestrator which prepends some prompts which describe fictional scenarios to attempt and elicit harmful responses * XPIAOrchestrator: Fix to BlobNotFound exception   Memory: * [BREAKING] All notebooks must explicitly initialize Central Memory through a new initialize_pyrit() function: https://github.com/Azure/PyRIT/pull/616. This puts ownership into the hands of the user to set where your prompts will be stored. Read more here: Memory * Ability to add memory labels on a per-prompt level, specifically useful in Multimodal scenarios * Conversation Scores now available when exporting Prompt Data * Filter Data by various queries (e.g. prompt ID, orchestrator ID, labels, etc) using get_prompt_request_pieces() * Consolidated method to Export Conversations using Filters: export_conversations() * SeedPrompts: Support for Multimodal Seed Prompts * [BREAKING] NormalizerRequestPieces replaced with SeedPrompts: https://github.com/Azure/PyRIT/pull/648

Scorers: * Add tasks by default to scorers to improve scorer accuracy

Misc: * Other changes include various maintenance improvements and bug fixes, addition of integration tests, new blog posts, and doc improvements.

Full list of changes

• MAINT Update release version to 0.5.3.dev0 by @rdheekonda in https://github.com/Azure/PyRIT/pull/592
• DOC: Multi-turn docs and blog post by @rlundeen2 in https://github.com/Azure/PyRIT/pull/593
• DOC: Fixing title by @rlundeen2 in https://github.com/Azure/PyRIT/pull/594
• MAINT: Update Memory Doc and Other Small Fixes by @jsong468 in https://github.com/Azure/PyRIT/pull/587
• FEAT Passing HTTP client kwargs from OllamaChatTarget by @rlundeen2 in https://github.com/Azure/PyRIT/pull/596
• MAINT: Refactoring Single-Turn by @rlundeen2 in https://github.com/Azure/PyRIT/pull/598
• DOC: Clarifying OpenAI docs by @rlundeen2 in https://github.com/Azure/PyRIT/pull/600
• FEAT - Adding optional kwargs to huggingface chat target by @perezbecker in https://github.com/Azure/PyRIT/pull/602
• FEAT: Ansi Escape Code Converter by @KutalVolkan in https://github.com/Azure/PyRIT/pull/597
• MAINT Update gcg_attack.py by @Tiger-Du in https://github.com/Azure/PyRIT/pull/606
• MAINT empty integration tests pipeline by @romanlutz in https://github.com/Azure/PyRIT/pull/603
• MAINT update integration-tests trigger to work with PRs by @romanlutz in https://github.com/Azure/PyRIT/pull/610
• FEAT: Playwright target by @AlexRRR in https://github.com/Azure/PyRIT/pull/583
• MAINT Add support for Local Multimodal Input Prompts When Using AzureSQLMemory by @rdheekonda in https://github.com/Azure/PyRIT/pull/613
• MAINT: Add Integration Test Directory + Refusal Scorer Eval Integration Test by @jsong468 in https://github.com/Azure/PyRIT/pull/605
• FEAT: Add Prepending Conversation Support to RedTeamingOrchestrator and CrescendoOrchestrator by @nina-msft in https://github.com/Azure/PyRIT/pull/578
• FIX: Adding SHA256 hashes to responses by @rlundeen2 in https://github.com/Azure/PyRIT/pull/615
• FEAT: binary converter by @AlexRRR in https://github.com/Azure/PyRIT/pull/611
• FIX: Update pyproject.toml for new versions for httpx, respx and openai by @jsong468 in https://github.com/Azure/PyRIT/pull/623
• FEAT Adding labels for individual prompts by @jbolor21 in https://github.com/Azure/PyRIT/pull/624
• FEAT Add Scores to Data Export with PromptRequestPiece data by @eugeniavkim in https://github.com/Azure/PyRIT/pull/617
• FEAT: Prompt Memory Consolidation and Filters by @rlundeen2 in https://github.com/Azure/PyRIT/pull/625
• FEAT: PDF Converter Updates by @KutalVolkan in https://github.com/Azure/PyRIT/pull/622
• FIX: small edits to populatepromptpiece_scores by @jsong468 in https://github.com/Azure/PyRIT/pull/626
• DOC: Updating contributor docs by @rlundeen2 in https://github.com/Azure/PyRIT/pull/627
• FEAT Consolidate Export Conversations into one method by @eugeniavkim in https://github.com/Azure/PyRIT/pull/628
• FEAT: Adding tasks to scorers by @rlundeen2 in https://github.com/Azure/PyRIT/pull/629
• FIX: sortrequestpieces bug by @rlundeen2 in https://github.com/Azure/PyRIT/pull/631
• FEAT: Allowing header SeedPrompt configuration by @rlundeen2 in https://github.com/Azure/PyRIT/pull/630
• FEAT: Add Support for Multimodal Seed Prompts and Update Data Type Serializer by @rdheekonda in https://github.com/Azure/PyRIT/pull/632
• FEAT: Explicitly Initialize Central Memory + Remove Defaults by @nina-msft in https://github.com/Azure/PyRIT/pull/616
• FIX Refactor to join queries for entries and scores by @eugeniavkim in https://github.com/Azure/PyRIT/pull/635
• MAINT: Cleanup Import Naming for initialize_pyrit func by @nina-msft in https://github.com/Azure/PyRIT/pull/636
• FEAT: Score Responses by Filters in ScoringOrchestrator by @nina-msft in https://github.com/Azure/PyRIT/pull/639
• MAINT infrastructure for integration tests by @romanlutz in https://github.com/Azure/PyRIT/pull/612
• MAINT: Add JSON Mode for Supported Targets and Scorers by @rdheekonda in https://github.com/Azure/PyRIT/pull/640
• DOC: Zero Day Quest blog post by @rlundeen2 in https://github.com/Azure/PyRIT/pull/643
• MAINT: Add Import Sorting (isort) Pre-Commit Hook by @nina-msft in https://github.com/Azure/PyRIT/pull/644
• FIX: Rerun Output for Audio Converter Notebook by @nina-msft in https://github.com/Azure/PyRIT/pull/645
• MAINT: Add Import Sorting for Docs and Jupyter Notebooks (isort/nbqa-isort) by @nina-msft in https://github.com/Azure/PyRIT/pull/646
• TEST: Converter Notebook Integration Tests by @nina-msft in https://github.com/Azure/PyRIT/pull/647
• FEAT: Replacing NormalizerRequestPieces with SeedPrompts by @rlundeen2 in https://github.com/Azure/PyRIT/pull/648
• MAINT: Remove Azure SQL Example from Audio Converters Notebook by @nina-msft in https://github.com/Azure/PyRIT/pull/649
• FIX: adding hashes to retrieved PromptRequestPiece by @rlundeen2 in https://github.com/Azure/PyRIT/pull/652
• DOC: Clarifying PromptTargets from PromptChatTargets by @rlundeen2 in https://github.com/Azure/PyRIT/pull/658
• DOC update pyrit.common API reference by @paulinek13 in https://github.com/Azure/PyRIT/pull/657
• FEAT - Realtime Target by @jbolor21 in https://github.com/Azure/PyRIT/pull/638
• MAINT: Updating getseedpromptgroups to include individual seedprompts by @rlundeen2 in https://github.com/Azure/PyRIT/pull/651
• DOC: Deleting extra doc by @rlundeen2 in https://github.com/Azure/PyRIT/pull/663
• FIX: Fixing circular import by @rlundeen2 in https://github.com/Azure/PyRIT/pull/665
• DOC Cleaning up Datasets and adding documentation for datasets and seed prompts by @eugeniavkim in https://github.com/Azure/PyRIT/pull/660
• DOC Adding NCC HTTPTarget Blog post by @jbolor21 in https://github.com/Azure/PyRIT/pull/664
• TEST Integration Tests for Target Notebooks by @jbolor21 in https://github.com/Azure/PyRIT/pull/667
• FEAT: Enhance PDFConverter to support text injection into existing PDFs by @KutalVolkan in https://github.com/Azure/PyRIT/pull/641
• FIX Target Integration test rename by @jbolor21 in https://github.com/Azure/PyRIT/pull/675
• FEAT: Adding Skip Criteria and Sending Prompts Cookbook by @rlundeen2 in https://github.com/Azure/PyRIT/pull/668
• FIX: http target bug by @ayeganov in https://github.com/Azure/PyRIT/pull/674
• FEAT add value hash columns and calc hash when committing seed prompt to memory by @jorisdg in https://github.com/Azure/PyRIT/pull/659
• TEST: Integration Tests for Python Notebooks (Auxiliary Attacks, Datasets, Memory) by @nina-msft in https://github.com/Azure/PyRIT/pull/670
• FIX: PDF Converter and Cookbook integration test by @rlundeen2 in https://github.com/Azure/PyRIT/pull/680
• FEAT: adding hex code converter (#666) by @millashin in https://github.com/Azure/PyRIT/pull/681
• FIX: Converter PDF Integration Build Pipeline by @rlundeen2 in https://github.com/Azure/PyRIT/pull/683
• TEST Integration Tests for Orchestrators & Scorers by @jbolor21 in https://github.com/Azure/PyRIT/pull/677
• FEAT add Discord link to the website by @paulinek13 in https://github.com/Azure/PyRIT/pull/682
• FIX: Integration Test Pipeline by @jbolor21 in https://github.com/Azure/PyRIT/pull/686
• FEAT: RolePlayingOrchestrator by @rlundeen2 in https://github.com/Azure/PyRIT/pull/679
• FEAT: Add Conversation/Prompt ID information to exceptions (for MultiTurnOrchestrators) by @jsong468 in https://github.com/Azure/PyRIT/pull/637
• FIX: Integration Pipeline by @jbolor21 in https://github.com/Azure/PyRIT/pull/689
• MAINT: Add Application Insights by @jsong468 in https://github.com/Azure/PyRIT/pull/692
• FIX Integration Test Pipeline by @jbolor21 in https://github.com/Azure/PyRIT/pull/693
• FEAT Role Play Trivia Game Template by @jsong468 in https://github.com/Azure/PyRIT/pull/700
• FIX: skip link checking in blog by @jbolor21 in https://github.com/Azure/PyRIT/pull/697
• DOC - Adding blog post for Seed Prompts by @eugeniavkim in https://github.com/Azure/PyRIT/pull/703
• FEAT POC for scanner with prompt sending orchestrator by @romanlutz in https://github.com/Azure/PyRIT/pull/673
• FEAT Add easier querying for supported input/output types by @paulinek13 in https://github.com/Azure/PyRIT/pull/688
• MAINT: address deprecation warnings in unit tests by @paulinek13 in https://github.com/Azure/PyRIT/pull/691
• Fix XPIAOrchestrator Blob Not Found Exception by @rdheekonda in https://github.com/Azure/PyRIT/pull/694
• FEAT: Add GroqChatTarget (#704) by @jsdlm in https://github.com/Azure/PyRIT/pull/705

New Contributors

• @perezbecker made their first contribution in https://github.com/Azure/PyRIT/pull/602
• @paulinek13 made their first contribution in https://github.com/Azure/PyRIT/pull/657
• @ayeganov made their first contribution in https://github.com/Azure/PyRIT/pull/674
• @millashin made their first contribution in https://github.com/Azure/PyRIT/pull/681
• @jsdlm made their first contribution in https://github.com/Azure/PyRIT/pull/705

Full Changelog: https://github.com/Azure/PyRIT/compare/v0.5.2...v0.6.0

- Python
Published by nina-msft about 1 year ago

pyrit - v0.5.2

What's Changed

• Pinned the httpx version to 0.27.2 and refactored the codebase to ensure compatibility.
• Fixed AzureSQLMemory authentication issues by adding token refresh, pool recycling, and pre-ping mechanisms.
• Redesigned PAIR attack technique to function as a specialized instance of TAP orchestrator, streamlining architecture.
• Added support for local Hugging Face model checkpoints.

Full list of changes

• [DOC] Updating README by @rlundeen2 in https://github.com/Azure/PyRIT/pull/579
• Fix Azure SQL Authentication Errors: Add Token Refresh, Pool Recycling, and Pre-Ping by @rdheekonda in https://github.com/Azure/PyRIT/pull/576
• FEAT: add support for local model checkpoints and trustremotecode in HuggingFaceChatTarget by @KutalVolkan in https://github.com/Azure/PyRIT/pull/574
• FEAT: Refactor PAIR to be a special instance of TAP by @rlundeen2 in https://github.com/Azure/PyRIT/pull/580
• FIX: httpx proxy arg fix, pinned httpx version by @jsong468 in https://github.com/Azure/PyRIT/pull/589
• FIX: Not raising exceptions on None responses by @rlundeen2 in https://github.com/Azure/PyRIT/pull/590
• Fix Test Prompt Response Error Values by @rdheekonda in https://github.com/Azure/PyRIT/pull/591

Full Changelog: https://github.com/Azure/PyRIT/compare/v0.5.0...v0.5.2

- Python
Published by rdheekonda about 1 year ago

pyrit - v0.5.0

What's Changed

• PyRIT now has a website

• We've been working on standardizing orchestrators in terms of naming and functionality:

  • The endpoint (of type PromptTarget) that PyRIT attacks will be referred to as objective_target.
  • The endpoint (of type PromptChatTarget) that helps us craft attacks will be referred to as adversarial_chat.
  • Beyond that, we've settled on a common interface for multi-turn orchestrators with a shared result object.
  • Instead of an attack_strategy arg we require a file path called adversarial_chat_system_prompt_path to make the connection to the adversarial_chat target clearer. Some orchestrators have a default for this, of course.
  • The initial prompt to the adversarial_chat is now called adversarial_chat_seed_prompt to also help with clarity and connection to adversarial_chat
  • Sometimes we use multiple scorers. For that reason, objective_scorer will be the scorer that decides if the objective has been achieved. Other scorers have similarly specific names, e.g., on_topic_scorer in the CrescendoOrchestrator
  • The new standard name for all orchestrators to execute an attack is run_attack_async.

The standardization is not fully completed yet but will continue in future releases. So far, CrescendoOrchestrator, TreeOfAttacksWithPruningOrchestrator, and RedTeamingOrchestrator have been adjusted. - Support for a centralized database using Azure SQL as an optional alternative to a local DuckDB database. - Introduced (multi-modal) SeedPrompts and SeedPromptDatasets as a starting point for red teaming ops with integration to our databases. - New orchestrators and auxiliary attacks: - FuzzerOrchestrator with 5 template converters - GCG support via Azure ML pipelines to optimize adversarial suffixes - FlipAttackOrchestrator - New targets: - HuggingFaceChatTarget - HTTPTarget - Open AI and Azure Open AI targets were refactored to simplify the logic. They now share a common interface OpenAITarget and you can decide between Azure vs. Open AI using is_azure_target=True or False. - New datasets: - HarmBench - PKU-SafeRLHF - wmdp-bio, wmdp-chem, and wmdp-cyber (now fetchable from the original data source) - AdvBench - Decoding Trust Stereotypes - LLM-LAT/harmful-dataset - tdc23 red teaming dataset - TrustAIRLab/forbiddenquestionset - LibrAI 'Do Not Answer' Dataset - New converters: - QRCodeConverter - AzureSpeechAudioToTextConverter - URLConverter - HumanInTheLoopConverter - ColloquialWordswapConverter - UnicodeConfusableConverter (updated with new functionality) - CharSwapGenerator - MaliciousQuestionGeneratorConverter - AsciiSmugglerConverter - MathPromptConverter - AudioFrequencyConverter - ZeroWidthConverter - DiacriticConverter - New scorers: - SelfAskRefusalScorer - HumanInTheLoopScorer - InsecureCodeScorer - We generally use a .env file to configure details of endpoints that PyRIT needs to execute. A new .env.local override file allow for further customization. - Finally, PyRIT now comes with several extras that you can install using pip install pyrit[<extra>] - dev includes developer dependencies that you shouldn't need unless you plan on contributing to the project. - torch includes just pytorch which is needed for some targets (e.g. Hugging Face) or auxiliary attacks (e.g., GCG) but not core functionality. This allows you to choose whether you want to install it. - gcg includes extra dependencies that are only needed for running GCG. Since this requires dedicated compute (ideally with GPU) you can choose whether it is required for you. - all includes all of the above.

Full list of changes

• MAINT Update release version to 0.4.1.dev0 by @rdheekonda in https://github.com/Azure/PyRIT/pull/342
• [FEAT] QRCodeConverter by @jsong468 in https://github.com/Azure/PyRIT/pull/339
• [MAINT] Delete output_filename arg in image/text and text/image converters by @jsong468 in https://github.com/Azure/PyRIT/pull/344
• MAINT Update Release Instructions by @rdheekonda in https://github.com/Azure/PyRIT/pull/345
• FEAT: Add Likert scoring definition and prompt templates for persuasion and deception by @saphirqi7 in https://github.com/Azure/PyRIT/pull/307
• [FEAT] Add "task" to the scoring memory entry by @jsong468 in https://github.com/Azure/PyRIT/pull/349
• FEAT: Add fetch function for datasets from HarmBench #270 by @KutalVolkan in https://github.com/Azure/PyRIT/pull/341
• FEAT Add SQL Entra Auth for Azure SQL Server by @elgertam in https://github.com/Azure/PyRIT/pull/330
• [MAINT] Fix typos in OllamaChatTarget by @riedgar-ms in https://github.com/Azure/PyRIT/pull/357
• [FEAT] Azure Speech Audio to Text Converter by @jsong468 in https://github.com/Azure/PyRIT/pull/352
• FEAT: Add Rate Limit (RPM) Threshold Parameter to Prompt Targets by @nina-msft in https://github.com/Azure/PyRIT/pull/331
• FIX: correct type of the top_p argument in various PromptTarget classes by @s-zanella in https://github.com/Azure/PyRIT/pull/366
• FEAT Add ability to fetch PKU-SafeRLHF Data by @enrajka in https://github.com/Azure/PyRIT/pull/374
• FEAT: Refusal Scorer by @rlundeen2 in https://github.com/Azure/PyRIT/pull/371
• FEAT Add ability to fetch wmdp-bio, wmdp-chem, and wmdp-cyber datasets by @mshirsekar1 in https://github.com/Azure/PyRIT/pull/380
• TEST skip failing auth test after the new azure.identity version was released by @romanlutz in https://github.com/Azure/PyRIT/pull/387
• FEAT Added AdvBench dataset by @enrajka in https://github.com/Azure/PyRIT/pull/383
• FEAT: Fuzzer orchestrator by @gseetha04 in https://github.com/Azure/PyRIT/pull/360
• FIX Crescendo Bug and Improve Scorer Metaprompt Handling by @rdheekonda in https://github.com/Azure/PyRIT/pull/389
• FEAT: Add Centralized DB Support Using Azure by @rdheekonda in https://github.com/Azure/PyRIT/pull/379
• FIX: Updating memory and fixing bugs by @rlundeen2 in https://github.com/Azure/PyRIT/pull/394
• FEAT: Handling duplicate memory for PromptRequestPiece/Score entries by @jsong468 in https://github.com/Azure/PyRIT/pull/369
• [FEAT] Decoding Trust Stereotypes Dataset by @jsong468 in https://github.com/Azure/PyRIT/pull/385
• FEAT Centralized DB Support for Azure Speech Converters by @rdheekonda in https://github.com/Azure/PyRIT/pull/402
• FEAT add additional template converters for fuzzer orchestrator (crossover, similar, rephrase) by @roeybc in https://github.com/Azure/PyRIT/pull/378
• DOC: Update Custom Targets Demo Docs by @nina-msft in https://github.com/Azure/PyRIT/pull/404
• FEAT New URL Converter by @jbolor21 in https://github.com/Azure/PyRIT/pull/399
• [FEAT] HumanInTheLoop Converter by @jsong468 in https://github.com/Azure/PyRIT/pull/401
• DOC: Updating RTO example to use gpt4o for scoring by @rlundeen2 in https://github.com/Azure/PyRIT/pull/408
• MAINT: Crescendo and Score Refactor by @rlundeen2 in https://github.com/Azure/PyRIT/pull/405
• FEAT: Colloquial Wordswap Attack by @eugeniavkim in https://github.com/Azure/PyRIT/pull/406
• FEAT emoji jailbreak by @romanlutz in https://github.com/Azure/PyRIT/pull/314
• MAINT: Add Refusal docs and Filter logic by @rlundeen2 in https://github.com/Azure/PyRIT/pull/431
• DOC: Moving rate limiting to target by @rlundeen2 in https://github.com/Azure/PyRIT/pull/433
• FEAT: optimized huggingface model support by @KutalVolkan in https://github.com/Azure/PyRIT/pull/354
• DOC Enhance Azure SQL Database Setup and Permissions Documentation by @rdheekonda in https://github.com/Azure/PyRIT/pull/434
• FIX Azure SQL DB Permissions by @rdheekonda in https://github.com/Azure/PyRIT/pull/440
• FIX: Handle JSON markdown format exceptions by @meisman-ms in https://github.com/Azure/PyRIT/pull/435
• FEAT: Add ability to send prepend to the conversation in PromptSendingOrchestrator by @rlundeen2 in https://github.com/Azure/PyRIT/pull/441
• FEAT: Homoglyph Attack by @KutalVolkan in https://github.com/Azure/PyRIT/pull/407
• FEAT: Charswap Attack by @KutalVolkan in https://github.com/Azure/PyRIT/pull/403
• Add Python option for generate docs scripts by @sf-msft in https://github.com/Azure/PyRIT/pull/375
• FEAT: Violent Durian Attack Strategy by @KutalVolkan in https://github.com/Azure/PyRIT/pull/398
• FEAT GCG algorithm and AML pipeline by @blakebullwinkel in https://github.com/Azure/PyRIT/pull/381
• MAINT: Adding original values as score metadata for Azure Safety and Likert Scorers by @rlundeen2 in https://github.com/Azure/PyRIT/pull/445
• [DOC] Note on notebooks by @riedgar-ms in https://github.com/Azure/PyRIT/pull/460
• FIX: Fixing pre-commit check_links by @rlundeen2 in https://github.com/Azure/PyRIT/pull/462
• FEAT: Adding Flip Attack by @rlundeen2 in https://github.com/Azure/PyRIT/pull/456
• [FIX] Allow AAD Auth for AzureContentFilterScorer by @riedgar-ms in https://github.com/Azure/PyRIT/pull/455
• FEAT: Adding New Generic HTTP Target by @jbolor21 in https://github.com/Azure/PyRIT/pull/446
• MAINT: Rounds in CrescendoOrchestrator are now "Turns" by @jsong468 in https://github.com/Azure/PyRIT/pull/470
• DOC Add doc changes for database setup by @eugeniavkim in https://github.com/Azure/PyRIT/pull/476
• FEAT: OpenAI Target Refactor by @rlundeen2 in https://github.com/Azure/PyRIT/pull/466
• DOC: Edit Image Text Converter Docs by @jbolor21 in https://github.com/Azure/PyRIT/pull/477
• FEAT: Malicious Question Generator by @KutalVolkan in https://github.com/Azure/PyRIT/pull/397
• FIX: Changed AzureSpeechTextToAudioConverter inputtype to text and added converter inputsupported tests by @jsong468 in https://github.com/Azure/PyRIT/pull/472
• FEAT added ascii smuggler converter by @gio-msft in https://github.com/Azure/PyRIT/pull/479
• DOC Fix Invalid MD File Referenced in Deploy HF Model to Azure ML Module by @rdheekonda in https://github.com/Azure/PyRIT/pull/485
• FIX: Re-Ran Jupytext on Crescendo Notebook by @jsong468 in https://github.com/Azure/PyRIT/pull/484
• FIX Warnings in pipelines (Issue #442) by @Tiger-Du in https://github.com/Azure/PyRIT/pull/481
• FEAT Add LLM-LAT/harmful-dataset #420 by @SnehaDharne in https://github.com/Azure/PyRIT/pull/437
• FIX: Small Notebook Fixes and env_example updates by @jsong468 in https://github.com/Azure/PyRIT/pull/487
• FEAT add tdc23 red teaming dataset by @Lakshmiaddepalli in https://github.com/Azure/PyRIT/pull/438
• MAINT Adding TrueFalseQuestion to initialize scorer more easily by @rlundeen2 in https://github.com/Azure/PyRIT/pull/488
• MAINT: Stripping json in llm scorers by @rlundeen2 in https://github.com/Azure/PyRIT/pull/489
• DOC: Adds citation section to README.md by @dlmgary in https://github.com/Azure/PyRIT/pull/491
• FIX Updating env variable for DALL E by @eugeniavkim in https://github.com/Azure/PyRIT/pull/492
• FIX: Remove Duplicate Import Statement in Documentation Examples by @douyipu in https://github.com/Azure/PyRIT/pull/495
• FIX changed OpenAIChatTarget default values by @blakebullwinkel in https://github.com/Azure/PyRIT/pull/496
• [DRAFT] FEAT: MathPromptConverter to Transform Prompts into Mathematical Problems by @KutalVolkan in https://github.com/Azure/PyRIT/pull/490
• FIX Set Unique Conversation IDs (RedTeamingOrchestrator) by @nina-msft in https://github.com/Azure/PyRIT/pull/468
• MAINT: Consolidate UnicodeConfusableConverter and HomoglyphGeneratorConverter by @jsong468 in https://github.com/Azure/PyRIT/pull/497
• Fix PromptMemoryEntry columns data types to support non-English values by @rdheekonda in https://github.com/Azure/PyRIT/pull/499
• FIX Added "Invalid prompt" OAI error to bad request exception handler by @blakebullwinkel in https://github.com/Azure/PyRIT/pull/500
• MAINT: Consistency Improvements by @rlundeen2 in https://github.com/Azure/PyRIT/pull/498
• [DRAFT] DOC: Add Skeleton Key Attack Demo by @KutalVolkan in https://github.com/Azure/PyRIT/pull/502
• FIX Include maxcompletiontokens argument for OpenAIChatTarget by @nina-msft in https://github.com/Azure/PyRIT/pull/501
• FEAT: Add audio frequency converter by @michellemorales in https://github.com/Azure/PyRIT/pull/478
• FIX: Separating OpenAIChatTarget Arguments by @rlundeen2 in https://github.com/Azure/PyRIT/pull/505
• MAINT: Refactor azure ml target by @jsong468 in https://github.com/Azure/PyRIT/pull/463
• MAINT: Adding MultiTurn Abstract Orchestrator Interface by @rlundeen2 in https://github.com/Azure/PyRIT/pull/504
• FEAT Add TrustAIRLab/forbiddenquestionset Dataset #453 by @ritikakumar0204 in https://github.com/Azure/PyRIT/pull/503
• FEAT: database connector to store and retrieve prompts, prompt templates, and prompt groups by @romanlutz in https://github.com/Azure/PyRIT/pull/396
• FIX fix references to renamed powershell files by @mhaoda in https://github.com/Azure/PyRIT/pull/510
• FEAT Add export for conversations and scores by @eugeniavkim in https://github.com/Azure/PyRIT/pull/517
• FIX: Removed unnecessary addresponseentriestomemory mocking and changed normalized target 'endpoint' param by @jsong468 in https://github.com/Azure/PyRIT/pull/521
• MAINT: Removing SeedPromptTemplate by @rlundeen2 in https://github.com/Azure/PyRIT/pull/520
• MAINT: Remove many shot Template by @rlundeen2 in https://github.com/Azure/PyRIT/pull/522
• FEAT: Add Zero-Width-Converter by @KutalVolkan in https://github.com/Azure/PyRIT/pull/519
• FEAT: Add Diacritics Converter by @KutalVolkan in https://github.com/Azure/PyRIT/pull/518
• MAINT: Standardizing Multi-Turn Orchestrators by @rlundeen2 in https://github.com/Azure/PyRIT/pull/509
• MAINT: Removing attack strategy by @rlundeen2 in https://github.com/Azure/PyRIT/pull/525
• FEAT add seed prompt dataset loading function for legacy datasets by @romanlutz in https://github.com/Azure/PyRIT/pull/524
• DOC Add jupyterbook project site page by @sf-msft in https://github.com/Azure/PyRIT/pull/430
• FIX outdated link by @romanlutz in https://github.com/Azure/PyRIT/pull/533
• FEAT: Functionality to update PromptMemoryEntries by @jsong468 in https://github.com/Azure/PyRIT/pull/531
• FEAT HITL Scorers by @jbolor21 in https://github.com/Azure/PyRIT/pull/493
• MAINT: Add Centralized Memory Management by @rdheekonda in https://github.com/Azure/PyRIT/pull/527
• MAINT Update DuckDB Memory Demo Notebook Documentation by @rdheekonda in https://github.com/Azure/PyRIT/pull/536
• FIX use cluster for compute by @romanlutz in https://github.com/Azure/PyRIT/pull/538
• FIX Remove aria2c dependency from HuggingFace Target by @nina-msft in https://github.com/Azure/PyRIT/pull/530
• [FIX] Fix broken azure_auth test by @jsong468 in https://github.com/Azure/PyRIT/pull/544
• FIX import tkinter only when using it to avoid import errors on ubuntu/macos by @romanlutz in https://github.com/Azure/PyRIT/pull/542
• DOC publish to GH pages when pushing changes to main by @romanlutz in https://github.com/Azure/PyRIT/pull/545
• FIX Fuzzer Converter Templates by @rdheekonda in https://github.com/Azure/PyRIT/pull/546
• FEAT: Add Insecure Code Scorer by @KutalVolkan in https://github.com/Azure/PyRIT/pull/523
• MAINT: Updating refusal scorer to work without tasks by @rlundeen2 in https://github.com/Azure/PyRIT/pull/547
• DOC bring back numbering for user guide, raise build issues as errors, and fix warnings by @romanlutz in https://github.com/Azure/PyRIT/pull/549
• FIX remove unnecessary threshold arg by @romanlutz in https://github.com/Azure/PyRIT/pull/550
• MAINT: Allowing prepending conversations in PSO from memory by @rlundeen2 in https://github.com/Azure/PyRIT/pull/555
• FEAT Enhance .env loading with optional .env.local overrides by @rdheekonda in https://github.com/Azure/PyRIT/pull/559
• MAINT update dependencies to separate torch into an extra, prune unnecessary ones, and related small fixes by @romanlutz in https://github.com/Azure/PyRIT/pull/556
• FIX remove timezone info, pass timestamp around when retrieving data from DB by @romanlutz in https://github.com/Azure/PyRIT/pull/560
• Fix TAP Orchestrator Invalid Argument by @rdheekonda in https://github.com/Azure/PyRIT/pull/561
• DOC: Relocate usehuggingfacechat_target notebook and script to targets directory by @KutalVolkan in https://github.com/Azure/PyRIT/pull/558
• FIX: Fixing bug in doc and adding repr to models by @rlundeen2 in https://github.com/Azure/PyRIT/pull/564
• MAINT: TAP Multi-turn refactor by @rlundeen2 in https://github.com/Azure/PyRIT/pull/562
• FEAT: Add LibrAI 'Do Not Answer' Dataset by @KutalVolkan in https://github.com/Azure/PyRIT/pull/565
• DOC: Add batch scoring example for SelfAskTrueFalseScorer by @KutalVolkan in https://github.com/Azure/PyRIT/pull/563
• FIX: Fixing and improving crescendo adversarial_chat prompt by @rlundeen2 in https://github.com/Azure/PyRIT/pull/570
• FIX repair component governance by @romanlutz in https://github.com/Azure/PyRIT/pull/557
• FEAT: Pass arguments to http client by @AlexRRR in https://github.com/Azure/PyRIT/pull/554
• [FEAT] Global Memory Labels by @jsong468 in https://github.com/Azure/PyRIT/pull/571
• FIX release related fixes by @romanlutz in https://github.com/Azure/PyRIT/pull/575

New Contributors

• @saphirqi7 made their first contribution in https://github.com/Azure/PyRIT/pull/307
• @riedgar-ms made their first contribution in https://github.com/Azure/PyRIT/pull/357
• @s-zanella made their first contribution in https://github.com/Azure/PyRIT/pull/366
• @enrajka made their first contribution in https://github.com/Azure/PyRIT/pull/374
• @mshirsekar1 made their first contribution in https://github.com/Azure/PyRIT/pull/380
• @gseetha04 made their first contribution in https://github.com/Azure/PyRIT/pull/360
• @roeybc made their first contribution in https://github.com/Azure/PyRIT/pull/378
• @eugeniavkim made their first contribution in https://github.com/Azure/PyRIT/pull/406
• @meisman-ms made their first contribution in https://github.com/Azure/PyRIT/pull/435
• @sf-msft made their first contribution in https://github.com/Azure/PyRIT/pull/375
• @gio-msft made their first contribution in https://github.com/Azure/PyRIT/pull/479
• @Tiger-Du made their first contribution in https://github.com/Azure/PyRIT/pull/481
• @SnehaDharne made their first contribution in https://github.com/Azure/PyRIT/pull/437
• @Lakshmiaddepalli made their first contribution in https://github.com/Azure/PyRIT/pull/438
• @douyipu made their first contribution in https://github.com/Azure/PyRIT/pull/495
• @michellemorales made their first contribution in https://github.com/Azure/PyRIT/pull/478
• @ritikakumar0204 made their first contribution in https://github.com/Azure/PyRIT/pull/503
• @mhaoda made their first contribution in https://github.com/Azure/PyRIT/pull/510
• @AlexRRR made their first contribution in https://github.com/Azure/PyRIT/pull/554

Full Changelog: https://github.com/Azure/PyRIT/compare/v0.4.0...v0.5.0

- Python
Published by romanlutz about 1 year ago

pyrit - v0.4.0

What's Changed

1. New Advanced Attack Techniques: Expanded orchestrators with advanced attack techniques, including PAIR, tree of attacks, and crescendo strategies.
2. New Targets: Crucible target, Prompt Shield Target, Azure OpenAI GPT-4o target
3. New Converters: Added Tense, Emoji, image to text, and Character Space converters.
4. New Scorers: Scale Scorer, Prompt Shield, and True/False Inverter Scorer
5. Automatic Scoring & Memory Labels: Introduced automatic scoring in the PromptSendingOrchestrator. Added support for scoring with user-provided memory labels.
6. Delegation SAS Authentication: Supported delegation SAS authentication for secure interactions with Azure Blob Storage targets.
7. Improved Resiliency: Enhanced the resiliency of targets, converters, and orchestrators with robust error handling mechanisms.
8. Bug Fixes & Performance: Various bug fixes, added support for Python 3.12, speedup unit tests
9. Fetch functionality: Introduced functionality to fetch adversarial datasets, such as SecLists, XStest etc.,
10. Updated Demo Codes: Replaced demo code examples with the GPT-4o target.

Full List of Changes

• FIX: Fixing policheck bug by @rlundeen2 in https://github.com/Azure/PyRIT/pull/261
• release v0.3.0 by @jbolor21 in https://github.com/Azure/PyRIT/pull/265
• DOC: Adding Guidance on Incorporating Research by @rlundeen2 in https://github.com/Azure/PyRIT/pull/268
• FEAT: Adding Tense Converter by @rlundeen2 in https://github.com/Azure/PyRIT/pull/273
• [FEAT] Add Scoring to PromptSendingOrchestrator by @nina-msft in https://github.com/Azure/PyRIT/pull/262
• FIX Fixed mypy Type Failures by @elgertam in https://github.com/Azure/PyRIT/pull/269
• FEAT: Adding Crucible Target by @rlundeen2 in https://github.com/Azure/PyRIT/pull/277
• FIX ValueError with Azure TTS Target in Single Turn Conversation Using PromptSendingOrchestrator by @nina-msft in https://github.com/Azure/PyRIT/pull/278
• FEAT: Converter Tokens by @rlundeen2 in https://github.com/Azure/PyRIT/pull/279
• [FIX] Add flake8-copyright check to pre-commit hooks by @nina-msft in https://github.com/Azure/PyRIT/pull/281
• FIX Exclude Morse Converter from Flake8 Precommit by @nina-msft in https://github.com/Azure/PyRIT/pull/284
• [DRAFT] [FIX] Replace Orchestrator ID with UUID by @nina-msft in https://github.com/Azure/PyRIT/pull/285
• DOC update citation for past tense paper by @romanlutz in https://github.com/Azure/PyRIT/pull/288
• FEAT Add scale scorer by @romanlutz in https://github.com/Azure/PyRIT/pull/274
• FEAT Add Delegation SAS-Based Auth, Update Storage Plugins, and Async Blob Download by @rdheekonda in https://github.com/Azure/PyRIT/pull/286
• FEAT add (back) Gandalf scorer by @romanlutz in https://github.com/Azure/PyRIT/pull/287
• MAINT clean up copyright by @romanlutz in https://github.com/Azure/PyRIT/pull/297
• FEAT: Add Error Handling to AML Chat Target by @nina-msft in https://github.com/Azure/PyRIT/pull/299
• FIX: bug with multi-modal image responses by @rlundeen2 in https://github.com/Azure/PyRIT/pull/301
• MAINT: Improving some LLM Converters by @rlundeen2 in https://github.com/Azure/PyRIT/pull/300
• [FIX][Issue #302] update language version enforcement to fix black-pre-commit installation incompatibility by @shivenchawla in https://github.com/Azure/PyRIT/pull/303
• FEAT return ID in conversation duplication code by @romanlutz in https://github.com/Azure/PyRIT/pull/296
• [FEAT] Implement PAIR by @dlmgary in https://github.com/Azure/PyRIT/pull/255
• FEAT add float scale threshold scorer by @romanlutz in https://github.com/Azure/PyRIT/pull/294
• FEAT: Add GPT4-o chat target by @shivenchawla in https://github.com/Azure/PyRIT/pull/293
• FEAT: Adding Emoji Converter by @rlundeen2 in https://github.com/Azure/PyRIT/pull/306
• DOC: Doc Reorg by @rlundeen2 in https://github.com/Azure/PyRIT/pull/304
• MAINT: Removing asyncio sleep by @rlundeen2 in https://github.com/Azure/PyRIT/pull/309
• MAINT add support for Python 3.12 and fix tests that started breaking by @romanlutz in https://github.com/Azure/PyRIT/pull/305
• FEAT Add print_conversation method to prompt sending orchestrator by @romanlutz in https://github.com/Azure/PyRIT/pull/312
• FEAT Add many-shot jailbreaking feature implementation by @KutalVolkan in https://github.com/Azure/PyRIT/pull/254
• FEAT: Add tree of attacks with pruning by @salmazainana in https://github.com/Azure/PyRIT/pull/210
• FEAT Add Space Converter by @rdheekonda in https://github.com/Azure/PyRIT/pull/316
• FEAT Add Flexible Memory Labels and Scoring to Orchestrators by @rdheekonda in https://github.com/Azure/PyRIT/pull/315
• FEAT: Crescendo Orchestrator by @SafwanA02 in https://github.com/Azure/PyRIT/pull/275
• Feat: Adding multi-turn promptSendingOrchestrator by @rlundeen2 in https://github.com/Azure/PyRIT/pull/317
• DOC Fix README.md link by @romanlutz in https://github.com/Azure/PyRIT/pull/319
• MAINT: Fixing data serializer ability to properly raise errors by @rlundeen2 in https://github.com/Azure/PyRIT/pull/318
• FEAT: Add fetch function for SecLists AI LLM Bias Testing datasets (#267) by @KutalVolkan in https://github.com/Azure/PyRIT/pull/280
• FEAT: Adding true_false inverter scorer by @rlundeen2 in https://github.com/Azure/PyRIT/pull/321
• FIX: fixing check links by @rlundeen2 in https://github.com/Azure/PyRIT/pull/323
• FEAT: Add Exception Handling to Azure TTS Target by @nina-msft in https://github.com/Azure/PyRIT/pull/322
• DOC - replacing gpt4 with gpt4o in example notebooks by @jsong468 in https://github.com/Azure/PyRIT/pull/313
• [MAINT] Changing Examples from stop signs by @jbolor21 in https://github.com/Azure/PyRIT/pull/325
• FEAT Prompt Shield by @ValbuenaVC in https://github.com/Azure/PyRIT/pull/271
• FEAT: add xstest dataset by @KutalVolkan in https://github.com/Azure/PyRIT/pull/320
• [FEAT] Created addimagetext_converter and unit tests by @jsong468 in https://github.com/Azure/PyRIT/pull/328
• DOC: Adding Notebook to document re-sending previous prompts by @rlundeen2 in https://github.com/Azure/PyRIT/pull/332
• MAINT: speeding up crescendo tests by @rlundeen2 in https://github.com/Azure/PyRIT/pull/333
• FIX Move pillow from dev to core dependency by @rdheekonda in https://github.com/Azure/PyRIT/pull/334
• FIX add sample image classifier file by @jbolor21 in https://github.com/Azure/PyRIT/pull/336
• FEAT: Add deterministic flag and custom substitutions to LeetspeakConverter by @KutalVolkan in https://github.com/Azure/PyRIT/pull/329
• MAINT Remove Duplicate Module by @rdheekonda in https://github.com/Azure/PyRIT/pull/337
• MAINT Restructure pyrit.models module and prune by @romanlutz in https://github.com/Azure/PyRIT/pull/338
• [MAINT] Speeding up unit tests by @jbolor21 in https://github.com/Azure/PyRIT/pull/335
• FIX Crescendo backtrack with same orchestrator ID and handling responses with markdown syntax by @romanlutz in https://github.com/Azure/PyRIT/pull/340

New Contributors

• @shivenchawla made their first contribution in https://github.com/Azure/PyRIT/pull/303
• @KutalVolkan made their first contribution in https://github.com/Azure/PyRIT/pull/254
• @salmazainana made their first contribution in https://github.com/Azure/PyRIT/pull/210
• @jsong468 made their first contribution in https://github.com/Azure/PyRIT/pull/313
• @ValbuenaVC made their first contribution in https://github.com/Azure/PyRIT/pull/271

Full Changelog: https://github.com/Azure/PyRIT/compare/v0.3.0...v0.4.0

- Python
Published by rdheekonda over 1 year ago

pyrit - v0.3.0

What's Changed

• New and improved scorers! Many new scorers have been added, and scorers can now be swapped out and made generic.
• Many new attack techniques and variations have been introduced. These include skeleton key, most of GPTFuzz, adding text to images, repeated token attack, cipherchat, shorten/expand, tone, CodeChameleon, and more. A total of 13 new converters have been added!
• Framework improvements:
  • Ability to duplicate conversations for reuse (this makes implementation easier for attacks like PAIR/TAP/crescendo).
  • Converters can be added to LLM responses.
  • All framework calls are now async and parallelizable.
  • Error handling and intelligent automatic retries in targets (e.g., for network errors) and converters/scorers (e.g., for JSON deserialization).

Full list of Changes

• FEAT: Refactoring and Standardizing Scores and Scorers by @rlundeen2 in https://github.com/Azure/PyRIT/pull/190
• FIX: Making RESULTS_PATH be simple in pip packages by @rlundeen2 in https://github.com/Azure/PyRIT/pull/191
• FIX: Minor Self-Ask Scorer Improvements by @rlundeen2 in https://github.com/Azure/PyRIT/pull/194
• FEAT: Adding Scores to the Database by @rlundeen2 in https://github.com/Azure/PyRIT/pull/195
• MAINT use context manager in XPIA notebook by @romanlutz in https://github.com/Azure/PyRIT/pull/198
• FEAT: Update score_async to add score to database by @rlundeen2 in https://github.com/Azure/PyRIT/pull/200
• FEAT support duplicating memory when cloning orchestrators by @romanlutz in https://github.com/Azure/PyRIT/pull/177
• MAINT: Likert Scoring Tweaks to Reduce False Positives by @rlundeen2 in https://github.com/Azure/PyRIT/pull/201
• FEAT add CSV support by @romanlutz in https://github.com/Azure/PyRIT/pull/197
• FEAT: Adding Human in the Loop Scorer by @rlundeen2 in https://github.com/Azure/PyRIT/pull/202
• FEAT: Azure content filter scorer by @cseifert1 in https://github.com/Azure/PyRIT/pull/206
• FEAT Adding Image Converter: add text on image by @jbolor21 in https://github.com/Azure/PyRIT/pull/205
• FEAT: Score Prompts Orchestrator by @rlundeen2 in https://github.com/Azure/PyRIT/pull/208
• MAINT: Deprecated send_prompt methods by @mart123p in https://github.com/Azure/PyRIT/pull/204
• FEAT Add image generation example with red teaming orchestrator and unify existing orchestrator definitions by @romanlutz in https://github.com/Azure/PyRIT/pull/189
• FEAT: self ask conversation objective and verifier scorer for crescendo by @cseifert1 in https://github.com/Azure/PyRIT/pull/209
• FEAT: Centralize Exception Handling and Implement in GPTv Target by @rdheekonda in https://github.com/Azure/PyRIT/pull/207
• MAINT Making Prompt Converters Async by @jbolor21 in https://github.com/Azure/PyRIT/pull/211
• Update .env_example Typo "Azure Open AI"→"Azure OpenAI" by @hyoshioka0128 in https://github.com/Azure/PyRIT/pull/214
• MAINT: Small scoring updates by @rlundeen2 in https://github.com/Azure/PyRIT/pull/215
• MAINT: Adding pretty print functionality and small RTO updates by @rlundeen2 in https://github.com/Azure/PyRIT/pull/217
• DOC: Re-organizing documentation by @rlundeen2 in https://github.com/Azure/PyRIT/pull/219
• FEAT: Add Suffix Converter by @NaijingGuo in https://github.com/Azure/PyRIT/pull/212
• MAINT: Updating GPT-V to use new exception guide by @rlundeen2 in https://github.com/Azure/PyRIT/pull/220
• FEAT: Add nesting and prepend/append jailbreaks from papers by @jl8771 in https://github.com/Azure/PyRIT/pull/216
• MAINT Adding Error Handling to OpenAIChatInterface by @jbolor21 in https://github.com/Azure/PyRIT/pull/218
• MAINT Add Exception Handling to DALLE Target by @rdheekonda in https://github.com/Azure/PyRIT/pull/221
• FEAT: Add repeated token attack converter by @jl8771 in https://github.com/Azure/PyRIT/pull/224
• MAINT Resolve Install Issues and Add Multiline Text Wrapping in AddTextImageConverter by @rdheekonda in https://github.com/Azure/PyRIT/pull/230
• MAINT: PromptRequestPiece SHA setting update by @rlundeen2 in https://github.com/Azure/PyRIT/pull/231
• FEAT: Implements Crescendo-style attack based on system prompt. by @dlmgary in https://github.com/Azure/PyRIT/pull/237
• MAINT add notebook version disclaimer by @romanlutz in https://github.com/Azure/PyRIT/pull/234
• FEAT: Adding Converters to Output by @rlundeen2 in https://github.com/Azure/PyRIT/pull/236
• DOC: Reorganizing MemoryDocs by @rlundeen2 in https://github.com/Azure/PyRIT/pull/239
• Added complex code jailbreak template by @petebryan in https://github.com/Azure/PyRIT/pull/238
• FEAT: Add prompt converters for atbash, caesar, morse and cipherchat from paper by @jl8771 in https://github.com/Azure/PyRIT/pull/223
• MAINT add test instructions to release guide by @romanlutz in https://github.com/Azure/PyRIT/pull/232
• FIX: Fixing doc links by @rlundeen2 in https://github.com/Azure/PyRIT/pull/245
• FEAT: Adding Master Key Jailbreak by @SafwanA02 in https://github.com/Azure/PyRIT/pull/248
• MAINT Adding Error Handling Code for converters by @jbolor21 in https://github.com/Azure/PyRIT/pull/247
• FIX: Fixing score conversation history by @rlundeen2 in https://github.com/Azure/PyRIT/pull/251
• FEAT: Add shorten/expand converters by @jl8771 in https://github.com/Azure/PyRIT/pull/246
• FEAT: Add CodeChameleon converter by @jl8771 in https://github.com/Azure/PyRIT/pull/240
• FEAT: Adding Noise and Tone Converters by @rlundeen2 in https://github.com/Azure/PyRIT/pull/252
• FEAT: Add persuasion converter with 5 persuasion techniques by @jl8771 in https://github.com/Azure/PyRIT/pull/253
• FEAT Implementation of SQL Server connectivity by @elgertam in https://github.com/Azure/PyRIT/pull/227
• MAINT Error Handling for Scorers by @jbolor21 in https://github.com/Azure/PyRIT/pull/256
• FIX: Skeleton Key Orchestrator by @SafwanA02 in https://github.com/Azure/PyRIT/pull/260
• MAINT upgrading AOAI version by @jbolor21 in https://github.com/Azure/PyRIT/pull/264

New Contributors

• @mart123p made their first contribution in https://github.com/Azure/PyRIT/pull/204
• @hyoshioka0128 made their first contribution in https://github.com/Azure/PyRIT/pull/214
• @jl8771 made their first contribution in https://github.com/Azure/PyRIT/pull/216
• @SafwanA02 made their first contribution in https://github.com/Azure/PyRIT/pull/248
• @elgertam made their first contribution in https://github.com/Azure/PyRIT/pull/227

Full Changelog: https://github.com/Azure/PyRIT/compare/v0.2.1...v0.3.0

- Python
Published by jbolor21 over 1 year ago

pyrit - v0.2.1

What's Changed

• added user authentication support for AOAI Chat Targets
• request validation in targets
• support for exporting conversations from the memory

Full list of changes

• Updating Release to 0.2.1.dev0 by @rlundeen2 in https://github.com/Azure/PyRIT/pull/181
• FEAT Add User AuthN Support to AOAI Chat Targets by @nina-msft in https://github.com/Azure/PyRIT/pull/182
• MAINT Add Request Validation for All Prompt Targets by @rdheekonda in https://github.com/Azure/PyRIT/pull/184
• FEAT Export Conversation by Orchestrator ID by @nina-msft in https://github.com/Azure/PyRIT/pull/183

Full Changelog: https://github.com/Azure/PyRIT/compare/v0.2.0...v0.2.1

- Python
Published by romanlutz almost 2 years ago

pyrit - v0.2.0

What's Changed

• Multi-modal support: You can now input/output various multi-modal targets.
• XPIA support: Enabling easier second order prompt injection attacks.
• A more robust local (duckDB) database: Allowing querying and inserting previous conversations.

Full list of Changes

• Added a ChatMessageNormalizer that formats messages in the template specified by a Hugging Face tokenizer by @blakebullwinkel in https://github.com/Azure/PyRIT/pull/128
• PromptMemoryEntry Table Added for more Extensible Target Logic by @rlundeen2 in https://github.com/Azure/PyRIT/pull/125
• Added prompt softener prompt converter by @cseifert1 in https://github.com/Azure/PyRIT/pull/132
• Dataset Organization and Adding Public Jailbreaks by @rlundeen2 in https://github.com/Azure/PyRIT/pull/131
• Adding Image Target by @jbolor21 in https://github.com/Azure/PyRIT/pull/118
• Adding more authentication methods, add capital letters converter by @pgrek001 in https://github.com/Azure/PyRIT/pull/139
• Add cross-domain prompt injection orchestrator by @romanlutz in https://github.com/Azure/PyRIT/pull/127
• Added support to target an Ollama endpoint as a prompt chat target by @uskr in https://github.com/Azure/PyRIT/pull/141
• Normalizer multi modal/flexible support refactor by @rlundeen2 in https://github.com/Azure/PyRIT/pull/143
• Adding Identifiers to Memory by @rlundeen2 in https://github.com/Azure/PyRIT/pull/145
• Adding Data Type Normalizer Helpers by @rlundeen2 in https://github.com/Azure/PyRIT/pull/147
• Updating run_jupytext to cache notebooks that previously passed by @rlundeen2 in https://github.com/Azure/PyRIT/pull/148
• Gandalf through level7 by @jorisdg in https://github.com/Azure/PyRIT/pull/152
• Adding Multi-Modal Output Support to Converters by @rlundeen2 in https://github.com/Azure/PyRIT/pull/155
• Adding TTS Target by @rlundeen2 in https://github.com/Azure/PyRIT/pull/161
• Updating Gandalf Target to be more clear by @rlundeen2 in https://github.com/Azure/PyRIT/pull/153
• Support python 3.11 by @romanlutz in https://github.com/Azure/PyRIT/pull/168
• New Converters: Replace Whitespace and Leetspeak by @jbolor21 in https://github.com/Azure/PyRIT/pull/162
• Refactored SelfAskGptClassifier into SelfAskScore class and added Likert scale scoring by @blakebullwinkel in https://github.com/Azure/PyRIT/pull/154
• Fix mypy issues, convert Azure completion class to target, fix AOAI and OAI tests, remove clip embedding class by @romanlutz in https://github.com/Azure/PyRIT/pull/172
• Converter for prompt text to audio by @pgrek001 in https://github.com/Azure/PyRIT/pull/149
• Updating PromptSendingOrchestrator to handle multi-modal by @rlundeen2 in https://github.com/Azure/PyRIT/pull/174
• Generalize XPIA orchestrator by @romanlutz in https://github.com/Azure/PyRIT/pull/163
• Add Several Content Classifiers by @nina-msft in https://github.com/Azure/PyRIT/pull/175
• Add AzureOpenAIGPTVChatTarget to Support MultiModal by @rdheekonda in https://github.com/Azure/PyRIT/pull/160
• Refactoring Dalle Target to support database by @jbolor21 in https://github.com/Azure/PyRIT/pull/156

New Contributors

• @blakebullwinkel made their first contribution in https://github.com/Azure/PyRIT/pull/128
• @cseifert1 made their first contribution in https://github.com/Azure/PyRIT/pull/132
• @libera826 made their first contribution in https://github.com/Azure/PyRIT/pull/133
• @NaijingGuo made their first contribution in https://github.com/Azure/PyRIT/pull/140
• @uskr made their first contribution in https://github.com/Azure/PyRIT/pull/141
• @jorisdg made their first contribution in https://github.com/Azure/PyRIT/pull/144

Full Changelog: https://github.com/Azure/PyRIT/compare/v0.1.2...v0.2.0

- Python
Published by rlundeen2 almost 2 years ago

pyrit - v0.1.2

What's Changed

Big changes this release include solidifying the orchestrator, converter, target model for attacks, and migrating the local memory storage from a JSON file to a DuckDB instance.

The first two demos have been updated with the new architecture, and two new demos have been added; send all prompts and using prompt converters

Full List of Changes

• FEAT: Adding StringJoinConverter by @rlundeen2 in https://github.com/Azure/PyRIT/pull/70
• DOC: Add release instructions by @romanlutz in https://github.com/Azure/PyRIT/pull/57
• FEAT: Chain Prompt Converters in Normalizer by @rlundeen2 in https://github.com/Azure/PyRIT/pull/73
• FEAT: Adding Support for 1:N PromptConverters by @rlundeen2 in https://github.com/Azure/PyRIT/pull/75
• FEAT: Adding NoOpTarget by @rlundeen2 in https://github.com/Azure/PyRIT/pull/79
• FEAT: Added converter for ascii art by @petebryan in https://github.com/Azure/PyRIT/pull/81
• FEAT: Add rot13 by @pgrek001 in https://github.com/Azure/PyRIT/pull/80
• FEAT: Adding Batch/Async Processing to PromptTargets by @rlundeen2 in https://github.com/Azure/PyRIT/pull/91
• FEAT: add support for chat messages dataset by @dlmgary in https://github.com/Azure/PyRIT/pull/90
• DOC: Release guidelines and PR template update by @romanlutz in https://github.com/Azure/PyRIT/pull/92
• DOC: Adding Docs for ChatMessageNormalizer by @rlundeen2 in https://github.com/Azure/PyRIT/pull/93
• FEAT: Prompt Variation Converter by @jbolor21 in https://github.com/Azure/PyRIT/pull/86
• DOC: Adding Converter Docs and Demos by @rlundeen2 in https://github.com/Azure/PyRIT/pull/100
• MAINT: Add red teaming orchestrators to replace RedTeamingBot by @romanlutz in https://github.com/Azure/PyRIT/pull/84
• FEAT: Making promptnoptarget into a stream target by @rlundeen2 in https://github.com/Azure/PyRIT/pull/99
• MAINT: Adding orchestrator abstract base class by @rlundeen2 in https://github.com/Azure/PyRIT/pull/102
• FIX: simplify flow in red teaming orchestrator code by @romanlutz in https://github.com/Azure/PyRIT/pull/105
• DOC: use google style docstrings by @romanlutz in https://github.com/Azure/PyRIT/pull/104
• DOC: add short guide on how to handle stale PRs & introduce standardized prefixes by @romanlutz in https://github.com/Azure/PyRIT/pull/101
• FEAT: Language Translation Converter by @rlundeen2 in https://github.com/Azure/PyRIT/pull/106
• FEAT: Add scalable and efficient memory by @rdheekonda in https://github.com/Azure/PyRIT/pull/97
• FEAT: add support for question answering benchmark by @dlmgary in https://github.com/Azure/PyRIT/pull/94
• FEAT: New prompt target: AzureBlobStorageTarget by @nina-msft in https://github.com/Azure/PyRIT/pull/95
• FEAT: Add UTR39 confusability converter by @yonatanzunger in https://github.com/Azure/PyRIT/pull/115
• MAINT: Refactoring AzureOpenAIChat to only be a promptTarget by @rlundeen2 in https://github.com/Azure/PyRIT/pull/114
• FEAT: Add support to OpenAI API to use official or custom endpoints by @friyin in https://github.com/Azure/PyRIT/pull/65
• FEAT: Migrating Azure ML to PromptTarget by @rlundeen2 in https://github.com/Azure/PyRIT/pull/113
• Various bug fixes and smaller documentation updates by the AI Red Team

New Contributors

• @eltociear made their first contribution in https://github.com/Azure/PyRIT/pull/67
• @petebryan made their first contribution in https://github.com/Azure/PyRIT/pull/81
• @pgrek001 made their first contribution in https://github.com/Azure/PyRIT/pull/80
• @yonatanzunger made their first contribution in https://github.com/Azure/PyRIT/pull/115
• @friyin made their first contribution in https://github.com/Azure/PyRIT/pull/65

Full Changelog: https://github.com/Azure/PyRIT/compare/v0.1.1...v0.1.2

- Python
Published by rlundeen2 almost 2 years ago

pyrit - v0.1.1

What's Changed

The previous release 0.1.0 did not include the datasets used in the example notebooks. Version 0.1.1 addresses this.

Full list of changes

• Cleanup notebook outputs by @rdheekonda in https://github.com/Azure/PyRIT/pull/56
• Fixing bug with pypi package directory by @rlundeen2 in https://github.com/Azure/PyRIT/pull/60
• Add Details to Contributor Guide About Forking/PRs by @nina-msft in https://github.com/Azure/PyRIT/pull/59
• Fix hyperlinks and updated package version by @rdheekonda in https://github.com/Azure/PyRIT/pull/58
• add datasets folder to wheel by @romanlutz (https://github.com/Azure/PyRIT/pull/61)

Full Changelog: https://github.com/Azure/PyRIT/commits/v0.1.1

- Python
Published by romanlutz almost 2 years ago

pyrit - v0.1.0

What's Changed

This is the first release of PyRIT to PyPI including basic documentation and notebooks.

Full list of changes

• Update notebook dependency due to vulnerabilities by @romanlutz in https://github.com/Azure/PyRIT/pull/26
• Add article in SECURITY.md by @romanlutz in https://github.com/Azure/PyRIT/pull/27
• Remove poetry by @romanlutz in https://github.com/Azure/PyRIT/pull/24
• add NOTICE file by @romanlutz in https://github.com/Azure/PyRIT/pull/28
• Change README content to say "prompt injection attacks" by @romanlutz in https://github.com/Azure/PyRIT/pull/29
• Update Demo Notebook Prerequisites by @rdheekonda in https://github.com/Azure/PyRIT/pull/30
• Update chat client & multiturn notebooks with more information by @romanlutz in https://github.com/Azure/PyRIT/pull/25
• Remove unused definitions from models.py by @romanlutz in https://github.com/Azure/PyRIT/pull/31
• Update pyproject.toml to find packages/modules by @romanlutz in https://github.com/Azure/PyRIT/pull/33
• Prompt Normalizer and Refactor by @rlundeen2 in https://github.com/Azure/PyRIT/pull/32
• Export requirements before component governance is run by @romanlutz in https://github.com/Azure/PyRIT/pull/35
• fixing bug with redteamingbot by @rlundeen2 in https://github.com/Azure/PyRIT/pull/34
• Fixing bug with RedTeamingBot attack strategy by @rlundeen2 in https://github.com/Azure/PyRIT/pull/36
• Standardize Endpoints and API Keys for AML Chat by @rlundeen2 in https://github.com/Azure/PyRIT/pull/37
• Added Flag character transformer for invisible prompts. by @ChristBian1 in https://github.com/Azure/PyRIT/pull/38
• Add Jupyter Dependencies + aiohttp to pyproject.toml by @nina-msft in https://github.com/Azure/PyRIT/pull/39
• AML Endpoint Bug Fix and Refactor by @rlundeen2 in https://github.com/Azure/PyRIT/pull/40
• Add jailbreakchat prompt templates and modify the structure of existing prompts by @romanlutz in https://github.com/Azure/PyRIT/pull/42
• Fix link to How to Guide by @romanlutz in https://github.com/Azure/PyRIT/pull/41
• fixing bug where default was set to debug by @rlundeen2 in https://github.com/Azure/PyRIT/pull/44
• build: update versions of pre-commit hooks by @dlmgary in https://github.com/Azure/PyRIT/pull/47
• Integrate Hugging Face Models with Azure ML: Download, Register, Deploy, and Test by @rdheekonda in https://github.com/Azure/PyRIT/pull/43
• Standardizing environment for completions and embeddings by @rlundeen2 in https://github.com/Azure/PyRIT/pull/46
• Adding Chat Message Normalizers so AML can work with more Models by @rlundeen2 in https://github.com/Azure/PyRIT/pull/45
• Improving multi-turn strategy by @rlundeen2 in https://github.com/Azure/PyRIT/pull/48
• Add ipykernel dependency package by @rdheekonda in https://github.com/Azure/PyRIT/pull/50
• Refactoring Docs to use JupyText by @rlundeen2 in https://github.com/Azure/PyRIT/pull/49
• Replace old graphics with new ones by @romanlutz in https://github.com/Azure/PyRIT/pull/53
• Add learn link to README by @romanlutz in https://github.com/Azure/PyRIT/pull/55
• Convert PyRIT Framework How to Guide to jupytext by @nina-msft in https://github.com/Azure/PyRIT/pull/51
• Move wiki contents into doc folder, update references by @romanlutz in https://github.com/Azure/PyRIT/pull/54
• Update HF Azure ML deployment managed compute to allow liveness probe configuration by @rdheekonda in https://github.com/Azure/PyRIT/pull/52

New Contributors

• @ChristBian1 made their first contribution in https://github.com/Azure/PyRIT/pull/38

Full Changelog: https://github.com/Azure/PyRIT/commits/v0.1.0

- Python
Published by romanlutz almost 2 years ago