MAINT Breaking: Rename Scenario Strategies to Techniques by rlundeen2 · Pull Request #2153 · microsoft/PyRIT

rlundeen2 · 2026-07-09T19:04:32Z

Summary

Renames the scenario-layer strategy → technique vocabulary for the 1.0 release. The CLI already selects techniques (the --strategies flag chose techniques), and the codebase already uses AttackTechnique / AttackTechniqueRegistry — ScenarioStrategy was the lone holdout. This aligns the naming end-to-end.

This is a hard break with no backward-compatibility aliases (intended for 1.0).

Key changes

ScenarioStrategy → ScenarioTechnique (base enum + module rename)
Scenario technique enums: Foundry/Cyber/Leakage/RapidResponse/Benchmark/Adversarial/TextAdaptive/Psychosocial*Strategy → *Technique
Params/registry: scenario_strategies/strategy_converters → scenario_techniques/technique_converters; _resolve_strategies_and_converters → _resolve_techniques_and_converters
Models/API (JSON fields): strategies, strategies_used, default_strategy, aggregate_strategies, all_strategies → techniques, techniques_used, default_technique, aggregate_techniques, all_techniques; ScenarioResult.get_strategies_used() → get_techniques_used()
CLI: --strategies/-s → --techniques/-t
Docs (scanner + scenarios + resiliency notebooks) and scenarios.instructions.md updated

Explicitly NOT touched (deferred)

Executor AttackStrategy/attack_strategy, WorkflowStrategy, PromptGeneratorStrategy, converter *SelectionStrategy, and StrategyResult are unrelated concepts and are preserved.

Verification

In-scope unit suite green; executor/converter safety suite green
ruff lint + format clean; ty check pyrit clean
CLI --help shows --techniques, -t; repo-wide sweep confirms no dangling old scenario symbols

End-to-end / live testing (nothing broken)

Ran the scanner path against a live OpenAI target and exercised the renamed API surface:

doc/scanner/1_pyrit_scan.ipynb — executed end-to-end (start-server → --list-scenarios/--list-initializers → real foundry.red_team_agent --techniques base64 scan → custom ScenarioTechnique subclass → stop-server). 0 cell errors; output shows the renamed vocabulary (scenario_techniques, technique_converters, techniques: N/N progress).
Direct CLI scan foundry.red_team_agent --target openai_chat --initializers target --techniques base64 — completed cleanly (Total Techniques: 2, per-group baseline / base64 breakdown).
doc/scanner/foundry.ipynb — executed the programmatic scenario + output_scenario_async rendering path. 0 cell errors (Scenario: RedTeamAgent, Atomic attacks: 2, Total Techniques: 2).
Renamed-enum smoke check — with the registry initialized, resolved every renamed technique enum used across the scanner notebooks and confirmed get_all_techniques() / get_aggregate_techniques() work: FoundryTechnique, RapidResponseTechnique, PsychosocialTechnique, CyberTechnique, JailbreakTechnique, LeakageTechnique, ScamTechnique, EncodingTechnique, DoctorTechnique, WebInjectionTechnique.

Also fixed a stale scanner-doc flag surfaced during testing: --scenario-techniques → --techniques (the doc referenced a non-existent flag).

Renames the scenario-layer 'strategy' vocabulary to 'technique' for 1.0, aligning with the pre-existing AttackTechnique/AttackTechniqueRegistry naming and the CLI (--techniques selects techniques). This is a hard break with no backward-compat aliases. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

The scanner walkthrough referenced a non-existent --scenario-techniques flag (carried forward from the earlier non-existent --scenario-strategies example). The actual CLI flag is --techniques / -t. Fixes the markdown examples in the pyrit_scan scanner guide. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: jsong468 <songjustin@microsoft.com>

… collisions Re-migrated onto current main (post technique rename microsoft#2153 / technique organization microsoft#2155). garak.encoding was already seam-migrated on main to EncodingTechnique(ScenarioTechnique); this layers the remaining #8380 standardization on top. Add an EncodingTechnique.DEFAULT aggregate (curated 8-scheme spread across base-N, cipher, and symbolic families) with a get_aggregate_tags() override, make it the default technique, and bump VERSION to 2. Follows the hand-written enum DEFAULT pattern used by scam/web_injection (not the factory default_technique_names path, which applies only to the core catalog). Fix atomic-attack name collisions by naming each converter variant f"{variant_slug}_{config_suffix}" and grouping variants under display_group=encoding_name so per-encoding results aggregate. Trim near-duplicate base64 variants (standard_b64encode is byte-identical to the default; b2a_base64 only appends a newline). Keep the default per-dataset cap at main's max_dataset_size=3 (no timing change).

… collisions Re-migrated onto current main (post technique rename microsoft#2153, technique organization microsoft#2155, PromptConverter->Converter microsoft#2161, seed-group rename microsoft#2165). garak.encoding was already seam/Technique-migrated on main; this layers the remaining #8380 standardization on top. Add an EncodingTechnique.DEFAULT aggregate (curated 8-scheme spread across base-N, cipher, and symbolic families) with a get_aggregate_tags() override, and make it the default technique (was the exhaustive ALL). Follows the hand-written enum DEFAULT pattern used by scam/web_injection (not the factory default_technique_names path, which applies only to the core catalog). Fix atomic-attack name collisions: previously every converter variant and decode config shared atomic_attack_name=encoding_name, so base64's variants collided and their result buckets (keyed by atomic_attack_name) collapsed. Each variant now gets a unique variant_slug -> atomic_attack_name=f"{variant_slug}_{config_suffix}", with display_group=encoding_name so variants still group per encoding in output. Trim near-duplicate base64 variants (standard_b64encode is byte-identical to the default; b2a_base64 only appends a newline). Bump VERSION 1->2. Keep main's per-dataset max_dataset_size=3 (no timing change).

…hods Rebuild airt.jailbreak on the mergeable-technique contract (post microsoft#2153/microsoft#2155/microsoft#2160/ microsoft#2161/microsoft#2165), replacing the bespoke TextJailbreakConverter + `match technique` dispatch with the pattern Rich described in microsoft#2045: a jailbreak template is a reusable technique, not a converter. Each jailbreak template becomes an AttackTechniqueSeedGroup (is_general_technique=True) that the base AtomicAttack merges into every objective via with_technique() at run time, without touching the SeedObjective. The technique enum is now pure delivery methods (JailbreakTechnique.System), each an AttackTechniqueFactory carrying the role-tagged framing seed. This mirrors the migrated core `flip`/role_play techniques. - Default objectives switched airt_harms -> harmbench (mirrors adversarial/red_team_agent). - num_templates / num_attempts / jailbreak_names are now declared run parameters (additional_parameters), so they are settable from the CLI / config, resolved from the parameter bag at build time. - Small curated default template set (aim, dan_11); full catalog opt-in via num_templates (random) or jailbreak_names (explicit), with mutual-exclusion validation. - Results group by jailbreak template (display_group); atomic-attack names are unique per (delivery method x template x attempt). - Resume-stable: the resolved template set is persisted to ScenarioResult metadata and replayed on resume so a random num_templates sample does not diverge. - VERSION 1 -> 2. Scope: ships the `system` delivery method (jailbreak framing as system prompt, objective sent as the user turn), which is the proven pattern. The `user`, `variation`, and `translation` delivery methods, and the baseline-skip optimization, are deferred: `user` currently drops the objective because a single user-role technique seed becomes the send's next_message (PromptSendingAttack sends next_message over the objective), so faithful user delivery needs objective-after-user-framing sequencing in the send path. Note: the default delivery semantics change from "objective embedded in the template as one user message" to "template as system prompt + objective as user turn".

… collisions Re-migrated onto current main (post technique rename microsoft#2153, technique organization microsoft#2155, PromptConverter->Converter microsoft#2161, seed-group rename microsoft#2165). garak.encoding was already seam/Technique-migrated on main; this layers the remaining #8380 standardization on top. Add an EncodingTechnique.DEFAULT aggregate (curated 8-scheme spread across base-N, cipher, and symbolic families) with a get_aggregate_tags() override, and make it the default technique (was the exhaustive ALL). Follows the hand-written enum DEFAULT pattern used by scam/web_injection (not the factory default_technique_names path, which applies only to the core catalog). Fix atomic-attack name collisions: previously every converter variant and decode config shared atomic_attack_name=encoding_name, so base64's variants collided and their result buckets (keyed by atomic_attack_name) collapsed. Each variant now gets a unique variant_slug -> atomic_attack_name=f"{variant_slug}_{config_suffix}", with display_group=encoding_name so variants still group per encoding in output. Trim near-duplicate base64 variants (standard_b64encode is byte-identical to the default; b2a_base64 only appends a newline). Bump VERSION 1->2. Keep main's per-dataset max_dataset_size=3 (no timing change).

…hods Rebuild airt.jailbreak on the mergeable-technique contract (post microsoft#2153/microsoft#2155/microsoft#2160/ microsoft#2161/microsoft#2165), replacing the bespoke TextJailbreakConverter + `match technique` dispatch with the pattern Rich described in microsoft#2045: a jailbreak template is a reusable technique, not a converter. Each jailbreak template becomes an AttackTechniqueSeedGroup (is_general_technique=True) that the base AtomicAttack merges into every objective via with_technique() at run time, without touching the SeedObjective. The technique enum is now pure delivery methods (JailbreakTechnique.System), each an AttackTechniqueFactory carrying the role-tagged framing seed. This mirrors the migrated core `flip`/role_play techniques. - Default objectives switched airt_harms -> harmbench (mirrors adversarial/red_team_agent). - num_templates / num_attempts / jailbreak_names are now declared run parameters (additional_parameters), so they are settable from the CLI / config, resolved from the parameter bag at build time. - Small curated default template set (aim, dan_11); full catalog opt-in via num_templates (random) or jailbreak_names (explicit), with mutual-exclusion validation. - Results group by jailbreak template (display_group); atomic-attack names are unique per (delivery method x template x attempt). - Resume-stable: the resolved template set is persisted to ScenarioResult metadata and replayed on resume so a random num_templates sample does not diverge. - VERSION 1 -> 2. Scope: ships the `system` delivery method (jailbreak framing as system prompt, objective sent as the user turn), which is the proven pattern. The `user`, `variation`, and `translation` delivery methods, and the baseline-skip optimization, are deferred: `user` currently drops the objective because a single user-role technique seed becomes the send's next_message (PromptSendingAttack sends next_message over the objective), so faithful user delivery needs objective-after-user-framing sequencing in the send path. Note: the default delivery semantics change from "objective embedded in the template as one user message" to "template as system prompt + objective as user turn".

… collisions Re-migrated onto current main (post technique rename microsoft#2153, technique organization microsoft#2155, PromptConverter->Converter microsoft#2161, seed-group rename microsoft#2165). garak.encoding was already seam/Technique-migrated on main; this layers the remaining #8380 standardization on top. Add an EncodingTechnique.DEFAULT aggregate (curated 8-scheme spread across base-N, cipher, and symbolic families) with a get_aggregate_tags() override, and make it the default technique (was the exhaustive ALL). Follows the hand-written enum DEFAULT pattern used by scam/web_injection (not the factory default_technique_names path, which applies only to the core catalog). Fix atomic-attack name collisions: previously every converter variant and decode config shared atomic_attack_name=encoding_name, so base64's variants collided and their result buckets (keyed by atomic_attack_name) collapsed. Each variant now gets a unique variant_slug -> atomic_attack_name=f"{variant_slug}_{config_suffix}", with display_group=encoding_name so variants still group per encoding in output. Trim near-duplicate base64 variants (standard_b64encode is byte-identical to the default; b2a_base64 only appends a newline). Bump VERSION 1->2. Keep main's per-dataset max_dataset_size=3 (no timing change).

…hods Rebuild airt.jailbreak on the mergeable-technique contract (post microsoft#2153/microsoft#2155/microsoft#2160/ microsoft#2161/microsoft#2165), replacing the bespoke TextJailbreakConverter + `match technique` dispatch with the pattern Rich described in microsoft#2045: a jailbreak template is a reusable technique, not a converter. Each jailbreak template becomes an AttackTechniqueSeedGroup (is_general_technique=True) that the base AtomicAttack merges into every objective via with_technique() at run time, without touching the SeedObjective. The technique enum is now pure delivery methods (JailbreakTechnique.System), each an AttackTechniqueFactory carrying the role-tagged framing seed. This mirrors the migrated core `flip`/role_play techniques. - Default objectives switched airt_harms -> harmbench (mirrors adversarial/red_team_agent). - num_templates / num_attempts / jailbreak_names are now declared run parameters (additional_parameters), so they are settable from the CLI / config, resolved from the parameter bag at build time. - Small curated default template set (aim, dan_11); full catalog opt-in via num_templates (random) or jailbreak_names (explicit), with mutual-exclusion validation. - Results group by jailbreak template (display_group); atomic-attack names are unique per (delivery method x template x attempt). - Resume-stable: the resolved template set is persisted to ScenarioResult metadata and replayed on resume so a random num_templates sample does not diverge. - VERSION 1 -> 2. Scope: ships the `system` delivery method (jailbreak framing as system prompt, objective sent as the user turn), which is the proven pattern. The `user`, `variation`, and `translation` delivery methods, and the baseline-skip optimization, are deferred: `user` currently drops the objective because a single user-role technique seed becomes the send's next_message (PromptSendingAttack sends next_message over the objective), so faithful user delivery needs objective-after-user-framing sequencing in the send path. Note: the default delivery semantics change from "objective embedded in the template as one user message" to "template as system prompt + objective as user turn".

… collisions Re-migrated onto current main (post technique rename microsoft#2153, technique organization microsoft#2155, PromptConverter->Converter microsoft#2161, seed-group rename microsoft#2165). garak.encoding was already seam/Technique-migrated on main; this layers the remaining #8380 standardization on top. Add an EncodingTechnique.DEFAULT aggregate (curated 13-of-17-scheme spread across base-N, byte-encoding, cipher, and symbolic families) with a get_aggregate_tags() override, and make it the default technique (was the exhaustive ALL). The niche/ lossy schemes (braille, ecoji, zalgo, ascii_smuggler) stay ALL-only. Follows the hand-written enum DEFAULT pattern used by scam/web_injection (not the factory default_technique_names path, which applies only to the core catalog). Fix atomic-attack name collisions: previously every converter variant and decode config shared atomic_attack_name=encoding_name, so base64's variants collided and their result buckets (keyed by atomic_attack_name) collapsed. Each variant now gets a unique variant_slug -> atomic_attack_name=f"{variant_slug}_{config_suffix}", with display_group=encoding_name so variants still group per encoding in output. Trim near-duplicate base64 variants (standard_b64encode is byte-identical to the default; b2a_base64 only appends a newline). Bump VERSION 1->2. Make the default run meatier per Rich's microsoft#2058 asks (06-20 'target of 10-20 minutes ... may finish too quickly'; 07-08 'change the dataset size a bit and adding more defaults'): raise per-dataset max_dataset_size 3->10 and expand DEFAULT from 8 to 13 schemes. ALL remains the exhaustive ceiling (adds the 4 niche/lossy schemes).

rlundeen2 and others added 2 commits July 9, 2026 12:03

jsong468 reviewed Jul 9, 2026

View reviewed changes

Comment thread doc/scanner/1_pyrit_scan.ipynb Outdated

jsong468 approved these changes Jul 9, 2026

View reviewed changes

Update doc/scanner/1_pyrit_scan.ipynb

6f20f05

Co-authored-by: jsong468 <songjustin@microsoft.com>

rlundeen2 enabled auto-merge July 9, 2026 22:47

rlundeen2 added this pull request to the merge queue Jul 9, 2026

romanlutz approved these changes Jul 9, 2026

View reviewed changes

Merged via the queue into microsoft:main with commit e151cce Jul 9, 2026
53 checks passed

rlundeen2 deleted the rlundeen2-rename-scenario-strategy-to-technique branch July 9, 2026 23:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

MAINT Breaking: Rename Scenario Strategies to Techniques#2153

MAINT Breaking: Rename Scenario Strategies to Techniques#2153
rlundeen2 merged 3 commits into
microsoft:mainfrom
rlundeen2:rlundeen2-rename-scenario-strategy-to-technique

rlundeen2 commented Jul 9, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

rlundeen2 commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key changes

Explicitly NOT touched (deferred)

Verification

End-to-end / live testing (nothing broken)

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rlundeen2 commented Jul 9, 2026 •

edited

Loading