Skip to content

[BREAKING] MAINT: Standardize garak.encoding defaults and fix atomic-attack name collisions#2058

Closed
varunj-msft wants to merge 1 commit into
microsoft:mainfrom
varunj-msft:varunj-msft/8380-Standardizing-Scenarios-Garak-Encoding-Defaults
Closed

[BREAKING] MAINT: Standardize garak.encoding defaults and fix atomic-attack name collisions#2058
varunj-msft wants to merge 1 commit into
microsoft:mainfrom
varunj-msft:varunj-msft/8380-Standardizing-Scenarios-Garak-Encoding-Defaults

Conversation

@varunj-msft

@varunj-msft varunj-msft commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Description

Part of the Standardizing Scenarios effort. This standardizes the garak.encoding
scenario so its default run is fast and representative instead of exhaustive, and
fixes a latent atomic-attack naming bug.

Two changes:

  1. Default strategy is now a curated DEFAULT aggregate (Base16, ROT13, MorseCode —
    one base-N, one substitution cipher, one symbolic alphabet) instead of ALL. This
    drops a default run from 96 to 16 atomic attacks. ALL is still available for an
    exhaustive run. The fast path is --strategies rot13 --max-dataset-size 1.

  2. Atomic-attack names were not unique: every converter variant of an encoding shared
    the encoding name (e.g. all four base64 variants × five prompt configs were named
    "base64"). Since results and the display map are keyed by atomic_attack_name, those
    collapsed to a single key, corrupting result tracking and --resume. Names are now
    unique per variant (e.g. base64_urlsafe_decode0), and display_group keeps the
    per-encoding grouping in reports.

The encoding axis here is the scheme, not an attack technique, so SINGLE_TURN/MULTI_TURN
aggregates don't apply and are intentionally not added.

Breaking: same constructor call now produces different default atomic attacks, so the
scenario VERSION is bumped 1 -> 2. On --resume against an old result this raises a clear
ValueError instead of silently merging incompatible runs. Public API and constructor
signatures are unchanged.

Also adds a backend regression test pinning the real EncodingDatasetConfiguration
round-trip, since the backend silently degrades a lost config subclass to a base config.

Tests and Documentation

Tests (tests/unit/scenario/garak/test_encoding.py, tests/unit/backend/test_scenario_run_service.py):

  • VERSION bump and DEFAULT-as-default-strategy
  • DEFAULT membership = {Base16, ROT13, MorseCode}, and DEFAULT ⊂ ALL
  • atomic counts: DEFAULT=16, ALL=96, ROT13 fast path=6
  • name uniqueness under ALL (96 unique) and across multi-variant encodings
  • display_group still groups variants by encoding
  • baseline-disabled and custom-template count edge cases
  • real EncodingDatasetConfiguration round-trips through the backend service

Full scenario unit suite passes (708). ruff, ruff format, and ty are clean.

Documentation (doc/scanner/garak.py + .ipynb):

  • documented DEFAULT (curated) vs ALL (exhaustive) and the rot13 fast path
  • ran jupytext to keep the .py and .ipynb in sync; notebook outputs preserved

"""

# Aggregate member
# Aggregate members

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we make this run more attacks? What's the run time, and how does it compare to other scanners?

I think target of 10-20 minutes is good and this may finish too qquickly

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Measured after curation: DEFAULT ~0.7 min, ALL ~3.3 min (seed-capped). Getting into the 10–20 min range means raising the per-dataset seed caps. Do you see that as a tuning/follow up or a blocker for this PR?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the biggest change in this PR is changing the defaults, so we might want to change the dataset size a bit and adding more defaults

Comment thread pyrit/scenario/scenarios/garak/encoding.py Outdated
@rlundeen2 rlundeen2 self-assigned this Jun 20, 2026
@varunj-msft
varunj-msft force-pushed the varunj-msft/8380-Standardizing-Scenarios-Garak-Encoding-Defaults branch 2 times, most recently from 8fe1194 to 2b54413 Compare July 1, 2026 02:55
@varunj-msft
varunj-msft force-pushed the varunj-msft/8380-Standardizing-Scenarios-Garak-Encoding-Defaults branch from 2b54413 to 411f918 Compare July 6, 2026 21:18

@rlundeen2 rlundeen2 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar to psychosocial this has a lot of conflicts with main. But the fixes here are all applicable.

Rebase onto main, then re-apply the three genuinely valuable changes inside the new structure:

  • EncodingStrategy: add DEFAULT = ("default", {"default"}), tag Base16/ROT13/MorseCode, and the get_aggregate_tags() override. This part is contract-agnostic and ports verbatim. Flip default_strategy=EncodingStrategy.DEFAULT and bump VERSION → 2.
  • Name-collision fix: add variant_slug to the converter table and set atomic_attack_name=f"{variant_slug}_{config_suffix}" + display_group=encoding_name — but inside main's _get_prompt_attacks(..., seed_groups=...) signature (seeds from context, not self._resolved_seed_groups).
  • Drop the manual baseline block entirely — baseline is now central. Several of the PR's tests assert on manual baseline behavior and will need reworking against the central path.
  • Tests: EncodingDatasetConfiguration is now a DatasetAttackConfiguration; the backend round-trip test and _get_prompt_attacks call sites need the new signatures.

And I recommend tweaking the defaults/dataset sizes further

@varunj-msft
varunj-msft deleted the varunj-msft/8380-Standardizing-Scenarios-Garak-Encoding-Defaults branch July 13, 2026 22:10
varunj-msft pushed a commit to varunj-msft/PyRIT that referenced this pull request Jul 14, 2026
… collisions

Re-migrated onto current main (post technique rename microsoft#2153, technique
organization microsoft#2155, PromptConverter->Converter microsoft#2161, seed-group rename microsoft#2165).
garak.encoding was already seam/Technique-migrated on main; this layers the
remaining #8380 standardization on top.

Add an EncodingTechnique.DEFAULT aggregate (curated 13-of-17-scheme spread across
base-N, byte-encoding, cipher, and symbolic families) with a get_aggregate_tags()
override, and make it the default technique (was the exhaustive ALL). The niche/
lossy schemes (braille, ecoji, zalgo, ascii_smuggler) stay ALL-only. Follows the
hand-written enum DEFAULT pattern used by scam/web_injection (not the factory
default_technique_names path, which applies only to the core catalog).

Fix atomic-attack name collisions: previously every converter variant and decode
config shared atomic_attack_name=encoding_name, so base64's variants collided and
their result buckets (keyed by atomic_attack_name) collapsed. Each variant now
gets a unique variant_slug -> atomic_attack_name=f"{variant_slug}_{config_suffix}",
with display_group=encoding_name so variants still group per encoding in output.

Trim near-duplicate base64 variants (standard_b64encode is byte-identical to the
default; b2a_base64 only appends a newline). Bump VERSION 1->2.

Make the default run meatier per Rich's microsoft#2058 asks (06-20 'target of 10-20 minutes
... may finish too quickly'; 07-08 'change the dataset size a bit and adding more
defaults'): raise per-dataset max_dataset_size 3->10 and expand DEFAULT from 8 to
13 schemes. ALL remains the exhaustive ceiling (adds the 4 niche/lossy schemes).
varunj-msft pushed a commit to varunj-msft/PyRIT that referenced this pull request Jul 14, 2026
… collisions

Re-migrated onto current main (post technique rename microsoft#2153, technique
organization microsoft#2155, PromptConverter->Converter microsoft#2161, seed-group rename microsoft#2165).
garak.encoding was already seam/Technique-migrated on main; this layers the
remaining #8380 standardization on top.

Add an EncodingTechnique.DEFAULT aggregate (curated 13-of-17-scheme spread across
base-N, byte-encoding, cipher, and symbolic families) with a get_aggregate_tags()
override, and make it the default technique (was the exhaustive ALL). The niche/
lossy schemes (braille, ecoji, zalgo, ascii_smuggler) stay ALL-only. Follows the
hand-written enum DEFAULT pattern used by scam/web_injection (not the factory
default_technique_names path, which applies only to the core catalog).

Fix atomic-attack name collisions: previously every converter variant and decode
config shared atomic_attack_name=encoding_name, so base64's variants collided and
their result buckets (keyed by atomic_attack_name) collapsed. Each variant now
gets a unique variant_slug -> atomic_attack_name=f"{variant_slug}_{config_suffix}",
with display_group=encoding_name so variants still group per encoding in output.

Trim near-duplicate base64 variants (standard_b64encode is byte-identical to the
default; b2a_base64 only appends a newline). Bump VERSION 1->2.

Make the default run meatier per Rich's microsoft#2058 asks (06-20 'target of 10-20 minutes
... may finish too quickly'; 07-08 'change the dataset size a bit and adding more
defaults'): raise per-dataset max_dataset_size 3->10 and expand DEFAULT from 8 to
13 schemes. ALL remains the exhaustive ceiling (adds the 4 niche/lossy schemes).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants