[BREAKING] MAINT: Standardize garak.encoding defaults and fix atomic-attack name collisions#2058
Conversation
| """ | ||
|
|
||
| # Aggregate member | ||
| # Aggregate members |
There was a problem hiding this comment.
Should we make this run more attacks? What's the run time, and how does it compare to other scanners?
I think target of 10-20 minutes is good and this may finish too qquickly
There was a problem hiding this comment.
Measured after curation: DEFAULT ~0.7 min, ALL ~3.3 min (seed-capped). Getting into the 10–20 min range means raising the per-dataset seed caps. Do you see that as a tuning/follow up or a blocker for this PR?
There was a problem hiding this comment.
I think the biggest change in this PR is changing the defaults, so we might want to change the dataset size a bit and adding more defaults
8fe1194 to
2b54413
Compare
2b54413 to
411f918
Compare
rlundeen2
left a comment
There was a problem hiding this comment.
Similar to psychosocial this has a lot of conflicts with main. But the fixes here are all applicable.
Rebase onto main, then re-apply the three genuinely valuable changes inside the new structure:
- EncodingStrategy: add DEFAULT = ("default", {"default"}), tag Base16/ROT13/MorseCode, and the get_aggregate_tags() override. This part is contract-agnostic and ports verbatim. Flip default_strategy=EncodingStrategy.DEFAULT and bump VERSION → 2.
- Name-collision fix: add variant_slug to the converter table and set atomic_attack_name=f"{variant_slug}_{config_suffix}" + display_group=encoding_name — but inside main's _get_prompt_attacks(..., seed_groups=...) signature (seeds from context, not self._resolved_seed_groups).
- Drop the manual baseline block entirely — baseline is now central. Several of the PR's tests assert on manual baseline behavior and will need reworking against the central path.
- Tests: EncodingDatasetConfiguration is now a DatasetAttackConfiguration; the backend round-trip test and _get_prompt_attacks call sites need the new signatures.
And I recommend tweaking the defaults/dataset sizes further
… collisions Re-migrated onto current main (post technique rename microsoft#2153, technique organization microsoft#2155, PromptConverter->Converter microsoft#2161, seed-group rename microsoft#2165). garak.encoding was already seam/Technique-migrated on main; this layers the remaining #8380 standardization on top. Add an EncodingTechnique.DEFAULT aggregate (curated 13-of-17-scheme spread across base-N, byte-encoding, cipher, and symbolic families) with a get_aggregate_tags() override, and make it the default technique (was the exhaustive ALL). The niche/ lossy schemes (braille, ecoji, zalgo, ascii_smuggler) stay ALL-only. Follows the hand-written enum DEFAULT pattern used by scam/web_injection (not the factory default_technique_names path, which applies only to the core catalog). Fix atomic-attack name collisions: previously every converter variant and decode config shared atomic_attack_name=encoding_name, so base64's variants collided and their result buckets (keyed by atomic_attack_name) collapsed. Each variant now gets a unique variant_slug -> atomic_attack_name=f"{variant_slug}_{config_suffix}", with display_group=encoding_name so variants still group per encoding in output. Trim near-duplicate base64 variants (standard_b64encode is byte-identical to the default; b2a_base64 only appends a newline). Bump VERSION 1->2. Make the default run meatier per Rich's microsoft#2058 asks (06-20 'target of 10-20 minutes ... may finish too quickly'; 07-08 'change the dataset size a bit and adding more defaults'): raise per-dataset max_dataset_size 3->10 and expand DEFAULT from 8 to 13 schemes. ALL remains the exhaustive ceiling (adds the 4 niche/lossy schemes).
… collisions Re-migrated onto current main (post technique rename microsoft#2153, technique organization microsoft#2155, PromptConverter->Converter microsoft#2161, seed-group rename microsoft#2165). garak.encoding was already seam/Technique-migrated on main; this layers the remaining #8380 standardization on top. Add an EncodingTechnique.DEFAULT aggregate (curated 13-of-17-scheme spread across base-N, byte-encoding, cipher, and symbolic families) with a get_aggregate_tags() override, and make it the default technique (was the exhaustive ALL). The niche/ lossy schemes (braille, ecoji, zalgo, ascii_smuggler) stay ALL-only. Follows the hand-written enum DEFAULT pattern used by scam/web_injection (not the factory default_technique_names path, which applies only to the core catalog). Fix atomic-attack name collisions: previously every converter variant and decode config shared atomic_attack_name=encoding_name, so base64's variants collided and their result buckets (keyed by atomic_attack_name) collapsed. Each variant now gets a unique variant_slug -> atomic_attack_name=f"{variant_slug}_{config_suffix}", with display_group=encoding_name so variants still group per encoding in output. Trim near-duplicate base64 variants (standard_b64encode is byte-identical to the default; b2a_base64 only appends a newline). Bump VERSION 1->2. Make the default run meatier per Rich's microsoft#2058 asks (06-20 'target of 10-20 minutes ... may finish too quickly'; 07-08 'change the dataset size a bit and adding more defaults'): raise per-dataset max_dataset_size 3->10 and expand DEFAULT from 8 to 13 schemes. ALL remains the exhaustive ceiling (adds the 4 niche/lossy schemes).
Description
Part of the Standardizing Scenarios effort. This standardizes the garak.encoding
scenario so its default run is fast and representative instead of exhaustive, and
fixes a latent atomic-attack naming bug.
Two changes:
Default strategy is now a curated DEFAULT aggregate (Base16, ROT13, MorseCode —
one base-N, one substitution cipher, one symbolic alphabet) instead of ALL. This
drops a default run from 96 to 16 atomic attacks. ALL is still available for an
exhaustive run. The fast path is
--strategies rot13 --max-dataset-size 1.Atomic-attack names were not unique: every converter variant of an encoding shared
the encoding name (e.g. all four base64 variants × five prompt configs were named
"base64"). Since results and the display map are keyed by atomic_attack_name, those
collapsed to a single key, corrupting result tracking and --resume. Names are now
unique per variant (e.g. base64_urlsafe_decode0), and display_group keeps the
per-encoding grouping in reports.
The encoding axis here is the scheme, not an attack technique, so SINGLE_TURN/MULTI_TURN
aggregates don't apply and are intentionally not added.
Breaking: same constructor call now produces different default atomic attacks, so the
scenario VERSION is bumped 1 -> 2. On --resume against an old result this raises a clear
ValueError instead of silently merging incompatible runs. Public API and constructor
signatures are unchanged.
Also adds a backend regression test pinning the real EncodingDatasetConfiguration
round-trip, since the backend silently degrades a lost config subclass to a base config.
Tests and Documentation
Tests (tests/unit/scenario/garak/test_encoding.py, tests/unit/backend/test_scenario_run_service.py):
Full scenario unit suite passes (708). ruff, ruff format, and ty are clean.
Documentation (doc/scanner/garak.py + .ipynb):