FEAT: Add remove_seeds_from_memory_async to memory by blahdeblahde · Pull Request #2243 · microsoft/PyRIT

blahdeblahde · 2026-07-21T22:22:57Z

Description

PyRIT memory can add seed prompt datasets (via add_seed_datasets_to_memory_async / add_seeds_to_memory_async) and query them with get_seeds, but there is no counterpart for removing seeds. Today users have to drop into raw SQL, which is error-prone and differs across the supported backends (DuckDB, SQLite, Azure SQL).

This PR adds remove_seeds_from_memory_async, which accepts the same filter parameters as get_seeds (dataset_name, dataset_name_pattern, added_by, harm_categories, authors, groups, source, value_sha256, data_types, seed_type, parameters, metadata, prompt_group_ids) and returns the number of seeds removed.

Recommended workflow — preview, then delete with the same filters:

# See what matches before deleting
to_delete = memory.get_seeds(dataset_name="illegal")

# Remove them; returns the count deleted
removed = await memory.remove_seeds_from_memory_async(dataset_name="illegal")

Implementation

Extracted the filter-building logic out of get_seeds into a shared private helper (_build_seed_filter_conditions) so both query and removal build conditions from a single source and cannot drift.
remove_seeds_from_memory_async deletes matching rows in a single transaction using the SQLAlchemy ORM (rollback on error), so it behaves consistently across DuckDB, SQLite, and Azure SQL.

Safety

At least one filter must be provided; a no-filter call raises ValueError to prevent accidentally wiping the entire seed database.

Tests and Documentation

Tests — tests/unit/memory/memory_interface/test_interface_remove_seeds.py covers:

exact dataset_name match
dataset_name_pattern (SQL LIKE)
multi-filter narrowing (dataset_name + added_by)
no-filter call raises ValueError
zero-match returns 0 and deletes nothing
harm_categories (list field) and source filters

Documentation — added a "Removing Seeds from the Database" section to doc/code/memory/8_seed_database.py (and the synced .ipynb) demonstrating the preview-then-remove workflow.

Copilot

Pull request overview

This PR adds first-class support for deleting seed prompts from PyRIT memory by introducing remove_seeds_from_memory_async, mirroring the existing get_seeds filter surface so users no longer need backend-specific SQL for seed cleanup.

Changes:

Extracts seed filter construction into a shared private helper (_build_seed_filter_conditions) used by both seed querying and deletion.
Adds remove_seeds_from_memory_async to delete matching SeedEntry rows transactionally and return the number removed (with a no-filter safety guard).
Adds unit tests and documentation demonstrating a “preview via get_seeds, then delete via remove_seeds_from_memory_async” workflow.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 13 comments.

File	Description
`pyrit/memory/memory_interface.py`	Adds shared seed-filter builder and new async seed removal API implemented via SQLAlchemy.
`tests/unit/memory/memory_interface/test_interface_remove_seeds.py`	Adds unit coverage for common filter cases and safety behavior.
`doc/code/memory/8_seed_database.py`	Documents seed removal usage and recommended preview-then-delete workflow.
`doc/code/memory/8_seed_database.ipynb`	Syncs the same documentation updates into the notebook format.

+@pytest.mark.asyncio
+async def test_remove_seeds_by_dataset_name(sqlite_instance: MemoryInterface):


+@pytest.mark.asyncio
+async def test_remove_seeds_by_dataset_name_pattern(sqlite_instance: MemoryInterface):


+@pytest.mark.asyncio
+async def test_remove_seeds_by_added_by(sqlite_instance: MemoryInterface):


+@pytest.mark.asyncio
+async def test_remove_seeds_multi_filter_narrowing(sqlite_instance: MemoryInterface):


+@pytest.mark.asyncio
+async def test_remove_seeds_no_filter_raises_value_error(sqlite_instance: MemoryInterface):


+        remove_seeds_from_memory_async stay in sync and cannot drift.

        Args:
            value (str): The value to match by substring. If None, all values are returned.


+
+        Args:
+            value (str): The value to match by substring. If None, all values are returned.
+            value_sha256 (str): The SHA256 hash of the value to match. If None, all values are returned.


+            prompt_group_ids (Sequence[uuid.UUID]): A list of prompt group IDs to filter by.
+
+        Returns:
+            Sequence[SeedPrompt]: A list of prompts matching the criteria.


+
+        Args:
+            value (str): The value to match by substring. If None, all values are considered.
+            value_sha256 (str): The SHA256 hash of the value to match. If None, all values are considered.


+                query = session.query(SeedEntry).filter(and_(*conditions))
+                count = query.delete(synchronize_session="fetch")
+                session.commit()


Adds an API to remove seed prompts from memory using the same filter parameters as get_seeds (dataset_name, dataset_name_pattern, added_by, harm_categories, authors, groups, source, value_sha256, etc.). Implementation: - Extract filter-building logic from get_seeds into shared _build_seed_filter_conditions helper to prevent drift - Add remove_seeds_from_memory_async that uses the shared helper - Require at least one filter (raises ValueError if none provided) - Single transaction with rollback on failure - Works across DuckDB, SQLite, Azure SQL (uses SQLAlchemy ORM) Unit tests cover: exact name filter, LIKE pattern, multi-filter narrowing, no-filter ValueError, zero-match return, harm_categories, and source filter. Docs: add a 'Removing Seeds from the Database' section to the seed database notebook (8_seed_database.py/.ipynb) demonstrating the preview-then-remove workflow. Resolves AB#5688 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9e3cc10-5c77-4384-92d9-d27dad9baea2

blahdeblahde · 2026-07-21T22:38:57Z

Thanks for the review! Addressed all points in the latest push:

Removed the @pytest.mark.asyncio decorators from the new tests (asyncio_mode = auto is set globally).
Fixed the value_sha256 docstring type to Sequence[str] | None in the shared helper, get_seeds, and remove_seeds_from_memory_async.
Corrected the get_seeds return docstring to Sequence[Seed] (not just SeedPrompt).
Changed the delete to synchronize_session=False to avoid the extra SELECT on a short-lived session.

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

+        # Delete matching rows in a single transaction so it is atomic across backends.
+        with closing(self.get_session()) as session:
+            try:
+                query = session.query(SeedEntry).filter(and_(*conditions))
+                count = query.delete(synchronize_session=False)
+                session.commit()
+                return count
+            except SQLAlchemyError as e:
+                session.rollback()
+                logger.exception(f"Failed to remove seeds from memory: {e}")
+                raise


blahdeblahde · 2026-07-21T22:47:17Z

@microsoft-github-policy-service agree company="Microsoft"

hannahwestra25 · 2026-07-22T18:11:46Z

+        prevent accidental deletion of all seeds. The deletion runs in a single transaction, so it works
+        consistently across DuckDB, SQLite, and Azure SQL.


DuckDB is no longer a supported memory backend. This should just say SQLite and Azure SQL.

hannahwestra25 · 2026-07-22T18:13:23Z

            logger.exception(f"Failed to retrieve prompts with dataset name {dataset_name} with error {e}")
            raise

+    async def remove_seeds_from_memory_async(


why async ?

hannahwestra25 · 2026-07-22T18:18:22Z

+    async def remove_seeds_from_memory_async(
+        self,
+        *,
+        value: str | None = None,


hmmm i'm kinda concerned about using value for deletion. It's much safer when we're just getting a seed but if a user inputs "the" for example or a commonly used word the substring matching would delete a lot of seeds.

hannahwestra25 · 2026-07-22T18:23:45Z

        metadata: dict[str, str | int] | None = None,
        prompt_group_ids: Sequence[uuid.UUID] | None = None,
-    ) -> Sequence[Seed]:
+    ) -> list[Any]:


Can we make this type stricter than list[Any]? The file already uses list[ColumnElement[bool]] for condition builders. Using Any here removes useful checking from both query and deletion paths.

hannahwestra25 · 2026-07-22T18:24:54Z

+        with closing(self.get_session()) as session:
+            try:
+                query = session.query(SeedEntry).filter(and_(*conditions))
+                count = query.delete(synchronize_session=False)


What are we doing to file-backed image, audio, and video seeds? This removes the database row but leaves the serialized file behind. We should either clean up owned artifacts or clearly document that this only removes database records.

hannahwestra25 · 2026-07-22T18:37:40Z

+            parameters (Sequence[str]): A list of parameters to filter by. Specifying parameters effectively targets
+                prompt templates instead of prompts.
+            metadata (dict[str, str | int]): A free-form dictionary for tagging prompts with custom metadata.
+            prompt_group_ids (Sequence[uuid.UUID]): A list of prompt group IDs to filter by.


so prompt_group_ids would remove entire groups ? I'm wondering if we should have a separate remove_seed_groups_from_memory (and similarly for datasets)

hannahwestra25 · 2026-07-22T18:40:03Z

+# %% [markdown]
+# ## Removing Seeds from the Database
+#
+# Just as you can add and query seeds, you can remove them using `remove_seeds_from_memory_async`. It accepts the same filter parameters as `get_seeds`, so the recommended workflow is to preview the matching seeds with `get_seeds(...)` first, then remove them with the same filters. The method returns the number of seeds removed.


just noting that there are some possible cases where deleting will cause unintended consequences:

Delete the sole objective, leave prompts: invalid AttackSeedGroup; scenario initialization raises ValueError.

Delete one modality: group remains valid but sends only text instead of text plus image/audio.

Delete one conversation turn: group may remain valid but its intended multi-turn context is incomplete.

Delete the only role-bearing prompt in a sequence: surviving multi-sequence group may fail role validation.

these are just some examples and for the most part can be classified as user errors but might be something good to document

Copilot AI review requested due to automatic review settings July 21, 2026 22:22

Copilot started reviewing on behalf of blahdeblahde July 21, 2026 22:23 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 21, 2026 22:38

blahdeblahde force-pushed the blahdeblahde-remove-seeds-from-memory-api branch from 2d45b57 to 0e3ae02 Compare July 21, 2026 22:38

Copilot started reviewing on behalf of blahdeblahde July 21, 2026 22:39 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

hannahwestra25 reviewed Jul 22, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

FEAT: Add remove_seeds_from_memory_async to memory#2243

FEAT: Add remove_seeds_from_memory_async to memory#2243
blahdeblahde wants to merge 1 commit into
microsoft:mainfrom
blahdeblahde:blahdeblahde-remove-seeds-from-memory-api

blahdeblahde commented Jul 21, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

blahdeblahde commented Jul 21, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

blahdeblahde commented Jul 21, 2026

Uh oh!

hannahwestra25 Jul 22, 2026

Uh oh!

hannahwestra25 Jul 22, 2026

Uh oh!

hannahwestra25 Jul 22, 2026

Uh oh!

hannahwestra25 Jul 22, 2026

Uh oh!

hannahwestra25 Jul 22, 2026

Uh oh!

hannahwestra25 Jul 22, 2026

Uh oh!

hannahwestra25 Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@pytest.mark.asyncio
		async def test_remove_seeds_by_dataset_name(sqlite_instance: MemoryInterface):

		@pytest.mark.asyncio
		async def test_remove_seeds_by_added_by(sqlite_instance: MemoryInterface):

		@pytest.mark.asyncio
		async def test_remove_seeds_multi_filter_narrowing(sqlite_instance: MemoryInterface):

		@pytest.mark.asyncio
		async def test_remove_seeds_no_filter_raises_value_error(sqlite_instance: MemoryInterface):

		prevent accidental deletion of all seeds. The deletion runs in a single transaction, so it works
		consistently across DuckDB, SQLite, and Azure SQL.

Uh oh!

Conversation

blahdeblahde commented Jul 21, 2026

Description

Implementation

Safety

Tests and Documentation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

blahdeblahde commented Jul 21, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

blahdeblahde commented Jul 21, 2026

Uh oh!

hannahwestra25 Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

hannahwestra25 Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

hannahwestra25 Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

hannahwestra25 Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

hannahwestra25 Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

hannahwestra25 Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

hannahwestra25 Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants