Skip to content

Make evaluation compatible with artifacts #2075

Description

@smcjones

Is your feature request related to a problem? Please describe.
I have some agents that require use of an artifact. I'd like to be able to unit test the agent independently of the workflow it falls within. I have to perform some hacks to get this to work because (1) session ID isn't an available value in the new schema, and (2) the artifact service is not injectable into AgentEvaluator.evaluate even though it looks like it's meant to be.

Describe the solution you'd like

  1. The addition of an id parameter in eval_case.SessionInput and apply that as the session_id in EvaluationGenerator._process_query if it exists.
  2. Allow the passing of an ArtifactService into AgentEvaluator.evaluate.

Describe alternatives you've considered
I'm currently able to run my tests by using this code:

session_id='test_session'

@pytest.fixture
def agent_evaluator(monkeypatch):
    artifact_service = TestArtifactService()
    MockArtifactService = lambda: artifact_service

    with unittest.mock.patch(
        'google.adk.artifacts.in_memory_artifact_service', {
            'InMemoryArtifactService': MockArtifactService,
        }
    ):
        monkeypatch.setattr('google.adk.evaluation.evaluation_generator.InMemoryArtifactService', MockArtifactService)
        
        from google.adk.evaluation.agent_evaluator import AgentEvaluator
        yield AgentEvaluator, MockArtifactService


class TestArtifactService(InMemoryArtifactService):
    SESSION_ID: ClassVar[str] = session_id

    def save_artifact(self, *, app_name, user_id, session_id, filename, artifact):
        return super().save_artifact(app_name=app_name, user_id=user_id, session_id=self.SESSION_ID, filename=filename, artifact=artifact)

    def load_artifact(self, *, app_name, user_id, session_id, filename, version = None):
        return super().load_artifact(app_name=app_name, user_id=user_id, session_id=self.SESSION_ID, filename=filename, version=version)

This works for my simple single eval case example, but would break when handling multiple different sessions in different eval cases. Applying session_id whenever an artifact is saved in order to load the last saved artifact would likely introduce many unwanted edge cases.

Metadata

Metadata

Labels

eval[Component] This issue is related to evaluationneeds review[Status] The PR/issue is awaiting review from the maintainer

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions