Is your feature request related to a problem? Please describe.
I have some agents that require use of an artifact. I'd like to be able to unit test the agent independently of the workflow it falls within. I have to perform some hacks to get this to work because (1) session ID isn't an available value in the new schema, and (2) the artifact service is not injectable into AgentEvaluator.evaluate even though it looks like it's meant to be.
Describe the solution you'd like
- The addition of an
id parameter in eval_case.SessionInput and apply that as the session_id in EvaluationGenerator._process_query if it exists.
- Allow the passing of an
ArtifactService into AgentEvaluator.evaluate.
Describe alternatives you've considered
I'm currently able to run my tests by using this code:
session_id='test_session'
@pytest.fixture
def agent_evaluator(monkeypatch):
artifact_service = TestArtifactService()
MockArtifactService = lambda: artifact_service
with unittest.mock.patch(
'google.adk.artifacts.in_memory_artifact_service', {
'InMemoryArtifactService': MockArtifactService,
}
):
monkeypatch.setattr('google.adk.evaluation.evaluation_generator.InMemoryArtifactService', MockArtifactService)
from google.adk.evaluation.agent_evaluator import AgentEvaluator
yield AgentEvaluator, MockArtifactService
class TestArtifactService(InMemoryArtifactService):
SESSION_ID: ClassVar[str] = session_id
def save_artifact(self, *, app_name, user_id, session_id, filename, artifact):
return super().save_artifact(app_name=app_name, user_id=user_id, session_id=self.SESSION_ID, filename=filename, artifact=artifact)
def load_artifact(self, *, app_name, user_id, session_id, filename, version = None):
return super().load_artifact(app_name=app_name, user_id=user_id, session_id=self.SESSION_ID, filename=filename, version=version)
This works for my simple single eval case example, but would break when handling multiple different sessions in different eval cases. Applying session_id whenever an artifact is saved in order to load the last saved artifact would likely introduce many unwanted edge cases.
Is your feature request related to a problem? Please describe.
I have some agents that require use of an artifact. I'd like to be able to unit test the agent independently of the workflow it falls within. I have to perform some hacks to get this to work because (1) session ID isn't an available value in the new schema, and (2) the artifact service is not injectable into
AgentEvaluator.evaluateeven though it looks like it's meant to be.Describe the solution you'd like
idparameter in eval_case.SessionInput and apply that as thesession_idin EvaluationGenerator._process_query if it exists.ArtifactServiceintoAgentEvaluator.evaluate.Describe alternatives you've considered
I'm currently able to run my tests by using this code:
This works for my simple single eval case example, but would break when handling multiple different sessions in different eval cases. Applying
session_idwhenever an artifact is saved in order to load the last saved artifact would likely introduce many unwanted edge cases.