Conversation
…ases (google#4896) Closes google#4896 - Add `events: Optional[list[Event]] = Field(default=None)` to `SessionInput` in `eval_case.py`. - Append initial session events to the newly created session in `_get_or_create_eval_session` within `evaluation_generator.py`. - Support `events` and `session_id` in `convert_eval_set_to_pydantic_schema` in `local_eval_sets_manager.py`. - Add unit tests in `test_eval_case.py`, `test_evaluation_generator.py`, and `test_local_eval_sets_manager.py`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please ensure you have read the contribution guide before creating a pull request.
Link to Issue or Description of Change
1. Link to an existing issue (if applicable):
SessionInputfor eval cases #4896Problem:
When evaluating LLM agents on mid-conversation turns (e.g. evaluating queries that rely on conversational context like "Send that information to my phone" or "Now book the second option"), there was previously no way to seed prior conversation events into the evaluation session.
SessionInputonly exposedapp_name,user_id,session_id, andstate.conversationforces those turns to be replayed through the LLM and evaluated/scored by autoraters, introducing non-deterministic variance and noise.statedoes not populatesession.events, which is what LLM agents and compaction flows inspect for chat history.Solution:
events: Optional[list[Event]] = Field(default=None)toSessionInputinsrc/google/adk/evaluation/eval_case.py._get_or_create_eval_sessioninsrc/google/adk/evaluation/evaluation_generator.py, iterate overinitial_session.eventsand append them viaawait session_service.append_event(session=session, event=event). This records events insession.events, applies any state deltas, and works seamlessly with both in-memory and persistent session services.eventsandsession_idinconvert_eval_set_to_pydantic_schemainsrc/google/adk/evaluation/local_eval_sets_manager.pyfor legacy JSON format compatibility.test_eval_case.py,test_evaluation_generator.py, andtest_local_eval_sets_manager.py.Testing Plan
Unit Tests:
pytest summary:
Manual End-to-End (E2E) Tests:
Verified round-tripping
SessionInputserialization to JSON and execution throughEvaluationGenerator._generate_inferences_from_root_agentwith pre-seededEventinstances insession.events.Checklist