Python: Use AI Foundry evaluators for self-reflection

[David Wu (YusakuNo1)]

Motivation and Context

Description

Demostrate how to implement self-reflection from agent framework

Contribution Checklist

• The code builds clean without any errors or warnings
• The PR follows the Contribution Guidelines
• All unit tests pass, and I have added new tests where possible
• Is this a breaking change? If yes, add "[BREAKING]" prefix to the title of the PR.

[Copilot]

Pull Request Overview

This PR adds a new Python sample demonstrating self-reflection capabilities for LLM responses using AI Foundry's groundedness evaluators. The sample shows how to iteratively improve LLM responses by evaluating them and providing feedback for refinement.

• New self-reflection sample using groundedness evaluation
• Batch processing capability for evaluating multiple prompts
• Integration with Azure OpenAI and AI Foundry evaluators

Reviewed Changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 7 comments.

File	Description
python/samples/getting_started/observability/self_reflection.py	New sample implementing self-reflection loop with groundedness evaluation for LLM responses
python/samples/getting_started/observability/resources/suboptimal_groundedness_prompts.parquet	Test data file containing prompts for self-reflection evaluation
python/samples/getting_started/observability/.env.example	Added Azure OpenAI configuration variables for the self-reflection sample
python/samples/README.md	Added reference to the new self-reflection sample

Comments suppressed due to low confidence (2)

python/samples/getting_started/observability/.env.example:21

• The model name "gpt-4.1" in the .env.example file doesn't appear to be a valid Azure OpenAI model deployment name. This should be updated to a valid model name like "gpt-4", "gpt-4o", or "gpt-4-turbo".
  python/samples/getting_started/observability/self_reflection.py:161
• This statement is unreachable.

        best_response = raw_response.choices[0].message.content

[Eduard van Valkenburg (eavanvalkenburg)]

Love the idea of this, but could you have a look at our ChatMiddleware and implement using that, that way it becomes a much more native way of working, and if you put the actual middleware function inside of a class then all the setup and best_score etc. can be captured and maintained, this samples shows classes, the same works for ChatMiddleware, but this could also be implemented with AgentMiddleware: https://github.com/microsoft/agent-framework/blob/main/python/samples/getting_started/middleware/class_based_middleware.py

[David Wu (YusakuNo1)]

Love the idea of this, but could you have a look at our ChatMiddleware and implement using that, that way it becomes a much more native way of working, and if you put the actual middleware function inside of a class then all the setup and best_score etc. can be captured and maintained, this samples shows classes, the same works for ChatMiddleware, but this could also be implemented with AgentMiddleware: https://github.com/microsoft/agent-framework/blob/main/python/samples/getting_started/middleware/class_based_middleware.py

Hi Eduard van Valkenburg (@eavanvalkenburg) , thanks for the suggestion! This approach is not added into middleware because it's slow and consumes a lot of tokens, we would like the AI developers to try it as "optional" instead of in the core path. What do you think?

[David Wu (YusakuNo1)]

Love the idea of this, but could you have a look at our ChatMiddleware and implement using that, that way it becomes a much more native way of working, and if you put the actual middleware function inside of a class then all the setup and best_score etc. can be captured and maintained, this samples shows classes, the same works for ChatMiddleware, but this could also be implemented with AgentMiddleware: https://github.com/microsoft/agent-framework/blob/main/python/samples/getting_started/middleware/class_based_middleware.py

Within the team, we discussed offline for the approach of using middeware. For the use case of middleware, it'll intercept the traffic and do something extra, but for our current use case, the self-reflection code will also modify the original user input, and then run again. With this, maybe it's not a good use case for middleware, plus, this is observability use case, maybe we can make it simple for the user for now. What do you think?

[Tao Chen (TaoChenOSU)]

I probably wouldn't put this sample in the observability samples because it does not have anything to do with observability.

Maybe a new evaluation folder?

[David Wu (YusakuNo1)]

I probably wouldn't put this sample in the observability samples because it does not have anything to do with observability.

Maybe a new evaluation folder?

For organization definition, our "observability" includes both tracing and evaluation... I think we can host the sample in observability for now?

[Tao Chen (TaoChenOSU)]

I probably wouldn't put this sample in the observability samples because it does not have anything to do with observability.
Maybe a new evaluation folder?

For organization definition, our "observability" includes both tracing and evaluation... I think we can host the sample in observability for now?

What do you think Eduard van Valkenburg (@eavanvalkenburg)? If we want to keep evaluation under observability, could we change the name of the sample to evaluation_with_self_reflection?

[Eduard van Valkenburg (eavanvalkenburg)]

Some small notes, but looks good overall