feat: add logging conformance scenarios (capability, invalid level, threshold filtering) - #449
Conversation
…, and threshold filtering Addresses modelcontextprotocol#435 by adding three new conformance scenarios that test logging/setLevel behavior beyond the existing basic happy-path check: - logging-capability-advertisement: verifies server advertises logging in capabilities before emitting notifications - logging-invalid-level: verifies -32602 rejection for unrecognized severity levels - logging-threshold-filtering: verifies notifications respect the configured severity threshold (suppresses below, allows above) All scenarios target 2025-06-18 through 2025-11-25 (removedIn draft).
|
@joy7758 would appreciate your review on this one when you get a chance. |
|
Thanks for the follow-up. I reviewed current head I found four scope/evidence points worth resolving before these checks are treated as conformance results:
The repository contribution guide also asks for a passing real-SDK run and a specific failing case. The PR currently documents the in-repo everything-server path, so adding the pinned real-SDK output plus a negative fixture/result would complete the evidence loop. The |
|
Thanks @joy7758, all four points are valid. I'll push a revision that:
|
- Downgrade threshold filtering from conformance assertions to interoperability observations (spec has no MUST/SHOULD for receiver-side filtering) - Make capability check conditional: SKIPPED when server does not evidence logging support (spec requires capability only for servers that emit notifications/message) - Use untestableCheck() for zero-notification case per repo convention (issue modelcontextprotocol#248) instead of SKIPPED - Change below-threshold violation from FAILURE to WARNING since there is no normative backing
When capabilities.logging is absent, open a probe connection and call test_tool_with_logging to check whether the server actually emits log notifications. If notifications are observed without capability advertised, report WARNING (MUST violation). If none observed, SKIPPED is correct.
Summary
Adds three new conformance scenarios for the
logging/setLevelendpoint (issue #435):logging-capability-advertisement— verifies the server advertisescapabilities.loggingin its initialize responselogging-invalid-level— verifies the server rejects unrecognized severity levels (e.g. "banana") with-32602(Invalid Params)logging-threshold-filtering— verifies thatnotifications/messagerespects the configured severity threshold (suppresses below, allows above)Scenarios 2 and 3 are registered in
pendingClientScenariosListbecause the everything-server does not implement invalid-level rejection or threshold-based filtering. They can be run targeted against a compliant fixture server vianpm start -- server --scenario logging-invalid-level --url <fixture>.Scenario 1 is in the active list and passes against the everything-server.
Spec references
Test plan
npm run buildpasseslogging-capability-advertisementpasses against everything-serverlogging-invalid-levelcorrectly skipped (pending) in default suitelogging-threshold-filteringcorrectly skipped (pending) in default suiteCloses #435