Reclaim connections orphaned by a cancelled request#1099
Open
wwmanidumaneesha wants to merge 1 commit into
Open
Reclaim connections orphaned by a cancelled request#1099wwmanidumaneesha wants to merge 1 commit into
wwmanidumaneesha wants to merge 1 commit into
Conversation
A request that is cancelled after the pool has assigned it a newly created connection, but before that connection has been established, leaves the connection behind in the pool. Such a connection cannot be reaped, since `is_closed()`, `has_expired()` and `is_idle()` all report False while it has yet to connect, and it cannot be reused, since `is_available()` reports False for HTTP/1.1. It therefore holds a pool slot that is never reclaimed. Once enough have accumulated the pool is permanently full, and every subsequent request fails with `PoolTimeout` against a perfectly healthy server, until the process is restarted. This restores the invariant the pool already relies on elsewhere: a connection that is neither available nor idle is reserved for the single request it was assigned to. Connections no longer referenced by any request, and which can no longer service one, are now closed and removed. Refs encode#1093, and the earlier reports in encode#658 and encode#830.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reclaim connections orphaned by a cancelled request
Closes #1093. Also covers the earlier reports in #658 and #830, which were closed but describe the same failure.
The problem
A request cancelled after the pool has assigned it a newly created connection, but before that connection has been established, leaves the connection behind in the pool.
That connection is then stuck:
self._connection is Noneand_connect_failedstillFalse,is_closed(),has_expired()andis_idle()all returnFalse, so no branch of the cleanup loop matches it.is_available()returnsFalsefor HTTP/1.1 while connecting.So it holds a pool slot that is never reclaimed. Once
max_connectionsof them accumulate, the pool is permanently full and every subsequent request fails withPoolTimeoutagainst a perfectly healthy server. Only recreating the pool recovers.The pool's own repr shows the inconsistency clearly — no requests, yet a connection still occupying the only slot:
Scope
Worth noting for reviewers: this needs cancellation delivered while the task is suspended in
wait_for_connection(), which is whatasyncio.Task.cancel()andasyncio.wait_for()do.Scope-based cancellation (
anyio.move_on_after, trio) is not affected — it is delivered at the next checkpoint, which lands insideAsyncHTTPConnection.handle_async_request(), where the existingexcept BaseExceptionsets_connect_failed = Trueand the connection is correctly reaped. I probed that path across several pool sizes and timings and saw no leak.The fix
Rather than patching the single cancellation path, this restores an invariant the pool already relies on. As the comment on
AsyncHTTP11Connection.is_available()puts it, a connection in this state "will not be acquired from the connection pool for any other request" — it belongs to exactly one request.So: a connection that is neither available nor idle is reserved for the one request it was assigned to. If no request references it any more, nothing can ever drive it forward, and it is closed and removed. This is self-healing regardless of how the connection came to be orphaned.
Tests
test_connection_pool_does_not_leak_slot_on_cancelled_requestasserts both that no connection is left behind and that the pool still works afterwards.Verified to fail on
masterand pass with the fix, 8 runs each way.