Skip to content

Reclaim connections orphaned by a cancelled request#1099

Open
wwmanidumaneesha wants to merge 1 commit into
encode:masterfrom
wwmanidumaneesha:fix/reclaim-orphaned-pool-connections
Open

Reclaim connections orphaned by a cancelled request#1099
wwmanidumaneesha wants to merge 1 commit into
encode:masterfrom
wwmanidumaneesha:fix/reclaim-orphaned-pool-connections

Conversation

@wwmanidumaneesha

Copy link
Copy Markdown

Reclaim connections orphaned by a cancelled request

Closes #1093. Also covers the earlier reports in #658 and #830, which were closed but describe the same failure.

The problem

A request cancelled after the pool has assigned it a newly created connection, but before that connection has been established, leaves the connection behind in the pool.

That connection is then stuck:

  • It cannot be reaped. With self._connection is None and _connect_failed still False, is_closed(), has_expired() and is_idle() all return False, so no branch of the cleanup loop matches it.
  • It cannot be reused. is_available() returns False for HTTP/1.1 while connecting.

So it holds a pool slot that is never reclaimed. Once max_connections of them accumulate, the pool is permanently full and every subsequent request fails with PoolTimeout against a perfectly healthy server. Only recreating the pool recovers.

The pool's own repr shows the inconsistency clearly — no requests, yet a connection still occupying the only slot:

<AsyncConnectionPool [Requests: 0 active, 0 queued | Connections: 1 active, 0 idle]>
pool.connections == [<AsyncHTTPConnection [CONNECTING]>]

Scope

Worth noting for reviewers: this needs cancellation delivered while the task is suspended in wait_for_connection(), which is what asyncio.Task.cancel() and asyncio.wait_for() do.

Scope-based cancellation (anyio.move_on_after, trio) is not affected — it is delivered at the next checkpoint, which lands inside AsyncHTTPConnection.handle_async_request(), where the existing except BaseException sets _connect_failed = True and the connection is correctly reaped. I probed that path across several pool sizes and timings and saw no leak.

The fix

Rather than patching the single cancellation path, this restores an invariant the pool already relies on. As the comment on AsyncHTTP11Connection.is_available() puts it, a connection in this state "will not be acquired from the connection pool for any other request" — it belongs to exactly one request.

So: a connection that is neither available nor idle is reserved for the one request it was assigned to. If no request references it any more, nothing can ever drive it forward, and it is closed and removed. This is self-healing regardless of how the connection came to be orphaned.

Tests

test_connection_pool_does_not_leak_slot_on_cancelled_request asserts both that no connection is left behind and that the pool still works afterwards.

Verified to fail on master and pass with the fix, 8 runs each way.

A request that is cancelled after the pool has assigned it a newly created
connection, but before that connection has been established, leaves the
connection behind in the pool.

Such a connection cannot be reaped, since `is_closed()`, `has_expired()`
and `is_idle()` all report False while it has yet to connect, and it cannot
be reused, since `is_available()` reports False for HTTP/1.1. It therefore
holds a pool slot that is never reclaimed. Once enough have accumulated the
pool is permanently full, and every subsequent request fails with
`PoolTimeout` against a perfectly healthy server, until the process is
restarted.

This restores the invariant the pool already relies on elsewhere: a
connection that is neither available nor idle is reserved for the single
request it was assigned to. Connections no longer referenced by any request,
and which can no longer service one, are now closed and removed.

Refs encode#1093, and the earlier reports in encode#658 and encode#830.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cancelling requests under pool contention permanently leaks connection slots (AsyncConnectionPool → PoolTimeout)

1 participant