Pass SSL Context to HTTP Connection

[glowww]

It seems with this version, the SSL context stopped being passed to the HTTP connection. Raising OSError: [Errno 24] Too many open files when sending over 500 async requests simultaneously.

[lovelydinosaur]

Okay, so there's a big jump here from "It seems with this version, the SSL context stopped being passed to the HTTP connection." ... to ... "Raising OSError: [Errno 24] Too many open files when sending over 500 async requests simultaneously."

Can you walk me through how you're getting an error, what the full traceback is, and what makes you think that this code change will resolve it?

This doesn't look correct to me, as it's passing the ssl context to the proxy connection, when actually it needs to be used when establishing the SSL tunnel. (So the ssl context is being between the client and the end server, rather than between the client and the proxy.)

[glowww]

If you go to the class HTTPConnection or AsyncHTTPConnection, you can see that they create a new SSL context if one is not provided. When you send over 500 simultaneous requests, it crashes, because it creates too many SSL contexts, even if you are already providing one to the client. You can try it yourself running this code:

import asyncio
import traceback
import httpx

PROXY = 'YOUR_PROXY_HERE'


async def send(ssl_context):
    print("Sending request")
    try:
        async with httpx.AsyncClient(verify=ssl_context, proxies=PROXY) as session:
            response = await session.get(url="https://api64.ipify.org/?format=json")
            print(response)
    except:
        traceback.print_exc()


async def main():
    ssl_context = httpx.create_ssl_context()
    await asyncio.gather(*[send(ssl_context) for _ in range(1000)], return_exceptions=True)

asyncio.run(main())

You will get this error:

Traceback (most recent call last):
  File "C:/Users/glowww/Desktop/bot/debug.py", line 10, in send
    response = await session.get(url="https://api64.ipify.org/?format=json")
  File "C:\Users\glowww\Desktop\bot\venv\lib\site-packages\httpx\_client.py", line 1736, in get
    return await self.request(
  File "C:\Users\glowww\Desktop\bot\venv\lib\site-packages\httpx\_client.py", line 1513, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "C:\Users\glowww\Desktop\bot\venv\lib\site-packages\httpx\_client.py", line 1600, in send
    response = await self._send_handling_auth(
  File "C:\Users\glowww\Desktop\bot\venv\lib\site-packages\httpx\_client.py", line 1628, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "C:\Users\glowww\Desktop\bot\venv\lib\site-packages\httpx\_client.py", line 1665, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "C:\Users\glowww\Desktop\bot\venv\lib\site-packages\httpx\_client.py", line 1702, in _send_single_request
    response = await transport.handle_async_request(request)
  File "C:\Users\glowww\Desktop\bot\venv\lib\site-packages\httpx\_transports\default.py", line 291, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "C:\Users\glowww\Desktop\bot\venv\lib\site-packages\httpcore\_async\connection_pool.py", line 225, in handle_async_request
    await self._attempt_to_acquire_connection(status)
  File "C:\Users\glowww\Desktop\bot\venv\lib\site-packages\httpcore\_async\connection_pool.py", line 180, in _attempt_to_acquire_connection
    connection = self.create_connection(origin)
  File "C:\Users\glowww\Desktop\bot\venv\lib\site-packages\httpcore\_async\http_proxy.py", line 107, in create_connection
    return AsyncTunnelHTTPConnection(
  File "C:\Users\glowww\Desktop\bot\venv\lib\site-packages\httpcore\_async\http_proxy.py", line 185, in __init__
    self._connection: AsyncConnectionInterface = AsyncHTTPConnection(
  File "C:\Users\glowww\Desktop\bot\venv\lib\site-packages\httpcore\_async\connection.py", line 38, in __init__
    ssl_context = default_ssl_context() if ssl_context is None else ssl_context
  File "C:\Users\glowww\Desktop\bot\venv\lib\site-packages\httpcore\_ssl.py", line 7, in default_ssl_context
    context = ssl.create_default_context()
  File "C:\Users\glowww\AppData\Local\Programs\Python\Python38\lib\ssl.py", line 755, in create_default_context
    context.keylog_filename = keylogfile
OSError: [Errno 24] Too many open files: 'C:\\Users\\glowww\\SSLKeys\\sslkeylog.log'

With my changes the issue disappears.

[lovelydinosaur]

Okay - I've taken an alternative tack on this.

The issue here is really that we're currently creating a bunch of SSLContext instances when we really don't need to.

Passing an ssl_context argument to the proxy connection does also suppress this, but it's misleading, because it's not actually used.

So instead I've changed things around so we only create instances at the point they're used.