Closing a httpx client results in a "RuntimeError: Event loop is closed"

[BabakAmini]

I initially raised this issue on Stack Overflow, but unfortunately, I haven't received a solution. Therefore, I'm bringing it up here again.

I need to maintain a persistent httpx client in my code to utilize its connection pool throughout the lifespan of my application. Below is a simplified version of my implementation:

import asyncio
import weakref
from threading import Lock

import nest_asyncio
from httpx import AsyncClient


class Request:
    _instance = None
    _lock = Lock()
    _loop: asyncio.AbstractEventLoop = None
    _new_loop = False
    _session = None

    def __new__(cls):
        if not cls._instance:
            with cls._lock:
                if not cls._instance:
                    instance = super().__new__(cls)
                    cls._instance = instance
                    weakref.finalize(instance, instance._close)

                    if cls._loop is None:
                        try:
                            cls._loop = asyncio.get_running_loop()
                        except RuntimeError:
                            cls._loop = asyncio.new_event_loop()
                            cls._new_loop = True
                            asyncio.set_event_loop(cls._loop)

                if cls._new_loop:
                    # nest_asyncio.apply()
                    cls._loop.run_until_complete(cls.create_client())
                else:
                    cls._loop.create_task(cls.create_client())

        return cls._instance

    @classmethod
    async def create_client(cls):
        cls._session = AsyncClient()

    @classmethod
    def _close(cls):
        cls._loop.run_until_complete(cls.close())

        if cls._new_loop:
            cls._loop.close()

    @classmethod
    async def close(cls):
        if cls._session:
            await cls._session.aclose()

    async def get(self, url):
        try:
            return await self._session.get(url)

        except Exception:
            return None


req = Request()


async def main():
    result = await req.get('https://www.google.com')
    if result:
        print(result.text[:100])


asyncio.run(main())

This results in a RuntimeError: Event loop is closed:

...
    await self._pool.aclose()
  File "/home/user/.cache/pypoetry/virtualenvs/app-gEFTwlce-py3.12/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 313, in aclose
    await self._close_connections(closing_connections)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gEFTwlce-py3.12/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 305, in _close_connections
    await connection.aclose()
  File "/home/user/.cache/pypoetry/virtualenvs/app-gEFTwlce-py3.12/lib/python3.12/site-packages/httpcore/_async/connection.py", line 171, in aclose
    await self._connection.aclose()
  File "/home/user/.cache/pypoetry/virtualenvs/app-gEFTwlce-py3.12/lib/python3.12/site-packages/httpcore/_async/http11.py", line 265, in aclose
    await self._network_stream.aclose()
  File "/home/user/.cache/pypoetry/virtualenvs/app-gEFTwlce-py3.12/lib/python3.12/site-packages/httpcore/_backends/anyio.py", line 55, in aclose
    await self._stream.aclose()
  File "/home/user/.cache/pypoetry/virtualenvs/app-gEFTwlce-py3.12/lib/python3.12/site-packages/anyio/streams/tls.py", line 201, in aclose
    await self.transport_stream.aclose()
  File "/home/user/.cache/pypoetry/virtualenvs/app-gEFTwlce-py3.12/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 1287, in aclose
    self._transport.close()
  File "/nix/store/gmx7bwrwy6s0kk89ij5yj8r8ayai95x1-python3-3.12.5/lib/python3.12/asyncio/selector_events.py", line 1210, in close
    super().close()
  File "/nix/store/gmx7bwrwy6s0kk89ij5yj8r8ayai95x1-python3-3.12.5/lib/python3.12/asyncio/selector_events.py", line 875, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "/nix/store/gmx7bwrwy6s0kk89ij5yj8r8ayai95x1-python3-3.12.5/lib/python3.12/asyncio/base_events.py", line 795, in call_soon
    self._check_closed()
  File "/nix/store/gmx7bwrwy6s0kk89ij5yj8r8ayai95x1-python3-3.12.5/lib/python3.12/asyncio/base_events.py", line 541, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

The only way to resolve this issue is to uncomment the line containing nest_asyncio.apply(). However, since the nest_asyncio package is heavily based on asyncio internal functions and is no longer maintained, I'm not interested in using it in my code.

My question is: Why is the loop closed when I call run_until_complete() (from within the _close() function), which creates a new event loop and immediately calls the AyncClient.aclose() function? How can I fix this?

For your reference, the same code works for aiohttp.ClientSession without the need to use nest_asyncio.

[widechaos]

There are two loops in this example. Request() creates/stores one before main(), then asyncio.run(main()) creates a different loop and closes it when main() returns. The pooled connections opened by get() belong to that second loop. Running aclose() on the stored first loop cannot make the connections' original loop usable again.

run_until_complete() runs the loop you call it on; it does not create a new loop. A weakref.finalize callback also cannot guarantee async cleanup happens before asyncio.run() shuts down.

Keep the shared client and all its users within one async application lifetime, and close it before leaving that lifetime:

import asyncio
import httpx

class Request:
    def __init__(self, client: httpx.AsyncClient):
        self.client = client

    async def get(self, url):
        return await self.client.get(url)

async def main():
    async with httpx.AsyncClient() as client:
        req = Request(client)
        # Pass this req/client to the components that need the shared pool.
        for _ in range(2):
            result = await req.get("https://www.google.com")
            print(result.status_code)

asyncio.run(main())

The client remains persistent for the whole main() scope, so this still allows connection pooling. In a framework, use its async startup/shutdown or lifespan mechanism to own that scope. Avoid creating a new event loop around each call to the same async client.

I checked the lifetime pattern on HTTPX 0.28.1 with a local plain-HTTP keep-alive server: two requests reused one connection and the client closed before loop exit. Using the client in one asyncio.run() and closing it in another reproduced Event loop is closed. I did not test the Google/TLS request above. HTTPX async client lifetime documentation.