async auth and yield from

[AstraLuma]

Ok, so, I'm using httpx in swiss-army-upload. Part of this is that I (will) have multiple httpx.Auth implementations for various backend systems. With such complexity, there's some common flows I'd like to break out, eg the stuff needed to get a real OIDC JWT token when running inside of GitHub Actions.

Which is fine, you can just yield from and it's pretty easy. Cool. Hardly a trifle.

But then I needed to start doing non-HTTP things inside the auth, namely talking to credentials storage (keyring) without blocking the async loop. Cool, fine, wrap it up in async wrappers, switch from auth_flow to async_auth_flow, easy enough.

Except that you can't use yield from inside of async functions.

Here's what I can think of:

1. Manually pump the generator (.send(), StopIteration, etc)
2. Make a class that lets me manually pump it slightly less painfully (eg, a for loop and .send())
3. Strip out all the async work and do them separately up front (which looses some refresh capabilities if the object is long lived, which I expect it is)
4. Run a refresh task and poll for changes on a schedule (which will probably mostly just waste work)

Is there a better way of handling this?

[AstraLuma]

I guess I could subclass httpx.Auth that adds a .delegate() which causes it to pump that until completion. Or something. Which is really half-way to an async loop.

[lux-liang]

For the linked TeahouseAuth, I would keep the nested flows as ordinary generators and put the async boundary in one shared driver. This is essentially your option 1, but implemented once for all backends. It preserves both yield from github_oidc() and the token returned by that helper, while allowing credential reads to happen on demand after a 403.

An async for forwarding loop is insufficient here: HTTPX sends each response back through asend(), and that response must reach the suspended response = yield request expression. Also, an async generator cannot return a value the way your current github_oidc() helper does. See PEP 525 and HTTPX's own adapter.

One application-level approach is to let your ordinary generators yield either a request or an explicit blocking operation. The shared adapter consumes the operations internally and yields only requests to HTTPX:

from collections.abc import Callable
from dataclasses import dataclass
from typing import Any

import anyio
import httpx


@dataclass(frozen=True)
class BlockingCall:
    fn: Callable[[], Any]


class ComposedAuth(httpx.Auth):
    def steps(self, request):
        # Override with a generator yielding Request or BlockingCall.
        yield request

    def sync_auth_flow(self, request):
        raise RuntimeError("This implementation requires AsyncClient")

    async def async_auth_flow(self, request):
        if self.requires_request_body:
            await request.aread()
        flow = self.steps(request)
        try:
            step = next(flow)
            while True:
                if isinstance(step, BlockingCall):
                    try:
                        value = await anyio.to_thread.run_sync(step.fn)
                    except Exception as exc:
                        step = flow.throw(exc)
                        continue
                elif isinstance(step, httpx.Request):
                    value = yield step
                    if self.requires_response_body:
                        await value.aread()
                else:
                    raise TypeError(f"Unexpected auth step: {type(step)!r}")
                step = flow.send(value)
        except StopIteration:
            return
        finally:
            flow.close()

Your backend then inherits ComposedAuth and renames its current auth_flow() to steps(). The existing yield from calls can remain. At the point where it actually needs a password, it can do:

from functools import partial
import keyring

# Inside steps(), or a regular helper delegated to with yield from:
password = yield BlockingCall(
    partial(keyring.get_password, "your-service", username)
)

BlockingCall and steps() are your application protocol, not HTTPX APIs. Keep blocking I/O inside those calls; the generator logic still runs on the event-loop thread. The worker-call exception is thrown back into the generator, so a helper can handle a missing credential, and closing the outer flow closes delegated generators too. Use a keyring backend that supports worker-thread access.

Keep requires_response_body = True from your current class if helpers parse response JSON. Overriding async_auth_flow() means the adapter must do that reading itself; the inherited flag alone does not perform it. HTTPX documents this override point, and AnyIO documents the worker-thread behavior.

Prepared with Codex and tested locally with Python 3.12 / HTTPX 0.28.1 using MockTransport: a 403 → credential lookup → streamed token response → retry succeeds twice on the same auth instance; the lookup runs off the event-loop thread; a helper catches a credential error; nested finalizers run on connection failure/cancellation; and a synchronous client is rejected. These checks use a fake credential store, not a live keyring or OIDC provider.