SCRAM: _generate_salted_password reimplements PBKDF2 in Python; hashlib.pbkdf2_hmac is ~30x faster and bit-identical

[danielaholl]

Summary

SCRAMAuthentication._generate_salted_password() implements PBKDF2-HMAC-SHA256 as a Python-level loop. hashlib.pbkdf2_hmac('sha256', ...) computes the identical value in C and is ~27–31× faster. Since this runs on the event loop on every new connection, it shows up as a measurable stall for workloads that open connections often (pool overflow, bursty traffic, short-lived tasks).

Where

asyncpg/protocol/scram.pyx (current master):

ui = hmac.new(p, s + b'\x00\x00\x00\x01', self.DIGEST)
u = ui.digest()
for x in range(iterations - 1):
    ui = hmac.new(p, ui.digest(), hashlib.sha256)
    u = self._bytes_xor(u, ui.digest())
return u

That is exactly the "Hi" function from RFC 5802, i.e. PBKDF2-HMAC-SHA256 with dkLen = hLen — which the stdlib already provides.

_bytes_xor is a Python generator over zip(), so each of the 4095 iterations allocates two digests and XORs 32 bytes one byte at a time.

Measurements

PostgreSQL's default scram_iterations is 4096.

Environment	Python loop	hashlib.pbkdf2_hmac	Factor
Debian container, x86_64, CPython 3.11	28.6 ms	0.915 ms	31×
macOS 26, arm64, CPython 3.12	9.28 ms	0.337 ms	27×

Output is bit-identical:

import hashlib, hmac, time

def asyncpg_way(p, s, it):
    ui = hmac.new(p, s + b'\x00\x00\x00\x01', hashlib.sha256)
    u = ui.digest()
    for _ in range(it - 1):
        ui = hmac.new(p, ui.digest(), hashlib.sha256)
        u = bytes(x ^ y for x, y in zip(u, ui.digest()))
    return u

p, s, iters = b"correct horse battery staple", b"0123456789abcdef", 4096
assert asyncpg_way(p, s, iters) == hashlib.pbkdf2_hmac("sha256", p, s, iters)

for name, fn in (("loop", asyncpg_way),
                 ("pbkdf2_hmac", lambda p, s, i: hashlib.pbkdf2_hmac("sha256", p, s, i))):
    fn(p, s, iters)
    t = time.perf_counter(); n = 0
    while time.perf_counter() - t < 2.0:
        fn(p, s, iters); n += 1
    print(f"{name:12} {(time.perf_counter()-t)/n*1000:7.3f} ms")

Why it matters in practice

We hit this while profiling event-loop stalls in a FastAPI/SQLAlchemy service with password_encryption = scram-sha-256. py-spy attributed ~16 % of total event-loop CPU to hmac.py with no application frame above it — the caller is invisible because scram.pyx is Cython-compiled, so only the Python hmac.new frames show up. It took a while to identify.

The trigger was connection churn: our SQLAlchemy pool was discarding overflow connections, so ~255 connections/minute were being established, each paying ~28.6 ms of PBKDF2 synchronously on the event loop — roughly 7 seconds of blocked loop per minute.

Fixing the churn on our side was the main remedy, and I'm not suggesting asyncpg is responsible for that. But a connection establishment costing 28.6 ms of CPU rather than 0.9 ms makes any such situation ~30× worse than it needs to be, and it's on the loop.

Suggested change

cdef _generate_salted_password(self, str password, bytes salt, int iterations):
    """This follows the "Hi" algorithm specified in RFC5802"""
    return hashlib.pbkdf2_hmac(
        'sha256', password.encode('utf8'), base64.b64decode(salt), iterations
    )

Note the loop already hardcodes hashlib.sha256 (while the first hmac.new uses self.DIGEST), so the function is SHA-256-only as it stands — no digest agility is lost by naming 'sha256' explicitly. If DIGEST should ever become configurable, pbkdf2_hmac takes the algorithm name as its first argument, so the change doesn't stand in the way.

_bytes_xor would become unused unless it's referenced elsewhere.

Happy to open a PR if the direction looks right.

Related

#378 (Support using pre-hashed passwords) would sidestep the derivation entirely, which is a broader change; this one is a drop-in replacement with identical output.