Skip to content

Use hashlib.pbkdf2_hmac for SCRAM salted password - #1359

Open
twelfthlabor wants to merge 2 commits into
MagicStack:masterfrom
twelfthlabor:fix/scram-pbkdf2
Open

twelfthlabor wants to merge 2 commits into
MagicStack:masterfrom
twelfthlabor:fix/scram-pbkdf2

Conversation

@twelfthlabor

@twelfthlabor twelfthlabor commented Sep 13, 2026

Copy link
Copy Markdown

SCRAMAuthentication._generate_salted_password() reimplements the RFC 5802
Hi() function as a Python-level PBKDF2 loop: one hmac.new() and a
Python-level XOR over zip() for each iteration. PostgreSQL's default
iteration count is 4096, so that loop runs 4095 times before every
password-authenticated connection can finish its handshake.

hashlib.pbkdf2_hmac("sha256", ...) computes exactly the same value in C.
Hi() is PBKDF2-HMAC-SHA256 with a single output block, so dkLen equals the
digest size (32 bytes) and the result is byte-identical to the loop.

The old code's first HMAC used self.DIGEST, and every iteration after that
hardcoded hashlib.sha256. Since SCRAM-SHA-256 is the only mechanism this
class handles, naming the algorithm explicitly removes no real digest
agility. _bytes_xor stays, because _generate_client_proof still uses it.

Fixes #1357.

Verification

The old loop and hashlib.pbkdf2_hmac were compared in a standalone battery
of 324 cases that reproduces the loop verbatim: published
PBKDF2-HMAC-SHA256 test vectors, iteration counts 1, 2, 3, 4096 and 10000,
empty, short, long and unicode passwords and salts, and 310 random
combinations. Every result is 32 bytes and byte-for-byte equal.

A build of the patch was then connected with a password to a local
PostgreSQL 16 server with password_encryption = scram-sha-256 and
scram-sha-256 in pg_hba.conf. Handshakes succeeded with stored
iteration counts 2, 4096 and 10000, a wrong password still failed with
InvalidPasswordError, and the server log recorded method=scram-sha-256.
Instrumenting hashlib.pbkdf2_hmac showed the compiled extension calling it
with ('sha256', ..., 4096) during a live handshake, while the same check on
the pre-fix build recorded no calls. The compiled path was exercised
end-to-end but was not separately timed.

A standalone microbenchmark of the old loop against the stdlib primitive at
4096 iterations, best of 7 runs, measured 12.9 ms against 0.85 ms in one run
(15.3x) and 18.8 ms against 1.08 ms in another (17.5x) on this machine (Apple
silicon, CPython 3.11).

Tests

tests/test_scram.py is new and runs without a server: RFC 7914 and RFC 7677
vectors, byte-equality against the old Hi() loop (including a unicode
password), and a spy asserting the compiled path calls
hashlib.pbkdf2_hmac("sha256", password, salt, iterations). All 4 pass on the fix. The pbkdf2-call test fails against the pre-fix code
(no call is recorded), so it pins the change. The server-free subset
(test_scram, test_record, test__sourcecode) is 31 tests OK, 2 skipped.

To make the compiled method reachable from a test without a server,
_generate_salted_password changes from cdef to cpdef in
scram.pyx/scram.pxd; internal call sites keep the C fast path. If you
would rather keep the method fully private, say so and I will move the test
behind a fake-server harness or drop it; the performance fix does not depend
on the visibility change.

tests.test_connect passes on both the pre-fix and post-fix build (51 tests,
2 environment skips), including test_auth_password_scram_sha_256 against a
managed cluster. tests.test__sourcecode and tests.test_test pass. The
full suite needs a PostgreSQL instance, so only the connect and auth subset
was run locally. CI runs the suite against PostgreSQL 9.5 through 18, and
test_auth_password_scram_sha_256 connects with a password on 10 and above,
so a wrong digest, dkLen or iteration handling fails in CI.

One edge case changes behavior. PostgreSQL never sends an iteration count of
zero and RFC 5802 does not allow it, but if a nonconforming server did, the
old loop quietly computed a one-iteration value while pbkdf2_hmac raises
ValueError and the connection fails. Validating i >= 1 in
parse_server_first_message would move that error into the auth layer; left
as a follow-up here.

_generate_salted_password becomes cpdef so tests can reach the compiled
method; internal call sites keep the C fast path.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SCRAM: _generate_salted_password reimplements PBKDF2 in Python; hashlib.pbkdf2_hmac is ~30x faster and bit-identical

1 participant