Fix: generate_surrogate_key renders valid hex strings on MySQL, T-SQL, StarRocks and Snowflake - #6033
Open
ptimizeroracle wants to merge 1 commit into
Open
Conversation
…l, tsql, starrocks, snowflake Signed-off-by: ptimizeroracle <contact@binblok.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
@GENERATE_SURROGATE_KEYpromises a hex string on every dialect, but on fouradapter dialects the rendered SQL is invalid or returns binary:
MySQL:
hash_function := 'SHA256'/'SHA512'rendered as bareSHA256(...)/SHA512(...). MySQL only hasSHA2(expr, digest_length),so the query fails at runtime:
(reproduced on MySQL 8.4.11). The cause: MySQL's sqlglot parser has no
entry for
SHA256, soexp.funcreturnsAnonymousand the existingSHA2Digest/MD5Digest conversions never fire.
T-SQL (MSSQL, Fabric): every hash renders as
HASHBYTES(...), whichreturns
VARBINARY, not a hex string. This includes the defaultMD5path, so every plain
@GENERATE_SURROGATE_KEY(a)on these adaptersproduced binary keys. Reproduced on Azure SQL Edge:
HASHBYTES('MD5','x')->0x9DD4E461...while DuckDB returns9dd4e461....StarRocks, Snowflake: same
Anonymousfall-through as MySQL; bareSHA256(...)is not a valid function on either engine (both spell itSHA2(expr, length), length optional on Snowflake).Fix, in
generate_surrogate_key:Anonymous->exp.SHA2(length)mapping is no longer scoped to thePresto family: any dialect whose parser hands back
Anonymousfor aknown SHA-2 name now gets the typed call with the canonical digest
length. Unknown hash names still pass through untouched (
MYHASHcasecovered by the existing test).
TO_HEXwrap:LOWER(CONVERT(VARCHAR(n), HASHBYTES(...), 2)), withnsized to the digest width. Like the Presto branch, a probe keeps it inert
if the sqlglot tsql generator ever emits the conversion itself.
End-to-end verification on live engines, fixed rendering executed and
compared against DuckDB / hashlib oracles (input
'x'):9dd4e461268c8034f5c8564e155c67a611f6ad8ec52a2984abaafd7c3b516503785c20722d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881a4abd4448c49562d...afe62(128 hex)The existing
test_generate_surrogate_key_hash_semanticssnowflake assertionwas pinning the bare
SHA256(...)form; since that is not a Snowflakefunction, it is updated to the mapped
SHA2(..., 256)form (Snowflakedefaults the digest size to 256).
Same invariant family as #5888 (Presto/Trino) and #5871 (Athena).
Test Plan
test_generate_surrogate_key_hex_string_on_mysql_tsql_starrockspinsexact rendered SQL for mysql SHA256/SHA512, starrocks SHA256, tsql
MD5/SHA1/SHA256 and fabric SHA512; fails on main, passes with the fix.
tests/core/test_macros.py(143 tests) and the macros doctests pass.containers, scripts in the PR discussion).
ruff check,ruff format, andmypy(pinned 1.13) clean on the changedfiles.
Checklist
make styleon the code (ruff + format + mypy equivalents on changed files)pytest tests/core/test_macros.pypasses (143 passed)