Python: cache exposed shared SSA adjacency relations - #22421
Draft
yoff wants to merge 2 commits into
Draft
Conversation
Exercise the three public AdjacentUses relations and compare them with their internal projection or recursive expansion contracts. The cache-placement defect changes evaluator specialization and work rather than semantic results, so a semantic contract snapshot is the stable red-state equivalent; wall-time assertions would be machine-dependent and flaky. This intentionally records no MISSING or SPURIOUS rows: the expected invariant is exact relation equality before and after cache placement changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 21ab8585-861f-42c9-a834-451604646c6b
The shared SSA module requires language adapters to cache predicates that they expose. The Python adapter exposed firstUse, adjacentUseUse, and useOfDef without restoring that cache boundary, unlike the legacy AdjacentUses implementation. On exact historical Salt, the missing boundary caused the same 6,313,793-row liveAtExit fixed point to be evaluated twice. The equivalent plans received distinct RA hashes (c6bc8xgji0uv6seurbhesjqd315 versus fabf1xs3jb6t67a2buq2iv8iof4 for unsafe deserialization, and c6bc8xgji0uv6seurbhesjqd315 versus 8270excv27ldlfrtk19ou81d206 for modification-of-default-value) because one inherited an unrelated cached-empty sentinel while the other used a literal empty base. Cache the three Python adapter relations rather than generic liveness. This restores the documented shared-SSA contract at the narrow language boundary and avoids imposing a 6.31M-row generic cache on every language instantiation. On current head 1a8e317 with exact saltstack/salt@d036b117, three matched prewarmed -j1 repeats reduced median evaluator time from 51.294s to 45.103s for unsafe deserialization and from 42.377s to 34.238s for modification-of-default-value. Median paired reductions were 6.428s and 8.247s. Joined tuples fell by 58,255,670 and 85,664,313; recursive pipeline runs fell by 1,999 and 3,015. Both queries retained the identical empty endpoint hash 2a514e093aae140a14f6bf77beebe1ad in every repeat. Historical exact controls also retained 483,922 definitions, 169,921 phi inputs, 390,548 first uses, 475,226 adjacent uses, and 123,231 semantic call edges with zero left-only or right-only rows. Historical Salt evaluator recovery was 16.5% and 19.9%. Cold prewarm evaluator time was neutral (106.609s to 106.620s), so this is a warm-query optimization rather than a claimed cold-cache speedup. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 21ab8585-861f-42c9-a834-451604646c6b
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The shared SSA implementation documents that language adapters must cache predicates they expose. Python's shared-SSA adapter exposed
AdjacentUses::firstUse,adjacentUseUse, anduseOfDefwithout restoring that cache boundary, unlike the legacy ESSA implementation.This two-commit dependent draft:
cachedto the PythonAdjacentUsesmodule and its three public predicates.The diagnostic intentionally has no
MISSING/SPURIOUSannotation and no wall-time assertion. The defect is duplicate evaluator specialization with identical semantic output, so relation equality is the stable red-state equivalent and timing would be machine-dependent.Why the adapter cache
On exact historical Salt, the same 6,313,793-row
liveAtExitfixed point was evaluated twice per endpoint query. Equivalent recursive plans received distinct RA hashes because one inherited an unrelated cached-empty sentinel while the other used a literal empty base.Caching at the Python adapter boundary restores the documented shared-SSA contract and is narrower than caching the 6.31M-row generic liveness relation for every language instantiation.
Current-head measurement
Base: exact #21925 head
1a8e317b4a328bea1059453a2ab3ba6eada09e3f.Database: exact
saltstack/salt@d036b1177efeec175164571e9cc07b52cddf7844.Mode: three matched, prewarmed, serial
-j1repeats.Every repeat retained zero endpoint rows and BQRS hash
2a514e093aae140a14f6bf77beebe1ad. Peak dependent relation size was unchanged at 15,216,060 tuples.Historical exact relation controls retained 483,922 definitions, 169,921 phi inputs, 390,548 first uses, 475,226 adjacent uses, and 123,231 semantic call edges, with zero left-only or right-only rows. Historical Salt recovery was 16.5% and 19.9%. Cold prewarm evaluator time was neutral (106.609s -> 106.620s), so this is a warm-query optimization rather than a claimed cold-cache speedup.
Tests
codeql query format --check-onlyfor the changed QL filespython/ql/test/library-tests/dataflow-new-ssa/AdjacentUsesContract.qlpython/ql/test/library-tests/dataflow-new-ssa/SsaTest.qlThe broader requested Airflow and Python test matrix was not started after the coordinator requested the final current-head result without additional experiments; this remains a draft for that reason.
Stack
This PR is based directly on
yoff/python-shared-cfg-dataflow-flip(#21925 head). It neither includes nor depends on sibling drafts #22116 (legacy CFG/ESSA unpin) or #22407 (capturedJumpStepbinding rewrite).No DCA was launched.