build: consolidate toolchain helpers via symlink

[leofang]

Summary

Follow-up to #2903 addressing the second condition of its approval (review comment):

A follow-up PR will be sent (by me or Ralf J) to use symlink or other approaches to ensure we have a single source of truth for the build hook (and its tests).

Also closes the audit in #1882 — the two backends' flag sets had drifted for legacy reasons; this PR unifies them, with a per-project injection point for the small handful of genuinely-different bits.

The two build_hooks.py files now share a single canonical file (cuda_bindings/_build_shared.py) via a symlink at cuda_core/_build_shared.py, and each backend's own build_hooks.py is a thin shell that declares only what is genuinely per-package. All the mechanics — CUDA path resolution, toolchain resolution, flag set, stamp read/compare/write, Cython cache — live in one place.

What's in _build_shared.py

• CUDA path resolution: _import_get_cuda_path_or_home, _get_cuda_path (the PEP 517 pathfinder-shadowing workaround from Known issue: cuda.pathfinder cannot be imported in PEP 517 in-tree build backends without a workaround #1824).
• Toolchain: constants (_TOOLCHAINS_LINUX/_WINDOWS, _TOOLCHAIN_COMPILERS), _resolve_toolchain_name, _apply_toolchain_env, _check_toolchain_available (the CUDA_PYTHON_TOOLCHAIN block from build: add CUDA_PYTHON_TOOLCHAIN override for compiler/linker selection #2903), resolve_toolchain(*, cxx_std, debug=False, compile_for_coverage=False, warnings_as_errors=False, tweak=None) (framework), and _build_flags(name, cxx_std, debug, coverage, warnings_as_errors) (shared flag set).
• Stamps: _BUILD_DIR, _abi_stamp_path (uses EXT_SUFFIX so 3.10 / 3.11 / freethreaded builds don't stomp each other), check_build_key(stamp, get_key) and record_build_key(stamp, get_key) (mechanics), force_build_ext (module-level bool, mutated by check_build_key).
• Cython cache (from build: add opt-in Cython generated-source cache #2933): _cython_cache_path, _stable_cython_alias.

Path(__file__).parent inside the shared file resolves per-package because Python does not dereference symlinks in __file__, so _BUILD_DIR and the alias location land under each package's own directory when loaded from its symlink (verified empirically before committing).

What each build_hooks.py still owns

Genuinely per-package:

• cuda_bindings/build_hooks.py: _BUILD_TOOLCHAIN_STAMP = _abi_stamp_path(".build-toolchain"), _current_toolchain_key() (re-derives from env for setup.py's post-build stamping), _tweak_flags (adds -Wno-deprecated-declarations on Linux, see below), _rename_architecture_specific_files, _prep_extensions, _build_cuda_bindings.
• cuda_core/build_hooks.py: _BUILD_CONFIG_STAMP = _abi_stamp_path(".build-config"), _build_config_key(cuda_major, toolchain, debug, coverage), WARNINGS_AS_ERRORS (reads CUDA_PYTHON_WERROR; from cuda.core: eliminate build warnings; set option to build with warnings as errors #2966), _determine_cuda_major_version, _relativize_extension_sources, _extension_sources, _extension_depends, _build_cuda_core, _add_cython_include_paths_to_pth (from Cython .pth file support for pixi path dependencies #1562), plus get_requires_for_build_* that pins the cuda-bindings runtime dep.

Both files have a module-level __getattr__ that re-exports _build_shared.force_build_ext transparently, so setup.py's existing build_hooks.force_build_ext attribute read keeps working unchanged.

Per-project knobs on resolve_toolchain

Three knobs let each backend pick its own defaults without forking the shared code:

• cxx_std (required kwarg) — each backend picks its own C++ standard. No shared default; the two backends have picked deliberately different values (see below), and forcing them onto a common default would either regress bindings or block core.
• warnings_as_errors=False — opts into -Werror (Linux) or /WX with C4551/C4244 exemptions for Cython's utility code (MSVC). Each backend gates this on its own env var so the two backends' source cleanups stay on independent timelines.
• tweak (optional callable) — post-hook that gets (name, extra_compile_args, extra_link_args) and returns the layered pair. Used for the small handful of flags a single backend needs but that don't belong in the shared set.

Current usage:

• cuda-bindings: resolve_toolchain(cxx_std=14, ..., tweak=_tweak_flags). cxx_std=14 because raising to c++17 costs a measured ~15% on launch.launch_{256,512}_args from gcc's c++17 variadic-template expansion (the aggregate ~3% geomean is inside the pyperf --fast noise floor, but the outlier is real; measured on origin/main vs. this PR under identical CTK 13.4 / conda-forge toolchain). _tweak_flags adds -Wno-deprecated-declarations on Linux: cudaMemcpy*Array* and cudaGetDriverEntryPoint are deprecated but still supported, and the ~38 resulting warnings would otherwise break a future -Werror build (the flag is order-independent since it disables the warning class entirely, so it composes cleanly with warnings_as_errors=True when bindings eventually opts in). Bindings does not currently pass warnings_as_errors=True.
• cuda-core: resolve_toolchain(cxx_std=17, ..., warnings_as_errors=WARNINGS_AS_ERRORS). c++17 is required by helper code under cuda/core/_cpp/ (structured bindings, if constexpr, etc.). WARNINGS_AS_ERRORS reads CUDA_PYTHON_WERROR at module load; CI's wheel builds set it (see cuda.core: eliminate build warnings; set option to build with warnings as errors #2966). No tweak.

Shared flag set

Per #1882, bindings and core carried different Linux and MSVC flags for legacy reasons. Both now go through _build_shared._build_flags:

• Linux compile: -std=c++{cxx_std}, -g0 -O2 (opt) or -g -O0 -D _GLIBCXX_ASSERTIONS (debug)
• Linux link: -fuse-ld=lld (llvm), -Wl,--strip-all (opt)
• MSVC compile: /std:c++{cxx_std}, /O2 (modern setuptools' MSVCCompiler no longer forces /Ox, so we set the opt level explicitly for symmetry with Linux -O2)
• Warnings-as-errors (when warnings_as_errors=True): -Werror on Linux; /WX /wd4551 /wd4244 on MSVC. The exemptions cover the two warning classes Cython's utility code emits in every module (C4551 "function call missing argument list" and C4244 narrowing from @cython.overflowcheck(True) helpers).
• Coverage: -DCYTHON_TRACE_NOGIL=1 -DCYTHON_USE_SYS_MONITORING=0

Bindings loses -fpermissive, -fno-var-tracking-assignments, -O3 (and keeps -Wno-deprecated-declarations via its tweak). CI will surface any compilation regression if -fpermissive was actually load-bearing for the Cython-generated C++.

Windows contributors

The symlink now matters at clone time on Windows outside of WSL: without the right git config, the "symlink" lands as a text stub. Handled in two places:

• CI: test-wheel-windows.yml, test-sdist-windows.yml, and build-wheel.yml gain a git config --global core.symlinks true step before checkout on Windows runners.
• CONTRIBUTING.md: new top-level Development on Windows section documenting (1) Activate Developer Mode (matching CuPy's contribution guide), and (2) git config --global core.symlinks true. Includes an existing-clone recovery recipe, with a note that re-cloning is generally cleaner than trying to fix symlinks in place. The prior Pre-commit on Windows / Pre-commit lychee workaround subsections are folded in so all Windows-only setup lives in one place. cuda_bindings/docs/source/install.rst and cuda_core/docs/source/install.rst cross-link the new section from their Installing from Source notes.

Not touched: contribute.rst in either package (per @leofang, they're being refactored separately), and cuda_pathfinder/ (pure Python, no build_hooks.py).

What else changes

• toolshed/check_build_hooks_sync.py and its .pre-commit-config.yaml entry — deleted. A single source of truth needs no drift check.
• ruff.toml — one line: adds _build_shared to known-first-party so isort groups the import with cuda.* after third-party.
• MANIFEST.in in both packages — include _build_shared.py so sdists carry the resolved content (setuptools' make_release_tree copies through symlinks).
• lychee.toml (new, at repo root) — the four link-checker excludes previously inlined in .github/workflows/build-docs.yml move to a proper config file; the workflow now just points at --config ./lychee.toml. Includes the self-referencing #development-on-windows anchor exclusion needed because the section only lands with this PR.
• cuda_python_test_helpers/build_shared.py (new) — shared test mixins for resolve_toolchain, _check_toolchain_available, and _abi_stamp_path. Both packages' test_build_hooks.py mix them in via subclass, so each shared assertion runs once in each package's env without duplicating the test source. Flag-content assertions (-std=c++14 vs c++17, presence of -Wno-deprecated-declarations) stay in the per-package test files since they encode the per-project choice.
• Merge with main: absorbs #2966 (CUDA_PYTHON_WERROR=1 toggle for CI wheel builds). The auto-merge conflict lived in cuda_core/build_hooks.py — HEAD had deleted the whole in-file toolchain block, main had added a -Werror branch inside it. Resolved by hoisting the toggle into the shared flag set (see Per-project knobs above) so bindings can adopt it later with a one-line flip.
• Test pruning — dropped tests whose behavior any successful CI wheel build already exercises (default no-touch, sccache preservation, default preflight noop, missing-stamp force, test_wheel_records_exact_prepared_config_after_success, etc.). Kept error paths, cross-configuration transitions, cross-Python-ABI scoping, failure paths, parametrized data variants, and llvm/edge cases wheel builds don't cover. Bindings 34 → 31, core 71 → 63.

Test plan

• pytest cuda_bindings/tests/test_build_hooks.py --noconftest — 31 pass / 2 skip locally.
• pytest cuda_core/tests/test_build_hooks.py --noconftest — 63 pass / 2 skip locally.
• pytest-randomly run of both suites — no order-dependent failures (this PR fixes one that had been latent: monkey-patching build_hooks.force_build_ext created a real attribute that shadowed the __getattr__ fallback and poisoned later stamp tests; fixed by patching sys.modules["_build_shared"] instead).
• ruff check + ruff format --check clean on all touched Python files.
• toolshed/check_spdx.py clean.
• python -m build --sdist for both packages: the extracted tarball contains _build_shared.py byte-identical to the canonical, and a from-scratch PEP 517 backend load (fresh sys.path) resolves every shared helper's __module__ to _build_shared.
• cuda-bindings latency-benchmark suite on CTK 13.4 (baseline vs. this PR vs. this PR + -O3): confirmed -O3 → -O2 is not the source of the ~15% launch_{256,512}_args outlier; -std=c++14 → c++17 is. That's why bindings stays on cxx_std=14.
• Full CI (waiting on /ok to test) — the flag change (drop -fpermissive etc.) makes CI the authoritative check.

Marked draft until CI confirms; ready for review otherwise.

-- Leo's bot

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[leofang]

/ok to test 9828a2f

[leofang]

@juenglin could you check if this PR works on your system?

[leofang]

/ok to test 727f231

[github-actions]

Doc Preview CI
Preview removed because the pull request was closed or merged.

[Andy-Jost]

Requesting changes for the two inline comments on cuda_core/tests/test_build_hooks.py: the stamp tests are order-dependent, and the force=True path through __getattr__ has no test. The comments on the shared mixin and CONTRIBUTING are small follow-ups.

One note with no single line to anchor: dropping -Wno-deprecated-declarations adds 38 warnings to the bindings build. In the linux-64 py3.12 build log: 18 each in _internal/runtime.cpp and _internal/runtime_ptds.cpp, one each in runtime.cpp and _internal/nvrtc.cpp, all cudaMemcpy*Array* and cudaGetDriverEntryPoint. Harmless today. #2966 adds -Werror in CI, and if that reaches the shared _build_flags, bindings stops building. Keep the flag for bindings, or account for it in #2966.

[leofang]

Keep the flag for bindings, or account for it in #2966.

This is now kept in commit 6477e98.

[leofang]

/ok to test 4949f56

[Andy-Jost]

looks good

[juenglin]

Closing this in favor of three smaller PRs that together re-implement it. I volunteered to take it over, and splitting it keeps each piece easier to review:

1. build: require symlinks on Windows; move lychee excludes to lychee.toml #2992 (merged): require symlinks on Windows and move the lychee excludes to lychee.toml. This covers the core.symlinks CI steps and the "Development on Windows" section of CONTRIBUTING.md.
2. build: unify compiler flags #2998 (merged): unify the compiler flags across cuda-bindings and cuda-core, closing the audit in cuda-core build_hooks.py missing compiler/linker flags present in cuda-bindings #1882.
3. build: consolidate toolchain helpers #3000 (open): consolidate the toolchain helpers into a single _build_shared.py (symlinked into cuda_core), including the Cython cache, the rebuild stamps, the CUDA path lookup and the shared test mixins.

Andy's review feedback on this PR is carried over in #3000: the stamp tests patch _build_shared instead of build_hooks, the force=True path has a test, there is an sccache CC test, the CONTRIBUTING.md recovery steps use the repo-local core.symlinks setting, and bindings keeps -Wno-deprecated-declarations.

Thanks @leofang for the original work; the commits here were the starting point for the three PRs above.