Cleanup concurrency problems in doc generation by jpascucci-nv · Pull Request #2914 · NVIDIA/cuda-python

jpascucci-nv · 2026-09-18T15:03:42Z

Refs #2197 - addressed by Leo Fang as not to fix now.

Description

Incorporates the force-free deployment approach from #2883.

The original version of this PR compacted gh-pages on every deployment. That required non-fast-forward pushes, conflicted with the active Protect gh-pages ruleset, and introduced additional operational risk around preserving the complete published site.

This revision does not rewrite gh-pages history. Instead, it addresses the concurrency problems identified while investigating #2197 and keeps the existing branch-protection rules intact.

Non-goal

This PR no longer compacts the existing gh-pages history and therefore does
not resolve the clone-size problem tracked by #2197. History compaction will
require a separate, ruleset-compatible design.

Changes

Serialize every automated gh-pages writer through the shared
cuda-python-gh-pages-publish concurrency group.
- Queue all waiting publishers with queue: max.
- Explicitly set cancel-in-progress: false.
Keep expensive documentation and coverage generation outside the serialized
section.
- Pass generated output to short deployment jobs through workflow artifacts.
Set force: false on all remaining github-pages-deploy-action invocations:
- PR previews.
- Latest and versioned documentation.
- Coverage reports.
Let the deploy action fetch, rebase, and retry if gh-pages advances before
its normal push completes.
Make the scheduled nightly workflow the sole owner of stale PR-preview
deletion.
- Remove immediate preview deletion from main and release builds.
- Remove the cleanup workflow's manual workflow_dispatch entry point.
- Restrict cleanup pushes to scheduled GitHub Actions runs holding the shared
  publishing lock.
Update the cleanup script to create a normal descendant commit and push
without force.
- Retry up to three times by fetching and rebasing if the branch advances.
- Work from a detached remote snapshot without modifying a user's local
  gh-pages branch.
Add targeted actionlint suppressions for GitHub's concurrency.queue
extension.
New or existing tests cover these changes.
The documentation is up to date with these changes.

The `gh-pages` branch has accumulated thousands of generated-site commits, substantially increasing the size of full repository clones. This PR keeps `gh-pages` at a single commit while preserving the existing documentation, PR previews, and coverage pages. It: - Enables `single-commit` publishing for documentation, PR previews, preview removal, and coverage. - Serializes every automated `gh-pages` writer using the shared `cuda-python-gh-pages-publish` concurrency group. - Separates expensive docs and coverage generation from the serialized deployment jobs, so builds can still run in parallel. - Passes generated output to the deployment jobs through workflow artifacts. - Updates stale-preview cleanup to: - Work from a detached snapshot without modifying a local `gh-pages` branch. - Create a parentless commit containing the updated complete tree. - Push using an explicit `--force-with-lease`. - Permit remote pushes only from the serialized cleanup workflow. - Adds the required targeted `actionlint` suppressions for GitHub's `concurrency.queue` extension. Preview and coverage pages remain intact because every deployment begins with the existing `gh-pages` tree and updates only its assigned target directory. This change intentionally performs a non-fast-forward update on every deployment. The active `Protect gh-pages` ruleset currently blocks these updates. Before exercising this workflow—including PR preview deployment—the authenticated publishing actor must have an `always` bypass for the ruleset, or the force-push restriction must be changed. `git-config-name: cuda-python-bot` is only commit metadata; as written, the authenticated actor is the GitHub Actions integration. The `Restrict deletions` rule is not otherwise a problem: the workflows delete files within commits, not the `gh-pages` branch ref. Granting the GitHub Actions integration a bypass is relatively broad. If that is undesirable, deployment should instead use a dedicated GitHub App with narrowly scoped repository permissions. - Drain any docs, coverage, and cleanup jobs started with the previous workflow before the first compacting deployment. Concurrency groups do not apply retroactively. - After the first deployment, verify that: - `gh-pages` contains one commit. - Latest and versioned documentation still exist. - Existing PR previews still exist. - Coverage pages still exist. - If a deployment-only job fails, use **Re-run all jobs** so its prerequisite job recreates the deployment artifact. - [x] New or existing tests cover these changes. - [x] The documentation is up to date with these changes.

copy-pr-bot · 2026-09-18T15:03:45Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

…ve single-commit.

github-actions · 2026-09-21T22:09:10Z

Doc Preview CI
Preview removed because the pull request was closed or merged.

Responding to Leo's comments. - detect preview builds using pull-request refs - honor deploy-docs for all documentation deployments - keep nightly cleanup as the sole gh-pages cleanup writer - add read-only manual preview inspection - retire sticky comments for closed PR previews

Andy-Jost

Requesting changes for one blocker and one small fix, plus two nice-to-haves.

Blocker: the comment update in ci/cleanup-pr-previews can stall the nightly cleanup. See the inline comment for details and a suggested fix.
The description says "Closes #2197". GitHub will close that issue on merge, but the PR does not fix the clone-size problem it tracks (see the Non-goal section). Please change it to "Refs #2197".

Nice to have:

Keep workflow_dispatch on the cleanup workflow, with the dry-run input, and let the script's --push guard accept both schedule and workflow_dispatch. Dispatching a workflow already requires write access, so removing it adds little protection, and it takes away the only way to run a cleanup on demand.
CI has only exercised the PR-preview path of the new deploy job. The main, release, and coverage deploys have not run yet. A release dry run dispatched from the pull-request/2914 branch with a gh-pages-dry-run docs branch would cover the release path before merge. The coverage nightly has been failing since Sept 24 for unrelated reasons, so that path cannot be checked right now.

The rest looks good to me. The shared concurrency group, force: false on every writer, the nightly job as sole owner of cleanup, and the artifact hand-off to short deploy jobs all match what the team agreed on.

jpascucci-nv · 2026-09-29T21:36:54Z

@Andy-Jost Ran release dry run here: https://github.com/NVIDIA/cuda-python/actions/runs/36633977438/job/109630709216

deleted the gh-pages-dry-run

Andy-Jost

LGTM!

leofang

Thanks, Jason/Andy!

leofang · 2026-09-30T04:12:18Z

+        gh api --method PATCH repos/"${REPOSITORY}"/issues/comments/"${comment_id}" \
+            --header "Accept: application/vnd.github+json" \
+            -f body="$comment_body" >/dev/null


My bot called this out and gave a 👍 because this edits in-place and does not send another notification.

Removed preview folders for the following PRs: - PR #2914

github-actions Bot added the CI/CD CI/CD infrastructure label Sep 18, 2026

jpascucci-nv changed the title ~~Closes #2197~~ Set gh-pages to single-commit and cleanup concurrency problems. Sep 18, 2026

jpascucci-nv mentioned this pull request Sep 18, 2026

Investigate gh-pages impact on full clone size and time #2197

Open

Make cancel-in-progress false explicitly.

6e8bb18

leofang reviewed Sep 18, 2026

View reviewed changes

Comment thread .github/workflows/build-docs.yml

add no force-push. Let nightly job be sole owner of doc cleanup. remo…

60adbd6

…ve single-commit.

jpascucci-nv changed the title ~~Set gh-pages to single-commit and cleanup concurrency problems.~~ Cleanup concurrency problems. Sep 21, 2026

jpascucci-nv marked this pull request as ready for review September 21, 2026 15:51

jpascucci-nv requested a review from leofang September 21, 2026 15:51

jpascucci-nv added this to the cuda.core 1.3.0 milestone Sep 21, 2026

jpascucci-nv self-assigned this Sep 21, 2026

Andy-Jost mentioned this pull request Sep 21, 2026

[doc-only] ci: rebase concurrent gh-pages deployments instead of force-pushing #2883

Closed

Merge branch 'main' into issue2197

c37a60c

jpascucci-nv changed the title ~~Cleanup concurrency problems.~~ Cleanup concurrency problems in doc generation Sep 23, 2026

leofang mentioned this pull request Sep 23, 2026

[Backport release/cuda-core-v1.2.1] [doc-only] docs(cuda.core): add 1.2.1 to the docs version switcher #2946

Merged

leofang reviewed Sep 24, 2026

View reviewed changes

Comment thread .github/actions/doc_preview/action.yml

Comment thread .github/workflows/cleanup-pr-previews.yml

Comment thread .github/workflows/build-docs.yml Outdated

Comment thread .github/workflows/build-docs.yml Outdated

somtri mentioned this pull request Sep 26, 2026

[BUG]: cleanup-pr-previews never removes previews of deleted PRs #2952

Open

1 task done

Andy-Jost requested changes Sep 29, 2026

View reviewed changes

Comment thread ci/cleanup-pr-previews Outdated

Address Andy's comments.

bb03ab8

Andy-Jost self-requested a review September 29, 2026 22:11

Andy-Jost approved these changes Sep 29, 2026

View reviewed changes

leofang approved these changes Sep 30, 2026

View reviewed changes

leofang merged commit 265959d into NVIDIA:main Sep 30, 2026
128 of 131 checks passed

github-actions Bot pushed a commit that referenced this pull request Sep 30, 2026

Clean up PR preview folders for 1 closed/merged PRs

8246957

Removed preview folders for the following PRs: - PR #2914

Navigation Menu

Cleanup concurrency problems in doc generation - #2914

Cleanup concurrency problems in doc generation#2914
leofang merged 6 commits into
NVIDIA:mainfrom
jpascucci-nv:issue2197

jpascucci-nv commented Sep 18, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Sep 18, 2026

Uh oh!

Uh oh!

github-actions Bot commented Sep 21, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Andy-Jost left a comment

Uh oh!

Uh oh!

jpascucci-nv commented Sep 29, 2026 •

edited

Loading

Uh oh!

Andy-Jost left a comment

Uh oh!

leofang left a comment

Uh oh!

leofang Sep 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

jpascucci-nv commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Non-goal

Changes

Uh oh!

copy-pr-bot Bot commented Sep 18, 2026

Uh oh!

Uh oh!

github-actions Bot commented Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Andy-Jost left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jpascucci-nv commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Andy-Jost left a comment

Choose a reason for hiding this comment

Uh oh!

leofang left a comment

Choose a reason for hiding this comment

Uh oh!

leofang Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jpascucci-nv commented Sep 18, 2026 •

edited

Loading

github-actions Bot commented Sep 21, 2026 •

edited

Loading

jpascucci-nv commented Sep 29, 2026 •

edited

Loading