direct: report resources as they are deployed - #6361
Merged
Merged
Conversation
Collaborator
Integration test reportCommit: 7984efb
Top 20 slowest tests (at least 2 minutes):
|
denik
force-pushed
the
denik/investigage-output-order
branch
from
August 25, 2026 07:29
e7f63e5 to
1552eda
Compare
denik
force-pushed
the
denik/investigage-output-order
branch
7 times, most recently
from
September 14, 2026 10:37
56dc3d1 to
67c8363
Compare
denik
marked this pull request as ready for review
September 14, 2026 10:45
shreyas-goenka
approved these changes
Sep 14, 2026
| b.DeploymentBundle.Apply(ctx, b.WorkspaceClient(ctx), plan) | ||
| // Not reported per resource: destroy names them up front for consent and then | ||
| // reports only a count, so there is no per-resource output to report into. | ||
| b.DeploymentBundle.Apply(ctx, b.WorkspaceClient(ctx), plan, false) |
Contributor
There was a problem hiding this comment.
why not report destroy as well? The progress is still useful to keep track of the progress, regardless of what the original plan was.
This is not about concent but a progress log right?
Contributor
Author
There was a problem hiding this comment.
No strong need to output this for destroy:
- the whole process is much faster
- already shows you list of resources in advance.
Those resources that had errors will be shown though, so you don't miss important stuff.
denik
force-pushed
the
denik/investigage-output-order
branch
from
September 14, 2026 11:00
d95ecf3 to
6f4e397
Compare
denik
enabled auto-merge
September 14, 2026 11:02
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Sep 14, 2026
denik
force-pushed
the
denik/investigage-output-order
branch
from
September 14, 2026 12:59
6f4e397 to
51b2882
Compare
denik
enabled auto-merge
September 14, 2026 13:05
denik
force-pushed
the
denik/investigage-output-order
branch
from
September 14, 2026 13:19
51b2882 to
ac4d042
Compare
The direct engine printed all per-resource lines after the whole deployment finished, so a deploy that failed part way through reported nothing about the resources it had already applied. Resources are applied in parallel, so the lines are no longer in a deterministic order. Add a SortLines/SortLinesOn acceptance config option that sorts runs of matching output lines before comparison, and enable it for these lines. Co-authored-by: Isaac
The blocks in the selftest sorted the same way whether or not the boundary was respected. Pick values so a merged sort would move lines across it, and add the matching unit test case. Co-authored-by: Isaac
Co-authored-by: Isaac
The migration ran inside deployCore, before the postdeploy script. Order is now deploy, then the postdeploy script, then the migration. The migration is still gated on the deploy succeeding and still not on the script: the resources are deployed either way, so the state describes the same deployment. Co-authored-by: Isaac
destroy does list resources individually, up front for consent; what it lacks is per-resource completion output. State the reason for the flag instead. Co-authored-by: Isaac
Picks up new direct-engine tests (dms, job_runs) that now show the per-resource line before an injected error, main's reworded migration warning, and the DMS matrix var in an out.test.toml. Co-authored-by: Isaac <no-reply@databricks.com>
main added tools/validate_nextchanges.py, which requires each fragment to start with "* " and carry a PR link in CI. Co-authored-by: Isaac <no-reply@databricks.com>
Per the "commit static test inputs" rule: the script now copies a committed empty-tfstate.json into place rather than generating it inline. Excluded from bundle sync so the uploaded-file count is unchanged. Co-authored-by: Isaac <no-reply@databricks.com>
The per-resource "Created" lines print as each resource is applied, which races the kill signal in the WAL crash tests — on Windows the faster kill loses the race, so which lines appear before [PROCESS_KILLED] was nondeterministic and task test (windows, direct) failed. These tests assert WAL content, not deploy stdout, so -q drops the racing lines. Sorting can't help: it cannot recover a line the dying process never printed. Co-authored-by: Isaac <no-reply@databricks.com>
main added the on_file_change* job_runs tests (and changed on_bundle_deploy) while this branch was behind. Their goldens predate the direct engine's per-resource "report as deployed" output, so regenerate them to carry the inline order (Created jobs.my_job before the job-run's streamed output). The order is deterministic — job_runs.my_run depends on jobs.my_job — verified byte-stable across 20 high-parallelism runs. Co-authored-by: Isaac <no-reply@databricks.com>
#6629 moved output.txt/out.requests.txt out of the bundle sync root and added a dedicated doComparison call for output.txt; pass sortLines to it so SortLines still applies to the deploy action lines there (also fixes the build). Regenerate two resource_deps error-path goldens for the new Files counts / inline line. Co-authored-by: Isaac <no-reply@databricks.com>
The prior post-rebase regen used a -run pattern with '/' inside alternation groups, which Go matches segment-by-segment, so it silently skipped job_runs and left the rebase auto-merge's main-order goldens (Created lines at the end). The deploy is deterministic (job_runs.my_run depends on its job, so the job is created and reported first), so a clean per-family regen produces stable goldens with the inline order. Verified 0 failures across 20 high-parallelism runs of basic, on_file_change_glob, and on_bundle_deploy. Co-authored-by: Isaac <no-reply@databricks.com>
The prior regen's -run pattern had '/' inside alternation groups, so Go's segment-by-segment matching skipped these families too. Clean per-family regen picks up #6629's lower Files counts (output.txt/out.requests.txt no longer synced) and the inline per-resource line. permissions/504/create verified stable across 10 runs. Co-authored-by: Isaac <no-reply@databricks.com>
…te line My per-resource reporting emits "Recreated schemas.foo" before the injected upload error; the committed golden predated it. Other dms multi-resource tests are unaffected because SortLines normalizes their blocks at compare time. Co-authored-by: Isaac <no-reply@databricks.com>
The injected 403 on the state-push (resources.json) races per-resource apply reporting: on some platforms "Created jobs.my_job" is emitted before the push fails, on others it is not (linux hit the former, macos the latter, same goldens). This test asserts that Files is reported and Resources is not on a push failure, so the per-resource line is incidental; -q drops it and makes the golden deterministic across platforms. Verified 0 failures across 15 runs. Co-authored-by: Isaac <no-reply@databricks.com>
This test has no resources (summary is all zeros, no per-resource lines) and runs on the terraform engine, so it exercises none of this PR's per-resource reporting. It covers only the deploy -> postdeploy script -> migration ordering, which already landed on main independently of this change; deploy.go here does not touch that order. Out of scope for this PR. Co-authored-by: Isaac <no-reply@databricks.com>
#6531 added this test on main; its golden was generated without this PR's per-resource reporting. Merged with this branch, the deploy now prints "Created cluster_policies.my_policy" inline before the dependent job-create fails (the policy is created first, in dependency order), so the golden needs that line. Deterministic (single resource before the failure), verified 0 failures across 10 runs of all five cluster_policy scenarios. Co-authored-by: Isaac <no-reply@databricks.com>
#6647 ("optional DoDelete in the direct engine") landed on main and now deletes the orphaned permissions child on the deploy path when its config block is removed. This PR's per-resource reporting prints that as an inline "Deleted <resource>.permissions" line, which the main-generated goldens lacked. Regenerated permissions/jobs/update and permissions/pipelines/update; grants and secret_scopes had no drift. Verified 0 failures across 5 runs each. Co-authored-by: Isaac <no-reply@databricks.com>
denik
force-pushed
the
denik/investigage-output-order
branch
from
September 14, 2026 14:44
978b1ff to
b05a192
Compare
A state-only delete (the resource implements no DoDelete, so the delete only
drops the state entry and calls no backend API) is not a real action: the
summary already excludes it (CountActions skips StateOnly) and the terraform
path in logDeploySummary already skips it. The direct engine's inline reporting
did not, so removing a child like jobs.<name>.permissions printed a misleading
"Deleted <resource>.permissions" line that the terraform variant of the same
shared golden never produced, breaking permissions/{jobs,pipelines}/update.
Skip the per-resource line for state-only deletes too, and revert those two
goldens. Verified both tests pass all engine variants.
Co-authored-by: Isaac <no-reply@databricks.com>
janniklasrose
pushed a commit
that referenced
this pull request
Sep 15, 2026
## Why The per-resource output added in #5720 (CLI v1.13.0) is printed after the whole deployment has finished, so a deploy that failed part way through reported nothing about the resources it had already applied. On the direct engine each resource is now reported as soon as it is applied. ## Changes Resources are applied in parallel, so these lines are no longer in a deterministic order. Added `SortLines` (name -> regexp) and `SortLinesOn` (name -> bool) to the acceptance config: each run of consecutive output lines matching a pattern is sorted before comparison, so a test pins which resources were applied rather than the order. Enabled once in `acceptance/test.toml`; an inner `test.toml` can replace a pattern by reusing its name or switch it off. `resource_deps/create_error` deploys with `-q` now: its `jobs.independent` has no dependency on the failing job, so where its line lands relative to the errors is not stable, and that test is about which errors a failing deploy reports. ## Tests New `selftest/sortlines` covers sorting, per-name override and disabling. 18 goldens updated: 12 failure and kill paths now report what succeeded, 6 are lines moving to their true chronological position. --------- Co-authored-by: Isaac <no-reply@databricks.com>
deco-sdk-tagging Bot
added a commit
that referenced
this pull request
Sep 16, 2026
## Release v1.17.0 ### Notable Changes * Bump the direct deployment state version to 3. Clients older than v1.8.0 will reject bundles deployed with this release. ([#6713](#6713)) ### CLI * Add an `INVALID_REFRESH_TOKEN` error code to `databricks auth token --output json` failures. ([#6684](#6684)) * Add experimental `databricks auth docker configure` to configure Docker credential helper access for Databricks Artifact Registry. ([#6700](#6700)) * Add experimental `databricks auth docker token` to generate Docker credentials for Databricks Artifact Registry. ([#6699](#6699)) * `databricks environments setup-local` now reports the `E_PROVISION_CONFLICT` error code instead of the generic `E_PROVISION` when `uv sync` fails to resolve a dependency conflict. ([#6666](#6666)) * Preserve SSH sessions across temporary tunnel disconnects, with bounded replay and backpressure for large transfers. ([#6650](#6650)) * Allow OAuth U2M logins to override the CLI client ID with `--client-id`, profile `client_id`, or `DATABRICKS_CLIENT_ID`. ([#6594](#6594)) ### Bundles * direct: Store a dashboard's `serialized_dashboard` in state as a content hash instead of its full contents. ([#6105](#6105)) * direct: Fix pipelines recreation when the whole `ingestion_definition` block is added or removed. ([#6589](#6589)) * `bundle plan`, `deploy`, and `destroy` no longer report removing `permissions`, `grants`, or secret scope ACLs from a bundle as a deletion, since it leaves the resource untouched. ([#6647](#6647)) * `bundle plan` and `deploy` no longer list or count a resource that was already deleted remotely as a deletion, matching `bundle destroy`; applying still cleans up its stale state entry. ([#6675](#6675)) * Fix `bundle run` failing with `expected an int, found a string` when an unrelated resource references another resource that is not deployed. `bundle run` now resolves `${resources.*}` references only within the resource being run. ([#6690](#6690)) * Add grants support for the AI Gateway `model_service`, `mcp_service`, and `model_provider_service` resources (direct engine). ([#6635](#6635)) * Add bundle support for the AI Gateway `mcp_service` resource (direct engine). ([#6633](#6633)) * Add bundle support for the AI Gateway `model_provider_service` resource (direct engine). ([#6634](#6634)) * Add bundle support for the AI Gateway `model_service` resource (direct engine). ([#6525](#6525)) * Prevent resource drift on catalogs if `storage_root` contained a trailing slash in the URL. ([#6622](#6622)) * Fixed a "lineage mismatch in state files" error that could occur after destroying a bundle and redeploying it from another machine. `bundle destroy` now removes the local state file so no stale lineage is left behind, and prunes the state directories it leaves empty (such as `.internal/` and `sync-snapshots/`). ([#6210](#6210), [#6685](#6685)) * direct: `bundle plan` no longer reports a permanent update on a cluster that uses a cluster policy: when the cluster spec sets `policy_id`, a field present in the remote but absent from the bundle config is not treated as drift. ([#6531](#6531)) * `bundle deploy` on the direct engine now reports each resource as soon as it is deployed, instead of listing them all after the deployment finishes. A deploy that fails part way through now reports the resources it did apply. ([#6361](#6361)) * Direct-engine bundles no longer flag phantom drift on server-populated nested fields under reused config types (e.g. `external_locations` file-event-queue resource IDs, `database_instances` parent-instance refs, `apps` git credential ID). ([#6618](#6618)) * `databricks bundle generate app` now reproduces a git-backed app's `git_repository` and `git_source` configuration instead of emitting a workspace `source_code_path`, so generating from a Git-deployed app no longer silently converts it to workspace source. ([#6656](#6656)) * Improved configuration load time for bundles with many included files. ([#6195](#6195)) * `bundle destroy` no longer deletes triggered job runs, leaving them untouched on the backend. ([#6672](#6672)) * direct: resources.job\_runs: new lifecycle.triggers.on\_file\_change setting to restart the run when monitored files change. Can be set to a series of paths or globs. ([#6309](#6309)) * Bundle summary now shows a name for Postgres branches, endpoints, databases, and roles instead of a blank Name field. ([#6663](#6663)) * Added PyDABs (Python) support for cluster policies, dashboards, and Genie spaces. ([#6585](#6585)) * CLI commands no longer imply that a resource whose type has no workspace URL is merely not deployed yet. ([#6583](#6583)) * Capture the implicit dependency a vector search index has on a catalog or schema defined in the same bundle, so the catalog and schema are deployed first. ([#6655](#6655)) ### Dependency Updates * Bump dependencies with known vulnerabilities. ([#6695](#6695)) * Bump `github.com/databricks/databricks-sdk-go` from v0.177.0 to v0.178.0. ([#6673](#6673)) * Bump Terraform provider from v1.131.0 to v1.132.0. ([#6671](#6671))
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The per-resource output added in #5720 (CLI v1.13.0)
is printed after the whole deployment has finished, so a deploy that failed part way
through reported nothing about the resources it had already applied. On the direct engine
each resource is now reported as soon as it is applied.
Changes
Resources are applied in parallel, so these lines are no longer in a deterministic order.
Added
SortLines(name -> regexp) andSortLinesOn(name -> bool) to the acceptanceconfig: each run of consecutive output lines matching a pattern is sorted before
comparison, so a test pins which resources were applied rather than the order. Enabled
once in
acceptance/test.toml; an innertest.tomlcan replace a pattern by reusing itsname or switch it off.
resource_deps/create_errordeploys with-qnow: itsjobs.independenthas nodependency on the failing job, so where its line lands relative to the errors is not
stable, and that test is about which errors a failing deploy reports.
Tests
New
selftest/sortlinescovers sorting, per-name override and disabling. 18 goldensupdated: 12 failure and kill paths now report what succeeded, 6 are lines moving to their
true chronological position.