Skip to content

Add sensitive_fields to resources.yml - #6192

Merged
andrewnester merged 19 commits into
mainfrom
acc/secrets-no-drift
Aug 12, 2026
Merged

andrewnester merged 19 commits into
mainfrom
acc/secrets-no-drift

Conversation

@andrewnester

@andrewnester andrewnester commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Changes

Add sensitive_fields to resources.yml
As a result it allows us to use Value field from SDK struct as is and do a proper drift detected

Why

The effective_value returned from the API after DoRead (with include_value=true) was not being used in the drift comparison, causing a false "update" action on every second plan.

Now we do not unset the fields and redact them instead which makes drift detection reliable.

Tests

Drift test pass

@andrewnester
andrewnester enabled auto-merge August 6, 2026 14:54
Comment thread bundle/direct/bundle_plan.go Outdated
@andrewnester
andrewnester requested a review from denik August 6, 2026 16:01
Comment thread acceptance/bundle/invariant/no_drift/test.toml
Comment thread bundle/direct/dresources/all_test.go Outdated
Comment thread bundle/direct/dresources/secret.go Outdated
Comment thread bundle/direct/bundle_plan.go Outdated
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: c0794e6

Run: 31596126861

Env 🔄​flaky 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
💚​ aws linux 5 3 296 1412 8:05
🔄​ aws windows 1 4 3 298 1410 10:14
💚​ azure linux 5 3 295 1412 8:46
💚​ azure windows 5 3 297 1410 6:30
💚​ gcp linux 2 4 296 1412 7:38
💚​ gcp windows 2 4 298 1410 7:44
8 interesting tests: 4 RECOVERED, 3 SKIP, 1 flaky
Test Name aws linux aws windows azure linux azure windows gcp linux gcp windows
🔄​ TestAccept 💚​R 🔄​f 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/invariant/no_drift 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🙈​ TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/ssh/connection 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
💚​ TestFetchRepositoryInfoAPI_FromRepo 💚​R 💚​R 💚​R 💚​R 🙈​S 🙈​S
💚​ TestFetchRepositoryInfoAPI_FromRepo/root 💚​R 💚​R 💚​R 💚​R
💚​ TestFetchRepositoryInfoAPI_FromRepo/subdir 💚​R 💚​R 💚​R 💚​R
Top 11 slowest tests (at least 2 minutes):
duration env testname
4:57 gcp windows TestAccept
3:17 azure windows TestAccept
3:03 azure windows TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=secret.yml.tmpl/READPLAN=1
2:50 azure windows TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=secret.yml.tmpl/READPLAN=
2:41 aws windows TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=secret.yml.tmpl/READPLAN=
2:39 azure linux TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=secret.yml.tmpl/READPLAN=
2:36 aws windows TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=secret.yml.tmpl/READPLAN=1
2:35 gcp windows TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=secret.yml.tmpl/READPLAN=1
2:33 aws windows TestAccept
2:16 aws linux TestAccept/bundle/resources/secrets/basic/DATABRICKS_BUNDLE_ENGINE=direct
2:05 gcp windows TestAccept/bundle/invariant/destroy_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=secret.yml.tmpl/READPLAN=

@andrewnester
andrewnester requested a review from denik August 7, 2026 10:07
Comment thread bundle/direct/dresources/secret.go Outdated
Comment thread acceptance/bundle/invariant/no_drift/test.toml
Comment thread bundle/direct/dresources/secret.go Outdated
Comment thread libs/structs/structdiff/diff.go Outdated
@andrewnester andrewnester changed the title Fix drift for UC secrets values Add sensitive_fields to resources.yml Aug 10, 2026
@andrewnester
andrewnester requested a review from denik August 10, 2026 14:28
Comment thread acceptance/bundle/invariant/delete_idempotent/test.toml Outdated
Comment thread acceptance/bundle/invariant/migrate/test.toml
Comment thread bundle/direct/dresources/redact_config.go Outdated
Comment thread bundle/direct/redact.go Outdated
return v, nil
}

const sensitiveRedactedMarker = "[redacted]"

@denik denik Aug 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One possible idea: instead of using "[redacted]", use "[redacted_uniquekey]" and keep map of unique key to secret. Then this uniquekey propagates all the way to resource where we can do search and replace post serialization.

@andrewnester
andrewnester requested a review from denik August 11, 2026 16:15
Comment thread bundle/direct/dresources/adapter.go Outdated
Comment thread bundle/direct/dresources/redact_config.go Outdated
Comment thread bundle/direct/redact.go Outdated
Comment thread bundle/direct/dresources/serialize_test.go Outdated
@andrewnester
andrewnester requested a review from denik August 12, 2026 09:53
Comment thread bundle/direct/apply.go Outdated
Comment thread bundle/direct/redact.go Outdated
Comment thread bundle/direct/dresources/secret.go Outdated
@andrewnester
andrewnester requested a review from denik August 12, 2026 12:22

@denik denik left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do any tests record bits of resources.json that show that redaction happened here?

@andrewnester
andrewnester added this pull request to the merge queue Aug 12, 2026
Merged via the queue into main with commit 0d0f12e Aug 12, 2026
25 checks passed
@andrewnester
andrewnester deleted the acc/secrets-no-drift branch August 12, 2026 13:04
@andrewnester

Copy link
Copy Markdown
Contributor Author

denik added a commit that referenced this pull request Aug 13, 2026
secrets/basic, secrets/update-value (#6192) and state/permission_level_migration
(#6222) landed on main while this branch was in review, so their goldens still
carried the old "Deploying resources..." output.
denik added a commit that referenced this pull request Aug 13, 2026
secrets/basic, secrets/update-value (#6192) and state/permission_level_migration
(#6222) landed on main while this branch was in review, so their goldens still
carried the old "Deploying resources..." output.
denik added a commit that referenced this pull request Aug 13, 2026
secrets/basic, secrets/update-value (#6192) and state/permission_level_migration
(#6222) landed on main while this branch was in review, so their goldens still
carried the old "Deploying resources..." output.
hejcman-enverus pushed a commit to hejcman-enverus/databricks_cli that referenced this pull request Sep 3, 2026
## Changes

Remove the `bundle:"sensitive"` tag and everything that read it:
`dyn.secretString`, `NewSensitiveValue`,
`Value.IsSensitive`/`MarkSensitive`, `convert.SensitiveFieldNames`, the
redaction branches in `jsonsaver`/`yamlsaver`, sensitivity propagation
in `dynvar.Resolve`, and `structwalk.RedactSensitiveFields`. Every hunk
restores the pre-databricks#5896 shape.

## Why

The tag has no declarations left in the tree — only test fixtures
declaring it to exercise the machinery that read it.

| PR | Merged | Effect on `bundle:"sensitive"` |
|---|---|---|
| [databricks#5896](databricks#5896) | 2026-07-22 |
Added the tag and the machinery reading it |
| [databricks#6179](databricks#6179) | 2026-08-05 |
Unwired `RedactSensitiveFields` — its reflection deep-clone dropped
unexported fields, corrupting every `duration.Duration` in direct-engine
state to `0s` |
| [databricks#5861](databricks#5861) | 2026-08-06 |
Added the only production declaration: `SecretState.SecretValue` |
| [databricks#6192](databricks#6192) | 2026-08-12 |
Replaced the tag with `sensitive_fields:` in `resources.yml`, deleting
that last declaration |

These are no-ops rather than behaviour changes: `dyn` sensitivity had a
single producer (`info.Sensitive[k]` in `FromTyped`, fed by the tag), so
`IsSensitive()` could never return true, and in `structdiff` `fieldName
== "-" && !bundleTag.Sensitive()` reduces to `fieldName == "-"`. No
golden file referenced `<redacted value>`.

The live `sensitive_fields:` mechanism (`bundle/direct/redact.go`,
`dresources/redact_config.go`) is untouched.

## Tests

Existing tests; the ones covering only the removed machinery are
deleted. No changelog fragment — pure internal removal, nothing
user-visible.

_This PR was written by Claude Code._
janniklasrose pushed a commit that referenced this pull request Sep 15, 2026
## Changes
Add sensitive_fields to resources.yml
As a result it allows us to use Value field from SDK struct as is and do
a proper drift detected

## Why
The effective_value returned from the API after DoRead (with
include_value=true) was not being used in the drift comparison, causing
a false "update" action on every second plan.

Now we do not unset the fields and redact them instead which makes drift
detection reliable.

## Tests
Drift test pass

<!-- If your PR needs to be included in the release notes for next
release,
add a changelog fragment: create .nextchanges/<section>/<name>.md with a
one-line description (e.g. .nextchanges/cli/quickstart.md). See
.nextchanges/README.md. -->
janniklasrose pushed a commit that referenced this pull request Sep 15, 2026
## Changes

Remove the `bundle:"sensitive"` tag and everything that read it:
`dyn.secretString`, `NewSensitiveValue`,
`Value.IsSensitive`/`MarkSensitive`, `convert.SensitiveFieldNames`, the
redaction branches in `jsonsaver`/`yamlsaver`, sensitivity propagation
in `dynvar.Resolve`, and `structwalk.RedactSensitiveFields`. Every hunk
restores the pre-#5896 shape.

## Why

The tag has no declarations left in the tree — only test fixtures
declaring it to exercise the machinery that read it.

| PR | Merged | Effect on `bundle:"sensitive"` |
|---|---|---|
| [#5896](#5896) | 2026-07-22 |
Added the tag and the machinery reading it |
| [#6179](#6179) | 2026-08-05 |
Unwired `RedactSensitiveFields` — its reflection deep-clone dropped
unexported fields, corrupting every `duration.Duration` in direct-engine
state to `0s` |
| [#5861](#5861) | 2026-08-06 |
Added the only production declaration: `SecretState.SecretValue` |
| [#6192](#6192) | 2026-08-12 |
Replaced the tag with `sensitive_fields:` in `resources.yml`, deleting
that last declaration |

These are no-ops rather than behaviour changes: `dyn` sensitivity had a
single producer (`info.Sensitive[k]` in `FromTyped`, fed by the tag), so
`IsSensitive()` could never return true, and in `structdiff` `fieldName
== "-" && !bundleTag.Sensitive()` reduces to `fieldName == "-"`. No
golden file referenced `<redacted value>`.

The live `sensitive_fields:` mechanism (`bundle/direct/redact.go`,
`dresources/redact_config.go`) is untouched.

## Tests

Existing tests; the ones covering only the removed machinery are
deleted. No changelog fragment — pure internal removal, nothing
user-visible.

_This PR was written by Claude Code._
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants