Add an experimental dbt-sql template by lennartkats-db · Pull Request #1059 · databricks/cli

lennartkats-db · 2023-12-12T15:01:58Z

Changes

This adds a new dbt-sql template. This work requires the new WorkspaceFS support for dbt tasks.

In this latest revision, I've hidden the new template from the list so we can merge it, iterate over it, and propertly release the template at the right time.

Blockers:

Tests

## Changes This enhances the template selection experience a bit as we add more and more built-in templates (like #1051 and #1059): ### New experience: <img width="661" alt="image" src="https://github.com/databricks/cli/assets/58432911/afe3b84d-8a77-47f3-b9c2-f827f7893cd7"> ### Current experience: <img width="265" alt="image" src="https://github.com/databricks/cli/assets/58432911/36f8d568-819f-4920-83b1-fb76109ea3d1"> --------- Co-authored-by: shreyas-goenka <88374338+shreyas-goenka@users.noreply.github.com>

pietern · 2024-02-05T08:03:40Z

+        },
+        "http_path": {
+            "type": "string",
+            "pattern": "^/sql/.\\../warehouses/[a-z0-9]+$",


How does this pattern work without + or *?

I'm not sure I understand this question. There is a + in there?

Or maybe you're referring to the \.\\.. part? That matches a version, like 1.0.

CLI: * Add support for UC Volumes to the `databricks fs` commands ([#1209](#1209)). Bundles: * Use dynamic configuration model in bundles ([#1098](#1098)). * Allow use of variables references in primitive non-string fields ([#1219](#1219)). * Add an experimental default-sql template ([#1051](#1051)). * Add an experimental dbt-sql template ([#1059](#1059)). Internal: * Add fork-user to winget release workflow ([#1214](#1214)). * Use `any` as type for data sources and resources in `tf/schema` ([#1216](#1216)). * Avoid infinite recursion when normalizing a recursive type ([#1213](#1213)). * Fix issue where interpolating a new ref would rewrite unrelated fields ([#1217](#1217)). * Use `dyn.Value` as input to generating Terraform JSON ([#1218](#1218)). API Changes: * Changed `databricks lakehouse-monitors update` command with new required argument order. * Added `databricks online-tables` command group. OpenAPI commit cdd76a98a4fca7008572b3a94427566dd286c63b (2024-02-19) Dependency updates: * Bump Terraform provider to v1.36.2 ([#1215](#1215)). * Bump github.com/databricks/databricks-sdk-go from 0.32.0 to 0.33.0 ([#1222](#1222)).

## Changes This enhances the template selection experience a bit as we add more and more built-in templates (like #1051 and #1059): ### New experience: <img width="661" alt="image" src="https://github.com/databricks/cli/assets/58432911/afe3b84d-8a77-47f3-b9c2-f827f7893cd7"> ### Current experience: <img width="265" alt="image" src="https://github.com/databricks/cli/assets/58432911/36f8d568-819f-4920-83b1-fb76109ea3d1"> --------- Co-authored-by: shreyas-goenka <88374338+shreyas-goenka@users.noreply.github.com>

## Changes This adds a new dbt-sql template. This work requires the new WorkspaceFS support for dbt tasks. In this latest revision, I've hidden the new template from the list so we can merge it, iterate over it, and propertly release the template at the right time. Blockers: - [x] WorkspaceFS support for dbt projects is in prod - [x] Move dbt files into a subdirectory - [ ] Wait until the next (>1.7.4) release of the dbt plugin which will have major improvements! - _Rather than wait, this template is hidden from the list of templates._ - [x] SQL extension is preconfigured based on extension settings (if possible) - MV / streaming tables: - [x] Add to template - [x] Fix databricks/dbt-databricks#535 (to be released with in 1.7.4) - [x] Merge databricks/dbt-databricks#338 (to be released with in 1.7.4) - [ ] Fix "too many 503 errors" issue (databricks/dbt-databricks#570, internal tracker: ES-1009215, ES-1014138) - [x] Support ANSI mode in the template - [ ] Streaming tables support is either ungated or the template provides instructions about signup - _Mitigation for now: this template is hidden from the list of templates._ - [x] Support non-workspace-admin deployment - [x] Make sure `data_security_mode: SINGLE_USER` works on non-UC workspaces (it's required to be explicitly specified on UC workspaces with single-node clusters) - [x] Support non-UC workspaces ## Tests - [x] Unit tests - [x] Manual testing - [x] More manual testing - [ ] Reviewer manual testing - _I'd like to do a small bug bash post-merging._ - [x] Unit tests

CLI: * Add support for UC Volumes to the `databricks fs` commands ([#1209](#1209)). Bundles: * Use dynamic configuration model in bundles ([#1098](#1098)). * Allow use of variables references in primitive non-string fields ([#1219](#1219)). * Add an experimental default-sql template ([#1051](#1051)). * Add an experimental dbt-sql template ([#1059](#1059)). Internal: * Add fork-user to winget release workflow ([#1214](#1214)). * Use `any` as type for data sources and resources in `tf/schema` ([#1216](#1216)). * Avoid infinite recursion when normalizing a recursive type ([#1213](#1213)). * Fix issue where interpolating a new ref would rewrite unrelated fields ([#1217](#1217)). * Use `dyn.Value` as input to generating Terraform JSON ([#1218](#1218)). API Changes: * Changed `databricks lakehouse-monitors update` command with new required argument order. * Added `databricks online-tables` command group. OpenAPI commit cdd76a98a4fca7008572b3a94427566dd286c63b (2024-02-19) Dependency updates: * Bump Terraform provider to v1.36.2 ([#1215](#1215)). * Bump github.com/databricks/databricks-sdk-go from 0.32.0 to 0.33.0 ([#1222](#1222)).

Add a dbt template

4fee665

lennartkats-db changed the title ~~[WIP] Add a dbt template~~ [WIP] Add a dbt-sql template Dec 12, 2023

lennartkats-db mentioned this pull request Dec 12, 2023

Improve experience for multiple builtin templates #1052

Merged

Use a template for VS Code settings

b3a5ef8

pietern reviewed Dec 15, 2023

View reviewed changes

Comment thread libs/template/templates/dbt-sql/databricks_template_schema.json

Comment thread libs/template/templates/dbt-sql/template/{{.project_name}}/profile_template.yml.tmpl Outdated

lennartkats-db added 2 commits December 19, 2023 10:27

Tweak message

c81d139

Update

c900cff

lennartkats-db changed the title ~~[WIP] Add a dbt-sql template~~ [WIP] [DONTMERGE] Add a dbt-sql template Dec 23, 2023

Add tests

16f26a3

lennartkats-db added 2 commits December 28, 2023 13:32

Merge remote-tracking branch 'databricks/main' into dbt-template

cd52c83

Fix test

d85d4c4

lennartkats-db marked this pull request as ready for review December 28, 2023 12:34

lennartkats-db changed the title ~~[WIP] [DONTMERGE] Add a dbt-sql template~~ [DONTMERGE] Add a dbt-sql template Dec 28, 2023

lennartkats-db requested a review from pietern December 28, 2023 12:35

lennartkats-db added 7 commits January 8, 2024 21:16

Merge remote-tracking branch 'databricks/main' into dbt-template

419bd27

Add template

9030f56

Improve catalog handling

45ea8db

Minor tweaks

0268c88

Update template to use materialized views & streaming tables

94ebd9a

Add conditional

14bc1fa

Improve template

1501298

lennartkats-db force-pushed the dbt-template branch from 62a0b69 to 1501298 Compare January 20, 2024 21:59

lennartkats-db added 4 commits January 22, 2024 09:23

Offer an option to use personal schemas

6fc5ed4

Merge remote-tracking branch 'databricks/main' into dbt-template

220a1ea

Fix ANSI mode

99f920e

Merge remote-tracking branch 'databricks/main' into dbt-template

af0dd6d

pietern added the do-not-merge label Jan 25, 2024

lennartkats-db added 2 commits January 25, 2024 21:17

Don't ask for a "production" schema, just assume "default"

1099eed

Explain mode: development

33c5e91

lennartkats-db added 3 commits January 28, 2024 09:11

Remove from list of templates for now

8e7c6a1

Update README.md

18c6b70

Merge remote-tracking branch 'databricks/main' into dbt-template

a660efa

lennartkats-db changed the title ~~[DONTMERGE] [WIP] Add a dbt-sql template~~ Add a dbt-sql template Jan 28, 2024

lennartkats-db changed the title ~~Add a dbt-sql template~~ Add an experimental dbt-sql template Jan 29, 2024

Mark as experimental

2f52ff1

lennartkats-db removed the do-not-merge label Feb 4, 2024

pietern approved these changes Feb 5, 2024

View reviewed changes

pietern mentioned this pull request Feb 5, 2024

Add an experimental default-sql template #1051

Merged

9 tasks

juliacrawf-db reviewed Feb 13, 2024

View reviewed changes

Comment thread libs/template/templates/dbt-sql/databricks_template_schema.json

juliacrawf-db reviewed Feb 19, 2024

View reviewed changes

Comment thread libs/template/templates/dbt-sql/template/{{.project_name}}/README.md.tmpl Outdated

juliacrawf-db reviewed Feb 19, 2024

View reviewed changes

Comment thread libs/template/templates/dbt-sql/template/{{.project_name}}/README.md.tmpl Outdated

juliacrawf-db reviewed Feb 19, 2024

View reviewed changes

Comment thread libs/template/templates/dbt-sql/template/{{.project_name}}/README.md.tmpl Outdated

lennartkats-db added 3 commits February 19, 2024 09:56

Restore sql-dbt template in hidden form

e041148

Merge remote-tracking branch 'databricks/main' into dbt-template

00bf2fe

Copy-editing

e5fb708

lennartkats-db force-pushed the dbt-template branch from aaec97d to e5fb708 Compare February 19, 2024 09:07

lennartkats-db enabled auto-merge February 19, 2024 09:10

lennartkats-db removed the request for review from PaulCornellDB February 19, 2024 09:12

lennartkats-db added this pull request to the merge queue Feb 19, 2024

Merged via the queue into databricks:main with commit 1c68012 Feb 19, 2024

lennartkats-db deleted the dbt-template branch February 19, 2024 09:20

lennartkats-db mentioned this pull request Feb 20, 2024

Add examples based on default-sql & dbt-sql templates databricks/bundle-examples#23

Merged

andrewnester mentioned this pull request Feb 20, 2024

Release v0.214.0 #1226

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add an experimental dbt-sql template#1059

Add an experimental dbt-sql template#1059
lennartkats-db merged 29 commits into
databricks:mainfrom
lennartkats-db:dbt-template

lennartkats-db commented Dec 12, 2023 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pietern Feb 5, 2024

Uh oh!

lennartkats-db Feb 19, 2024

Uh oh!

lennartkats-db Feb 19, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

lennartkats-db commented Dec 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Tests

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pietern Feb 5, 2024

Choose a reason for hiding this comment

Uh oh!

lennartkats-db Feb 19, 2024

Choose a reason for hiding this comment

Uh oh!

lennartkats-db Feb 19, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lennartkats-db commented Dec 12, 2023 •

edited

Loading