Support serverless mode in default-python template by denik · Pull Request #2348 · databricks/cli

denik · 2025-02-12T21:48:00Z

Add 'serverless' setting to default-python template. Default is currently set "no" and the setting is not prompted for.

If set to "auto", it will call /settings-api/ to detect serverless support. It falls back to classic in case of errors on that API.

Depends on #2352.

Tests

Split default-python into two tests default-python/classic and default-python/serverless.
Add acceptance tests for 'auto' mode: returning yes/no/error.
Manually check that "bundle init default-python" does not prompt for serverless.
Manually check that "bundle init default-python" with serverless=auto can be deployed and run on dogfood.

This is needed for b.WorkspaceClient().CurrentWorkspaceID(ctx) which is used by initialize_urls.go mutator ("bundle summary") (#2316). It also also needed for to call serverless endpoint (#2348). Builds on top of #2338

This is needed for b.WorkspaceClient().CurrentWorkspaceID(ctx) which is used by initialize_urls.go mutator ("bundle summary") #2316 It also also needed for to call serverless detection endpoint #2348 Builds on top of #2338

diff.py is like "diff -r -U2" but it applies replacements first to the argument. This allows comparing different output files and directories but ignore differences that are going to be replaced by placeholders. This is useful for tests that record large amount of files, specifically "bundle init" with standard templates. In those tests, changing one parameter results in a small diff so recording the full directory is not helpful, because it's hard to see what changed there. I'm using it in implementation of serverless mode for templates that need it: #2348 Related small changes: add [TESTROOT] replacement for absolute path to acceptance directory in git repo. Add $TESTDIR env var for absolute path to a given test in git repo.

## Changes diff.py is like "diff -r -U2" but it applies replacements first to the argument. This allows comparing different output files and directories but ignore differences that are going to be replaced by placeholders. This is useful for tests that record large amount of files, specifically "bundle init" with standard templates. In those tests, changing one parameter results in a small diff so recording the full directory is not helpful, because it's hard to see what changed there. I'm using it in implementation of serverless mode for templates that need it: #2348 The serverless templates are slightly different from classic, capturing the diff helps to see exactly where. Related small changes: - Add [TESTROOT] replacement for absolute path to acceptance directory in git repo. - Add $TESTDIR env var for absolute path to a given test in git repo. ## Tests - New test acceptance/selftest/diff to test the helper. - Via #2348 which makes use of this feature.

It's not interesting since it just dumps what is in the repo. This is especially annoying with bundle/templates tests with a lot of files.

… Windows

denik · 2025-02-25T15:22:50Z

We should rely on the serverless_jobs_ws_nb_enable settings name, see the slack thread I sent you in a DM

After offline discussion, these APIs may have edge cases so for now the safe thing to do is explicit prompt: #2377

…2377) ## Changes - Add 'serverless' prompt to default-python template (default is currently set to "no"). - This is a simplified version of #2348 with 'auto' functionality removed. ## Tests - Split default-python into default-python/classic, default-python/serverless, default-python/serverless-customcatalog. - Manually check that "bundle init default-python" with serverless=yes can be deployed and run on dogfood and test env.

…atabricks#2377) ## Changes - Add 'serverless' prompt to default-python template (default is currently set to "no"). - This is a simplified version of databricks#2348 with 'auto' functionality removed. ## Tests - Split default-python into default-python/classic, default-python/serverless, default-python/serverless-customcatalog. - Manually check that "bundle init default-python" with serverless=yes can be deployed and run on dogfood and test env.

This is needed for b.WorkspaceClient().CurrentWorkspaceID(ctx) which is used by initialize_urls.go mutator ("bundle summary") #2316 It also also needed for to call serverless detection endpoint #2348 Builds on top of #2338

## Changes diff.py is like "diff -r -U2" but it applies replacements first to the argument. This allows comparing different output files and directories but ignore differences that are going to be replaced by placeholders. This is useful for tests that record large amount of files, specifically "bundle init" with standard templates. In those tests, changing one parameter results in a small diff so recording the full directory is not helpful, because it's hard to see what changed there. I'm using it in implementation of serverless mode for templates that need it: #2348 The serverless templates are slightly different from classic, capturing the diff helps to see exactly where. Related small changes: - Add [TESTROOT] replacement for absolute path to acceptance directory in git repo. - Add $TESTDIR env var for absolute path to a given test in git repo. ## Tests - New test acceptance/selftest/diff to test the helper. - Via #2348 which makes use of this feature.

…2377) ## Changes - Add 'serverless' prompt to default-python template (default is currently set to "no"). - This is a simplified version of #2348 with 'auto' functionality removed. ## Tests - Split default-python into default-python/classic, default-python/serverless, default-python/serverless-customcatalog. - Manually check that "bundle init default-python" with serverless=yes can be deployed and run on dogfood and test env.

denik temporarily deployed to test-trigger-is February 12, 2025 21:48 — with GitHub Actions Inactive

denik mentioned this pull request Feb 12, 2025

acc: Set X-Databricks-Org-Id on scim/v2/Me endpoint #2349

Merged

denik force-pushed the denik/serverless-template branch from 7a261c9 to 5ac7251 Compare February 12, 2025 21:53

denik temporarily deployed to test-trigger-is February 12, 2025 21:53 — with GitHub Actions Inactive

denik changed the base branch from main to denik/acc-x-databricks-org February 12, 2025 21:53

denik temporarily deployed to test-trigger-is February 12, 2025 21:58 — with GitHub Actions Inactive

denik temporarily deployed to test-trigger-is February 12, 2025 22:08 — with GitHub Actions Inactive

denik temporarily deployed to test-trigger-is February 12, 2025 22:10 — with GitHub Actions Inactive

Base automatically changed from denik/acc-x-databricks-org to main February 13, 2025 07:31

denik mentioned this pull request Feb 13, 2025

acc: add a helper to diff with replacements #2352

Merged

denik force-pushed the denik/serverless-template branch from fda87dd to 7d35ea0 Compare February 13, 2025 09:27

denik temporarily deployed to test-trigger-is February 13, 2025 09:27 — with GitHub Actions Inactive

denik changed the base branch from main to denik/acc-diff February 13, 2025 09:29

denik force-pushed the denik/serverless-template branch from 7d35ea0 to d78d857 Compare February 13, 2025 10:09

denik temporarily deployed to test-trigger-is February 13, 2025 10:09 — with GitHub Actions Inactive

denik temporarily deployed to test-trigger-is February 13, 2025 10:23 — with GitHub Actions Inactive

denik temporarily deployed to test-trigger-is February 13, 2025 10:28 — with GitHub Actions Inactive

denik temporarily deployed to test-trigger-is February 13, 2025 12:23 — with GitHub Actions Inactive

denik temporarily deployed to test-trigger-is February 14, 2025 09:38 — with GitHub Actions Inactive

denik force-pushed the denik/serverless-template branch from 7c272e9 to ca39cea Compare February 14, 2025 10:16

denik temporarily deployed to test-trigger-is February 14, 2025 10:16 — with GitHub Actions Inactive

denik temporarily deployed to test-trigger-is February 14, 2025 10:19 — with GitHub Actions Inactive

denik force-pushed the denik/serverless-template branch from 6a3a987 to 5a5b59c Compare February 14, 2025 10:47

denik temporarily deployed to test-trigger-is February 14, 2025 10:47 — with GitHub Actions Inactive

denik temporarily deployed to test-trigger-is February 14, 2025 11:03 — with GitHub Actions Inactive

Base automatically changed from denik/acc-diff to main February 14, 2025 11:08

denik temporarily deployed to test-trigger-is February 24, 2025 09:23 — with GitHub Actions Inactive

denik temporarily deployed to test-trigger-is February 24, 2025 09:39 — with GitHub Actions Inactive

denik added 17 commits February 24, 2025 12:36

Support serverless in default-python

dcee3db

It's not interesting since it just dumps what is in the repo. This is especially annoying with bundle/templates tests with a lot of files.

tweak whitespace

6530657

update default-python-errorserverless

a9c6716

update with regard to libraries

44bdd35

reorg tests

2d65e71

change default to no

94b4700

use_serverless --> serverless

97caa1c

Fix tests; record classic output fully

734021c

add README

e88ee45

rebase & get rid of SaveRepls option

41ffe48

clean up

dca31a7

do not add 'trace' in front of diff.py to avoid slashes difference on…

406cd3f

… Windows

More diagnostics

5d29d84

add 'serverless: true' to pipeline

71b6394

adjust whitespace

654832e

rebase and update acceptance tests

06f0c52

update tests

5706d1e

denik force-pushed the denik/serverless-template branch from cdd4311 to 5706d1e Compare February 24, 2025 11:37

denik temporarily deployed to test-trigger-is February 24, 2025 11:38 — with GitHub Actions Inactive

denik mentioned this pull request Feb 25, 2025

Support serverless mode in default-python template (explicit prompt) #2377

Merged

denik closed this Feb 25, 2025

pietern deleted the denik/serverless-template branch March 3, 2025 09:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support serverless mode in default-python template#2348

Support serverless mode in default-python template#2348
denik wants to merge 17 commits into
mainfrom
denik/serverless-template

denik commented Feb 12, 2025 •

edited

Loading

Uh oh!

denik commented Feb 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

denik commented Feb 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tests

Uh oh!

denik commented Feb 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

denik commented Feb 12, 2025 •

edited

Loading