Translate code_source git and include_paths in convert-to-dabs

[vinchenzo-db]

Summary

experimental air convert-to-dabs previously rejected a code_source.snapshot that
pinned a git ref or narrowed to include_paths — it errored and told the user to work
around it. Now it translates them into a tgz artifact (added in #6428): DABs builds
the tarball from the git ref / include subset at deploy, and code_source_path points
at the built tarball.

This closes the conversion gaps for the two cases convert used to refuse.

Mapping

For a snapshot that pins a git ref and/or include_paths, convert emits a tgz
artifact and points code_source_path at its output:

• snapshot.git.{branch,commit} → artifact git.{branch,commit}
• snapshot.include_paths → artifact include

Archive entries must nest under the code directory's basename (the runtime extracts to
/databricks/code_source/<dir>), so convert emits path = the code dir's parent and
include = basename-prefixed subpaths:

• root_path: ./src → path: ".", include: ["src"]
• root_path: ./src, include_paths: [foo] → path: ".", include: ["src/foo"]

A plain snapshot (no git / include_paths) is unchanged: code_source_path stays the
source directory, packaged at deploy. remote_volume is still rejected — a per-source
volume isn't representable in a bundle (set workspace.artifact_path instead).

Testing

Unit tests assert the emitted artifact (type, path, git/include, files) and
the rewritten code_source_path for both the git-ref and include_paths cases; the
plain-snapshot path is covered by the existing convert acceptance golden.

E2E test:
TEST INCLUDE
Setup

$ rm -rf /tmp/pr2-include && mkdir -p /tmp/pr2-include/src && cd /tmp/pr2-include
printf 'print("include ok")\n' > src/train.py
printf 'python train.py\n'      > src/command.sh
printf 'excluded\n'             > src/extra.txt
cat > train.yaml <<'YAML'
experiment_name: pr2_include_demo
command: python train.py
compute: {accelerator_type: GPU_1xA10, num_accelerators: 1}
code_source:
  type: snapshot
  snapshot:
    root_path: ./src
    include_paths: [train.py, command.sh]
YAML

convert to dabs

$ /tmp/cli-pr2 experimental air convert-to-dabs train.yaml
Wrote a Databricks Asset Bundle to .:
  databricks.yml
  generated_artifacts/training_config.yaml
  generated_artifacts/command.sh

To deploy and run this workload as a bundle:
  1. /tmp/cli-pr2 bundle validate
  2. /tmp/cli-pr2 bundle deploy
  3. /tmp/cli-pr2 bundle run pr2_include_demo --no-wait

bundle deploy uploads the code source and launch scripts automatically.
To see what it deployed and where: /tmp/cli-pr2 bundle summary

Unlike `air run` (which submits an ephemeral run), bundle deploy creates a
persistent job that is not garbage-collected. When you are done, remove the
job and its uploaded files with:
  /tmp/cli-pr2 bundle destroy

investigate artifacts:

$ cat databricks.yml
cat generated_artifacts/command.sh
bundle:
  name: pr2_include_demo
sync:
  paths:
    - generated_artifacts
artifacts:
  code_source:
    type: tgz
    path: .
    include:
      - src/train.py
      - src/command.sh
    files:
      - source: ./dist/code_source.tgz
targets:
  dev:
    mode: development
    default: true
resources:
  jobs:
    pr2_include_demo:
      name: pr2_include_demo
      tasks:
        - task_key: pr2_include_demo
          environment_key: default
          max_retries: 3
          ai_runtime_task:
            experiment: pr2_include_demo
            deployments:
              - command_path: ./generated_artifacts/command.sh
                compute:
                  accelerator_type: GPU_1xA10
                  accelerator_count: 1
            code_source_path: ./dist/code_source.tgz
      environments:
        - environment_key: default
          spec:
            environment_version: "4"
cd /databricks/code_source/src
python train.py%  

deploy:

$ /tmp/cli-pr2 bundle validate && /tmp/cli-pr2 bundle deploy
Name: pr2_include_demo
Target: dev
Workspace:
  User: v.chen@databricks.com
  Path: /Workspace/Users/v.chen@databricks.com/.bundle/pr2_include_demo/dev

Validation OK!
Building code_source...
Uploading dist/code_source.tgz...
Uploading bundle files to /Workspace/Users/v.chen@databricks.com/.bundle/pr2_include_demo/dev/files...
Files: 2 uploaded, 0 deleted
Resources: 0 created, 0 changed, 0 deleted, 1 unchanged
# v.chen at ip-10-90-20-219 in /tmp/pr2-include (git:) [19:22:29]
$ tar tzf dist/code_source.tgz
src/command.sh
src/train.py

run:

# v.chen at ip-10-90-20-219 in /tmp/pr2-include (git:) [19:22:34]
$ /tmp/cli-pr2 bundle summary
Name: pr2_include_demo
Target: dev
Workspace:
  User: v.chen@databricks.com
  Path: /Workspace/Users/v.chen@databricks.com/.bundle/pr2_include_demo/dev
Resources:
  Jobs:
    pr2_include_demo:
      Name: [dev v_chen] pr2_include_demo
      URL:  https://e2-dogfood.staging.cloud.databricks.com/jobs/212600874211444?w=6051921418418893
# v.chen at ip-10-90-20-219 in /tmp/pr2-include (git:) [19:22:39]
$ /tmp/cli-pr2 bundle run pr2_include_demo --no-wait
Run URL: https://e2-dogfood.staging.cloud.databricks.com/jobs/212600874211444/runs/882449706897243?o=6051921418418893

TEST GIT
Setup dirty git:

# v.chen at ip-10-90-20-219 in /tmp/pr2-include (git:) [19:22:48]
$ rm -rf /tmp/pr2-git && mkdir -p /tmp/pr2-git/src && cd /tmp/pr2-git
git init -q -b main
printf 'print("git v1 committed")\n' > src/train.py
printf 'python train.py\n'           > src/command.sh
git add -A && git commit -qm "v1"
Databricks pre-commit Git Hook V2.5.0
Running secret scanning on changes staged for commit.
secret-scan hook completed in 74 ms
Unknown project name: None, skipping linting.
pre-commit-total hook completed in 90 ms
Databricks commit-msg Git Hook V2.5.0
Running secret scanning on commit message.
secret-scan hook completed in 23 ms
commit-msg-total hook completed in 29 ms
# v.chen at ip-10-90-20-219 in /tmp/pr2-git (git:main) [19:26:30]
$ printf 'print("v2 UNCOMMITTED - should NOT appear")\n' > src/train.py

cat > train.yaml <<'YAML'
experiment_name: pr2_git_demo
command: python train.py
compute: {accelerator_type: GPU_1xA10, num_accelerators: 1}
code_source:
  type: snapshot
  snapshot:
    root_path: ./src
    git: {branch: main}
YAML

convert to dabs and investigate artifact:

$ /tmp/cli-pr2 experimental air convert-to-dabs train.yaml
cat databricks.yml
Wrote a Databricks Asset Bundle to .:
  databricks.yml
  generated_artifacts/training_config.yaml
  generated_artifacts/command.sh

To deploy and run this workload as a bundle:
  1. /tmp/cli-pr2 bundle validate
  2. /tmp/cli-pr2 bundle deploy
  3. /tmp/cli-pr2 bundle run pr2_git_demo --no-wait

bundle deploy uploads the code source and launch scripts automatically.
To see what it deployed and where: /tmp/cli-pr2 bundle summary

Unlike `air run` (which submits an ephemeral run), bundle deploy creates a
persistent job that is not garbage-collected. When you are done, remove the
job and its uploaded files with:
  /tmp/cli-pr2 bundle destroy
bundle:
  name: pr2_git_demo
sync:
  paths:
    - generated_artifacts
artifacts:
  code_source:
    type: tgz
    path: .
    git:
      branch: main
    include:
      - src
    files:
      - source: ./dist/code_source.tgz
targets:
  dev:
    mode: development
    default: true
resources:
  jobs:
    pr2_git_demo:
      name: pr2_git_demo
      tasks:
        - task_key: pr2_git_demo
          environment_key: default
          max_retries: 3
          ai_runtime_task:
            experiment: pr2_git_demo
            deployments:
              - command_path: ./generated_artifacts/command.sh
                compute:
                  accelerator_type: GPU_1xA10
                  accelerator_count: 1
            code_source_path: ./dist/code_source.tgz
      environments:
        - environment_key: default
          spec:
            environment_version: "4"

deploy:

$ /tmp/cli-pr2 bundle validate && /tmp/cli-pr2 bundle deploy
Name: pr2_git_demo
Target: dev
Workspace:
  User: v.chen@databricks.com
  Path: /Workspace/Users/v.chen@databricks.com/.bundle/pr2_git_demo/dev

Validation OK!
Building code_source...
Uploading dist/code_source.tgz...
Uploading bundle files to /Workspace/Users/v.chen@databricks.com/.bundle/pr2_git_demo/dev/files...
Created jobs.pr2_git_demo
Files: 2 uploaded, 0 deleted
Resources: 1 created, 0 changed, 0 deleted, 0 unchanged

Prove we properly exclude dirty git:

# v.chen at ip-10-90-20-219 in /tmp/pr2-git (git:main) [19:28:02]
$ mkdir -p /tmp/pr2-git-check && tar xzf dist/code_source.tgz -C /tmp/pr2-git-check
grep -R "committed" /tmp/pr2-git-check/src/train.py    # -> "git v1 committed"
grep -R "UNCOMMITTED" /tmp/pr2-git-check/src/train.py || echo "OK: uncommitted change correctly excluded"
print("git v1 committed")
OK: uncommitted change correctly excluded

run:

$ /tmp/cli-pr2 bundle summary
Name: pr2_git_demo
Target: dev
Workspace:
  User: v.chen@databricks.com
  Path: /Workspace/Users/v.chen@databricks.com/.bundle/pr2_git_demo/dev
Resources:
  Jobs:
    pr2_git_demo:
      Name: [dev v_chen] pr2_git_demo
      URL:  https://e2-dogfood.staging.cloud.databricks.com/jobs/235795448342543?w=6051921418418893
# v.chen at ip-10-90-20-219 in /tmp/pr2-git (git:main) [19:28:47]
$ /tmp/cli-pr2 bundle run pr2_git_demo --no-wait
Run URL: https://e2-dogfood.staging.cloud.databricks.com/jobs/235795448342543/runs/137864060324526?o=6051921418418893

[ben-hansen-db]

if there's git: but not branch or commit what is the behavior? Do we error in CLI? Does translation fail?

[vinchenzo-db]

It would fail when user runs convert-to-dabs before any sort of dab command

[vinchenzo-db]

throws git: must specify either 'branch' or 'commit'

[ben-hansen-db]

can we make sure that include_paths tests directories? That is main use case not individual files

[ben-hansen-db]

I am confused by this. Why is it needed?

[ben-hansen-db]

Not clear to me what this is fixing

[vinchenzo-db]

The launcher runs command.sh from the launch directory, but code_source is extracted to /databricks/code_source/<dir>. If no cd, a relative command like python train.py runs from the launch dir and can't find the file.

I think running python train.py is what most users would assume and they would be confused why they have to cd into /databricks/code_source/<dir> before running.

^ This was a requirement raised by Pieter in the previous pr because we are making tgz a generic artifact type, we need to follow DABs semantics for what they do with the other artifact types which I agree with. This comment

[ben-hansen-db]

isn't that what we do currently in CLI? Why is there a difference with DABs

[ben-hansen-db]

from claude:

Issue: With root_path: ., codeDirRel is ".", so path.Base → "." and path.Dir → ".", yielding include: ["."] and an injected cd /databricks/code_source/..
Fix: Guard/normalize the root_path == "." case or confirm it's unreachable.