v0.275.0 is Flagging "CAN_MANAGE_RUN" as an invalid job permission

[barryjpodonovan]

Describe the issue

When running databricks bundle deploy a new error has started appearing that says the only valid permissions are CAN_MANAGEandIS_OWNERfor a job even thoughCAN_MANAGE_RUNandCAN_VIEW` are valid as per:

1. API Reference
2. DABs Job Permission Documentation
3. What can actually be assigned in Databricks UI

Configuration

Extract from databricks.yml

targets:
  feature_branch:
    resources:
      jobs:
        interim_gold_layer_job:
          permissions:
            - group_name: <group name>
              level: CAN_MANAGE
  green:
    resources:
      jobs:
        interim_gold_layer_job:
          permissions:
            - user_name: <user email>
              level: CAN_MANAGE_RUN

Steps to reproduce the behavior

Please list the steps required to reproduce the issue, for example:

1. Run databricks bundle deploy ...
2. See error

Expected Behavior

The expectation is for all permissions as per the API Reference that are applicable to the object as per the DABs Job Permission Documentation to be allowed.

Actual Behavior

Example from my GitHub Workflow log below

Error: cannot create permissions: cannot remove management permissions for the current user for job, allowed levels: CAN_MANAGE, IS_OWNER
with databricks_permissions.job_interim_gold_layer_job,
on bundle.tf.json line 1975, in resource.databricks_permissions.job_interim_gold_layer_job:
1975: },

OS and CLI version

• CLI: v0.275.0
• OS: Linux

Is this a regression?

Yes, this wasn't an issue in v0.275.0

[barryjpodonovan]

This is preventing our upgrade to v0.273.0 as the permissions are causing error

[denik]

Thanks for the report, I added your test case there https://github.com/databricks/cli/pull/3865/files#top

Note, I can only reproduce it if user_email is set to current user email.

Before v0.273 DABs filtered out all permissions for current user. So we actually just dropped the permission CAN_MANAGE_RUN and terraform gave your user IS_OWNER permission.

Now, we keep your permissions in, which creates redundant permissions, which terraform rejects (this seems like a validation is too strict on terraform side).

If you remove CAN_MANAGE_RUN entry, it should work exactly as pre 0.273.

[barryjpodonovan]

@denik -- I might have left out a detail in my issue. The job is owned (and deployed by) our service principal, so we are adding can_manage_run for certain users (referring to them by their email address) as additional permissions for the job.

So I can't remove the can_manage_run in this use case

[denik]

As far as I can see in the code, this error can only happen if you deploy as that user matching CAN_MANAGE_RUN permission.

Can you email full config and debug logs to dabs-feedback@databricks.com?

[denik]

@denik -- I might have left out a detail in my issue. The job is owned (and deployed by) our service principal, so we are adding can_manage_run for certain users (referring to them by their email address) as additional permissions for the job.

Is it possible that the job is deployed both by service principal and also (e.g. in dev targets) by the actual users? So CAN_MANAGE_RUN would work when it's SP deploying it and does not when it's actual user deploying it.

If that's the case, then the right fix would be to remove permissions from the target that is deployed by the user and keep it on target that is run by SP.

Doing some more testing I see that we should not send CAN_MANAGE_RUN and IS_OWNER permissions for the current user, as this will cause permanent drift. So terraform is correct to reject this.

[barryjpodonovan]

@denik No, all our jobs to that workspace are SP deployed and run as. I was off yesterday so I'll see what I can pull today. I don't think I'll have the full logs, but I can pull the ymls

[barryjpodonovan]

@denik While pulling together the info for you today, I figured out the offending line in our bundle (see below). We have two service principals in our workspace (don't get me started on why), but we want both to have VIEW access on all DAB targets. So we give both that permission at the top level, then we specify CAN_MANAGE to the relevant SP in the relevant target. I guess this is somewhat redundant as the SP we are giving CAN_MANAGE to also deploys to the target, but I think the CLI will warn if you don't specify a CAN_MANAGE...

Either way, previously this wasn't an issue, as again I thought the permission of the target overrides the permissions at the top level.

Happy to be corrected on any of this, but this is what causes my error in cli 0.275...

permissions:
  - group_name: <user_group_name>
    level: CAN_VIEW

  # Causes issue in CLI 0.275.0 start
  - service_principal_name: ${var.service_principal_green}
    level: CAN_VIEW
  # Causes issue in CLI 0.275.0 end

  - service_principal_name: ${var.service_principal_blue}
    level: CAN_VIEW

targets:
  feature_branch:
    default: true
    mode: production
    variables:
      schema_prefix: zz_${var.sanitised_branch_name}__
      workflow_prefix: "zz ${var.sanitised_branch_name} | "
      catalog: nonprod_orx_ironsh_ptr
    presets:
      # name_prefix: zz_${var.sanitised_branch_name}__ #[TODO: Re-test this in future to see if they have fixed the bug the opmits the underscore in deployed schema names]
      # pipelines_development: true #as of databricks cli version 0.247.0, this setting produces the following error: Error: target with 'mode: production' cannot include a pipeline with 'development: true'
      trigger_pause_status: PAUSED
      tags:
        branch: ${var.branch_name}
        schema_prefix: zz_${var.sanitised_branch_name}__
        dab_target: Feature Branch
    workspace:
      root_path: ~/.bundle/${bundle.name}/${bundle.target}/${var.sanitised_branch_name}
    permissions:
      - service_principal_name: ${var.service_principal_green}
        level: CAN_MANAGE
    run_as:
      service_principal_name: ${var.service_principal_green}

[barryjpodonovan]

@denik Did the above comment make sense? This seems like a cut and dry bug as the permissions in the target should supersede the top-level permissions, like it did in previous versions of the cli

[denik]

@barryjpodonovan, thanks for the additional info.

So what’s happening is terraform rejects setting multiple permissions for a given principal. The reason they do that is because backend only stores single permission level per principal, so if we allow setting multiple permissions level, we’ll get permanent drift.

Before 0.273. DABs used to simply delete all permissions for current user (and rely on terraform to add IS_OWNER). This made cases like yours work, because it essentially ignored all permissions set by config. This was changed in #3850

Given that’s it’s very easy with DABs to have situation where multiple levels of permissions are set on the same resource/principal, we’ve decided we need to support that and clean up all but the highest level. This would fix your situation. It’s more general than previous “clean up everything for current user” since it also fixes users other than current.

I plan to make a PR before release next week.

[barryjpodonovan]

Awesome. I'll leave my bundle as-is and wait for the fix in a future version.

If it's in anyway related, cleaning up the list of permissions allowed for schemas (issue: v0.273.0 warns of Invalid Schema Permissions that ARE valid or any other resources would be great.

The logs for large bundles are useless given the number of warning generated.