Demystifying secrets, scopes and acls in Databricks CLI
Introduction
As a data engineer, you’ll eventually hit a point where the Databricks UI just isn’t enough. Maybe you need to automate something, manage environments at scale, or work with features hidden behind the API — like secret scopes or ACLs.
That’s where the Databricks CLI becomes a superpower.
In this guide, I’ll walk you through:
- installing the CLI,
- securely authenticating,
- configuring profiles,
- and using it to manage secrets and scopes properly
- last but not least, use those secrets within Databricks.
This is the exact setup I use in real projects, and I’ll share a few tips that saved me a lot of headaches along the way.
To Begin: Install the Databricks CLI
On Windows, the simplest method is winget. Open PowerShell or CMD and run:
Check that the Databricks CLI is working correctly by running:
If you see a version number, you’re ready to go.
Authenticating with Databricks CLI
Set up authentication next. While Personal Access Tokens (PATs) are quick to generate and work for basic workspace tasks, they’re less secure and can’t reach account-level resources such as the Databricks Account Console—so account admins should avoid PATs for that purpose.
User‑to‑Machine (U2M) authentication is the recommended option: it’s more secure, supports account-level operations, and is required if you plan to access the Account Console. If you’re an account admin, follow steps 1–5 in the official U2M guide with your Databricks workspace URL and then run the provided command in your terminal.
If you still need a PAT (for example, for a short-lived script in a single workspace), generate one in the workspace UI, capture the workspace URL and token value, and run the same CLI login command, choosing the PAT flow when prompted.
After you run the authentication command, your browser opens to the Databricks workspace so you can confirm your identity. If you aren’t signed in yet, you’ll be prompted to log in there. Once that completes, the CLI drops a .databrickscfg file in your user folder root, storing the profile and credentials it will reuse for future commands.
Configuring the CLI Profile
Open the .databrickscfg file and you’ll see the profile that was just created during authentication, including the workspace URL, credentials, and any other settings the CLI needs. You can add multiple profiles in this file—dev, staging, prod, or dedicated sandboxes such as: [info] —and then target whichever one you need by passing –profile
