December 7, 2023 | By Advait Bhadane

In today’s data-driven world a cutting-edge platform is required that seamlessly integrates with the cloud, embraces open-source innovation and prioritises robust data security. Databricks is a pioneer in this field. Not only does it provide a unified lake house platform, but it takes data protection to the next level with its Security Analysis Tool (SAT).

In this blog, we will unravel the power of Databricks’ SAT, focusing on the pivotal role it plays in generating daily health reports for your workspaces. It will also walk you through the step-by-step process of setting SAT in your workspace.

Security Analysis Tool Features

Databricks’ Security Analysis Tool has a set of features which helps in providing the highest level of data security within your workspaces. Let’s look at some key features of Databricks Security Analysis Tool:

  • Routine Scans: SAT conducts routine scans of your Databricks workspace, identifying potential threats.
  • Daily Health Reports: The Daily Health Reports helps in giving us insights of the security status of our workspace and areas that require attention.
  • Alerting System: The customisable alerting system gives us real-time notifications, empowering you to respond promptly.
  • Automated insights: SAT helps in identifying security issues and giving us automated insights and addresses security concerns effectively.
  • Seamless Integration with Databricks Platform: SAT integration ensures that security measures are embedded seamlessly into your existing processes without any disturbance.

https://cms.databricks.com/sites/default/files/inline-images/db-381-blog-img-1.png

Why should we implement Databricks Security Analysis Tool?

To implement the Databricks Security Analysis Tool, consider the following key points:

  • The Databricks workspace Security Analysis Tool is pivotal in enhancing the security posture of your data environment.
  • It demonstrates strong protection against security risks and vulnerabilities.
  • Receive instant alerts in case of issues and make sure that your data remains secure.

Implementation

Admin access of the Databricks and Premium pricing tier is required to set up SAT. Here’s a step-by-step guide for implementing SAT and using its powerful features.

  • Login into your Azure account and to your Databricks Workspace.
  • Setup a single user cluster according to your requirements.
  • From SQL Datawarehouse note down the warehouse ID.

  • Import SAT git repo into the Databricks repos: Security Analysis Tool
  • Confirm if PyPl access is available.
  • Create Secrets Scopes:
    • Setup Databricks CLI.
    • Create a separate profile e2 which will communicate with your workspace:

    • You can also list down all the folders in your workspace by using following command:

    • Set up the secret scope:

  • Your config in <SAT Project>/notebooks/Utils/initialize CMD 4 should look like this if you are not using the secrets.

  • Your config in <SAT Project>/notebooks/Utils/initialize CMD 4 should look like this if you are using the secrets (Required for TF deployments), no need to edit the cell:

  •  Add a service principle to each workspace using the admin console. This is so, the security analysis tool can reach the specific workspace and access the configuration.

Security Analysis Initializer

  • First make sure your cluster is running. After running the notebook, you will see the see the security analysis analyser complete processing and configure the workspaces so that it is ready for analysis. After looking at the sub jobs that are executed by this notebook:
  • List the account workspaces: it uses the service principle that was provided as a part of the configuration, contacts the azure APIs, gets all the workspace information and configures the workspaces under the configuration called configs.csv.
  • It runs a test connection to those workspaces and enables them to successfully connect using that service principle and ignores all other workspaces from analysis. It also imports the dashboard template.
  • It configures the alerts for generating any type of deviations when a specific best practice is configured for alerts.

Security Analysis Driver

  • After executing the analysis driver notebook. You can look at the dashboard. To verify the report the dashboard uses a SQL warehouse. Once the SQL Warehouse is online you will be able to see the SAT report for configured workspace.

Final Thoughts 

Databricks SAT is helpful in protecting your data. By using its features, you can avoid probable risks, receive alerts in case of issues and ensure that your data and workspace always remain secure.