Add a Salesforce-Managed Private Network Route for Databricks on Azure

Prepare for Databricks Integration

To make sure you’re prepared to integrate Data 360 with Databricks, verify you have completed all of the required steps.

• A workspace running on the Azure Databricks Premium Tier.
• An Owner or Contributor role on the Azure subscription that contains the Databricks workspace and on the workspace's managed resource group.
• Workspace admin access in Azure Databricks.
• Your Azure Databricks workspace can accept inbound front-end Private link connections. Front-end Private Link secures all external connections to the workspace, including JDBC connections from Data 360. It doesn't require VNet injection or back-end (classic compute plane) Private Link. For more information, see Configure Inbound Private Link in the Azure Databricks documentation.
• The Databricks Workspace Storage Account is accessible. The Databricks JDBC driver uses Cloud Fetch to download SQL warehouse query results. Databricks temporarily stages query result data in the workspace's Databricks workspace storage account, a separate location from where your data persists. Private Connect Multicloud provisions a separate private endpoint to that storage account so that result data returns over the private connection. For more information about Cloud Fetch, see How We Achieved High-bandwidth Connectivity With BI Tools on the Databricks blog. For more information on allowing access to the storage account, see Connect services outside Azure Databricks to the storage account.
• A SQL warehouse for Data 360 to connect to. See Create an SQL Warehouse in the Azure Databricks documentation.

Step 1: Gather Databricks Information

Gather the Azure Databricks configuration values that Data 360 requires to establish the private network connection.

1. In the Azure portal, as a user with read access to the Databricks workspace and its managed resource group, open your Azure Databricks workspace resource. Select the Overview page and record:
   1. The workspace URL, in the format adb-{workspace-id}.{n}.azuredatabricks.net. Don't include https:// or a trailing /.
   2. The Managed Resource Group Name. Azure Databricks creates this resource group when you create the workspace and then stores the workspace's Databricks workspace storage account inside it. You need the Managed Resource Group name to locate the storage account resource in step 1c of this section.
2. Open the JSON view of the workspace resource and record the workspace's full Azure resource ID.
3. If your Databricks workspace storage account root is private, perform this step. If it is publicly accessible, you can skip this step. Open the workspace's managed resource group. Find the Storage account resource, which holds the Databricks workspace root storage account for the workspace. Open the storage account resource and record:
   1. The Storage account name from the Overview page.
   2. The full Azure resource ID from the storage account's JSON view.
4. In Azure Databricks, as a workspace admin:
   1. Select SQL Warehouses and select the SQL warehouse that Data 360 connects to.
   2. On the warehouse details page, select the Connection details tab and record the HTTP path.

Step 2: Create the Private Network Route in Data 360

Create the Salesforce-managed private network route in Data 360.

1. From the App Launcher, select Data Cloud.
2. From Setup, in the Quick Find box, enter Data Cloud Setup, and then select Data Cloud Setup.
3. Expand Admin Tools, select Private Connect, and click New.
4. On the Add Private Network Route page, select the Databricks tile.
5. In the Select a Network panel, select Azure, and then click Add Private Network Route
6. From the Choose Management menu, select Salesforce-Managed, and then click Next.
7. On the Configure Databricks Private Network Route page, enter these values:
   1. Route Name: Enter a name that identifies the route.
   2. Route API Name: Data 360 autopopulates this field based on the route name.
   3. Description: Optionally enter a description of the connection.
8. Under Step 1: Account, enter these values:
   1. Databricks Workspace URL: Enter the workspace URL you recorded earlier Enter the URL in the format adb-{workspace-id}.{n}.azuredatabricks.net. Don't include https://. This is the standard workspace URL; Azure Private DNS resolves it to a private IP when front-end Private Link is configured. There's no separate privatelink.azuredatabricks.net URL to use.
   2. Databricks Workspace Azure Resource ID: Enter the workspace's full Azure resource ID you recorded earlier.
9. If your Databricks workspace storage account is private, perform this step. If it is publicly accessible, you can skip this step. Under Step 2: Internal Stage, enter these values:
   1. Storage Account URL: Enter the storage account URL in the format {storage-account-name}.blob.core.windows.net. Don't include https://. Azure Private DNS resolves this to a private IP when the private endpoint is configured. There's no separate privatelink URL.
   2. Databricks Internal Stage Azure Resource ID: Enter the storage account's full Azure resource ID you recorded earlier.
10. Save your configuration settings.

Salesforce starts provisioning two Azure private endpoints if you didn’t opt out of private network storage—one to the Databricks workspace front end, and one to the workspace's Databricks workspace root storage account. Data 360 creates two private network route records: one named after your route (the workspace endpoint) and one named {route-name}_internalstage (the storage account endpoint). Each PNR shows Waiting for the Azure Resource ID to be provisioned until provisioning completes. Provisioning is complete when each private network route details page shows an Azure Resource ID value. Provisioning takes about 10 minutes. The only way to get the latest status state is to click the on-screen refresh button along the top right corner of the grid.

Note After you create the PNR, you can edit only the name and description fields.

Step 3: Authorize Endpoints and Validate the Connection

After provisioning completes, authorize the private endpoints in Azure and validate the connection.

1. Follow these steps to approve the Databricks workspace private endpoint.
   1. On the workspace private network route details page in Data 360, copy the Azure Resource ID.
   2. In the Azure portal, open your Azure Databricks workspace resource.
   3. In the left navigation, select Networking, and then select the Private endpoint connections tab.
   4. Locate the pending connection corresponding to Data 360. Select Approve.
   5. Optionally, enter a description.
2. Follow these steps to approve the Databricks internal stage private endpoint. Note that this step is only required if your Databricks workspace root storage account has public network access disabled. If your storage account allows public access, you can skip this step.
   1. In Data 360, from the {route-name}_internalstage private network route details page, copy the Azure Resource ID for the endpoint.
   2. In the Azure portal, open the Databricks workspace root storage account in the workspace's managed resource group.
   3. In the left navigation, select Networking, and then select the Private endpoint connections tab.
   4. Locate the pending connection corresponding to Data 360. Select Approve.

Validate the Connection

Follow these steps to confirm that all Salesforce-managed private endpoints are approved and operational. Depending on your setup, you may have one or two endpoints.

1. In the Azure portal, on the Databricks workspace's Private endpoint connections page, the Salesforce connection shows Connection state: Approved and Provisioning state: Succeeded.
2. In the Azure portal, on the Databricks workspace storage account’s Private endpoint connections page (if applicable), the Salesforce connection shows Connection state: Approved and Provisioning state: Succeeded.
3. In Data Cloud, both PNR records show Status: Ready.

Step 4 (Optional): Security Hardening to Restrict Workspace Access to Private Connections

Private Connect Multicloud works whether or not your Azure Databricks workspace and Databricks workspace storage account allow public network access. In other words, you can use Private Connect without disabling public access. The steps in this section are optional security hardening for teams that want to make sure all traffic to the workspace or storage account flows exclusively over private endpoints. Plan and coordinate these changes with your Azure Databricks and data lake administrators. These changes affect every user and application, private or public, that accesses your workspace or storage account.

1. In the Azure portal, open your Azure Databricks workspace resource.
2. Under Settings, select Networking.
3. Set Public Network Access to Disabled, and Save your changes.

Step 5 (Optional): Security Hardening to Restrict Databricks Workspace Storage Account Access to Private Connections

To enforce that traffic to the workspace's Databricks workspace storage account flows only over private endpoints, disable public network access on the storage account. This change affects all compute resources, users, and tools that access the storage account, not just Salesforce. Coordinate with your Azure Databricks and data lake administrators before proceeding. The workspace's compute resources need a permitted network path to reach this storage account.

1. In the Azure portal, open the workspace's Databricks workspace storage account in the managed resource group
2. Under Security + networking, select Networking.
3. Set Public Network Access to Disabled, and Save your changes.

Important Don't disable storage account key access on the Databricks workspace storage account. Disabling that setting causes unexpected behavior and errors in Azure Databricks. See Recommendations for working with DBFS root in the Azure Databricks documentation.