Need to connect service principal with certificate to authorize using pyspark - pyspark

Question

I want to connect service principal with certificate to authorize using pyspark. I could see the code in scala in this link - https://github.com/Azure/azure-event-hubs-spark/blob/master/docs/use-aad-authentication-to-connect-eventhubs.md
I have client _id and tenant_id and certificate details. Could some please share me the code in pyspark for same?

Answer 1

You add the Azure AD service principal to the Azure Databricks workspace using the SCIM API 2.0. Authentication using Pyspark isn't available.
To authenticate using service principal, you need to follow below steps:
As you already have clientID and tenantID, so the service principal ID already created.
Create the Azure Databricks personal access token
You’ll use an Azure Databricks personal access token (PAT) to authenticate against the Databricks REST API. To create a PAT that can be used to make API requests:
Go to your Azure Databricks workspace.
Click the user icon in the top-right corner of the screen and click User Settings.
Click Access Tokens > Generate New Token.
Copy and save the token value.
Add the service principal to the Azure Databricks workspace
You add the Azure AD service principal to a workspace using the SCIM API 2.0. You must also give the service principal permission to launch automated job clusters. You can grant this through the allow-cluster-create permission. Open a terminal and run the following command to add the service principal and grant the required permissions:
curl -X POST 'https://<per-workspace-url>/api/2.0/preview/scim/v2/ServicePrincipals' \
--header 'Content-Type: application/scim+json' \
--header 'Authorization: Bearer <personal-access-token>' \
--data-raw '{
"schemas":[
"urn:ietf:params:scim:schemas:core:2.0:ServicePrincipal"
],
"applicationId":"<application-id>",
"displayName": "test-sp",
"entitlements":[
{
"value":"allow-cluster-create"
}
]
}'
Replace with the unique per-workspace URL for your Azure Databricks workspace.
Replace with the Azure Databricks personal access token.
Replace with the Application (client) ID for the Azure AD application registration.
Create an Azure Key Vault-backed secret scope in Azure Databricks
Secret scopes provide secure storage and management of secrets. You’ll store the secret associated with the service principal in a secret scope. You can store secrets in a Azure Databricks secret scope or an Azure Key Vault-backed secret scope. These instructions describe the Azure Key Vault-backed option:
Create an Azure Key Vault instance in the Azure portal.
Create the Azure Databricks secret scope backed by the Azure Key Vault instance.
Step 1: Create an Azure Key Vault instance
In the Azure portal, select Key Vaults > + Add and give the key vault a name.
Click Review + create.
After validation completes, click Create .
After creating the key vault, go to the Properties page for the new key vault.
Copy and save the Vault URI and Resource ID.
Step 2: Create An Azure Key Vault-backed secret scope
Azure Databricks resources can reference secrets stored in an Azure key vault by creating a Key Vault-backed secret scope. To create the Azure Databricks secret scope:
Go to the Azure Databricks Create Secret Scope page at https:///#secrets/createScope. Replace per-workspace-url with the unique per-workspace URL for your Azure Databricks workspace.
Enter a Scope Name.
Enter the Vault URI and Resource ID values for the Azure key vault you created in Step 1: Create an Azure Key Vault instance.
Click Create.
Save the client secret in Azure Key Vault
In the Azure portal, go to the Key vaults service.
Select the key vault created in Step 1: Create an Azure Key Vault instance.
Under Settings > Secrets, click Generate/Import.
Select the Manual upload option and enter the client secret in the Value field.
Click Create.
Grant the service principal read access to the secret scope
You’ve created a secret scope and stored the service principal’s client secret in that scope. Now you’ll give the service principal access to read the secret from the secret scope.
Open a terminal and run the following command:
curl -X POST 'https://<per-workspace-url/api/2.0/secrets/acls/put' \
--header 'Authorization: Bearer <personal-access-token>' \
--header 'Content-Type: application/json' \
--data-raw '{
"scope": "<scope-name>",
"principal": "<application-id>",
"permission": "READ"
}'
Replace with the unique per-workspace URL for your Azure Databricks workspace.
Replace with the Azure Databricks personal access token.
Replace with the name of the Azure Databricks secret scope that contains the client secret.
Replace with the Application (client) ID for the Azure AD application registration.
Create a job in Azure Databricks and configure the cluster to read secrets from the secret scope
You’re now ready to create a job that can run as the new service principal. You’ll use a notebook created in the Azure Databricks UI and add the configuration to allow the job cluster to retrieve the service principal’s secret.
Go to your Azure Databricks landing page and select Create Blank Notebook. Give your notebook a name and select SQL as the default language.
Enter SELECT 1 in the first cell of the notebook. This is a simple command that just displays 1 if it succeeds. If you have granted your service principal access to particular files or paths in Azure Data Lake Storage Gen 2, you can read from those paths instead.
Go to Workflows and click the + Create Job button. Give the job a name, click Select Notebook, and select the notebook you just created.
Click Edit next to the Cluster information.
On the Configure Cluster page, click Advanced Options.
On the Spark tab, enter the following Spark Config:
fs.azure.account.auth.type.acmeadls.dfs.core.windows.net OAuth
fs.azure.account.oauth.provider.type.acmeadls.dfs.core.windows.net org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
fs.azure.account.oauth2.client.id.acmeadls.dfs.core.windows.net <application-id>
fs.azure.account.oauth2.client.secret.acmeadls.dfs.core.windows.net {{secrets/<secret-scope-name>/<secret-name>}}
fs.azure.account.oauth2.client.endpoint.acmeadls.dfs.core.windows.net https://login.microsoftonline.com/<directory-id>/oauth2/token
Replace with the name of the Azure Databricks secret scope that contains the client secret.
Replace with the Application (client) ID for the Azure AD application registration.
Replace with the name associated with the client secret value in the secret scope.
Replace with the Directory (tenant) ID for the Azure AD application registration.
Transfer ownership of the job to the service principal
A job can have exactly one owner, so you’ll need to transfer ownership of the job from yourself to the service principal. To ensure that other users can manage the job, you can also grant Can Manage permissions to a group. In this example, we use the Permissions API to set these permissions.
Open a terminal and run the following command:
curl -X PUT 'https://<per-workspace-url>/api/2.0/permissions/jobs/<job-id>' \
--header 'Authorization: Bearer <personal-access-token>' \
--header 'Content-Type: application/json' \
--data-raw '{
"access_control_list": [
{
"service_principal_name": "<application-id>",
"permission_level": "IS_OWNER"
},
{
"group_name": "admins",
"permission_level": "CAN_MANAGE"
}
]
}'
Replace with the unique per-workspace URL for your Azure Databricks workspace.
Replace with the Azure Databricks personal access token.
Replace with the Application (client) ID for the Azure AD application registration.
The job will also need read permissions to the notebook. Run the following command to grant the required permissions:
curl -X PUT 'https://<per-workspace-url>/api/2.0/permissions/notebooks/<notebook-id>' \
--header 'Authorization: Bearer <personal-access-token>' \
--header 'Content-Type: application/json' \
--data-raw '{
"access_control_list": [
{
"service_principal_name": "<application-id>",
"permission_level": "CAN_READ"
}
]
}'
Replace with the unique per-workspace URL for your Azure Databricks workspace.
Replace with the ID of the notebook associated with the job. To find the ID, go to the notebook in the Azure Databricks workspace and look for the numeric ID that follows notebook/ in the notebook’s URL.
Replace with the Azure Databricks personal access token.
Replace with the Application (client) ID for the Azure AD application registration.
You can now test the job. You run jobs with a service principal the same way you run jobs as a user, either through the UI, API, or CLI.