We are getting error in Azure Data Factory Data Flow using Azure Hosted Integration runtime and Private Link linked to Private Link Service and the target is SQL Server behind load balancers.
Job failed due to reason: java.lang.Throwable: DataFlowManagerClient.getPayload fail to getPayload request for run:90e9cb52-f2ba-48fb-ba9f-2a5de66454ab, exception:java.net.UnknownHostException: dmeus2aksig1aks.svc.datafactory.azure.com: Temporary failure in name resolution
What could cause this?
Related
I am trying to create a linked service in Azure Data Factory to an Azure Data Lake Storage Gen2 data store. Below is my linked service configuration:
I get the following error message when I test the connection:
Error code 24200 Details ADLS Gen2 operation failed for: Storage
operation '' on container 'testconnection' get failed with 'Operation
returned an invalid status code 'Forbidden''. Possible root causes:
(1). It's possible because some IP address ranges of Azure Data
Factory are not allowed by your Azure Storage firewall settings. Azure
Data Factory IP ranges please refer
https://learn.microsoft.com/en-us/azure/data-factory/azure-integration-runtime-ip-addresses..
I have found a very similar question here, but I'm not using Managed Identity as my authentication method. Perhaps I should be using that method. How can I overcome this error?
I tried to create a linked service to my Azure Data Lake storage and when I test its connection, it gives me the same error.
Error code 24200 Details ADLS Gen2 operation failed for: Storage
operation '' on container 'testconnection' get failed with 'Operation
returned an invalid status code 'Forbidden''. Possible root causes:
(1). It's possible because some IP address ranges of Azure Data
Factory are not allowed by your Azure Storage firewall settings. Azure
Data Factory IP ranges please refer
https://learn.microsoft.com/en-us/azure/data-factory/azure-integration-runtime-ip-addresses
As indicated by the Possible root causes in the error details, this occurred because of the Azure data lake storage account firewall settings.
Navigate to your data lake storage account, go to Networking -> Firewalls and virtual networks.
Here, when the public network access is either disabled or enabled from selected virtual networks and IP addresses, the linked service creation fails with the above specified error message.
Change it to Enabled from all networks save the changes and try creating the linked service again.
When we test the connection before creating the linked service, it will be successful, and we can proceed to create it.
UPDATE:
In order to proceed with a data lake storage with public access enabled from selected virtual netowrks and IP addresses to create a successful connection via linked service, you can use the following approach.
Assuming your data lake storage has public network access enabled from selected virtual netowrks and IP addresses, first create an integration runtime in your azure data factory.
In your data factory studio, navigate to Manage -> Integration Runtime -> New. Select Azure,self hosted as the type of integration runtime.
Select Azure in the next window and click continue. Enter the details for integration runtime
In the virtual network tab, enable the virtual network configuration and check the interactive authoring checkbox.
Now continue to create the Integration runtime. Once it is up and running, start creating the linked service for data lake storage.
In Connect via integration runtime, select the above created IR. In order to complete the creation, we also need to create a managed private endpoint (It will be prompted as shown in the image below).
Click Create new, with account selection method as From azure subscription, select the data lake storage you are creating the linked service to and click create.
Once you create this, a private endpoint request will be sent to your data lake storage account. Open the storage account, navigate to Networking -> Private endpoint connections. You can see a pending request. Approve this request.
Once this is approved, you can successfully create the linked service where your data lake storage allows access on selected virtual networks and IP addressess.
The error has occurred because of firewall and network access restriction. One way to overcome this error is by adding your client ip to the firewall and network setting of your storage account. Navigate to your data lake storage account, go to Networking -> Firewalls and virtual networks. Under firewall option click on "Add your client ip address"
I was working on connecting dedicated sql pool(formerly sql DWH) to synapse spark notebooks. I was using spark.read.synapsesql(). I'm able to write data as table but not able to read data from the table.
val df:DataFrame = spark.read.option(Constants.SERVER, "XXXXX.database.windows.net")
.option(Constants.USER, "XXXXX")
.option(Constants.PASSWORD, "XXXXX")
.option(Constants.TEMP_FOLDER,"abfss://xxxxx#xxxx.dfs.core.windows.net/Tempfolder/")
.synapsesql("dedicated-poc.dbo.customer"
com.microsoft.spark.sqlanalytics.SQLAnalyticsConnectorException: com.microsoft.sqlserver.jdbc.SQLServerException: External file access failed due to internal error: 'Error occurred while accessing HDFS: Java exception raised on call to HdfsBridge_Connect.
Java exception message: Configuration property XXXXXXXX.dfs.core.windows.net not found.' at com.microsoft.spark.sqlanalytics.ItemsScanBuilder$PlanInputPartitionsUtilities$.extractDataAndGetLocation(ItemsScanBuilder.scala:183)
Permission: we have owner, storage data blob contributor access for synapse and specific user
To resolve the above exception, please try the below:
Try updating the code by adding below:
spark._jsc.hadoopConfiguration().set("fs.azure.account.key.xxxxx.dfs.core.windows.net", "xxxx==")
To read data from table, try including date data type in SQL Pool and then read.
Note:
Synapse RBAC roles do not grant permissions to create or manage SQL pools, Apache Spark pools, and Integration runtimes in Azure Synapse workspaces. Azure Owner or Azure Contributor roles on the resource group are required for these actions.
Give Azure owner role to resource group instead of synapse and specific user.
Check if there is any firewall rule that is blocking the connectivity and disable it.
If still the issue persists, raise a Azure support request
For more in detail, please refer below links:
Azure Synapse RBAC roles - Azure Synapse Analytics | Microsoft Docs
azure databricks - File read from ADLS Gen2 Error - Configuration property xxx.dfs.core.windows.net not found - Stack Overflow
I'm following tutorial to deploy ARM with Azure DevOps pipeline, but getting error. I wonder what is wrong? Error happens with "Azure Resource Manager connection". It find all subsc correctly, but cannot move forward.
https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/deployment-tutorial-pipeline
Failed to obtain the Json Web Token(JWT) using service principal client ID.
Exception Message:
AADSTS700016: Application with identifier '111117a0-1c4f-486f-8765-e19669693333' was not found in the directory '11111041-ba57-4f49-866b-06c297c12222'. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You may have sent your authentication request to the wrong tenant. Trace ID: 1174e46d-22fb-456e-9c18-450c95080b00
Correlation ID: 333c3a0e-42f4-41d7-83c1-f8e3e3a83274 Timestamp: 2020-04-07 10:07:14Z
I created Service Principal automatically and now it works!
I've been getting this "useful" error message when deploying an experimental data factory pipeline that has a DotNetActivity within it.
Entity provisioning failed: The blob supplied is not valid. Blob:
customactivity/package.zip. Error code: . Error message: .
I've been reading trouble shooting tips and double checking connection strings, but to no avail.
After double checking everything in particular blob storage connection strings, I was eventually able to resolve it by copying the connection string from the azure portal -> storage account -> access keys into the azure storage linked service settings.
I can only theorize that there is a subtlety in the connection string format which results in a poor error message from the pipeline provisioning.
Interestingly, I can now put a completely wrong access key into the storage account linked service, redeploy it along with the the pipeline and it doesn't complain about provisioning. So perhaps there is additional checks done on the first provision.
I have added code for Azure datafactory custom activity in Azure batch service and pointed the datafactory pipeline to the bacth service. When I execute the code in local environment, it works fine. But when I upload it to run in azure batch service, it throws and sqlexception
System.Data.SqlClient.SqlException: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections.
Today, Custom activity cannot access on-prem resource. The data movement gateway can only be used on copy\Sproc activity scenario, and it doesn’t have interface to execute customer’s code.
The solution here is:
Try copy activity to copy your data to azure storage or other public cloud can access. Then run custom activity.
Otherwise you can try vNet and ExpressRoute to connect your Azure public cloud with your onprem environment.