error connecting to azure data lake in azure data factory - azure-data-factory

Question

I am trying to create a linked service in Azure Data Factory to an Azure Data Lake Storage Gen2 data store. Below is my linked service configuration:
I get the following error message when I test the connection:
Error code 24200 Details ADLS Gen2 operation failed for: Storage
operation '' on container 'testconnection' get failed with 'Operation
returned an invalid status code 'Forbidden''. Possible root causes:
(1). It's possible because some IP address ranges of Azure Data
Factory are not allowed by your Azure Storage firewall settings. Azure
Data Factory IP ranges please refer
https://learn.microsoft.com/en-us/azure/data-factory/azure-integration-runtime-ip-addresses..
I have found a very similar question here, but I'm not using Managed Identity as my authentication method. Perhaps I should be using that method. How can I overcome this error?

Answer 1

I tried to create a linked service to my Azure Data Lake storage and when I test its connection, it gives me the same error.
Error code 24200 Details ADLS Gen2 operation failed for: Storage
operation '' on container 'testconnection' get failed with 'Operation
returned an invalid status code 'Forbidden''. Possible root causes:
(1). It's possible because some IP address ranges of Azure Data
Factory are not allowed by your Azure Storage firewall settings. Azure
Data Factory IP ranges please refer
https://learn.microsoft.com/en-us/azure/data-factory/azure-integration-runtime-ip-addresses
As indicated by the Possible root causes in the error details, this occurred because of the Azure data lake storage account firewall settings.
Navigate to your data lake storage account, go to Networking -> Firewalls and virtual networks.
Here, when the public network access is either disabled or enabled from selected virtual networks and IP addresses, the linked service creation fails with the above specified error message.
Change it to Enabled from all networks save the changes and try creating the linked service again.
When we test the connection before creating the linked service, it will be successful, and we can proceed to create it.
UPDATE:
In order to proceed with a data lake storage with public access enabled from selected virtual netowrks and IP addresses to create a successful connection via linked service, you can use the following approach.
Assuming your data lake storage has public network access enabled from selected virtual netowrks and IP addresses, first create an integration runtime in your azure data factory.
In your data factory studio, navigate to Manage -> Integration Runtime -> New. Select Azure,self hosted as the type of integration runtime.
Select Azure in the next window and click continue. Enter the details for integration runtime
In the virtual network tab, enable the virtual network configuration and check the interactive authoring checkbox.
Now continue to create the Integration runtime. Once it is up and running, start creating the linked service for data lake storage.
In Connect via integration runtime, select the above created IR. In order to complete the creation, we also need to create a managed private endpoint (It will be prompted as shown in the image below).
Click Create new, with account selection method as From azure subscription, select the data lake storage you are creating the linked service to and click create.
Once you create this, a private endpoint request will be sent to your data lake storage account. Open the storage account, navigate to Networking -> Private endpoint connections. You can see a pending request. Approve this request.
Once this is approved, you can successfully create the linked service where your data lake storage allows access on selected virtual networks and IP addressess.

Answer 2

The error has occurred because of firewall and network access restriction. One way to overcome this error is by adding your client ip to the firewall and network setting of your storage account. Navigate to your data lake storage account, go to Networking -> Firewalls and virtual networks. Under firewall option click on "Add your client ip address"