I have added code for Azure datafactory custom activity in Azure batch service and pointed the datafactory pipeline to the bacth service. When I execute the code in local environment, it works fine. But when I upload it to run in azure batch service, it throws and sqlexception
System.Data.SqlClient.SqlException: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections.
Today, Custom activity cannot access on-prem resource. The data movement gateway can only be used on copy\Sproc activity scenario, and it doesn’t have interface to execute customer’s code.
The solution here is:
Try copy activity to copy your data to azure storage or other public cloud can access. Then run custom activity.
Otherwise you can try vNet and ExpressRoute to connect your Azure public cloud with your onprem environment.
Related
So I want to create a copy activity in Azure Data factory. From PostgreSQL to Azure blob storage.
As my vm (postgreSQL) is in a private subnet in Azure.
So my question is, is it possible to create a pipeline from a vm which is in a private subnet?
updates
So currently this is the situation.
I have created a private endpoint postgresql-2-data-storage and now I want to connect datafactory to my vm which is in a Azure vNet with a private ip address 172.16.101.4
when I click on create new Linked service, I don't see Azure resource(vNet) or private endpoint.
PostgreSQL to blob storage if PostgreSQL is in a private subnet
AFAIK, to Assess the database from On-premises or from Azure private network you need to configure a self-hosted integration runtime to connect to it.
Using Azure Private Link, you can connect to various platform as a service (PaaS) deployment in Azure via a private endpoint and to access data from Private network, you need to Create private endpoint on Azure data factory an add that endpoint to same virtual network where your VM is present.
Go to your ADF settings >> Networking >> Private endpoint connection >> Private endpoint.
Then fill all details an configure it. after this install SHIR in your VM and connect your PostgreSQL to Data factory
Follow this document To Install Self-Hosted Integration Runtime on Azure VM by using Private EndPoint for more information.
I am trying to create a linked service in Azure Data Factory to an Azure Data Lake Storage Gen2 data store. Below is my linked service configuration:
I get the following error message when I test the connection:
Error code 24200 Details ADLS Gen2 operation failed for: Storage
operation '' on container 'testconnection' get failed with 'Operation
returned an invalid status code 'Forbidden''. Possible root causes:
(1). It's possible because some IP address ranges of Azure Data
Factory are not allowed by your Azure Storage firewall settings. Azure
Data Factory IP ranges please refer
https://learn.microsoft.com/en-us/azure/data-factory/azure-integration-runtime-ip-addresses..
I have found a very similar question here, but I'm not using Managed Identity as my authentication method. Perhaps I should be using that method. How can I overcome this error?
I tried to create a linked service to my Azure Data Lake storage and when I test its connection, it gives me the same error.
Error code 24200 Details ADLS Gen2 operation failed for: Storage
operation '' on container 'testconnection' get failed with 'Operation
returned an invalid status code 'Forbidden''. Possible root causes:
(1). It's possible because some IP address ranges of Azure Data
Factory are not allowed by your Azure Storage firewall settings. Azure
Data Factory IP ranges please refer
https://learn.microsoft.com/en-us/azure/data-factory/azure-integration-runtime-ip-addresses
As indicated by the Possible root causes in the error details, this occurred because of the Azure data lake storage account firewall settings.
Navigate to your data lake storage account, go to Networking -> Firewalls and virtual networks.
Here, when the public network access is either disabled or enabled from selected virtual networks and IP addresses, the linked service creation fails with the above specified error message.
Change it to Enabled from all networks save the changes and try creating the linked service again.
When we test the connection before creating the linked service, it will be successful, and we can proceed to create it.
UPDATE:
In order to proceed with a data lake storage with public access enabled from selected virtual netowrks and IP addresses to create a successful connection via linked service, you can use the following approach.
Assuming your data lake storage has public network access enabled from selected virtual netowrks and IP addresses, first create an integration runtime in your azure data factory.
In your data factory studio, navigate to Manage -> Integration Runtime -> New. Select Azure,self hosted as the type of integration runtime.
Select Azure in the next window and click continue. Enter the details for integration runtime
In the virtual network tab, enable the virtual network configuration and check the interactive authoring checkbox.
Now continue to create the Integration runtime. Once it is up and running, start creating the linked service for data lake storage.
In Connect via integration runtime, select the above created IR. In order to complete the creation, we also need to create a managed private endpoint (It will be prompted as shown in the image below).
Click Create new, with account selection method as From azure subscription, select the data lake storage you are creating the linked service to and click create.
Once you create this, a private endpoint request will be sent to your data lake storage account. Open the storage account, navigate to Networking -> Private endpoint connections. You can see a pending request. Approve this request.
Once this is approved, you can successfully create the linked service where your data lake storage allows access on selected virtual networks and IP addressess.
The error has occurred because of firewall and network access restriction. One way to overcome this error is by adding your client ip to the firewall and network setting of your storage account. Navigate to your data lake storage account, go to Networking -> Firewalls and virtual networks. Under firewall option click on "Add your client ip address"
I have an Azure SQL server where I wish to deploy my database via dacpac using GitHub CI/CD. I am using the Azure SQL Deploy action with Service Principal for Azure login action
Due to policy restrictions on the client side, the "Deny Public Network Access" is always enabled and therefore while deploying even though the service principal login works, the GitHub action is unable to add the IP Address to the firewall rule.
We are using Self-Hosted GitHub runners. Is there any workaround to deploying the database via CI/CD under such circumstances where we cannot add the firewall rule to whitelist the agent/runners IP Address?
The solution was to do away with Azure Login action and add self-hosted runner virtual network in the Azure SQL Firewall settings:
The Azure Login action attempts to add IP Address of the runner to the Azure SQL Firewall. Hence, this action must not be used. I removed this action and relied on the second step for seamlessly accessing Azure SQL Database instead.
The Azure SQL Deploy action requires either above login step or "allow access to azure services" to be turned ON. Both of which were not an option for me. I decided to go with an action that runs sqlpackage.exe. https://learn.microsoft.com/en-us/sql/tools/sqlpackage/sqlpackage-publish?view=sql-server-ver15
I am using self-hosted runners which are hosted within the virtual network configured within Azure. However, the point that I had missed was to add the virtual network to the Firewall settings in Azure SQL. Once, I did that, I did not require an explicit login or whitelisting runner ip addresses.
I am trying to create a service connection from Azure Devops on premise to my Azure connection. But I get this error:
Failed to obtain the Json Web Token(JWT) using service principal client ID.
Exception Message: An error occurred while sending the request.
The user is owner in the subscription.
Problem connecting Azure DevOps on-site with Azure
According to the error, it seems that the Service principal key has expired.
You could try to run below script on the Azure DevOps OnPrem machine and can verify if able to generate the access token:
GetAccessToken.ps1
And try to create a new service principal and service principal key to create the new service connection.
BTW, you could check if it works with Azure DevOps service when using the same service connection configuration.
I am trying to copy files from FTPServer to Azure Blob store.I have reffered below link :
https://learn.microsoft.com/en-us/azure/data-factory/v1/data-factory-ftp-connector
The FTPLinked Service gets deployed but when I execute the pipeline, the output dataset fails giving below error:
Even after trying with wizard "Copy Data(preview)" from Azure portal, I am not able to connect to FTPServer.
I am able to connect FTPServer via FileZilla. But when I am trying to access file from Azure Datafactory then it fails.
How to resolve this issue?