I am trying to copy files from FTPServer to Azure Blob store.I have reffered below link :
https://learn.microsoft.com/en-us/azure/data-factory/v1/data-factory-ftp-connector
The FTPLinked Service gets deployed but when I execute the pipeline, the output dataset fails giving below error:
Even after trying with wizard "Copy Data(preview)" from Azure portal, I am not able to connect to FTPServer.
I am able to connect FTPServer via FileZilla. But when I am trying to access file from Azure Datafactory then it fails.
How to resolve this issue?
Related
I need to automate pg_dump for a postress server in Azure. I preferably want to use github actions and azure powershell to do this and store the file on an azure storage account
I cant seem to find any docs online which documents the use of github actions and azure powershell to do this and save the exported postgress file to an azure storage account
I would like to copy files with Azure File Copy with Azure Pipeline.
I'm following instruction of https://praveenkumarsreeram.com/2021/04/14/azure-devops-copy-files-from-git-repository-to-azure-storage-account/
I'm using automatically created Service Connection named "My Sandbox (a1111e1-d30e-4e02-b047-ef6a5e901111)"
I'm getting error with AzureBlob File Copy:
INFO: Authentication failed, it is either not correct, or
expired, or does not have the correct permission ->
github.com/Azure/azure-storage-blob-go/azblob.newStorageError,
/home/vsts/go/pkg/mod/github.com/!azure/azure-storage-blob-
go#v0.10.1-0.20201022074806-
8d8fc11be726/azblob/zc_storage_error.go:42
RESPONSE Status: 403 This request is not authorized to perform
this operation using this permission.
I'm assuming that Azure Pipeline have no access to Azure Storage.
I wonder how do find service principal which should get access to Azure Storage.
I can also reproduce your issue on my side, as different Azure file copy task versions use different versions of AzCopy in behind, then they use different auth ways to call the API to do the operations.
There are two ways to fix the issue.
If you use the automatically created service connection, it should have Contributor role in your storage account, you could use Azure file copy task version 3.* instead of 4.*, then it will work.
If you want to use Azure file copy task version 4.*, navigate to your storage account -> Access Control (IAM) -> add your service principal used in the service connection as a Storage Blob Data Contributor role, see detailed steps here. It will also work.
I am trying to implement CICD for MDS via Azure Devops.
Below are the steps which I have followed :
Created a self hosted agent which has SSDT installed
Tried executing the MDS Cmd queries from the VM which I created as a self hosted agent and it was a success :
In Azure Devops, used the self hosted agent and CMD task:
And added the inline query :
cd C:\Program Files\Microsoft SQL Server\110\Master Data Services\Configuration
MDSModelDeploy createpackage -service MDS1 -model SalesBI -package "C:\MDS\CICD\SalesBI_CICD.pkg"
And after execution of the pipeline, I am getting the error :
The web configuration file for the Master Data Services instance
cannot be loaded. An error occurred loading a configuration file:
Access to the path 'C:\Program Files\Microsoft SQL Server\110\Master
Data Services\WebApplication\web.config' is denied. (C:\Program
Files\Microsoft SQL Server\110\Master Data
Services\WebApplication\web.config)
So is there anything which I am missing out while setting up the self hosted agent in Devops because executing the same commands by manually logging into the same VM is getting executed successfully.
In the Azure DevOps, If you want to access local file in the azure devops pipeline via self-hosted agent, It accesses the file via service account instead of Personal account， so you can run the cmd locally from your computer, and get error in the azure devops pipeline.
Workaround
We could open check the file permission and configure the service account permission. service account format User/Administrator/Administrators({Agent.ComputerName}\User/Administrator/Administrators)
Also, we could change the agent service account to your owner account.
Steps: Open service on the agent machine and search the agent service account, check the pic below, just change the account name and password to yours, then It will use this account to perform the operation.
I would like to use a .MDF SQL Server database file on which my unit tests are performed during a Visual Studio Test when deploying via a DevOps pipeline.
I added the .MDF and .LDF files to the unit test project and I am able to execute locally. I have confirmed that the files are properly deployed to the pipeline using a PowerShell script that lists the folder contents. I have also confirmed case-sensitivity by ensuring everything is upper-case.
When adding a Visual Studio Test step to an existing working pipeline, I initially received this error when attempting to save to the database during test initialization:
Microsoft.Data.SqlClient.SqlException: Failed to update database "D:\A\1\S\APPCORETESTS\BIN\RELEASE\NETCOREAPP3.1\MYTESTDATABASE.MDF" because the database is read-only
I added the following call during test initialize:
ALTER DATABASE [MYTESTDATABASE.MDF] SET READ_WRITE
and am now getting the following error:
Unable to open the physical file "D:\a\1\s\AppCoreTests\bin\Release\netcoreapp3.1\MYTESTDATABASE.mdf". Operating system error 5: "5(Access is denied.)"
Unable to open the physical file "D:\a\1\s\AppCoreTests\bin\Release\netcoreapp3.1\myTestDatabase_log.ldf". Operating system error 5: "5(Access is denied.)".
I am able to open a connection to the .MDF file using the following connection string (from a config, hence the two backslash "\"):
"Server=(LocalDB)\\MSSQLLocalDB;AttachDbFileName='|DataDirectory|\\MYTESTDATABASE.MDF'"
and I am able to run a query that returns several databases including mine (as well as tempdb, master, etc):
select name from sys.databases
If this error was occurring locally, I would set the folder permissions for the folder in which the .MDF / .LDF reside, but do not know if that is possible in a DevOps pipeline or if it is the correct approach to solving this problem.
Which agent are you using, hosted agent or self hosted agent?
Azure DevOps Pipeline accesses file via service account, we should check the service account permission and ensure it has enough permission.
If you are using hosted agent, it access the file via this account test Build Service(Org name), we need open project settings->Repositories->select the repo and check the service account test Build Service ({Org name}) permission, and also check the Pipeline->Settings
If you are using self-hosted agent and access local file, you should check the file permission. Steps: Select the local->Properties->Security, the service account name should be Administrators(Agent service name\Administrators) or Users(Agent service name\Users).
By the way, we can change the agent service account to your owner account.
Steps: Open service on the agent machine and search the agent service account, check the pic below, just change the account name and password to yours, then It will use this account to perform the operation.
Step to reproduce:
I created a Copy Data first in the pipeline to simple transfer CSV files frol Azure VM to Azure Blob storage. I always use IRPOC1 as a connection via integration runtime and connect using SAS URI and SAS Token to my Blob Storage
After validate and run my first Copy Data, I successfully have CSV file transfer from my VM to Blob storage
I tried to add a new Data Flow after the Copy Data activity
In my Data Flow, my source is the Blob storage containing the CSV files transferred from VM, my Sink is my Azure SQL Database with successful connection
However, when I ran validation, I got the error message on my Data Flow Source:
Linked Service with self-hosted integration runtime is not supported in data flow.
I saw someone replied on Microsoft Azure Document issue Github that I need to use Copy Data to transfer data to Blob first. Then use the source from this blob with data. This is what I did but I still have the same error. Could you please let me know how I can fix this?
The Data Flow source dataset must use a Linked Service that uses an Azure IR, not a self-hosted IR.
Go to the dataset in your data flow Source, click "Open". In the dataset page, click "Edit" next to Linked Service.
In the Linked Service dialog, make sure you are using an Azure Integration Runtime, not a Self-hosted IR.