I am trying to delete a file that is located in a sharepoint directory after successful copy activity. The Delete Activity is having the following properties:
Linked Service : HTTP
DataSet : Excel
Additional Header: #{concat('Authorization: Bearer ',activity('GetToken').output.access_token)}
Here, GetToken is the Web Activity in ADF that generates a token number for accessing SharePoint.
When I am running the pipeline, I am getting the below error:
Invalid delete activity payload with 'folderPath' that is required and cannot be empty.
I have no clue on how to tackle this.
As per my understanding you are trying to delete a file in Sharepoint online using Azure Data Factory.
Currently delete activity in ADF only supports the below data stores and not sharepoint online. which is why you are receiving the above error.
Azure Blob storage
Azure Data Lake Storage Gen1
Azure Data Lake Storage Gen2
Azure Files
File System
FTP
SFTP
Amazon S3
Amazon S3 Compatible Storage
Google Cloud Storage
Oracle Cloud Storage
HDFS
Image: Delete activity Supported Data stores
Ref: Delete activity supported data sources
As a workaround you may try exploring HTTP connector. OR you can use custom activity and write your own code to delete files from SharePoint.
Hope this info helps.
Related
I need to pull the data from Azure Sentinel in an Incremental manner.
E.g:
step 1: Need a daily login details to my UI from Sentinel(using KQL)
step 2: create a pipeline from ADF
step 3: Load the data in tables
Is there any Lync Services or connectors available for Azure Sentinel or Azure Log Analytics to connect Azure Data Factory?
There are two methods witch depend on the method of authentication to the API.
The first is with a service principle, High level steps:
Import Azure Monitor log data into Azure Data Factory
Blog on the topic: https://datasavvy.me/2020/12/24/retrieving-log-analytics-data-with-data-factory/comment-page-1/#comment-28467
Second is with Managed Identity:
first give ADF access to Log Analytics using IAM How can I use this API in Azure Data Factory
Then connect to Log Analytic API with Web activity or a copy activity (these are the two i got working).
Web Activity
URL: https://api.loganalytics.io/v1/workspaces/[Workspace ID]/query
Body: {"query":"search '*'| where TimeGenerated >= datetime(#{pipeline().parameters.it_startDate}) and TimeGenerated < datetime(#{pipeline().parameters.it_endDate}) | distinct $table "}
Copy Activity
First the linked service.
ADF Datasets:
Base URL: URL: https://api.loganalytics.io/v1/workspaces/[Workspace ID]/
Copy Source:
Body: {
"query": "#{item()[0]} | where TimeGenerated >= datetime(#{pipeline().parameters.it_startDate}) and TimeGenerated < datetime(#{pipeline().parameters.it_endDate})"
}
Additional:
The body code above, gets a list of the table names in log analytics using the web activity. Which I then pass to the Copy Activity to exports copy of the data for each table.
We have used the below Kusto query , to pull the list of login that where happened on a particular computer during the last one day
SecurityEvent
| where TimeGenerated >= ago(1d)
| where Computer == "<vmname>"
You can refer this documentation , for more sample kusto query queries.
Is there any Lync Services or connectors available for Azure Sentinel
or Azure Log Analytics to connect Azure Data Factory?
No we don't have any direct connectors available to connect Azure log analytics workspace or Azure Sentinel with Azure data factory.
if you want to use the log analytics workspace/Azure sentinel data in ADF you need to export the data to either storage account(blob storage) or Azure Events hubs & load that data from blob storage to data factory as explained in this documentation.
You can refer these documentations to export the data from the Azure log analytics workspace to storage account , Azure sentinel data to azure storage account
I have a process that dumps json files into Azure Storage Blobs. I'm using an Azure Data Factory to Upsert the blobs to a Azure CosmosDB.
Once the json is upserted to CosmosDB I want it removed from storage.
I can add a Delete action to the workflow, but that risks deleting a json file that has not yet been processed.
Is it possible to set the Data Factory to 'Delete after Upsert' or simlar?
Any other suggestions?
EDIT: Adding image.
Step1: Create Pipeline
Step2: Select Dataflow activity
Step3: Here you will see Delete Source files after completion option. As shown in below screenshot.
In the ADF activity, you'll see this option on your Source transformation:
I have an internal document (Excel) that has an Azure information Protection / O365 Unified Sensitivity Labelling applied to it.
Im trying to extract that data, but I'm getting an Encryption Error because and rightly so the information is encrypted.
The process:
The document is pulled from Sharepoint into a blob storage container and then Azure Data factory picks up the file using the Copy activity and reads the contents into an Azure SQL Database
Error message:
ErrorCode=EncryptedExcelIsNotSupported,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Encrypted excel file 'dummy file.xlsx' is not supported, please remove its password.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=ICSharpCode.SharpZipLib.Zip.ZipException,Message=Wrong Local header signature: 0xE011CFD0,Source=ICSharpCode.SharpZipLib,'
I have a Linked Service using a Service principal that can connect to the file, but previewing the data results in a message saying the file is encrypted.
I presume I would need to give permissions to the Service Principal, but im still stuck what those would be.
I tried adding Azure Rights Management read/create in the API permissions but that still hasn't worked
Data Factory can't read the file data which is protected by other service.
If you want to copy data from the encrypted files, you must have the permission to access it.
Step to reproduce:
I created a Copy Data first in the pipeline to simple transfer CSV files frol Azure VM to Azure Blob storage. I always use IRPOC1 as a connection via integration runtime and connect using SAS URI and SAS Token to my Blob Storage
After validate and run my first Copy Data, I successfully have CSV file transfer from my VM to Blob storage
I tried to add a new Data Flow after the Copy Data activity
In my Data Flow, my source is the Blob storage containing the CSV files transferred from VM, my Sink is my Azure SQL Database with successful connection
However, when I ran validation, I got the error message on my Data Flow Source:
Linked Service with self-hosted integration runtime is not supported in data flow.
I saw someone replied on Microsoft Azure Document issue Github that I need to use Copy Data to transfer data to Blob first. Then use the source from this blob with data. This is what I did but I still have the same error. Could you please let me know how I can fix this?
The Data Flow source dataset must use a Linked Service that uses an Azure IR, not a self-hosted IR.
Go to the dataset in your data flow Source, click "Open". In the dataset page, click "Edit" next to Linked Service.
In the Linked Service dialog, make sure you are using an Azure Integration Runtime, not a Self-hosted IR.
Is there any way to copy the data from azure data lake storage to database present in azure analysis server using azure data factory ?
I am trying to use copy activity to do the same task but I don't know how to specify the Analysis Server Database as the destination in output dataset.
For the data connector not in data factory support list, you can write custom activity to access your data
Here is the doc: https://learn.microsoft.com/en-us/azure/data-factory/data-factory-use-custom-activities
Thanks,
Charles