I tried to setup scheduled triggers within Azure Data Factory, but they are not getting executed and are not logged either. They used to work but since a couple of days nothing happens anymore, even when I delete them and setup new ones.
When I hit "trigger now", everything is fine.
So basically I setup scheduled triggers (activate them), deploy them and wait for the automatic call but nothing happens at all.
Do you have an idea?
lot of configured triggers
emtpy trigger execution logging
eg:
{
"name": "Trigger_Hourly3",
"properties": {
"annotations": [],
"runtimeState": "Started",
"pipelines": [
{
"pipelineReference": {
"referenceName": "Pipe_02_xy",
"type": "PipelineReference"
}
}
],
"type": "ScheduleTrigger",
"typeProperties": {
"recurrence": {
"frequency": "Hour",
"interval": 1,
"startTime": "2020-06-30T13:43:00.000Z",
"timeZone": "UTC"
}
}
}
}
Thanks, Lukas
Resolution: Fixed an error that was raised during Azure DevOps deployment that was not raised by ADF validation and not sent back as error to ADF UI. Then the triggers were activated again in the following task and everything runs smoothly again.
Related
I am deploying Azure Data Factory pipeline along with linked services and triggers via json templates. Even though the trigger has the "runtime" property as "started" in the template, the triggers are always deployed in the stopped state and have to be activated manually.
Code
{
"name": "trigger-name-started",
"properties": {
"description": "some connection",
"annotations": [],
"runtimeState": "Stopped",
"pipelines": [
{
"pipelineReference": {
"referenceName": "pipeline-new",
"type": "PipelineReference"
}
}
],
"type": "ScheduleTrigger",
"typeProperties": {
"recurrence": {
"frequency": "Minute",
"interval": 15,
"startTime": "2022-12-03T09:53:00Z",
"timeZone": "UTC"
}
}
}
}
Unable to activate triggers in Azure Data Factory as part of CI/CD
deployment ?
If your trigger stopped before you start the deployment . Restart after deployment is complete.
Please follow continuous integration using Azure Pipelines and also, you could use the PowerShell script for stopping triggers and update your trigger settings.
For more in detail, please refer below links:
https://social.technet.microsoft.com/wiki/contents/articles/53153.azuredevops-cicd-for-azure-data-factory.aspx#Existing_CICD_Process_ARM_Template
http://datanrg.blogspot.com/2019/02/continuous-integration-and-delivery.html
I'm doing some initial ADF deployment from my adf-dev to adf-staging environment. In the MS docs it says:
Deployment can fail if you try to update active triggers. To update active triggers, you need to manually stop them and then restart them after the deployment.
Does this mean I need to turn off my dev or staging triggers pre/post deployment?
2nd issue. I need to schedule the same set of triggers to run on different days in dev (sat) vs staging (sun). Do I need to make a separate set of triggers for each environment then or can I rewrite the trigger schedules for the existing triggers during deployment?
You will need your staging triggers stopped before you start the deployment, and restarted after deployment is complete.
this page have a PowerShell script for stopping triggers: https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment#updating-active-triggers
Also you could use the custom petameters configuration file to update your trigger settings: https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment#triggers
To parametrise the trigger deployment in Arm template, first here is a sample weekly trigger that runs on a specific day:
{
"name": "OnceAWeekTrigger",
"properties": {
"annotations": [],
"runtimeState": "Stopped",
"pipelines": [],
"type": "ScheduleTrigger",
"typeProperties": {
"recurrence": {
"frequency": "Week",
"interval": 1,
"startTime": "2021-05-25T22:59:00Z",
"timeZone": "UTC",
"schedule": {
"weekDays": [
"Sunday"
]
}
}
}
}
}
Create an arm-template-parameters-definition.json file as follow:
{
"Microsoft.DataFactory/factories/triggers": {
"properties": {
"typeProperties": {
"recurrence": {
"schedule": {
"weekDays": "=:-weekDays:array"
}
}
}
}
}
}
this file specifies that you want to prarametrise schedule_weekDays property.
after running ADFUtilities export function:
npm run build export c:\git\adf /subscriptions/<subscriptionid>/resourceGroups/datafactorydev/providers/Microsoft.DataFactory/factories/<datafactory_name> "ArmTemplate"
You now get arm template for trigger properties parametrised as follows:
... {
"name": "[concat(parameters('factoryName'), '/OnceAWeekTrigger')]",
"type": "Microsoft.DataFactory/factories/triggers",
"apiVersion": "2018-06-01",
"properties": {
"annotations": [],
"runtimeState": "Stopped",
"pipelines": [],
"type": "ScheduleTrigger",
"typeProperties": {
"recurrence": {
"frequency": "Week",
"interval": 1,
"startTime": "2021-05-25T22:59:00Z",
"timeZone": "UTC",
"schedule": {
"weekDays": "[parameters('OnceAWeekTrigger_weekDays')]"
}
}
}
}, ...
and the parameters file ArmTemplate\ARMTemplateParametersForFactory.json looks as follows:
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"factoryName": {
"value": "factory_name"
},
"OnceAWeekTrigger_weekDays": {
"value": [
"Sunday"
]
}
}
}
you could then create different parameter files for dev and staging with different days of the week by modifying the array value for OnceAWeekTrigger_weekDays
I have a custom activity in Azure Data Factory, which attempts to execute the following command:
PowerShell.exe -Command "Write-Host 'Hello, world!'"
However, when I debug (run) this command from within Azure Data Factory, it runs for a long time, and finally fails.
I guess it fails because perhaps it could not locate "PowerShell.exe". How can I ensure that the ADF Custom Activity has access to PowerShell.exe?
Some sites say about specifying a package (.zip file) that contains everything needed for the exe to execute. However, since PowerShell is from Microsoft, I think it would be inappropriate to ZIP the PowerShell directory, and specify it as a package to the Custom Activity.
Please suggest as to how I can execute PowerShell command from Custom Activity of an Azure Data Factory. Thanks!
Whenever I search "Execute PowerShell from Custom Activity in Azure Data Factory", the search results are talking more about which Az PowerShell command to use to trigger start an ADF pipeline.
I saw two threads in Stackoverflow.com, where the answer just specifies to use a Custom Activity, and the answer is not specific to PowerShell command call from ADF
Here is the JSON for the task:
{
"name": "ExecutePs1CustomActivity",
"properties": {
"activities": [
{
"name": "ExecutePSScriptCustomActivity",
"type": "Custom",
"dependsOn": [],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"command": "PowerShell.exe -Command \"Write-Host 'Hello, world!'\"",
"referenceObjects": {
"linkedServices": [],
"datasets": []
}
},
"linkedServiceName": {
"referenceName": "Ps1CustomActivityAzureBatch",
"type": "LinkedServiceReference"
}
}
],
"annotations": []
}
}
I see "In Progress" for 3 minutes (180 seconds), and then it shows as "Failed."
I would suggest you to move all you scripting task in a powershell file and copy it to a storage account linked with your custom activity. . Once done try to call it like below:
powershell .\script.ps1
You can also provide the path of the script in json like below:
{
"name": "MyCustomActivityPipeline",
"properties": {
"description": "Custom activity sample",
"activities": [{
"type": "Custom",
"name": "MyCustomActivity",
"linkedServiceName": {
"referenceName": "AzureBatchLinkedService",
"type": "LinkedServiceReference"
},
"typeProperties": {
"command": "helloworld.exe",
"folderPath": "customactv2/helloworld",
"resourceLinkedService": {
"referenceName": "StorageLinkedService",
"type": "LinkedServiceReference"
}
}
}]
}
}
Please try it and see if it helps. Also i would suggest you to troubleshoot the pipeline steps to look for detailed error.
Also to your second point "Some sites say about specifying a package (.zip file) that contains everything needed for the exe to execute." This is required when you are building a custom activity using dot net then it is must copy all the Dll's and Exe's for execution.
Hope it helps.
I'm attemping to create a LinkedService via the powershell command
New-AzureRmDataFactoryV2LinkedService -ResourceGroupName rg -DataFactoryName df -Name n -DefinitionFile n.json
the result is that the LinkedService is created, however the reference to the password type of AzureKeyVaultSecret is removed rendering it non-operational
The config file n.json was extracted from the DataFactory code tab and has the syntax below...
{
"name": "<name>",
"type": "Microsoft.DataFactory/factories/linkedservices",
"properties": {
"type": "Oracle",
"typeProperties": {
"connectionString": "host=<host>;port=<port>;serviceName=<serviceName>;user id=<user_id>",
"password": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "Prod_KeyVault",
"type": "LinkedServiceReference"
},
"secretName": "<secretname>"
}
},
"connectVia": {
"referenceName": "<runtimename>",
"type": "IntegrationRuntimeReference"
}
}
}
When the new LinkedService is created, the code looks exactly the same except properties->typeProperties->password is removed and requires manual configuration - which I'm trying to avoid if possible.
Any thoughts?
If you have tried using "Update-Module -Name AzureRm.DataFactoryV2" to update your powershell to the latest version, and it is still the same behavior, then the possible root cause is that password is not support as Azure Key Value yet in Powershell. As far as I know, it is a new feature added recently. So it may take some time to rollout it to Powershell.
In that case, the workaround is to use UI to create linked service for now.
I'm facing an issue when running a U-SQL Script with Azure Data Factory V2.
This U-SQL Script work fine in the portal or vs :
#a =
SELECT * FROM
(VALUES
("Contoso", 1500.0, "2017-03-39"),
("Woodgrove", 2700.0, "2017-04-10")
) AS D( customer, amount );
#results =
SELECT
customer,
amount
FROM #a;
OUTPUT #results
TO "test"
USING Outputters.Text();
But don't work when started in Azure Data Factory V2 Activity, bellow my ADF scripts.
Creating or updating linked service ADLA [adla] ...
{
"properties": {
"type": "AzureDataLakeAnalytics",
"typeProperties": {
"accountName": "adla",
"servicePrincipalId": "ea4823f2-3b7a-4c-78d29cffa68b",
"servicePrincipalKey": {
"type": "SecureString",
"value": "jKhyspEwMScDAGU0MO39FcAP9qQ="
},
"tenant": "41f06-8ca09e840041",
"subscriptionId": "9cf053128b749",
"resourceGroupName": "test"
}
}
}
Creating or updating linked service BLOB [BLOB] ...
{
"properties": {
"type": "AzureStorage",
"typeProperties": {
"connectionString": {
"type": "SecureString",
"value": "DefaultEndpointsProtocol=https;AccountName=totoblob;AccountKey=qZqpKyGtWMRXZO2CNLa0qTyvLTMko4lzfgsg07pjloIPGZtJel4qvRBkoVOA==;EndpointSuffix=core.windows.net"
}
}
}
}
Creating or updating pipeline ADLAPipeline...
{
"properties": {
"activities": [
{
"type": "DataLakeAnalyticsU-SQL",
"typeProperties": {
"scriptPath": "src/test.usql",
"scriptLinkedService": {
"referenceName": "blob",
"type": "LinkedServiceReference"
},
"degreeOfParallelism": 1
},
"linkedServiceName": {
"referenceName": "adla",
"type": "LinkedServiceReference"
},
"name": "Usql-toto"
}
]
}
}
1 - I checked the connection to the blob storage, the u-sql script is successfully found (if i rename it, it throw a not found error)
2 - I checked the connection to azure data lake analytics, it seems to connect, if i set a wrong credential it throw an other error
3 - When running the pipeline I Have the following error : "Activity Usql-toto failed: User is not able to access to datalake store" and actually I don't provide Data Lake Store Credential but there is a default datalake store account attached ADLAnalytics.
Any hint?
Finally found help in this post: U-SQL Job Failing in Data Factory
Folders system and catalog doesn't inherit from parent permissions ... So I had to reassign permission on this two folders.
I was also having this problem. What helped me was running the "Add User Wizard" from Data Lake Analytics. Using this wizard I added the service principal that I use in the linked service for Data Lake Analytics as an Owner with R+W permissions.
Before using the wizard, I tried to configure this manually by setting appropriate permissions from the Explore data screen, but that didn't resolve the problem. (SP was already contributor on the service)