I'm new to using Azure release pipelines and have been fighting issues trying to deploy a database project to a new Azure SQL database. Currently the pipeline is giving me the following error...
TargetConnectionString argument cannot be used in conjunction with any other Target database arguments
I've tried deploying with and without the TargetConnectionString included in my publish profile. Any suggestions or something else to try? I'm out of ideas.
TargetConnectionString
Specifies a valid SQL Server/Azure connection string to the target database. If this parameter is specified it shall be used exclusively of all other target parameters. (short form /tcs)
So please remove all other TargetXXX arguments.
(if you don't have them can you show what arguments you have inline and in publish profile - of course without data)
Related
I have an Azure DevOps pipeline which is failing to run because it seems to be using an old connection string.
The pipeline is for a C# project, where a FileTransform task updates an appsettings.json file with variables set on the pipeline.
The variables were recently updated to use a new connection string, however, when running a Console.PrintLn before using it and viewing it on the pipeline, it shows an outdated value.
Many updates similar to this have been run in the past without issue.
I've also recently added a Powershell task to echo what the value is in the variables loaded while the pipeline is running, which does display the new value.
I've checked the order of precedence of variables and there shouldn't be any other variables being used.
There is no CacheTask being used in this pipeline.
Does anyone have any advice to remedy this? It seems that the pipeline itself is just ignoring the variables set on the pipeline.
There is a problem with the recent File transform task version v1.208.0.
It will shows the warning message and not update the variable value correctly.
Warning example:
Resource file haven't been set, can't find loc string for key: JSONvariableSubstitution
Refer to this ticket: File transform task failing to transform files, emitting "Resource file haven't been set" warnings
The issue is from Task itself instead of the Pipeline configuration. Many users have the same issue.
Workaround:
You can change to use the File Transform task Version 2 to update the appsettings.json file.
Here is an example: Please remove the content in XML Transformation rules field and set the JSON file path
I am testing the Azure data factory deployment using ARM Templates, where I am deploying the all the ADF Instances (Data factory pipeline, linked services, data sets, data flow, Trigger etc.) from Development to UAT and Production, but before deploying to UAT and production. I included below activity in Azure DevOps pipeline-
Stopping the ADF Trigger using 'Azure PowerShell' in built task of azure Devops release pipeline. In this task I am using the Inline script to stop trigger before deployment to UAT/PROD environment.
ARM Template deployment = using ARM Template I configured following value in it.
Template -> the ARM template of the pipeline ARMTemplateForFactory.json
Template Parameters -> the ARM \TemplateParametrsForFactory.json
Override template Parameter-> When I tried to enter the Values for UAT/Prod environment. some parameter are showing double.
Q-1. PFA, the details about each step and the logs. Please guide me why trigger not taking proper format. what could be the reason behind this? How should I correct this so that it can take trigger parameters only once.
Q-2. Also Please let me know Do I need Azure function app key for UAT/Prod environment to enter that value while override template parameter? Can someone please take a look and guide me what I am missing here?
Thanks
This question is answered in Microsoft Q&A platform. Thank you #sachingupta-1921 and #MartinJaffer-MSFT for the workable solution, posting it as an answer here to help the other community members.
A-1: Changing the name of the triggers without spaces resolved the problem mentioned in Q-1.
A-2: If you have different keys for dev / uat/ prod, then you would definitely need to include that.
Since you have a Key Vault, you could put the Function App key in there, and let it be taken care of by the change of key vaults. Then you would not need to enter the Function App key in the parameters.
All, when running a build pipeline using Azure Devops with ARM template, the process is consistently failing when trying to deploy a dataset or a reference to a dataset with this error:
ARM Template deployment: Resource Group scope (AzureResourceManagerTemplateDeployment)
BadRequest: The document creation or update failed because of invalid reference 'dataset_1'.
I've tried renaming the dataset and also recreating it to see if that would help.
I then deleted the dataset_1.json file from the repo and still get the same message so it's some reference to this dataset and not the dataset itself I think. I've looked through all the other files for references to this but they all look fine.
Any ideas on how to troubleshoot this?
thanks
try this
Looks like you have created 'myTestLinkedService' linked service, tested connection but haven't published it yet and trying to reference that linked service in the new dataset that you are trying to create using Powershell.
In order to reference any data factory entity from Powershell, please make sure those entities are published first. Please try publishing the linked service first from the portal and then try to run your Powershell script to create the new dataset/actvitiy.
I think I found the issue. When I went into the detailed logs I found that in addition to this error there was an error message about an invalid SQL connection string, so I though it may be related since the dataset in question uses Azure SQL database linked service.
I adjusted the connection string and this seems to have solved the issue.
I want to create a task group where Azure Resource Manager Connection is filled with a parameter:
However, this is not possible to do in portal as a validation force to fill it with working value. So I tried to export the task group as json and them modify it and import but then I got this message saving release pipeline:
Is there a way to overcome this? I understood that this is security check (which btw doesn't work in yaml pipelines becauce there you can use Azure Reource Manager connection even if you not allowed). However, in this way it limits usage of task group to a single connection.
EDIT:
Kevin thank you for your anser. I tired it but it didn't work for me.
So I have the connection rg-the-code-manual:
I created a variablewith it:
But when I tried to use it I have a validation error:
Based on my test, when I set the variable as the Azure Resource Manager Connection name, I could reproduce the same issue.
For example:
To solve this issue, you need to set the variable value in release pipeline.
Then you could save the release pipeline successfully.
On the other hand, you could also set the default value for the variable in Task Group.
In this case, the task group will use the default value in release pipeline. And the parameter will also exist in the task group task, you could directly select the value in the drop downlist.
Note: you need to make sure that the Service connection name is valid.
As Earlier Posted a thread for syncing Data from Premises Mysql to Azure SQL over here referring this article, and found that lookup component for watermark detection is only available for SQL Server Only.
So tried a work Around, that while using "Copy" Data Flow task ,will pick data greater than last watermark stored from Mysql.
Issue:
Able to validate package successfully but not able to publish same.
Question :
In Copy Data Flow Task i'm using below query to get data from MySql greater than watermark available.
Can't we use Query like below on other relational sources like Mysql
select * from #{item().TABLE_NAME} where #{item().WaterMark_Column} > '#{activity('LookupOldWaterMark').output.firstRow.WatermarkValue}'
CopyTask SQL Query Preview
Validate Successfully
Error With no Details
Debug Successfully
Error After following steps mentioned by Franky
Azure SQL Linked Service Error (Resolved by re configuring connection /edit credentials in connection tab)
Source Query got blank (resolved by re-selection source type and rewriting query)
Could you verify if you have access to create a template deployment in the azure portal?
1) Export the ARM Template: int he top-right of the ADFv2 portal, click on ARM Template -> Export ARM Template, extract the zip file and copy the content of the "arm_template.json" file.
2) Create ARM Template deployment: Go to https://portal.azure.com/#create/Microsoft.Template and log in with the same credentials you use in the ADFv2 portal (you can also get to this page going in the Azure portal, click on "Create a resource" and search for "Template deployment"). Now click on "Build your own template in editor" and paste the ARM template from the previous step in the editor and Save.
3) Deploy template: Click on existing resource group and select the same resource group as the one where your Data Factory is. Fill out the parameters that are missing (for this testing it doesn't really matter if the values are valid); Factory name should already be there. Agree the terms and click purchase.
4) Verify the deployment succeeded. If not let me know the error, it might be an access issue which would explain why your publish fails. (ADF team is working on giving a better error for this issue).
Did any of the objects publish into your Data Factory?