I am trying to connect to snowflake using linked service and copy data from adls to SF using adf pipeline. I created the linked service and tested the connection. It works fine. Even the debug over the pipeline works fine. althogh when I manually try to trigger the pipeline I get "Required property 'connectionString' is not provided in connection properties"
Thanks in advance.
This looks like a configuration issue and this behavior is noticed when the linked service is configured to use parameters (linked service parameterization), or a key vault is used, and if the connection string value isn't passed to the linked serviced during runtime.
Since it is working in debug mode, I would recommend publishing your linked service and in case if it is parameterized or using AKV, then please make sure that the connection string value is being passed and evaluated at runtime.
Related
All, when running a build pipeline using Azure Devops with ARM template, the process is consistently failing when trying to deploy a dataset or a reference to a dataset with this error:
ARM Template deployment: Resource Group scope (AzureResourceManagerTemplateDeployment)
BadRequest: The document creation or update failed because of invalid reference 'dataset_1'.
I've tried renaming the dataset and also recreating it to see if that would help.
I then deleted the dataset_1.json file from the repo and still get the same message so it's some reference to this dataset and not the dataset itself I think. I've looked through all the other files for references to this but they all look fine.
Any ideas on how to troubleshoot this?
thanks
try this
Looks like you have created 'myTestLinkedService' linked service, tested connection but haven't published it yet and trying to reference that linked service in the new dataset that you are trying to create using Powershell.
In order to reference any data factory entity from Powershell, please make sure those entities are published first. Please try publishing the linked service first from the portal and then try to run your Powershell script to create the new dataset/actvitiy.
I think I found the issue. When I went into the detailed logs I found that in addition to this error there was an error message about an invalid SQL connection string, so I though it may be related since the dataset in question uses Azure SQL database linked service.
I adjusted the connection string and this seems to have solved the issue.
I want to create a task group where Azure Resource Manager Connection is filled with a parameter:
However, this is not possible to do in portal as a validation force to fill it with working value. So I tried to export the task group as json and them modify it and import but then I got this message saving release pipeline:
Is there a way to overcome this? I understood that this is security check (which btw doesn't work in yaml pipelines becauce there you can use Azure Reource Manager connection even if you not allowed). However, in this way it limits usage of task group to a single connection.
EDIT:
Kevin thank you for your anser. I tired it but it didn't work for me.
So I have the connection rg-the-code-manual:
I created a variablewith it:
But when I tried to use it I have a validation error:
Based on my test, when I set the variable as the Azure Resource Manager Connection name, I could reproduce the same issue.
For example:
To solve this issue, you need to set the variable value in release pipeline.
Then you could save the release pipeline successfully.
On the other hand, you could also set the default value for the variable in Task Group.
In this case, the task group will use the default value in release pipeline. And the parameter will also exist in the task group task, you could directly select the value in the drop downlist.
Note: you need to make sure that the Service connection name is valid.
So I've been trying to publish my first project to azure. I've got everything set-up, a service app and a sql database.
My initial page loads properly(It's the standard view for a .net core web application).
The first thing I need to do is register a new user. Whenever I try through my azure app (myapp.azurewebsites.net) it fails and the logs says it's db related.
However I try the same thing by running the application on my machine in production environment, again connected to the azure sql server and everything works perfectly. I can register users, I can create posts, I can edit them. The allow access to azure services option is turned on. This error is from the eventlogs. I have not included the stacktrace.
Category: Microsoft.EntityFrameworkCore.Query EventId: 10100 RequestId: 800001be-0000-ba00-b63f-
84710c7967bb RequestPath: /Identity/Account/Register SpanId: |1e5a93ae-43f424904f38ea9f. TraceId:
1e5a93ae-43f424904f38ea9f ParentId: ActionId: c3430236-e61c-4785-a3c3-4f60ba115b6e ActionName:
/Account/Register An exception occurred while iterating over the results of a query for context type
'MyApp.Data.ApplicationDbContext'. Microsoft.Data.SqlClient.SqlException (0x80131904): Server
name cannot be determined. It must appear as the first segment of the server's dns name
(servername.database.windows.net). Some libraries do not send the server name, in which case the
server name must be included as part of the user name (username#servername). In addition, if both
formats are used, the server names must match.
Those are the different ways I tried to add the connection string to the appsettings.json file. (Server name, catalog, user and password have been replaced, they are written correctly in the appsettings file)
Server=tcp:servername.database.windows.net,1433;Initial Catalog=db;Persist Security Info=False;
User ID=user#server;Password=mypassword;
MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;
Server=tcp:servername.database.windows.net,1433;Initial Catalog=db;Persist Security Info=False;
User ID=user;Password=mypassword;
MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;
Data Source=tcp:server.database.windows.net,1433;
Initial Catalog=db;User Id=#server.database.windows.net;Password=password;
Alright so after a day and a half, I finally managed to fix it. The solution is rather simple and it is most likely my newbie mistake, that caused so much trouble.
I was following a tutorial for setting up the application and database connection after that. In the tutorial, the connection string that was being used, was the default one, found in the "myApp -> Configuration -> Connection strings", the format was:
Data Source=tcp:server.database.windows.net,1433;
Initial Catalog=db;User Id=#server.database.windows.net;Password=password;
This one was working in the guide, but not for me. So what I did, was go to my "sqldb -> connection strings" and copied the one provided there. I then went back to the app configuration and added it as a new configuration string using SqlServer as the Type.
This string was in the format:
Server=tcp:servername.database.windows.net,1433;Initial Catalog=db;Persist Security Info=False;
User ID=user;Password=mypassword;
MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;
After that, the app started working properly.
When I try to upload the zip file to an azure function app using kudu REST API, it throws an error while I try to view the c# code in Function App editor in the browser. The error is:
"Error:
Function ($Source) Error: Microsoft.Azure.WebJobs.Host: Error indexing method 'Functions.Source'. Microsoft.Azure.WebJobs.Host: Value cannot be null.
Parameter name: hostAccount.
Session Id: xxxxxxxxxxx
Timestamp: 2016-12-02T18:35:00.083Z"
Please note that I have automated end to end of Application Insights starting from creation of a resource group till exporting the multi-setep web test results to our Splunk - All using Powershell.
In this process of automation, I am forming a storage connection string and setting it to the app settings of the function app and then providing that key in my function.json binding.
But still I get this error.
Here is the issue I created in the Azure Function App - Git: https://github.com/Azure/azure-webjobs-sdk-script/issues/1020
The error points to missing host configuration (e.g. the host storage account).
If you're automating the creation process, you need to make sure the required environment variables are properly set. The recommended way would be to use an ARM template manage that for you.
I have some details on how you can get the ARM template for a resource (which you could use to look at the required settings for your Function App) here.
We also have some sample templates you can use linked here
I hope this helps!
I am using ASP.NET MVC4 deployed on WebApps, EF6, Azure SQL Database
I have two connectionstrings, one direct to the DB and one that is needed as I am using Entity Framework. If I setup the first via Azure and leave the EF string in web.config, all works fine. Once I try to move the EF connectionstring to Azure, it breaks.
I have tried to follow all the posts on how to setup an EF connection string in Azure Management Portal, and the advice seems to be to use the following:
In Management Portal:
Name:
EFConnectionString
Value:
metadata=res://MyModel/model.csdl|res://MyModel/model.ssdl|res://MyModel/model.msl;provider=System.Data.SqlClient;provider connection string="Data Source=tcp:myserver.database.windows.net,1234;Initial Catalog=mydatabase;User ID=Admin#myserver.database.windows.net;Password=12345678"
SQL Database:
Custom
Note:
" is replaced by " in the Azure string.
In Web.config I still need the following:
<add name="EFConnectionString" connectionString="" providerName="System.Data.EntityClient"/>
I have changed certain details in the connectionstring for security reasons, but the true string does work fine in Web.config.
However when I do attempt the above, I get :
The page cannot be displayed because an internal server error has occurred.
Further Log error message:
The specified named connection is either not found in the configuration, not intended to be used with the EntityClient provider, or not valid.
Any help appreciated. I am trying to do this to enhance the security of stored passwords for accessing DB on Azures, as I understand that setting them up via the portal encrypts them.
Thanks.
First check your deployment. I've run in a similar problem, and after a couple of hours struggling I found out that I was deploying without actually sending connection strings, falling back to whatever was in the portal config... the others keys in the we.config were being sent over, but the conn strings weren't... after I properly set it up on the portal, everything went back to normal...