I'm trying to deploy a dacpac to an Azure Sql Database with Always encrypted enabled. The Devops agent is running in a self-hosted VM with sqlpackage.exe version 19 with build 16.0.5400.1 installed on it.
I've been able to trace down the issues by adding /diagnostics as an argument to the task and the exception that is raised is:
Unexpected exception executing KeyVault extension 'Object reference not set to an instance of an object.' at Microsoft.SqlServer.Dac.KeyVault.DacKeyVaultAuthenticator.Validate(IList`1 keyVaultUrls, CancellationToken cancelToken)
Anybody have a suggestion on how to solve this?
Please check below points
Microsoft.SqlServer.Dac.KeyVault.DacKeyVaultService Provides a service for discovering and configuring a Microsoft.SqlServer.Dac.KeyVault.KeyVaultAuthenticator to handle key vault access requests.
These requests will occur during deployment if an encrypted table is being altered. It also supports initialization of general key vault support in an application
If you store your column master keys in a key vault and you are using access policies for authorization:
Your application's identity needs the following access policy permissions on the key vault: get, unwrapKey, and verify.
A user managing keys for Always Encrypted needs the following access policy permissions on the key vault: create, get, list, sign, unwrapKey, wrapKey, verify.
SEE Create & store column master keys for Always Encrypted - SQL Server | Microsoft Docs
3.
To publish DAC package if Always Encrypted is set up in the DACPAC
or/and in the target database, you might need some or all of the below
permissions, depending on the differences between the schema in the
DACPAC and the target database schema.
ALTER ANY COLUMN MASTER KEY, ALTER ANY COLUMN ENCRYPTION KEY, VIEW ANY COLUMN > MASTER KEY DEFINITION, VIEW ANY COLUMN ENCRYPTION KEY
DEFINITION
we need to enable that Azure virtual machine check box
References:
Configure column encryption using Always Encrypted with a DAC package - SQL Server | Microsoft Docs
azure-sql-advanced-deployment-part4.
KeyVaultAuthenticator.Validate(IList, CancellationToken) >> Microsoft.SqlServer.Dac.KeyVault Namespace | Microsoft Docs
I managed to find a solution. I downgraded the sqlpackage.exe version. If I understand it correctly apparently version 19 seems to be targeted for SQL Server compatibility level 160 which is shipped with SQL Server 2022. When using version 18 it seems to be working with the current 150 that my Azure DB is set to.
Related
In our organization, we are having common credentials to access the postgres databases, which every developers know, as it is hardcoded in application's connection string. Due to which, whenever a DML/DDL changes happens on databases, it is hard for us trace, as the developers use to make changes on their own. We can't have individual logins for each developers which is tedious to manage.
Note: Also, we can't ensure that the credentials won't be shared with the peer developers.
To get rid of this, we thought of integrating Postgres with Azure Active Directory, for Authentication.
If we can map Azure AD group/users to Postgres, security will be tightened as well as maintenance overhead will also reduce.
But, I couldn't find a article to implement this, since most of the articles says the integration for Azure managed postgresql with Azure AD, and not for the postgres running on VMs.
Can anyone guide me or share a detailed article to implement the Azure AD integration for Postgres running on a VM(IaaS)
In Azure portal go to the postgresql database select Authentication and set active directory admin.
You can specify an Azure AD group instead of an individual user to have multiple administrators.
Connecting to postgresql :
1.Login to Azure subscription.
2.Get the access token of the postgresql serverusing below command:
az account get-access-token --resource https://ossrdbms-aad.database.windows.net
3.Use that token as password for login with postgresql server.
Creating user
CREATE USER "user1#yourtenant.onmicrosoft.com" IN ROLE azure_ad_user;
Token validation:
Token is signed by Azure AD and has not been tampered with
Token was issued by Azure AD for the tenant associated with the server
Token has not expired
Token is for the Azure Database for PostgreSQL resource (and not another Azure resource)
Reference Link: Use Azure Active Directory - Azure Database for PostgreSQL - Single Server | Microsoft Learn
Using Azure Active Directory is a great idea for the reasons you specified, but unfortunately there's no native support for connection to Azure Active Directory with a local Postgres database (which is essentially what you have with Postgres in a VM). It can be done through the LDAP protocol, however.
FULL DISCLOSURE: I haven't actually done this part myself (or used the steps in the tutorial link), but this is my understanding from working with system operators. Use LDAP to connect to Azure AD then Postgres to connect via LDAP. More information on LDAP authentication in Postgres can be found here.
Bhavani's answer is about Azure Database for PostgreSQL, which is a Azure-native database service. This part I have used and I highly recommend it; you get Azure AD integration and can manage the database performance and connectivity specifically without having to also manage VM performance. Note that their screenshot is for the Flexible Server while the reference link says 'Single Server'; I recommend Flexible Server.
I've created new ADF instance on Azure with Managed Virtual Network integration enabled.
I planned to connect to Azure Key Vault to retrieve credentials for my pipeline’s source and sink systems using Key Vault Private Endpoint. I was able to successfully create it using Azure Data Factory Studio. I have also created Azure Key Vault linked service.
However, when I try to configure another Linked Services for source and destination systems the only option available for retrieving credentials from Key Vault is AVK Linked Service. I'm not able to select related Private Endpoint anywhere (please see below screen).
Do I miss something?
Are there any additional configuration steps required? Is the scenario I've described possible at all?
Any help will be appreciated!
UPDATE: Screen comparing 2 linked services (one with managed network and private endpoint selected and another one where I'm not able to set this options up):
Managed Virtual Network integration enabled, Make sure check which region you are using unfortunately ADF managed virtual network is not supported for East Asia.
I have tried in my environment even that option is not available
So, I have gathered some information even if you create a private endpoint for Key Vault, this column is always shown as blank .it validates URL format but doesn't do any network operation
As per official document if you want to use new link service, instead of key vault try to create other database services like azure sql, azure synapse service like as below
For your Reference:
Store credentials in Azure Key Vault - Azure Data Factory | Microsoft Docs
Azure Data Factory and Key Vault - Tech Talk Corner
I have an SQL database in Azure with always encryption enabled (master key in Azure key vault) to encrypt the data in a particular table. Azure DB is on version Azure SQL Database v12.
Somehow, I am not able to deploy my dacpac DB project into it using Azure DevOps release pipeline. I am using 'Azure SQL Database deployment' task version 1.*
During the execution of the release pipeline, I am getting an error message “The element MPCMK1 is not supported in Microsoft Azure SQL Database v12.” MPCMK1 is the column master key already created in the target server (stored in key azure vault). Other than the deployment, the functionalities such as encryption/decryption and all working fine. I am not understanding why it says not supported in this version.
I added publish profile and configured to skip deploying column master keys, column encryption keys etc but still I am getting this message on the key that is in the target server.
I removed the key from the target server just for testing, and this error went away. But that doesn't help me as removing the key will require to remove encryptions from the tables and all. But this was just to make sure the error is on the target but not on the source.
Also tried the arguments /p:AllowIncompatiblePlatform=True /p:BlockOnPossibleDataLoss=false but no luck.
It would be great if someone can help me on this.
I am using an open source tool for deployment of schema for my warehouse snowflake. I have successfully done it for tables, views and procedures. Currently I'm facing an issue, I have to deploy snowflake stages same way. But stages required url and azure saas token when you define it in your sql file like this:
CREATE or replace STAGE myStage
URL = 'azure://xxxxxxxxx.blob.core.windows.net/'
CREDENTIALS = ( AZURE_SAS_TOKEN = 'xxxxxxxxxxxxxxxxxxxx' )
file_format = myFileFormat;
As it is not encouraged to use your credentials in file that will be published on version control and access by others. Is there a way/task in azure devOps so I can just pass a template SQL file in repo and change it before compilation and execution(may be via azure key vault) and change back to template? So these credentials and token always remain secure.
Have you considered using a STORAGE INTEGRATION, instead? If you use the storage integration credentials and grant that to your Blob storage, then you'd be able to create STAGE objects without passing any credentials at all.
https://docs.snowflake.net/manuals/sql-reference/sql/create-storage-integration.html
For this issue ,you can use credential-less stages to secure your cloud storage without sharing secrets.
Here agree with Mike, storage integrations, a new object type, allow a Snowflake administrator to create a trust policy between Snowflake and the cloud provider. When Snowflake connects to the organization’s cloud storage, the cloud provider authenticates and authorizes access through this trust policy.
Storage integrations and credential-less external stages put into the administrator’s hands the power of connecting to storage in a secure and manageable way. This functionality is now generally available in Snowflake.
For details ,please refer to this document. In addition, you can also via azure key vault, key vault provides a secure place for accessing and storing secrets.
I am trying to configure Azure Key Vault and setup Managed Identities for use in CI/CD pipeline for Azure Dev Ops.
I have looked around in MSDN documentation but I only specific links for use with Azure SQL and we are using Azure SQL Managed Instances.
If I did not make any misunderstand, in fact, you want to use Managed Identity work with Azure SQL Managed Instance? If this, unfortunately to say, the Managed Identity could not work with Azure SQL Managed Instance. Please see this doc: Services that support managed identities for Azure resources. It list all of the Azure services name which support work with the Managed identities in great detail.
You can see for SQL database, it only support the integration with Azure SQL instead of Azure SQL Managed Instance. That's why you only see the doc link for the usage with Azure SQL.
Until now, the Azure SQL managed instance only support two authentication method:
SQL Authentication:
This authentication method uses a username and password.
Azure Active Directory Authentication:
This authentication method uses identities managed by Azure Active
Directory and is supported for managed and integrated domains. Use
Active Directory authentication (integrated security) whenever
possible.
You can refer to this thread: Managed Identity with Azure SQL Managed Instance?. In this thread, out engineer provided some work around if you trying to configure the app with Managed Identity.