I am using an open source tool for deployment of schema for my warehouse snowflake. I have successfully done it for tables, views and procedures. Currently I'm facing an issue, I have to deploy snowflake stages same way. But stages required url and azure saas token when you define it in your sql file like this:
CREATE or replace STAGE myStage
URL = 'azure://xxxxxxxxx.blob.core.windows.net/'
CREDENTIALS = ( AZURE_SAS_TOKEN = 'xxxxxxxxxxxxxxxxxxxx' )
file_format = myFileFormat;
As it is not encouraged to use your credentials in file that will be published on version control and access by others. Is there a way/task in azure devOps so I can just pass a template SQL file in repo and change it before compilation and execution(may be via azure key vault) and change back to template? So these credentials and token always remain secure.
Have you considered using a STORAGE INTEGRATION, instead? If you use the storage integration credentials and grant that to your Blob storage, then you'd be able to create STAGE objects without passing any credentials at all.
https://docs.snowflake.net/manuals/sql-reference/sql/create-storage-integration.html
For this issue ,you can use credential-less stages to secure your cloud storage without sharing secrets.
Here agree with Mike, storage integrations, a new object type, allow a Snowflake administrator to create a trust policy between Snowflake and the cloud provider. When Snowflake connects to the organization’s cloud storage, the cloud provider authenticates and authorizes access through this trust policy.
Storage integrations and credential-less external stages put into the administrator’s hands the power of connecting to storage in a secure and manageable way. This functionality is now generally available in Snowflake.
For details ,please refer to this document. In addition, you can also via azure key vault, key vault provides a secure place for accessing and storing secrets.
Related
In our organization, we are having common credentials to access the postgres databases, which every developers know, as it is hardcoded in application's connection string. Due to which, whenever a DML/DDL changes happens on databases, it is hard for us trace, as the developers use to make changes on their own. We can't have individual logins for each developers which is tedious to manage.
Note: Also, we can't ensure that the credentials won't be shared with the peer developers.
To get rid of this, we thought of integrating Postgres with Azure Active Directory, for Authentication.
If we can map Azure AD group/users to Postgres, security will be tightened as well as maintenance overhead will also reduce.
But, I couldn't find a article to implement this, since most of the articles says the integration for Azure managed postgresql with Azure AD, and not for the postgres running on VMs.
Can anyone guide me or share a detailed article to implement the Azure AD integration for Postgres running on a VM(IaaS)
In Azure portal go to the postgresql database select Authentication and set active directory admin.
You can specify an Azure AD group instead of an individual user to have multiple administrators.
Connecting to postgresql :
1.Login to Azure subscription.
2.Get the access token of the postgresql serverusing below command:
az account get-access-token --resource https://ossrdbms-aad.database.windows.net
3.Use that token as password for login with postgresql server.
Creating user
CREATE USER "user1#yourtenant.onmicrosoft.com" IN ROLE azure_ad_user;
Token validation:
Token is signed by Azure AD and has not been tampered with
Token was issued by Azure AD for the tenant associated with the server
Token has not expired
Token is for the Azure Database for PostgreSQL resource (and not another Azure resource)
Reference Link: Use Azure Active Directory - Azure Database for PostgreSQL - Single Server | Microsoft Learn
Using Azure Active Directory is a great idea for the reasons you specified, but unfortunately there's no native support for connection to Azure Active Directory with a local Postgres database (which is essentially what you have with Postgres in a VM). It can be done through the LDAP protocol, however.
FULL DISCLOSURE: I haven't actually done this part myself (or used the steps in the tutorial link), but this is my understanding from working with system operators. Use LDAP to connect to Azure AD then Postgres to connect via LDAP. More information on LDAP authentication in Postgres can be found here.
Bhavani's answer is about Azure Database for PostgreSQL, which is a Azure-native database service. This part I have used and I highly recommend it; you get Azure AD integration and can manage the database performance and connectivity specifically without having to also manage VM performance. Note that their screenshot is for the Flexible Server while the reference link says 'Single Server'; I recommend Flexible Server.
I've created new ADF instance on Azure with Managed Virtual Network integration enabled.
I planned to connect to Azure Key Vault to retrieve credentials for my pipeline’s source and sink systems using Key Vault Private Endpoint. I was able to successfully create it using Azure Data Factory Studio. I have also created Azure Key Vault linked service.
However, when I try to configure another Linked Services for source and destination systems the only option available for retrieving credentials from Key Vault is AVK Linked Service. I'm not able to select related Private Endpoint anywhere (please see below screen).
Do I miss something?
Are there any additional configuration steps required? Is the scenario I've described possible at all?
Any help will be appreciated!
UPDATE: Screen comparing 2 linked services (one with managed network and private endpoint selected and another one where I'm not able to set this options up):
Managed Virtual Network integration enabled, Make sure check which region you are using unfortunately ADF managed virtual network is not supported for East Asia.
I have tried in my environment even that option is not available
So, I have gathered some information even if you create a private endpoint for Key Vault, this column is always shown as blank .it validates URL format but doesn't do any network operation
As per official document if you want to use new link service, instead of key vault try to create other database services like azure sql, azure synapse service like as below
For your Reference:
Store credentials in Azure Key Vault - Azure Data Factory | Microsoft Docs
Azure Data Factory and Key Vault - Tech Talk Corner
I would like to copy files with Azure File Copy with Azure Pipeline.
I'm following instruction of https://praveenkumarsreeram.com/2021/04/14/azure-devops-copy-files-from-git-repository-to-azure-storage-account/
I'm using automatically created Service Connection named "My Sandbox (a1111e1-d30e-4e02-b047-ef6a5e901111)"
I'm getting error with AzureBlob File Copy:
INFO: Authentication failed, it is either not correct, or
expired, or does not have the correct permission ->
github.com/Azure/azure-storage-blob-go/azblob.newStorageError,
/home/vsts/go/pkg/mod/github.com/!azure/azure-storage-blob-
go#v0.10.1-0.20201022074806-
8d8fc11be726/azblob/zc_storage_error.go:42
RESPONSE Status: 403 This request is not authorized to perform
this operation using this permission.
I'm assuming that Azure Pipeline have no access to Azure Storage.
I wonder how do find service principal which should get access to Azure Storage.
I can also reproduce your issue on my side, as different Azure file copy task versions use different versions of AzCopy in behind, then they use different auth ways to call the API to do the operations.
There are two ways to fix the issue.
If you use the automatically created service connection, it should have Contributor role in your storage account, you could use Azure file copy task version 3.* instead of 4.*, then it will work.
If you want to use Azure file copy task version 4.*, navigate to your storage account -> Access Control (IAM) -> add your service principal used in the service connection as a Storage Blob Data Contributor role, see detailed steps here. It will also work.
I have tried integrating my azure blobs with Apache nifi and failed. I have used listazureblobStorage and FetchAzureBlobStorage processors.
The other blind you mention have the same Storage Credentials, Storage Account Name, and Storage Account Key. If those values are the same used in the other processors for azure listed above and are shown to work, it could either be an issue with the specific access tokens validity, access to the blob, or something else with the specific NiFi processor.
I am trying to configure Azure Key Vault and setup Managed Identities for use in CI/CD pipeline for Azure Dev Ops.
I have looked around in MSDN documentation but I only specific links for use with Azure SQL and we are using Azure SQL Managed Instances.
If I did not make any misunderstand, in fact, you want to use Managed Identity work with Azure SQL Managed Instance? If this, unfortunately to say, the Managed Identity could not work with Azure SQL Managed Instance. Please see this doc: Services that support managed identities for Azure resources. It list all of the Azure services name which support work with the Managed identities in great detail.
You can see for SQL database, it only support the integration with Azure SQL instead of Azure SQL Managed Instance. That's why you only see the doc link for the usage with Azure SQL.
Until now, the Azure SQL managed instance only support two authentication method:
SQL Authentication:
This authentication method uses a username and password.
Azure Active Directory Authentication:
This authentication method uses identities managed by Azure Active
Directory and is supported for managed and integrated domains. Use
Active Directory authentication (integrated security) whenever
possible.
You can refer to this thread: Managed Identity with Azure SQL Managed Instance?. In this thread, out engineer provided some work around if you trying to configure the app with Managed Identity.