I have tried integrating my azure blobs with Apache nifi and failed. I have used listazureblobStorage and FetchAzureBlobStorage processors.
The other blind you mention have the same Storage Credentials, Storage Account Name, and Storage Account Key. If those values are the same used in the other processors for azure listed above and are shown to work, it could either be an issue with the specific access tokens validity, access to the blob, or something else with the specific NiFi processor.
Related
I've created new ADF instance on Azure with Managed Virtual Network integration enabled.
I planned to connect to Azure Key Vault to retrieve credentials for my pipeline’s source and sink systems using Key Vault Private Endpoint. I was able to successfully create it using Azure Data Factory Studio. I have also created Azure Key Vault linked service.
However, when I try to configure another Linked Services for source and destination systems the only option available for retrieving credentials from Key Vault is AVK Linked Service. I'm not able to select related Private Endpoint anywhere (please see below screen).
Do I miss something?
Are there any additional configuration steps required? Is the scenario I've described possible at all?
Any help will be appreciated!
UPDATE: Screen comparing 2 linked services (one with managed network and private endpoint selected and another one where I'm not able to set this options up):
Managed Virtual Network integration enabled, Make sure check which region you are using unfortunately ADF managed virtual network is not supported for East Asia.
I have tried in my environment even that option is not available
So, I have gathered some information even if you create a private endpoint for Key Vault, this column is always shown as blank .it validates URL format but doesn't do any network operation
As per official document if you want to use new link service, instead of key vault try to create other database services like azure sql, azure synapse service like as below
For your Reference:
Store credentials in Azure Key Vault - Azure Data Factory | Microsoft Docs
Azure Data Factory and Key Vault - Tech Talk Corner
I am using an open source tool for deployment of schema for my warehouse snowflake. I have successfully done it for tables, views and procedures. Currently I'm facing an issue, I have to deploy snowflake stages same way. But stages required url and azure saas token when you define it in your sql file like this:
CREATE or replace STAGE myStage
URL = 'azure://xxxxxxxxx.blob.core.windows.net/'
CREDENTIALS = ( AZURE_SAS_TOKEN = 'xxxxxxxxxxxxxxxxxxxx' )
file_format = myFileFormat;
As it is not encouraged to use your credentials in file that will be published on version control and access by others. Is there a way/task in azure devOps so I can just pass a template SQL file in repo and change it before compilation and execution(may be via azure key vault) and change back to template? So these credentials and token always remain secure.
Have you considered using a STORAGE INTEGRATION, instead? If you use the storage integration credentials and grant that to your Blob storage, then you'd be able to create STAGE objects without passing any credentials at all.
https://docs.snowflake.net/manuals/sql-reference/sql/create-storage-integration.html
For this issue ,you can use credential-less stages to secure your cloud storage without sharing secrets.
Here agree with Mike, storage integrations, a new object type, allow a Snowflake administrator to create a trust policy between Snowflake and the cloud provider. When Snowflake connects to the organization’s cloud storage, the cloud provider authenticates and authorizes access through this trust policy.
Storage integrations and credential-less external stages put into the administrator’s hands the power of connecting to storage in a secure and manageable way. This functionality is now generally available in Snowflake.
For details ,please refer to this document. In addition, you can also via azure key vault, key vault provides a secure place for accessing and storing secrets.
My Java backend server has to upload files to the Google Cloud Storage (GCS).
Right now I just run
public void store(MultipartFile multipartFile) throws IOException {
Storage storage = StorageOptions.getDefaultInstance().getService();
storage.create(
BlobInfo.newBuilder(
BUCKET_NAME,
Objects.requireNonNull(multipartFile.getOriginalFilename()))
.build(),
multipartFile.getBytes()
);
}
Having set GOOGLE_APPLICATION_CREDENTIALS=$PROJECT_DIR$/project-1234-abcdefg.json in my environment.
However, this makes things complicated for my deployment setup. I don't know how I would go about making this file available to my service.
Is there another way to get access to GCS for my service account?
Background
I am deploying my server to Heroku as a compiled jar file and I don't know how to make the credentials available to my server during deployment.
You need a Google Account to access to GCS, either personal or technical. Technical is a service account.
However, you have another solution, but not really easy to implement. I wrote an article for securing serverless product with Cloud Endpoint with and API Key. Here your serverless solution can be Cloud Storage. But that implies that you call GCS with REST API and not with the java library, not very fun. That also implies additional cost for the hosting and the processing time of Cloud Endpoint.
Note: you can improve the authorization from API Key to Firebase auth or something else if you prefer. Check the Cloud Endpoint authentication capabilities
Note2: Google is working on another authentication mechanism but I don't know at which stage are the developments, and if it's plan for 2020. In any case, your constraint is known and addressed by Google
I am a new user of the Azure platform, and am having trouble understanding how differents parts are conected. I have data in a Storage blob that I would like to use to make HTTPS POST requests to a web service. My question therfore is as follows: How can I send data from my Azure storage blob to a REST API endpoint?
First, let's start with a little background:
Azure Resource Manager (ARM)
ARM is the REST API that you interface with, using the Azure Portal, PowerShell module, or cross-platform (xPlat) CLI tool, in order to provision and manage cloud resources inside your Azure subscription (account). In order to provision resources, you must first create a Resource Group, essentially a management container for various cloud resource instances.
Azure Storage (Blob)
Microsoft Azure Storage offers several different services:
Blob (unstructured, flat data storage)
Files (cloud-based SMB share for Azure VMs)
Queue (FIFO / LIFO queues, similar to Azure Service Bus)
Table (NOSQL partitioned storage)
Of these types of storage, Blob storage is arguably the most common. In order to utilize any of these storage services, you must first provision a Storage Account inside an ARM Resource Group (see above). To specifically utilize blob storage, you create a Blob Container inside your Storage Account, and then create or upload blobs into this container(s). Once data is stored in an Azure Blob Container, it does not move unless a service explicitly requests the data.
Azure App Service
If you're deploying a Web App (with a front end) or a REST API App (no front end), you'll be using Microsoft Azure's App Service offering. One unique feature of Azure App Service's Web App (I know, it's a mouthful) offering is WebJobs. WebJobs essentially allow you to run arbitrary code in the cloud, kind of like a background worker process. You can trigger WebJobs when blobs are created or uploaded, using this document.
Essentially, you use the [BlobTrigger()] .NET attribute, from the Azure WebJobs SDK, to designate code that will be executed inside Azure WebJobs whenever a new blob is created. The code that executes could grab the blob data, and send it off to your REST API endpoint.
In Amazon cloud API there is the possibility to get identity data, meaning data from the running instance - on which region it is, dns ....
is there the same option in Azure? as I am creating management system in which the server is installed on a virtual machine and I need to know to which region it is related, all this using REST API
In Azure you can use Azure API Management REST API to get all sort of information for Azure:
ex:
Lists all of the resources in a subscription:
https://management.azure.com/subscriptions/{subscription-id}/resources?$top={top}$skiptoken={skiptoken}&$filter={filter}&api-version={api-version}
For the complete documentation look at this page here:
https://msdn.microsoft.com/en-us/library/azure/dn776326.aspx
You can do similar things using Powershell scripts as well.