I have created a method using GETFILE() service of azure. Reference: https://learn.microsoft.com/en-us/rest/api/storageservices/get-file
public void getImage(){
string storageKey = 'xxxxStorageKeyxxx';
string storageName = '<storageName>';
Datetime dt = Datetime.now();
string formattedDate = dt.formatGMT('EEE, dd MMM yyyy HH:mm:ss')+ ' GMT';
string CanonicalizedHeaders = 'x-ms-date:'+formattedDate+'\nx-ms-version:2016-05-31';
string CanonicalizedResource = '/' + storageName + '/<shareName>/<dirName>/<File Name>\ntimeout:20';
string StringToSign = 'GET\n\n\n\n\napplication/octet-stream\n\n\n\n\n\n\n' + CanonicalizedHeaders+'\n'+CanonicalizedResource;
Blob temp = EncodingUtil.base64Decode(storageKey);
Blob hmac = Crypto.generateMac('HmacSHA256',Blob.valueOf(StringToSign),temp ); //StringToSign
system.debug('oo-'+EncodingUtil.base64Encode(hmac));
HttpRequest req = new HttpRequest();
req.setMethod('GET');
req.setHeader('x-ms-version','2016-05-31' );
req.setHeader('x-ms-date', formattedDate);
req.setHeader('content-type','application/octet-stream');
string signature = EncodingUtil.base64Encode(hmac);
string authHeader = 'SharedKey <storageName>'+':'+signature;
req.setHeader('Authorization',authHeader);
req.setEndpoint('https://<storageName>.file.core.windows.net/<shareName>/<dirName>/<file Name>?timeout=20');
Http http = new Http();
HTTPResponse res;
res = http.send(req);
}
The above was working fine and giving the 200 as response code. But, my main goal is to display/download the respective image which i retrieved through REST API. How can i achieve that?
So a few things before I answer your question:
File storage is not really suitable for what you're trying to accomplish (it's possible though).
You should look at Blob storage for this as blob storage is more suitable for this kind of scenario.
Assuming you go with Blob storage, there are a few things you could do:
If the blob container (equivalent to a share in file storage) has an ACL is Blob or Container (i.e. blobs in a container are publicly available), you could simply return the blob's URL (Same is your request URL in code above) in your response and then create a link in your application with href set to this URL.
If the blob container has an ACL as Private (i.e. blobs are not publicly available), you would need to create a Shared Access Signature (SAS) token on that blob with at least Read permission and then create a SAS URL. A SAS URL is simply blob URL + SAS token and return this SAS URL in your response and then create a link in your application with href set to this URL.
Since an Azure File Share is always private, if you were to use Azure File service to serve a file, you would do the same thing as 2nd option I listed above. You will create a SAS token on the file with at least Read permission and then return the SAS URL in the response and then create a link in your application with href set to this URL.
To read about Shared Access Signature, you may find this link helpful: https://learn.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1.
To create a Shared Access Signature using REST API, you may find this link helpful: https://learn.microsoft.com/en-us/rest/api/storageservices/Constructing-a-Service-SAS?redirectedfrom=MSDN
Related
I want to upload (and eventually, download) a file to Azure Blob Storage. I am not allowed to use the Storage SDK, and can only use preloaded libraries (aka requests, json, etc).
So far, I am able to get the token using this method. Once that is done, I build another requests with the required headers. However, I keep getting a 401 Error.
Code to get Token:
# Set the request url
url = f'https://login.microsoftonline.com/{tenant}/oauth2/token'
# Set the request headers
headers = {'Content-Type': 'application/x-www-form-urlencoded'}
resource = 'https://management.azure.com/'
# Set the request body
body = {
'grant_type': 'client_credentials',
'client_id': client_id,
'client_secret': client_secret,
'resource': resource
}
# Make the POST request to the authentication endpoint
response = requests.post(url, headers=headers, data=body)
# Parse the JSON response
response_json = json.loads(response.text)
# Save to variable
oauth_token = response_json['access_token']
Code works, and return a token. Also, to clarify, I am able to upload a file using the SDK library (which means the app registration and permissions are fine), but not manually. PUT request to Azure Blob Storage is as follows:
# Code to upload file to blob storage
# Set the request url
endpoint = f'https://{storage_account_name}.blob.core.windows.net/{container_name}/{blob_name}'
# Open the file to be uploaded
with open(blob_name, "rb") as f:
# Get file size
file_size = str(len(f.read()))
# Get date
now = datetime.datetime.utcnow().strftime('%a, %d %b %Y %H:%M:%S GMT')
# Move the pointer back to the beginning of the file
f.seek(0)
# Set the headers for the request
headers = {
"Authorization": f"Bearer {oauth_token}",
"x-ms-version":"2020-04-08",
"Content-Length": file_size,
"x-ms-blob-type": "BlockBlob",
"Date": now
}
# Send the PUT request to upload the file
response = requests.put(endpoint, headers=headers, data=f)
response.status_code
Response Status is 401.
Any help is appreciated.
I've tried filling in more than the required header fields. I've deleted and recreated the app registration. Still unable to access the resource.
For those that might be wondering, I should have specified the resource variable as https://<storage_account_name>.blob.core.windows.net. Code now works.
I’m trying to upload zip using put request to azure url.
as a result of PUT request i’m getting 201 (created) as result of request.
Snaplogic Response
Here is the details i’m using to upload a file
Blob Url: https://symphnytstcjgrk509.blob.core.windows.net/dmf/GHD%20Forecast%20Interface%20BY.xml?sv=2014-02-14&sr=b&sig=XXXst=2022-11-30T12%3A59%3A44Z&se=2022-11-30T13%3A34%3A44Z&sp=rw
Content-Type : application/zip
Upload transfer request type: Calculate content length
x-ms-blob-type: BlockBlob
The same is getting successfully executed from Post man and the file is uploaded
Do you know what's happening here?
I have uploaded the zipfile using code and able to fetch the file using postman and also using urls.
Create a storage in Azure
Create a container
Code Snippet:
string connection_String = Environment.GetEnvironmentVariable("AzureWebStorage");
string container_Name = Environment.GetEnvironmentVariable("ContainerName");
var file = req.Form.Files["File"];
var filecontent = file.OpenReadStream();
var blobClient = new BlobContainerClient(connection_String, container_Name);
var blob = blobClient.GetBlobClient(file.FileName);
byte[] byteArray = Encoding.UTF8.GetBytes(filecontent.ToString());
using (Stream myblob = new MemoryStream(byteArray))
{
await blob.UploadAsync(myblob);
}
return new OkObjectResult("file uploaded successfylly");
Zip File Uploaded
And also, able to upload and download using URLs.
I have a blob created using the google cloud storage API, and have saved its path using blob.path. The path is of the form
/b/bucketname/o/some%2Fobject%2Fid
How do I recreate the blob from this URL?
It's unfortunate that the GCS API doesn't provide a factory method to go from the path URL back to a blob, since saving blob paths in databases etc. is quite common.
Here is a factory method that allows you to go from a blob.path back to a blob:
def blob_from_blobpath(blob_path):
import google.cloud.storage as gcs
blob_path = blob_path[3:] # /b/
slash_loc = blob_path.index('/')
bucket_name = blob_path[:slash_loc]
blob_name = blob_path[(slash_loc+3):] # /o/
bucket = gcs.Client().get_bucket(bucket_name)
return bucket.blob(blob_name)
I'm having trouble writing to an Azure Block Blob from C++ using a SAS (Shared Access Signature). I'm using the Blob REST API and Poco. The HTTP request returns error 404 (resource does not exist), but I can't figure out what I'm doing wrong.
I generate the SAS on the server in C# like this (seems to work fine):
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString"));
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer container = blobClient.GetContainerReference("my-blob");
container.CreateIfNotExists();
SharedAccessBlobPolicy sasConstraints = new SharedAccessBlobPolicy();
sasConstraints.SharedAccessExpiryTime = DateTime.UtcNow.AddMinutes(40);
sasConstraints.Permissions = SharedAccessBlobPermissions.Write | SharedAccessBlobPermissions.List;
string sasContainerToken = container.GetSharedAccessSignature(sasConstraints);
return Request.CreateResponse(HttpStatusCode.OK, container.Uri + sasContainerToken);
In the Azure portal I can indeed see the Blob container being created as expected. I receive this SAS in C++ using an HTTP request. What I get looks like this (some names and signature replaced for security reasons):
https://myname.blob.core.windows.net/my-blob?sv=2012-02-12&se=2016-06-07T11%3A13%3A19Z&sr=c&sp=wl&sig=%%%%%%%%%%%%%%%%%%%%%%%
Then I try to create the file using Poco and the Blob REST API. That looks like this:
std::string cloudUrl = sasURI + "&restype=container";
std::string fileName = "fname.ext";
Poco::URI* uri = new Poco::URI(cloudUrl.c_str());
std::string* path = new std::string(uri->getPathAndQuery());
Poco::Net::HTTPSClientSession* session = new Poco::Net::HTTPSClientSession(uri->getHost(), uri->getPort());
std::string method = Poco::Net::HTTPRequest::HTTP_PUT;
Poco::Net::HTTPRequest* request = new Poco::Net::HTTPRequest(method, *path, Poco::Net::HTTPMessage::HTTP_1_1);
request->add("x-ms-blob-content-disposition", "attachment; filename=\"" + fileName + "\"");
request->add("x-ms-blob-type", "BlockBlob");
request->add("x-ms-meta-m1", "v1");
request->add("x-ms-meta-m2", "v2");
Poco::Net::HTTPResponse* httpResponse = new Poco::Net::HTTPResponse();
int fileContent = 42;
request->setContentLength(sizeof(int));
request->setKeepAlive(true);
std::ostream& outputStream = session->sendRequest(*request);
outputStream << fileContent;
std::istream &is = session->receiveResponse(*httpResponse);
Poco::Net::HTTPResponse::HTTPStatus status = httpResponse->getStatus();
std::ostringstream outString;
Poco::StreamCopier::copyStream(is, outString);
if (status != Poco::Net::HTTPResponse::HTTP_OK)
{
Logger::log("Connection failed\nstatus:", status, "\nreason:", httpResponse->getReason(), "\nreasonForStatus:", httpResponse->getReasonForStatus(status), "\nresponseContent:", outString.str());
}
I've looked up here how the REST API works. I found here that when using a SAS I don't need to do regular authentication.
What am I doing wrong here? Why am I getting error 404?
I believe most of your code is correct, all you need to do is insert the file name in your SAS URL.
Now that I have seen this question more carefully, this is what is happening:
You're creating a SAS on a blob container (my-blob) and using this SAS to upload a file (let's call it fname.ext). However you're not including the file name in the SAS URL so Azure Storage Service is assuming that you're trying to upload a file called my-blob in a $root container so on the service side when Azure Blob Service tries to validate the SAS, it validates it against $root container. Because you created the SAS for my-blob container and Azure Service is using $root container, the SAS does not match and that's why you're getting 403 error.
What you need to do is insert the file name in your SAS URL. So your SAS URL (or Request URL) would be something like (notice that I added fname.ext there):
https://myname.blob.core.windows.net/my-blob/fname.ext?sv=2012-02-12&se=2016-06-07T11%3A13%3A19Z&sr=c&sp=wl&sig=%%%%%%%%%%%%%%%%%%%%%%%
Also, you don't need the following two lines of code:
request->add("x-ms-version", "2015-02-21");
request->add("x-ms-date", "2016-06-07");
As these are not really needed when using SAS.
I've finally figured out what was going wrong here. :)
There were two problems in the above code. The first is that the filename needed to be inserted into the URL, as Gaurav Mantri explained. This does the trick:
int indexOfQuestionMark = cloudUrl.find('?');
cloudUrl = cloudUrl.substr(0, indexOfQuestionMark) + "/" + fileName + cloudUrl.substr(indexOfQuestionMark);
The other problem is that I wasn't uploading enough bytes. sizeof(int) is 4 bytes while pushing 42 into a stream turns it into characters, making it only 2 bytes. The server then keeps waiting for the remaining 2 bytes. That makes this the correct line in the example code above:
request->setContentLength(2);
Also, it works without these three lines so I suppose they're not needed:
request->add("x-ms-blob-content-disposition", "attachment; filename=\"" + fileName + "\"");
request->add("x-ms-meta-m1", "v1");
request->add("x-ms-meta-m2", "v2");
Similarly, adding this doesn't seem needed: "&restype=container".
Finally, for writing the SharedAccessBlobPermissions.List rights aren't needed so those can be left out in SAS generation on the server side.
One possible reason for your error could be the request date being too old. You're setting the request date as Midnight UTC tonight. Azure Storage allows about 15 minutes of clock skewness. Request date/time being "too old" is one of the major reasons for this 403 error (apart from incorrect account key and expired token in case of a SAS).
This is how you're setting x-ms-date request header.
request->add("x-ms-date", "2016-06-07");
This header's value should be formatted in the following format:
request->add("x-ms-date", "Sun, 11 Oct 2009 21:49:13 GMT");
Usually in C# world, we would do a DateTime.UtcNow.ToString("R") to get the date/time in correct format.
Please change your code accordingly and see if that solves the problem.
Rest client.
Can I add a local variable for value into URL string for a Rest client ?
Example
URL testurl = new URL("http://X.X.X.X:7001/lab2.local.rest1/api/v1/status/database?rxnum=1111");
The above works if I provide literal value for rxnum (i.e. 1111).
But I need rest client to utilize value of a local variable. exam
int rxvalue = 1111;
URL testurl = new URL("http://X.X.X.X:7001/lab2.local.rest1/api/v1/status/database?rxnum=+(rxvalue)+");
this doesn't work, obvious my URL string is incorrect. What is correct syntax to allow URL string to use value of local variable rxvalue?
thanks in advance
URL testurl = new URL("http://X.X.X.X:7001/lab2.local.rest1/api/v1/status/database?rxnum=" +rxvalue);
Simple String concatenation.
You are not building the URL string correctly. It is always a good idea to log url/print to be sure that you are creating the correct url. The problem lies in the way you are trying to concatenate the rxvalue, here is the correction in your code :
String urlString = "http://X.X.X.X:7001/lab2.local.rest1/api/v1/status/database?rxnum=" + rxvalue;
URL testurlWithString = new URL(urlString);
System.out.println(testurlWithString);