I want to upload (and eventually, download) a file to Azure Blob Storage. I am not allowed to use the Storage SDK, and can only use preloaded libraries (aka requests, json, etc).
So far, I am able to get the token using this method. Once that is done, I build another requests with the required headers. However, I keep getting a 401 Error.
Code to get Token:
# Set the request url
url = f'https://login.microsoftonline.com/{tenant}/oauth2/token'
# Set the request headers
headers = {'Content-Type': 'application/x-www-form-urlencoded'}
resource = 'https://management.azure.com/'
# Set the request body
body = {
'grant_type': 'client_credentials',
'client_id': client_id,
'client_secret': client_secret,
'resource': resource
}
# Make the POST request to the authentication endpoint
response = requests.post(url, headers=headers, data=body)
# Parse the JSON response
response_json = json.loads(response.text)
# Save to variable
oauth_token = response_json['access_token']
Code works, and return a token. Also, to clarify, I am able to upload a file using the SDK library (which means the app registration and permissions are fine), but not manually. PUT request to Azure Blob Storage is as follows:
# Code to upload file to blob storage
# Set the request url
endpoint = f'https://{storage_account_name}.blob.core.windows.net/{container_name}/{blob_name}'
# Open the file to be uploaded
with open(blob_name, "rb") as f:
# Get file size
file_size = str(len(f.read()))
# Get date
now = datetime.datetime.utcnow().strftime('%a, %d %b %Y %H:%M:%S GMT')
# Move the pointer back to the beginning of the file
f.seek(0)
# Set the headers for the request
headers = {
"Authorization": f"Bearer {oauth_token}",
"x-ms-version":"2020-04-08",
"Content-Length": file_size,
"x-ms-blob-type": "BlockBlob",
"Date": now
}
# Send the PUT request to upload the file
response = requests.put(endpoint, headers=headers, data=f)
response.status_code
Response Status is 401.
Any help is appreciated.
I've tried filling in more than the required header fields. I've deleted and recreated the app registration. Still unable to access the resource.
For those that might be wondering, I should have specified the resource variable as https://<storage_account_name>.blob.core.windows.net. Code now works.
Related
I’m trying to upload zip using put request to azure url.
as a result of PUT request i’m getting 201 (created) as result of request.
Snaplogic Response
Here is the details i’m using to upload a file
Blob Url: https://symphnytstcjgrk509.blob.core.windows.net/dmf/GHD%20Forecast%20Interface%20BY.xml?sv=2014-02-14&sr=b&sig=XXXst=2022-11-30T12%3A59%3A44Z&se=2022-11-30T13%3A34%3A44Z&sp=rw
Content-Type : application/zip
Upload transfer request type: Calculate content length
x-ms-blob-type: BlockBlob
The same is getting successfully executed from Post man and the file is uploaded
Do you know what's happening here?
I have uploaded the zipfile using code and able to fetch the file using postman and also using urls.
Create a storage in Azure
Create a container
Code Snippet:
string connection_String = Environment.GetEnvironmentVariable("AzureWebStorage");
string container_Name = Environment.GetEnvironmentVariable("ContainerName");
var file = req.Form.Files["File"];
var filecontent = file.OpenReadStream();
var blobClient = new BlobContainerClient(connection_String, container_Name);
var blob = blobClient.GetBlobClient(file.FileName);
byte[] byteArray = Encoding.UTF8.GetBytes(filecontent.ToString());
using (Stream myblob = new MemoryStream(byteArray))
{
await blob.UploadAsync(myblob);
}
return new OkObjectResult("file uploaded successfylly");
Zip File Uploaded
And also, able to upload and download using URLs.
I'm using postman to memic a restful api call and trying to access google sheets API end point. When I try to access my endpoint it returns:
{
"error": {
"code": 403,
"message": "The request is missing a valid API key.",
"status": "PERMISSION_DENIED"
}
}
which is fair enough as I did not use my API key. I created a service account and got a json file, but I plan to access using a rest endpoint so need to pass token in header but I'm not sure how.
I looked at the json file and wasn't sure what to extract in order to pass it for my rest call.
Has anyone been able to do this successfully?
Before calling Google Services from Postman, you would need to re-create the flow for getting an access token form service account credentials :
build and encode the JWT payload from the data from credentials files (to populate aud, iss, sub, iat and exp)
request an access token using that JWT
make the request to the API using this access token
You can find a complete guide for this flow is located here: https://developers.google.com/identity/protocols/oauth2/service-account#authorizingrequests
Here is an example in python. You will need to install pycrypto and pyjwt to run this script :
import requests
import json
import jwt
import time
#for RS256 you may need this
#from jwt.contrib.algorithms.pycrypto import RSAAlgorithm
#jwt.register_algorithm('RS256', RSAAlgorithm(RSAAlgorithm.SHA256))
token_url = "https://oauth2.googleapis.com/token"
credentials_file_path = "./google.json"
#build and sign JWT
def build_jwt(config):
iat = int(time.time())
exp = iat + 3600
payload = {
'iss': config["client_email"],
'sub': config["client_email"],
'aud': token_url,
'iat': iat,
'exp': exp,
'scope': 'https://www.googleapis.com/auth/spreadsheets'
}
jwt_headers = {
'kid': config["private_key_id"],
"alg": 'RS256',
"typ": 'JWT'
}
signed_jwt = jwt.encode(
payload,
config["private_key"],
headers = jwt_headers,
algorithm = 'RS256'
)
return signed_jwt
with open(credentials_file_path) as conf_file:
config = json.load(conf_file)
# 1) build and sign JWT
signed_jwt = build_jwt(config)
# 2) get access token
r = requests.post(token_url, data= {
"grant_type": "urn:ietf:params:oauth:grant-type:jwt-bearer",
"assertion": signed_jwt.decode("utf-8")
})
token = r.json()
print(f'token will expire in {token["expires_in"]} seconds')
at = token["access_token"]
print(at)
Note the value of the scope: https://www.googleapis.com/auth/spreadsheets
Probably, you can do all the above flow using Google API library depending on what
programming language you prefer
The script above will print the access token :
ya29.AHES67zeEn-RDg9CA5gGKMLKuG4uVB7W4O4WjNr-NBfY6Dtad4vbIZ
Then you can use it in Postman in Authorization header as Bearer {TOKEN}.
Or using curl :
curl "https://sheets.googleapis.com/v4/spreadsheets/$SPREADSHEET_ID" \
-H "Authorization: Bearer $ACCESS_TOKEN"
Note: you can find an example of using service account keys to call Google translate API here
I have created a method using GETFILE() service of azure. Reference: https://learn.microsoft.com/en-us/rest/api/storageservices/get-file
public void getImage(){
string storageKey = 'xxxxStorageKeyxxx';
string storageName = '<storageName>';
Datetime dt = Datetime.now();
string formattedDate = dt.formatGMT('EEE, dd MMM yyyy HH:mm:ss')+ ' GMT';
string CanonicalizedHeaders = 'x-ms-date:'+formattedDate+'\nx-ms-version:2016-05-31';
string CanonicalizedResource = '/' + storageName + '/<shareName>/<dirName>/<File Name>\ntimeout:20';
string StringToSign = 'GET\n\n\n\n\napplication/octet-stream\n\n\n\n\n\n\n' + CanonicalizedHeaders+'\n'+CanonicalizedResource;
Blob temp = EncodingUtil.base64Decode(storageKey);
Blob hmac = Crypto.generateMac('HmacSHA256',Blob.valueOf(StringToSign),temp ); //StringToSign
system.debug('oo-'+EncodingUtil.base64Encode(hmac));
HttpRequest req = new HttpRequest();
req.setMethod('GET');
req.setHeader('x-ms-version','2016-05-31' );
req.setHeader('x-ms-date', formattedDate);
req.setHeader('content-type','application/octet-stream');
string signature = EncodingUtil.base64Encode(hmac);
string authHeader = 'SharedKey <storageName>'+':'+signature;
req.setHeader('Authorization',authHeader);
req.setEndpoint('https://<storageName>.file.core.windows.net/<shareName>/<dirName>/<file Name>?timeout=20');
Http http = new Http();
HTTPResponse res;
res = http.send(req);
}
The above was working fine and giving the 200 as response code. But, my main goal is to display/download the respective image which i retrieved through REST API. How can i achieve that?
So a few things before I answer your question:
File storage is not really suitable for what you're trying to accomplish (it's possible though).
You should look at Blob storage for this as blob storage is more suitable for this kind of scenario.
Assuming you go with Blob storage, there are a few things you could do:
If the blob container (equivalent to a share in file storage) has an ACL is Blob or Container (i.e. blobs in a container are publicly available), you could simply return the blob's URL (Same is your request URL in code above) in your response and then create a link in your application with href set to this URL.
If the blob container has an ACL as Private (i.e. blobs are not publicly available), you would need to create a Shared Access Signature (SAS) token on that blob with at least Read permission and then create a SAS URL. A SAS URL is simply blob URL + SAS token and return this SAS URL in your response and then create a link in your application with href set to this URL.
Since an Azure File Share is always private, if you were to use Azure File service to serve a file, you would do the same thing as 2nd option I listed above. You will create a SAS token on the file with at least Read permission and then return the SAS URL in the response and then create a link in your application with href set to this URL.
To read about Shared Access Signature, you may find this link helpful: https://learn.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1.
To create a Shared Access Signature using REST API, you may find this link helpful: https://learn.microsoft.com/en-us/rest/api/storageservices/Constructing-a-Service-SAS?redirectedfrom=MSDN
I'm creating a Google Drive service using the Drive REST Api v3 in Angular 2. Most of the functionality is in place: view file, download, create etc.. but I cannot find how to name a file (either when creating a file or updating).
I'm using the following docs pages: create and update. They say the file name should be part of the request body. The relevant code from my Google Drive service is bellow.
createFile(name :string, content :string) :Promise<Object> {
let headers = new Headers({
'Content-Type': 'text/markdown',
'Authorization': 'Bearer ' + this.token,
'name': name //TODO name not working!
});
let options = new RequestOptions({ headers: headers }); // Create a request option
return this.http
.post('https://www.googleapis.com/upload/drive/v3/files' + '?uploadType=multipart', content, options)
.toPromise();
}
updateFile(id :string, content :string, name :string) :Promise<Object> {
let headers = new Headers({
'Content-Type': 'text/markdown',
'Authorization': 'Bearer ' + this.token,
'id': id,
'name': name //TODO name not working!
}); //generate headers
let options = new RequestOptions({ headers: headers }); // Create a request option
return this.http
.patch('https://www.googleapis.com/upload/drive/v3/files/' + id + '?uploadType=multipart', content, options)
.toPromise();
}
To summarise files are being created and updated fine (including content) but naming and renaming a file doesn't work at all.
Thanks for any help.
Try placing name in the request body and not in the request header as described in the Files: create:
Request body
In the request body, supply a Files resource with the following properties as the metadata. For more information, see the document on media upload.
To test it, try using API Explorer to help you explore various Google APIs interactively.
Sample Request:
POST https://www.googleapis.com/drive/v3/files?key={YOUR_API_KEY}
{
"name": "My File"
}
Response:
200
{
"kind": "drive#file",
"id": "fileID",
"name": "My File"
}
There is also a related SO post that explain how to insert file to Google Drive through API.
Hope this helps.
I also faced that problem. I think there is 3 solutions:
Use multipart upload https://developers.google.com/drive/v3/web/multipart-upload with different headers for file metadata and actual file. Me myself stuck there, didn't found how to add boundaries to separate request headers in Angular 2+
Upload file in two requests. First to create empty file with metadata (response will provide id of the file) and second to actually "update" the file.
Use resumable upload. First request to "setup metadata" (will not even create empty file) and get "special link" where to send request to upload actual file. And this approach have some other features, like uploading in chunks.https://developers.google.com/drive/v3/web/resumable-upload
Here is the link to another Question with implementation of resumable upload in Angular 2 and DRIVE REST API V3
Angular 2+ HTTP POST and GDrive API. Resumable file upload with name
I hope it might be useful.
You are trying to set the name using an http header. This is wrong. I can't begin to understand how you thought that was the way to do it, so you need to go back and reread the Drive API documentation.
In short, the name: "name" should be a JSON object passed in the body of the request, not in an http header.
Is there a way to achieve downloading via. the google-php-api? I have tried the following:
using the medialink and trying to curl the object (Returns "Login Required")
reading the guzzle response stream (comes back empty even though all the headers have the correct data)
I am able to see everything but the body of the file via. the API.
Edit:
I am of course able to download the file via the medialink, taken it is set to public - however that will not work for this situation.
The solution is as follows...
You must make an authorized HTTP request, to do this you must:
$object = $service->objects->listObjects(BUCKET, OBJECT);
$http = $client->authorize();
$request = new GuzzleHttp\Psr7\Request('GET', $object->getMediaLink());
$response = $http->send($request);
$body = $response->getBody()->read($object->getSize());
The above is a small snippet but the jist of what you need to get the contents of a file.