I am trying to figure out how to use Google Cloud Storage for my app. The app should allow any user to POST (or PUT) objects into a Bucket and also let them read any files from said bucket. I am confused as to how I am supposed to form my POST requests in order to make this work.
I have just been playing around, sending requests to see if I can upload a file into a bucket. I have tried forming a request according to this example:
https://cloud.google.com/storage/docs/json_api/v1/how-tos/upload
POST /upload/storage/v1/b/<bucketname>/o?uploadType=media&name=<objectname> HTTP/1.1
Host: www.googleapis.com
Content-Type: image/jpeg
Content-Length: 100
Authorization: Bearer <your_auth_token>
I am confused as to what exactly the 'auth_token' is. I have tried going into the developers console and generating a 'Public API access' key and attempted to use this, but I got a response saying it was unauthorized.
Am I generating the right type of key? I understand OAuth keys are used when you need access to a Google user's data. I don't need this: I simply need to allow users of my app to add files to a bucket and read from that bucket. Any help to point me in the right direction is appreciated.
It is like an ID to identify you have permission to perform your actions, as you haven´t provide which language you are using, I will provide a golang example
https://github.com/johnbalvin/google-cloud-go/blob/master/storage/resumableUpload.go
Related
I'm following step by step guide on Microsoft's site (https://learn.microsoft.com/en-us/azure/active-directory/develop/v2-oauth2-auth-code-flow#refresh-the-access-token), everything is working correctly but every time I try to refresh the access token, I get this error: AADSTS70000: Provided grant is invalid or malformed. error_codes: 70000.
I've created a Postman collection for testing, also downloaded the official postman collection from the Microsoft's site, everything is working correctly until the access token does not expire. When it expire, trying to refresh the token always lead to an error and I'm pretty stuck with it. I've double and triple checked correspondence between redirect_url, permission, grant, copy/paste errors, waited for the access_token to expire before trying to refresh... I've done almost 100 tests, and every time I'm stuck at the refresh part!
I start with doing the normal call to Microsoft Login API in my browser, and getting the code in query string from the browser (no problems here) (please note that client_id is URL encoded because, in my test environment, client id is an URL due to the configuration of the Drupal portan we're using, I'm truing to recreate the same behaviour in postman)
https://login.microsoftonline.com/{tenant_guid}/oauth2/v2.0/authorize?client_id={myclient_id_urlencoded}&response_type=code&redirect_uri={redirect_uri_urlencoded}&scope=offline_access%20https%3A%2F%2Fgraph.microsoft.com%2Fuser.read&state=12345
Then with the code in query string, i POST to the token endpoint:
POST /{tenant_guid}/oauth2/v2.0/token HTTP/1.1
Host: login.microsoftonline.com
Content-Type: application/x-www-url-form-urlencoded
cache-control: no-cache
Postman-Token: a0456a8d-6979-491f-b61e-86b5d614c577
client_id={myclient_id_urlencoded}
scope=https%3A%2F%2Fgraph.microsoft.com%2Fuser.read
redirect_uri={redirect_uri_urlencoded}
grant_type=authorization_code
client_secret={client_secret_urlencoded}
code=OAQABAAIAAADCoMpjJXrxTq9VG9te-7FXujKZhF...
I receive back an accesso token (that is working like a charm in accessing https://graph.microsoft.com/v1.0/me for an hour) and a refresh token. I would love to get a new pair of access/refresh token when the original access token expires, using the refresh_token grant_type
POST /tenant_guid/oauth2/v2.0/token HTTP/1.1
Host: login.microsoftonline.com
Content-Type: application/x-www-url-form-urlencoded
cache-control: no-cache
Postman-Token: 5d71f813-768e-476c-a97f-c109fba3165e
client_id={myclient_id_urlencoded}
scope=https%3A%2F%2Fgraph.microsoft.com%2Fuser.read
redirect_uri={redirect_uri_urlencoded}
grant_type=refresh_token
cclient_secret={client_secret_urlencoded}
refresh_token=OAQABAAAAAADCoMpjJXrxTq9VG9te-7FX8m6YMg-.....
But no matter if I try before access token expiration or after, closing and reopening postman, I always receive that error back. I've done almost 50 tests (always with the full round of login/authorization to use always a fresh refresh token) with no luck.
Seems like I'm missing something really stupid here because I can't imagine that everybody else is behaving correctly... but really can't find a way out!
So I just got mine working! Here are the required parameters I needed:
client_id = your client id
refresh_token = the refresh token here
grant_type = refresh_token,
client_secret = secret
NOTE: Everything I read told me to URLEncode the values. I found it worked with them UNENCODED - no idea if it will really make a difference or not. Since it is going in the body of the post, which means it is TLS encrypted.
The other important thing was the url I posted to. There seem to be so many examples and none seem to be consistent. I used this format:
https://login.microsoftonline.com/{tenantId}/oauth2/v2.0/token
The last thing is to be sure you are using the correct app id. In my case I was using the appId for the wrong app and it didnt have consent. Hope this helps. I do wish Microsoft would make a concerted effort to spell out things consistently and think like someone who doesn't do security for a living.
Finally resolved thanks yo the Azure Support.
The problem is the client id: as I supposed before, Microsoft allow you to define another application name, but always want to use the GUID client id to submit any request. Unfortunately, it was warning me when I didn't url-encoded it, but did not alert me that it was not correct until I tried to use the refresh token.
So just read very very well the documentation: client_id: The Application (client) ID that the Azure portal – App registrations experience assigned to your app (so not the one you choose).
https://learn.microsoft.com/en-us/azure/active-directory/develop/v2-oauth2-auth-code-flow
Maybe including a format validation in the documentation would help!
I have a .pdf object stored in Google Cloud Storage with Content-Type = application/octet-stream.
When giving temporary access through a signed URL, I extend the URL with:
&response-content-type=application%2Fpdf
Nevertheless, the response coming back from Google Cloud Storage still contains Content-Type = application/octet-stream
Inspecting the request + response through the browser confirms this behaviour.
According to the documentation (https://cloud.google.com/storage/docs/xml-api/reference-headers#responsecontenttype) the response-content-type should ensure Content-Type = application/pdf in my example.
For another usecase, I am succesfully making use of the Content-Disposition override via response-content-disposition, so I am very curious why the response-content-type is not working for me.
Anyone any idea what I am missing to make this work?
Thanks!
As per documentation signed URLs are not authenticated GET requests that support content-type override.
Query String Parameters like response-content-disposition and response-content-type are not verified by the signature. To force a Content-Disposition or Content-Type in the response, set those parameters in the object metadata using gsutil or the XML/JSON API.
I'm building REST API, and when resource is created normally I return HTTP 201 Created along with Location header to specify where that resource is located. But from some reason http client is not redirecting.
I'm using Postman for this. Does anyone have idea on this problem?
In short, a Location header is not sufficient to trigger a client redirect. It must be used in conjunction with a 3xx HTTP status code.
References:
https://en.m.wikipedia.org/wiki/HTTP_location
Redirecting with a 201 created
This is one of those things where the expectation does not meet what actually happens, and the first thing people think is "well that doesn't work properly", as has been suggested in other comments.
The Location is just a random header, and clients, such as Postman or curl or anything else need to be instructed to follow them. Most won't do this by default, as that is an unreasonable default.
YouTube for example returns a body for some responses and a Location tag too. One example would be video uploads. They respond to your original meta-data for the video is sent with a POST, and they shove a Location URL which is the endpoint to upload the video too. If clients just randomly redirected to that you'd be having a bad time.
You can use Paw to make a "sequence", which I believe will let you take values from headers to reuse. This is also possible with Runscope Ghostinspector.
I have been trying to write a bucket policy that will allow (X-HTTP-Method-Override) because my research shows that Facebook loads HTTPS hosted iframe apps via HTTP POST instead of HTTP GET which causes S3 and CloudFront errors.
Can anyone please help me with this problem?
This is what's returned from S3 if I served my Facebook app directly from S3:
<?xml version="1.0" encoding="UTF-8" ?>
- <Error>
<Code>MethodNotAllowed</Code>
<Message>The specified method is not allowed against this resource.</Message>
<ResourceType>OBJECT</ResourceType>
<Method>POST</Method>
<RequestId>B21565687724CCFE</RequestId>
<HostId>HjDgfjr4ktVxqlIBeIlvXT3UzBNuPg8b+WbhtNHOvNg3cDNpfLH5GIlyUUpJKZzA</HostId>
</Error>
This is what's returned from CloudFront if I served my Facebook app from CloudFront with S3 as the origin:
ERROR
The request could not be satisfied.
Generated by cloudfront (CloudFront)
I think the solution should be to write a bucket policy that makes use of X-HTTP-Method-Override... Probably I am wrong though. A solution to this problem would be highly appreciated.
After trying many different ways to get this to work, it turns out that it simply is not possible to make the POST to static content work on S3 as things stand. Even if you allow POST through Cloudfront, enable CORS, change the bucket policy so that the Cloudfront origin identity can GET/PUT etc. it will still throw an error.
As an aside, S3 is not the only thing that balks at responding to such a POST request to static content. If you configure nginx as an origin for a Facebook iframe you will get the same 405 error, though you can work around that problem in a couple of ways (essentially rewriting it to a GET under the covers). You can also change the page (though still static) to be a dynamic extension (.aspx or .php) to work around the issue with nginx.
You can host all your other content on S3 of course, and just move the page that you POST to onto a different origin. With a decent cache time you should see minimal traffic, but it will mean keeping your content in two places. What I ended up doing was:
Creating EC2 instances in an autoscaling group (just in case) to serve the content
They used a cron job to sync the content from S3 every 5 minutes
No change in workflow was required (still just upload content to S3)
It's not ideal, nor is it particularly efficient, but hopefully it will save others a lot of fruitless testing trying to get this to work on S3 alone.
You can set your Cloudfront distribution to allow POST methods.
If you go into your dashboard and edit the Behavior for the distribution
- Then select Allowed HTTP Methods - GET, HEAD, PUT, POST, PATCH, DELETE, OPTIONS
This allows the POST from Facebook to go through to your origin.
I was fighting with S3 and CloudFront for last couple of days. and I confirm that with any bucket policy we cannot redirect POST calls from Facebook to S3 static (JS enriched) contents.
The only solution seems to be the one Adam Comerford mentioned in this thread:
Having a light application which receives Facebook calls then fetching the content from S3 or CloudFront.
If anyone has any other solution or idea it will be appreciated.
you can't change POST to GET - that's the way Facebook loads app page because it also sends data about the current user as POST body (see signed_request for more details). I would suggest you look into fixing your app to make sure it properly responds to POST request.
I'm designing a public API to my company's data. We want application developers to sign up for an API key so that we can monitor use and overuse.
Since the API is REST, my initial thought is to put this key in a custom header. This is how I've seen Google, Amazon, and Yahoo do it. My boss, on the other hand, thinks the API is easier to use if the key becomes merely a part of the URL, etc. "http://api.domain.tld/longapikey1234/resource". I guess there is something to be said for that, but it violates the principle of the URL as a simple address of what you want, and not how or why you want it.
Would you find it logical to put the key in the URL? Or would you rather not have to manually set HTTP headers if writing a simple javascript frontend to some data?
It should be put in the HTTP Authorization header. The spec is here https://www.rfc-editor.org/rfc/rfc7235
If you want an argument that might appeal to a boss: Think about what a URL is. URLs are public. People copy and paste them. They share them, they put them on advertisements. Nothing prevents someone (knowingly or not) from mailing that URL around for other people to use. If your API key is in that URL, everybody has it.
It is better to use API Key in header, not in URL.
URLs are saved in browser's history if it is tried from browser. It is very rare scenario. But problem comes when the backend server logs all URLs. It might expose the API key.
In two ways, you can use API Key in header
Basic Authorization:
Example from stripe:
curl https://api.stripe.com/v1/charges -u sk_test_BQokikJOvBiI2HlWgH4olfQ2:
curl uses the -u flag to pass basic auth credentials (adding a colon after your API key will prevent it from asking you for a password).
Custom Header
curl -H "X-API-KEY: 6fa741de1bdd1d91830ba" https://api.mydomain.com/v1/users
passing api key in parameters makes it difficult for clients to keep their APIkeys secret, they tend to leak keys on a regular basis.
A better approach is to pass it in header of request url.you can set user-key header in your code .
For testing your request Url you can use Postman app in google chrome by setting user-key header to your api-key.
I would not put the key in the url, as it does violate this loose 'standard' that is REST. However, if you did, I would place it in the 'user' portion of the url.
eg: http://me#example.com/myresource/myid
This way it can also be passed as headers with basic-auth.