I am trying to collect data from amazon web services. Every time I make the call I get back a 403 Forbidden.
This is what my code looks like (the link is jumbled):
Invoke-RestMethod -Uri "https://hosted-data-work.s3.amazonaws.com/20161121T220310.324/dw_split/73610000000000001/assignment_fact/part00101.gzAWSAccessKeyId=ASIAJVX3JXfd5dfdfRKJNGM74Q&Expires=1479839499&Signature=J4JdyX53AwH6wExVmoVAtkweCEI%3D&resp222onse-contentdisposition=inline%3B%20filename%3D%22assignment_fact-00000-095582fd.gz%22%3B&x-amz-security-token=bluh" -Method Get
The link above is a download file. I just want to get the data the simplest way possible. What else do I need to add in the call? I have no clue about aws!
How did you generate that URL? It looks like a presigned URL, which means the authorization for accessing the object will be granted based on the credentials used when presigning. There are a couple of possible reasons that could be giving you a not authorized response:
The credentials used to generate the presigned URL do not actually have permissions to read the object. Double check your IAM policies and/or ACLs for the bucket and the IAM user which generated the URL.
The signature got truncated/corrupted between the the time you generated the presigned URL and the time you tried to use it. Try logging the url when you generate it and again when you use it and compare to make sure they match exactly.
Presigned URLs expire after a specified validity period which cannot be longer than 1 week. Make sure you are generating a fresh URL when needed and setting the expiration appropriately.
Any of those could be causing the result you're seeing.
I was misinformed and took a bad approach at this problem. I did not know I can simply download the file to my computer. I though it had to be transferred from bucket to bucket, then to my computer.
Related
I'm trying to download a CSV file from a website in Data Factory using the HTTP connector as my source linked service in a copy activity. It's basically a web call to a url that looks like https://www.mywebsite.org/api/entityname.csv?fields=:all&paging=false.
The website uses basic authentication. I have manually tested by using the url in a browser and entering the credentials, and everything works fine. I have used the REST connector in a copy activity to download the data as a JSON file (same url, just without the ".csv" in there), and that works fine. But there is something about the authentication in the HTTP connector that is different and causing issues. When I try to execute my copy activity, it downloads a csv file that contains the HTML for the login page on the source website.
While searching, I did come across this Github issue on the docs that suggests that the basic auth header is not initially sent and that may be causing an issue.
As I have it now, the authentication is defined in the linked service. I'm hoping that maybe I can add something to the Additional Headers or Request Body properties of the source in my copy activity to make this work, but I haven't found the right thing yet.
Suggestions of things to try or code samples of a working copy activity using the HTTP connector and basic auth would be much appreciated.
The HTTP connector expects the API to return a 401 Unauthorized response after the initial request. It then responds with the basic auth credentials. If the API doesn't do this, it won't use the credentials provided in the HTTP linked service.
If that is the case, go to the copy activity source, and in the additional headers property add Authorization: Basic followed by the base64 encoded string of username:password. It should look something like this (where the string at the end is the encoded username:password):
Authorization: Basic ZxN0b2njFasdfkVEH1fU2GM=`
It's best if that isn't hard coded into the copy activity but is retrieved from Key Vault and passed as secure input to the copy activity.
I suggest you try to use the REST connector instead of the HTTP one. It supports Basic as authentication type and I have verified it using a test endpoint on HTTPbin.org
Above is the configuration for the REST linked service. Once you have created a dataset connected to this linked service you can include it in you copy activity.
Once the pipeline executes the content of the REST response will be saved in the specified file.
I have an application which is uploading objects to Google Cloud Storage using signed URLs and I'd like to know if it's possible to make the object public during the sign/upload step.
I know it's possible to make the object publicly visible by setting the policy on its bucket or by using the client library/making a REST request after it's been uploaded, but in order to minimize the impact on my workflow, I'd like to do it all in one go. Is this possible? If it can be done, I'm assuming it's by setting a header when signing the URL or when making the REST request using the signed URL but I haven't been able to find documentation which covers this.
UPDATE:
I've just found the Extension/Custom Headers section of the XML API docs which claims that this can be achieved using the x-goog-acl header (e.g. x-goog-acl: public-read). Unfortunately, this does not work. The object is not publicly visible after setting the header when signing the URL and when uploading the file.
Quoting the Cloud Storage documenation regarding Signed URLs:
When specifying the name:value pairs for headers, keep in mind the following:
Remove any whitespace around the colon that appears after the header name.
For example, using the custom header x-goog-acl: private without removing the space after the colon returns a 403 Forbidden error, because the request signature you calculate does not match the signature Google calculates.
So the solution could be setting the header value as x-goog-acl:public-read instead of x-goog-acl: public-read.
Amazon has an administration page for content sent to your Kindle. This page uses an undocumented HTTP API that sends requests like this:
{
"csrfToken":"gEABCzVR2QsRk3F2QVkLcdKuQzYCPcpGkFNte0SAAAAAJAAAAAFkUgW5yYXcAAAAA",
"data":{"param":{"DeleteContent":{"asinDetails":{"3RSCWFGCUIZ3LD2EEROJUI6M5X63RAE2":{"category":"KindlePDoc"},"375SVWE22FINQY3FZNGIIDRBZISBGJTD":{"category":"KindlePDoc"},"4KMPV2CIWUACT4QHQPETLHCVTWEJIM4N":{"category":"KindlePDoc"}}}}}
}
I made a wrapper library for the previous API they used, but this time they have added CSRF tokens, making each session unique. That is a bit of a show stopper, and I was wondering how I can get hold of these tokens. I did not find it in the cookies. This is for use in a Chrome Extension, so issues like CORS is not an issue.
Actually, after manually going searching the Response tab of each request in the "XHR" and "Doc" tab, I was able to find out that this token is set in an inline script in the myx.html (main page):
var csrfToken = "gPNABCIemSqEWBeXae3l1CqMPESRa4bXBq0W7rCIAAAAJAAAAAFkUlo1yYXcAAAAA";
This means it is set on the window object, making it available for all there. I guess this means a Chrome extension would need to fetch this page and manually parse the html to retrieve this token. Sad, but doable, although highly fragile :-(
So when using Facebook login, it turns out that if you set response_type to token or code%20token then the "response data is included as a URL fragment" on the URL that redirects to your app. Seriously, why? When would that ever be useful to anyone?
Anyway, I'm hoping that this is somehow security related and that it's there for a reason other than to annoy people, but I would otherwise like to know if there's a way to disable it, since I'm using Angular and it's a real pain to handle urls with hashbangs.
If I set a response_type of code only, the code is returned as a nice and clean query parameter, but I would like to receive both in order to perform additional validations. Is there a way to change this behaviour? Thanks.
EDIT:
Well it turns out that setting a response_type of code returns #/= at the end of the url, so there's no way to get a clean querystring. It has already been established that there is no way to change this behaviour, but I'm still interested in finding out why Facebook is doing it. Is it security related? What is the purpose of these url fragments?
Answers
To your first question, I imagine you would use token when you are handling everything in-browser and not processing at the server.
To your main question, as far as I know, you can not change how Facebook redirects successful authorizations. Facebook doesn't give you the token as a clean query parameter. If you use token or code token, what you want will be in the hash fragment. Sorry.
NOTE: This may not be a problem for you. I'm not sure what you want with "additional validations", but when you use code token, exchanging the resulting code gets you a different token than the one you just got embedded in the URI. Both are now valid and will expire separately. Really, you probably need either the code or the token; both won't help you since they're not linked.
Examples of using all three methods:
If you set response_type to code you get redirected to something like:
http://example.com/redirect_uri?state=thestatepassed&code=AQDN9E9GYjA8NbyCt
87_jV5vHnCQylNxmBswo6Z1BsrR7lmTPom6wjrzfan6P4GBLDt3EQrfPg0xSLoMLxBBfscsyfSY
JNM2vu9OoqEQXXSJCTUq_fMpUwqkYbCHp-GAqL4H1ymbMz7zPKAG61V9BtKTSuez39yhawOu7l-
6ww4thP41Ka9PVcknTQ6fPjPXKYSyxEmANps9zevCPFsXpBZCO7_dms65-ZZuG2wVBd16gFnBZH
q8EY0qih6-9o61wXh7bBvVPVSZ2im7Oj1nx47YgDpbD3X0XdlVhUoGYmBdER9hNmIC2PmmY7VAo
PlYCZc#_-_
From there, you need to exchange the code through Facebook's OAuth endpoint to receive an access token. The access token can then be used against the Facebook APIs.
If you set response_type to token you get redirected to something like:
http://example.com/redirect_uri#access_token=CAACYnSxGEhsBAJBg0ohZBhAf7pKEU
sm5ytZAZBzKjISFuRun2ZByZCqEsxrVIgtiO7iIlJZBBbGm6fRPQXItZCX6YgjPknUBsr78tJtv
W6fySULrUo9vdW57ZCMUUIlNaeZAcU8DzUXKmFpgotOyhE3jXYz1c3eu00Aii0AZBsPrtrwjpwQ
mV8VYQNiqKZBIsqOrIwZD&expires_in=4168&state=thestatepassed
You have everything you need to call the Facebook APIs. The access token returned is valid, but should probably be checked against Facebook's token inspection endpoint if you're doing something server-side. (Really, at that point, just use code. I've never done it this way, so good luck.)
As you noted, the access token is now contained within the hash fragment of the url.
If you set response_type to code token you get redirected to something like:
http://example.com/redirect_uri#code=AQAtzsjPivFPsJ538KFlPuhLaK6pDMlrGDiwmi
KDcpgNfWrO1EdX5i6zK_Op2D0QDEXZLyifXxh4TSeBZCWhnkl7YV1LMyEkbPURAWSoqRoeG7tfM
4nB4nDAHOK0H9umb0KnoypRT1pP05FJKhl2QjpCJrPPFDHl6y-1X9ZMj1uVHtmPNi4tG_6QAbuL
RaGadBkekb22uJ0iwSrWc9OKi6ET70lCTYb18hbwUkzHXtTq12nNEdsDJ7Ku2wEBwMygFwErYDX
CrnPoFoah_z0UPCfv3XZLy98Dhlzw_lnx8nnCB-PCppOWRqmydvQJehPd86k&access_token=C
AACYnSxGEhsBALXHRQwfm4UoauRlZBJDVpZCiM6ZCuM3bE965F5JVBfBB8inTFdhfJ5obnonSqa
m3v8FbWhHXrhRSx4ugwAmmDaWyxmPELWqSrkrDO5ueTUXhhjiEZBTd7HjCVCSOXXhOSo3DjEVSC
lOaZBfqmXsprYyc6LJC39sroCcHYCZCv&expires_in=5183426&state=thestatepassed
Now you have both an access token and a code (that expiration applies to the token, not the code). As stated earlier, that code can be exchanged for an access token in the usual way, but the returned access token will be different from the one you just got embedded in the URI.
But, again, what you care about is in the hash fragment.
See the AngularJS doc on $location for accessing the hash. Someone more knowledgable than me can speak to how that works with routes and how best to scrape the parameters.
Well today I had the same situation and managed to resolve it in a way I could get the access token from my server.
After getting the code using response_type=code I called via GET the following url:
https://graph.facebook.com/v3.2/oauth/access_token?app_id=MY_APP_ID&code=MY_CODE&client_id=MY_APP_ID&client_secret=MY_APP_SECRET&redirect_uri=YOUR_LOGIN_REDIRECT_URI
This route returns a JSON response, which has the access_token parameter, in this way:
{
"access_token": "ABAECAEFAEFAEA...",
"type": "bearer",
"expires_in": 838383838
}
So there you go :) I hope this works for you too
There are many other question related to this, but they didn't help me fix my problem.
I'm using the Facebook server-side login for a website, which I want to test locally. The path that initiates the login action is [http://localhost:8080/fblogin] (this redirects to the Facebook login dialogue, and goes from there).
I can successfully get the code, but when I try to exchange that for an access token, I get the following error:
{"error":{"message":"Missing redirect_uri parameter.","type":"OAuthException","code":191}}
I am providing the redirect_uri, url encoded and it is the same as the one I use to get the first code. Here is the url I'm using to request the access token (with the all-caps query string parameters replaced with their actual values, of course):
https://graph.facebook.com/oauth/access_token?client_id=CLIENT_ID&redirect_uri=http%3A%2F%2Flocalhost%3A8080%2Ffblogin&client_secret=CLIENT_SECRET&code=CODE_FROM_FB
I suspect this might have to do with how my app is set up on Facebook. Here are the values I have set:
Display Name: (an actual display name here)
App Domains: localhost
Contact email: (an actual email here)
Site URL: [http://localhost:8080/fblogin]
What do I need to tweak in the settings to get this to work? Or does this look correct?
By the way, if it makes any difference, I am using the Play! framework, version 2.0.1
After digging around a little more, I found that it was necessary for me to use POST when sending the request from my server to get the access token.
Interesting that using POST worked for you as this didn't for me.
In any case, did you add the query parameters using setQueryParameter()? (see How to make multiple http requests in play 2?)