I am trying to query the Workfront REST Services from PowerShell
I am using a URL like this
https://ourcompany.attask-ondemand.com/attask/api/v4.0/project/search?apiKey=XYZetc
This returns JSON in both IE and Chrome and works in my Web Service tester.
All this runs behind a corporate proxy obviously.
The PowerShell I am using is
$postResult = Invoke-RestMethod -Uri $URI -Method "GET" -Proxy
http://internalproxyname:80 -ProxyUseDefaultCredentials
This fails with an Error
{"error":{"class":"com.attask.common.AuthenticationException","message":"You
are not currently logged in"}}
This looks like an Error at the attask END not the proxy at our end (I get different errors running this as a non auth user or with mangled credentials passed to the Proxy
The docs suggest I don't need to be logged in if I was using an apiKey. I am not logged in in the browsers I am using (I don't even have a user account on the workfront instance)
I have trawled various blogs and stack answers to no avail. Can anyone point me in the right direction for figuring out what is going on? or what I might be doing wrong.
I have Enabled a trust all certs policy and set the validation callback to Ignore within the powershell
but equally I've tried this with these turned off and also investigated various properties on the ServicePointManager. I can produce any number of different errors/issues but the closest I get seems to be the above.
Oh and the Workfront API docs and examples being wrong didn't help me when I was getting started :-)
many thanks
Steve
OK this was me being stupid. There was a bug in the code generating the URI (an extra slash) and the attask default error response is auth error not mangled request.
For reference the URL needs to be in the form shown in my original post. Don't miss off the api version number and don't use a port number as the code samples show.
Always look for the simple things first (I should remember that)
Doh!
Related
I need to scrape a webpage.
I specifically need to extract the section "USDT Funding Market" on the page.
$WebExtract = Invoke-WebRequest -Uri "https://www.kucoin.com/margin/lend/USDT"
Although $WebExtract.content or $WebExtract.AllElements does not contain the "USDT Funding Market".
The reason for this is that your initial web request is only returning you the HTML of the page, however a lot of modern web applications like this have some JavaScript under the hood that will reach out to their API's and fill the page with data after the initial HTML is loaded.
The good news is, usually with public websites like this you can hit the underlying API's without worrying about authentication - which actually makes parsing the data easier, as it's typically in JSON format.
Try opening the debug console in your browser (usually F12), open the network tab and refresh the page - you can typically find these API requests/endpoints in there, e.g;
Simply copy that URL out and invoke the web request to the API instead, for example;
$request = Invoke-WebRequest -Uri 'https://www.kucoin.com/_api/currency/prices?base=USD&targets='
$json = $request.Content | ConvertFrom-Json
$json.data
This might not be the specific API endpoint/data you're after, but hopefully this points you in the right direction. If you poke around in the console you should be able to find the one you're looking for.
EDIT:
Apologies for the misleading information here, I didn't notice that these numbers were updating in real time on the site. The above is still pretty common for a lot of websites so I'll leave it, however on sites that are this responsive/live another common technology used is Web Sockets.
If you have a look at the last lines of the console, you should be able to see a request ending with 'transport=websocket' to a URL like this;
wss://ws-web.kucoin.com/socket.io/?token=...%3D%3D&format=json&acceptUserMessage=false&connectId=connect_welcome&EIO=3&transport=websocket
If you select this line and head over to the Messages tab in your browser console/debugger, you'll be able to see the web socket messages being returned;
This looks like the data you're after, but I'm not hugely familiar with querying Web Sockets through PowerShell.
Simply invoking a web request won't work here, as it's using the web sockets protocol to handle the communication. You would also need to find a way to get a valid token for opening a web sockets connection (likely just a web request for that).
Perhaps these posts will help;
http://wragg.io/powershell-slack-bot-using-the-real-time-messaging-api/
How to use a Websocket client to open a long-lived connection to a URL, using PowerShell V2?
The login page is this: https://login.procore.com/
I feel like I'm close to getting it to work, but have hit a brick wall due to a lack of understanding of login procedures. Here is the code so far, without the actual sign in information.
$r=Invoke-WebRequest https://login.procore.com/ -SessionVariable fb
$form = $r.Forms[0]
$form.Fields["session_email"] = "xxxxxxxxx"
$form.Fields["session_password"] = "xxxxxxxx"
$r=Invoke-WebRequest ('https://login.procore.com/' + $form.Action) -WebSession $fb -Method $form.Method -Body $form.Fields
Could someone help me understand what is missing? I did notice that $form.Fields contains an empty field named: session_sso_target_url, but honestly have no clue what it means, or how to use it.
You've given me insufficient info to provide a complete answer, because I don't have a login and don't see a way to sign up for a free trial, and you haven't stated what kind of error you are getting.
I hazard a guess that session_sso_target_url relates to federation, which is semantically related to single sign-on (SSO). In federation, an application is configured to accept logins from another login domain. The obvious example in corporateland is ADFS, but any time you see an app that says Login with Facebook or Login with Google, that's the same thing. Federation is a big topic. The meaning of having a target URL is that the browser is often redirected to the identity provider (ADFS / FB / GOOG etc) with the callback URL that the browser should come back to once it is authenticated.
Suffice it to say that I suspect that you need do nothing with this field! And the reason I say this is because I hit it with Fiddler.
You should know about Fiddler. It is a cost-free debugging proxy from Telerik. I am not affiliated with Telerik, but I owe them hours of saved time when web scraping. (It is not the only tool for the job, and if any moderator deems that I am violating site rules, I will be happy to sanitise this post.)
Do this:
install Fiddler
Set it up to listen on 127:0.0.1:whatever and to be your system proxy
in Tools > Options > HTTPS, set it to decrypt HTTPS (this will replace all certs with auto-generated self-signed ones, so do not leave this running while you perform other tasks)
Set your filters to only include traffic to *.procore.com
Log in through your browser - you should now see web traffic in the left-hand pane. This captured traffic is your baseline.
Select any one web request and look at the Inspectors tab in the right-hand pane. You can look at Raw, Forms, Cookies, etc. This gives you a low-level view of what your client is doing.
Run your code snippet. You can now compare the differences between the baseline and your code, and adjust accordingly.
TL;DR
I need to submit multiple forms on a site that reloads pages but the server closes session after every request on a single page app that only has one URL and I think this is preventing my POST method from going through. The main problem is every request uses the first state of the page and I can't get to states farther in the process.
What I'm trying to accomplish
I am trying to automate a process that requires me to go to take the following steps in order:
Navigate to webpage. (GET)
Click on a button that reloads the page with new data but uses same URL. (POST)
Enter text into a field on new page.
Click on the form to submit the text. (POST)
... perform unrelated admin tasks....
I'm trying to automate this process using the Invoke-WebRequest cmdlet in PowerShell following steps similar to those found in the PowerShell Cookbook regarding FaceBook login.
When I run an Invoke-WebRequest from a completely fresh PowerShell Session I get a response but I can't reuse that session ever again. To make another request I need to create a new -SessionVariable or use -DisableKeepAlive.
The server will always return a connection close in the response no matter what even though it is using http 1.1 and it's not my site so I can' change this.
So how can I go about establishing a connection to the server that I can reuse to POST the form data? I feel like it should be doable because it is clearly happening on the WebPage itself.
When I go to the WebPage, open the Developer Tools in Chrome and step through the process the header contains this in the Form Data field:
RAW
ggt_textbox%2810007%29=&action=adminloginbtn&ggt_hidden%2810008%29=2
PARSED and DECODED
ggt_textbox(10007):
action:adminloginbtn
ggt_hidden(10008):2
If I try to do something like this:
Invoke-WebRequest $uri -SessionVariable session -Verbose -Method POST -Body "ggt_textbox%2810007%29=&action=adminloginbtn" -DisableKeepAlive
It returns the page I'm expecting in step 2. So I performed steps 3 and 4 in Chrome to try and do the same thing. I get the the following Form Data in Chrome Dev Tools:
RAW
ggt_textbox%2810006%29=textIentered&action=loginbtn&ggt_hidden%2810008%29=3
PARSED
ggt_textbox(10006):textIentered
action:loginbtn
ggt_hidden(10008):3
So that made me think I could do something like this:
Invoke-WebRequest $uri -WebSession $session -Verbose -Method POST -Body "ggt_textbox%2810006%29=textIentered&action=loginbtn&ggt_hidden%2810008%29=3"# -DisableKeepAlive
But since the main page and the login page use the same URI it tries to POST to a form that doesn't exist because it's looking at the very first page.
I did some more digging and found when I perform this same action from the webpage itself it returns a 302 Moved Temporarily status code the response header actually has a cookie in it (still closes the connection) which is a first and then appears to do a GET request using the new cookie and I'm now logged into the admin page.
So I think I have two problems I need to get around:
How can I get to the form that exists after I click the first button since they use the same URI?
How can I get around the 302 status since I'm only getting back a header and nothing else. I think I need to do a GET request using the cookie from the header but I'm not sure how to specify a cookie with Invoke-WebRequest. I think I would need to use the -Header parameter and specify Cookie: COOKIENAME=CookieID
I think most of all I need to get through my first question and then from there I can start working towards my second.
All help is appreciated and I can provide any header/source needed but the web page is super simple so there is not a whole lot going on in the front end other than a couple of buttons and a logo with a little bit of inline JavaScript.
EDIT
After doing some additional reading about 302 and redirects I found out that shouldn't be a problem. The reason for this is explained in this question.
I figured out my problem. The inline JavaScript is validating to make sure the length of the string is greater than 0 before submitting the login. I don't think there is away to by pass the client side validation easily in a script.
I am trying to collect data from amazon web services. Every time I make the call I get back a 403 Forbidden.
This is what my code looks like (the link is jumbled):
Invoke-RestMethod -Uri "https://hosted-data-work.s3.amazonaws.com/20161121T220310.324/dw_split/73610000000000001/assignment_fact/part00101.gzAWSAccessKeyId=ASIAJVX3JXfd5dfdfRKJNGM74Q&Expires=1479839499&Signature=J4JdyX53AwH6wExVmoVAtkweCEI%3D&resp222onse-contentdisposition=inline%3B%20filename%3D%22assignment_fact-00000-095582fd.gz%22%3B&x-amz-security-token=bluh" -Method Get
The link above is a download file. I just want to get the data the simplest way possible. What else do I need to add in the call? I have no clue about aws!
How did you generate that URL? It looks like a presigned URL, which means the authorization for accessing the object will be granted based on the credentials used when presigning. There are a couple of possible reasons that could be giving you a not authorized response:
The credentials used to generate the presigned URL do not actually have permissions to read the object. Double check your IAM policies and/or ACLs for the bucket and the IAM user which generated the URL.
The signature got truncated/corrupted between the the time you generated the presigned URL and the time you tried to use it. Try logging the url when you generate it and again when you use it and compare to make sure they match exactly.
Presigned URLs expire after a specified validity period which cannot be longer than 1 week. Make sure you are generating a fresh URL when needed and setting the expiration appropriately.
Any of those could be causing the result you're seeing.
I was misinformed and took a bad approach at this problem. I did not know I can simply download the file to my computer. I though it had to be transferred from bucket to bucket, then to my computer.
There are many other question related to this, but they didn't help me fix my problem.
I'm using the Facebook server-side login for a website, which I want to test locally. The path that initiates the login action is [http://localhost:8080/fblogin] (this redirects to the Facebook login dialogue, and goes from there).
I can successfully get the code, but when I try to exchange that for an access token, I get the following error:
{"error":{"message":"Missing redirect_uri parameter.","type":"OAuthException","code":191}}
I am providing the redirect_uri, url encoded and it is the same as the one I use to get the first code. Here is the url I'm using to request the access token (with the all-caps query string parameters replaced with their actual values, of course):
https://graph.facebook.com/oauth/access_token?client_id=CLIENT_ID&redirect_uri=http%3A%2F%2Flocalhost%3A8080%2Ffblogin&client_secret=CLIENT_SECRET&code=CODE_FROM_FB
I suspect this might have to do with how my app is set up on Facebook. Here are the values I have set:
Display Name: (an actual display name here)
App Domains: localhost
Contact email: (an actual email here)
Site URL: [http://localhost:8080/fblogin]
What do I need to tweak in the settings to get this to work? Or does this look correct?
By the way, if it makes any difference, I am using the Play! framework, version 2.0.1
After digging around a little more, I found that it was necessary for me to use POST when sending the request from my server to get the access token.
Interesting that using POST worked for you as this didn't for me.
In any case, did you add the query parameters using setQueryParameter()? (see How to make multiple http requests in play 2?)