If website pops out a 503 then webclient will just throw an exception.
For example, go to http://www.google.com/sorry/?continue=http%3A%2F%2Fwww.google.com%2Fsearch%3Fq%3Dkucing
If we open it in internet explorer it'll open a page. If we use livehttpheader it returns a 503 rather than 200. However, it still show something.
Now try curl the same page.
http://www.google.com/sorry/?continue=http%3A%2F%2Fwww.google.com%2Fsearch%3Fq%3Dkucing
The curl will just stop failing. So how to make curl treat 503 like 200?
Samples are when we try to search something at google. Google sometimes require captcha. Well, just pop that out so I can fill the captcha. But webclient simply throw an exception without assigning the content to a file. That's not good.
How to ensure that webclient do not throw things out.
Same goes for curl
No body is answering it so I'll just tell CURLOPT_FAILONERROR
Set that to false.
I think there is something similar for webclient. Will set this as the answer unless somebody comes up with something better.
For WebClient you need to process the WebException.Response. E.g. this LINQPad query dumps the HTML provided by my web server's "Not Found" error web page:
Dim wc = New System.Net.WebClient
Try
Dim rd = wc.DownloadData(New Uri("http://localhost/test"))
rd.Dump
Catch Ex As System.Net.WebException
Dim rs = Ex.Response
Call (New StreamReader(rs.GetResponseStream)).ReadToEnd.Dump
End Try
Now actually using WebClient.
Related
Is it possible to intercept the request going through Charles and immediately return 500 error code without sending this request to the server?
Can't find any information on this. All resources suggest to wait for the response and then change HTTP response code to 500.
I assume you have already tried adding a rewrite rule to make the request to be returned with the 500 status. Have you tried combining this with a map local, to an empty file on your disk, for instance? It may work.
If this doesn't work too, I think I would do a Map Remote to another path on my localhost (for instance: http://localhost:8081/exected-response-500) and make that URL to return the 500 status error (in my case I would use a basic Spring Boot app to achieve this).
Hi I have a REST web service.
The service method throws java.lang.Exception but the client always receive Http Code 200 which is success.
Shouldn't the rest service return other code than 200?
Probably yes. 200 OK would imply that the request would have succeeded. It sounds like this was not a success so it should have been something else.
If your code is throwing an Exception, it's supposed to be broken and not delivering whatever your request wanted. That indeed should not return a "200 OK" response.
Here's a few things to do:
Don't throw java.lang.Exception. Try to be more specific about the exceptions on your code so it's easier to identify the problem when something crashes.
Determine WHAT is happening that is throwing that exception.
In most cases, send a 400 Bad Request when something is wrong and the client is guilt for that. If it's a server-side problem, it's usually a 500 Internal Server Error response. Do a little research on HTTP codes here.
Make sure to document any rules that need to be followed so this problem don't occur in the future.
TL;DR
I need to submit multiple forms on a site that reloads pages but the server closes session after every request on a single page app that only has one URL and I think this is preventing my POST method from going through. The main problem is every request uses the first state of the page and I can't get to states farther in the process.
What I'm trying to accomplish
I am trying to automate a process that requires me to go to take the following steps in order:
Navigate to webpage. (GET)
Click on a button that reloads the page with new data but uses same URL. (POST)
Enter text into a field on new page.
Click on the form to submit the text. (POST)
... perform unrelated admin tasks....
I'm trying to automate this process using the Invoke-WebRequest cmdlet in PowerShell following steps similar to those found in the PowerShell Cookbook regarding FaceBook login.
When I run an Invoke-WebRequest from a completely fresh PowerShell Session I get a response but I can't reuse that session ever again. To make another request I need to create a new -SessionVariable or use -DisableKeepAlive.
The server will always return a connection close in the response no matter what even though it is using http 1.1 and it's not my site so I can' change this.
So how can I go about establishing a connection to the server that I can reuse to POST the form data? I feel like it should be doable because it is clearly happening on the WebPage itself.
When I go to the WebPage, open the Developer Tools in Chrome and step through the process the header contains this in the Form Data field:
RAW
ggt_textbox%2810007%29=&action=adminloginbtn&ggt_hidden%2810008%29=2
PARSED and DECODED
ggt_textbox(10007):
action:adminloginbtn
ggt_hidden(10008):2
If I try to do something like this:
Invoke-WebRequest $uri -SessionVariable session -Verbose -Method POST -Body "ggt_textbox%2810007%29=&action=adminloginbtn" -DisableKeepAlive
It returns the page I'm expecting in step 2. So I performed steps 3 and 4 in Chrome to try and do the same thing. I get the the following Form Data in Chrome Dev Tools:
RAW
ggt_textbox%2810006%29=textIentered&action=loginbtn&ggt_hidden%2810008%29=3
PARSED
ggt_textbox(10006):textIentered
action:loginbtn
ggt_hidden(10008):3
So that made me think I could do something like this:
Invoke-WebRequest $uri -WebSession $session -Verbose -Method POST -Body "ggt_textbox%2810006%29=textIentered&action=loginbtn&ggt_hidden%2810008%29=3"# -DisableKeepAlive
But since the main page and the login page use the same URI it tries to POST to a form that doesn't exist because it's looking at the very first page.
I did some more digging and found when I perform this same action from the webpage itself it returns a 302 Moved Temporarily status code the response header actually has a cookie in it (still closes the connection) which is a first and then appears to do a GET request using the new cookie and I'm now logged into the admin page.
So I think I have two problems I need to get around:
How can I get to the form that exists after I click the first button since they use the same URI?
How can I get around the 302 status since I'm only getting back a header and nothing else. I think I need to do a GET request using the cookie from the header but I'm not sure how to specify a cookie with Invoke-WebRequest. I think I would need to use the -Header parameter and specify Cookie: COOKIENAME=CookieID
I think most of all I need to get through my first question and then from there I can start working towards my second.
All help is appreciated and I can provide any header/source needed but the web page is super simple so there is not a whole lot going on in the front end other than a couple of buttons and a logo with a little bit of inline JavaScript.
EDIT
After doing some additional reading about 302 and redirects I found out that shouldn't be a problem. The reason for this is explained in this question.
I figured out my problem. The inline JavaScript is validating to make sure the length of the string is greater than 0 before submitting the login. I don't think there is away to by pass the client side validation easily in a script.
I am using Perl with Selenium. I have set $sel->set_timeout("86400000");.
When opening a website with large content, 500 read timeout message is displayed. Can someone please help me?
It seems to me that not the Selenium webdriver (the client) has issued the timeout, but the webserver has been waiting too long.
What do you want to accomplish? Maybe you can just make a HTTP HEAD request to check that your URL is valid? (A HEAD request does not give you any content back, just the HTTP header with the http status code and, optionally, the "Content-Length" header, among other fields. The HEAD request is much faster that a GET or POST request and yo won't have problems with timeouts. You might get more than one HEAD respnses e.g. if your request is redirected to another server.
Or do you want to check the large content itself. Then I cannot help you at this point. There is not enough information.
You can use a Test::WWW::Mechanize object to create the HEAD request (it is a subclass of LWP::Request). NOt sure if selenium supports head requests.
i'm developing an application for blackbery, i'm displaying a webpage using Eclipse and net.rim.device.api.browser.field.* api when i click a submit buttom in a form i get this error "Unknown host http:80", can anyone helpme?
Don't know anything about Blackberries, but it looks like you're entering a URL where your program is only expecting a host name.
It sounds like form on the web page is not properly set up, causing the post action to post to an invalid URL. It would help if you included the app code and the form HTML.
In this 2005 forum thread people complain about getting that kind of error on their Blackberries.
I'm on the server side and I can see some Proxy servers trying to access my server with either HTTP/1.0 and no HTTP_HOST (which my app requires) or using the wrong HTTP_HOST.
For example, I am getting requests for widgets.twimg.com , www.google-analytics.com , servedby.jumpdisplay.com . My server doesn't host those domains so the response is obviously not any of the sites on the server, and instead I'm giving back an error.
So, it might be that your Blackberry is not providing the right HTTP_HOST to the server (or none at all) and the server doesn't know what to do with it.
To me, that's Blackberry (or whatever proxy that might exist between you and the server) 's fault.