First off thanks for reading!
Second off YES I have tried to find the answer! :) Perhaps I haven't found it because I'm not using the right words to describe my problem, but it's been about 4 hours that I've been trying to figure it out now and I'm getting a little loopy trying to piece it together on my own.
I am very new to programming. Python is my first language. I am on my third Python course. I have an assignment to use the socket library (not urllib library - I know how to do that) to make a socket and use GET to receive information. The problem is that the program needs to take raw input for the URL in question.
I have everything else the way I want it, but I need to know the syntax that I'm supposed to be using INSIDE my "GET" request in order for the HTTP message to include the requested document path.
I have tried (obviously not all together lol):
mysock.send('GET (url) HTTP/1.0\n\n')
mysock.send( ('GET (url) HTTP:/1.0\n\n'))
mysock.send(('GET (url) HTTP:/1.0\n\n'))
mysock.send("GET (url) HTTP/1.0\n\n")
mysock.send( ("'GET' (url) HTTP:/1.0\n\n"))
mysock.send(("'GET' (url) 'HTTP:/1.0\n\n'"))
and:
basically every other configuration of the above (, ((, ( (, ', '' combinations listed above.
I have also tried:
-Creating a string using the 'url' variable first, and then including it inside mysock.send(string)
-Again with the "string-first" theory, but this time I used %r to refer to my user input (so 'GET %r HTTP/1.0\n\n' % url basically)
I've read questions here, other programming websites, the whole chapter in the book and the whole lectures/notes online, I've read articles on the socket library and the .send(), and of course articles on GET requests... but I'm clearly missing something. It seems most don't use socket library when they can use urllib and I don't blame them!!
Thank you again...
Someone from the university posted back to me that the url variable can concatenated with the GET syntax and assigned to a string variable which can then be called with .send(concatenatedvariable) - I had mentioned trying that but had missed that GET requires a space after the word 'GET' so of course concatenating didn't include a space and that blew it. In case anyone else wants to know :)
FYI: A fully quallified URL is only allowed in HTTP/1.1 requests. It is not the norm, though, as HTTP/1.1 requires setting the Host header. The relevant piece of reading would've been RFC 7230, sec. 3.1.1 and possibly RFC 3986. The syntax of the parameters is largely borrowed from the CGI format. It is in no way enforced, however. In a nutshell, everything put together would look like this on the wire:
GET /path?param1=value1¶m2=value2 HTTP/1.1
Host: example.com
As a final note: The line delimiter in HTTP is CRLF (\r\n). For robustness, a simple linefeed is acceptable as well but not recommended.
Is it possible to use Kdb+ http client to access pages protected by login? I am using https://github.com/KxSystems/cookbook/blob/master/yahoo.q as example of basic GET/POST. Does anyone have an example how to extract a cookie and use it in the following requests?
It is probably a bit crude, but the following will extract headers from an http, then cookies, parse and return as a dictionary:
x:"HTTP/1.0 200 OK\r\nContent-type: text/html\r\nSet-Cookie: theme=light\r\nSet-Cookie: sessionToken=abc123; Expires=Wed, 09 Jun 2021 10:18:14 GMT\r\n\r\n";
left:{(first y ss x)#y};
vs1:{{(y#x;(count[z]+y)_x)}[y;;x](first y ss x)};
headers:{{(`$x[0];x[1])} flip vs1[": "] each 1_"\r\n" vs left["\r\n\r\n"]x};
cookies:{(!). {(`$x[0];x[1])} flip vs1["="] each {x[1]#where x[0]=`$"Set-Cookie"} x};
cookies headers[x]
Whilst you might be able to various bits and bobs from an http response, the fact that you won't be able to manipulate http methods means that q can't be your tool to do this - well, not without some vigorous effort.
I would use something like Beautiful Soup in conjunction with q. Soup has some great tools for handling this kind of thing (e.g. cookies etc). There are various other similar projects too.
System call for Beautiful Soup that make relevant get/post/put calls and download required data
system"/path/to/code.py"
Where the code dumps the result somewhere or puts it into kdb directly. Then do whatever you like with it.
For example, I might want to return the current sequence number of the resource with a 409 response to a HEAD request to a versioned resource, but I may not supply it in the response entity because HEAD doesn't allow any. Another example: suppose a POST request to a commit endpoint fails due to version conflict. I can respond with 409, but sometimes I may want to additionally inform the client that the transaction it was committing has exceeded the maximum retry count and further attempts will not be successful. I might return e.g. HTTP/1.1 409 Conflict/final instead of just HTTP/1.1 409 Conflict in this case. My question is, is this acceptable practice? HTTP 1.1 RFC does not supply a clear-cut answer to this question.
I know I can put such information in a X-... HTTP header, or include this information in the response entity in some manner (additional XML tag or JSON property etc.) Assume I can't or don't want to do so if I can help it.
The reason phrase is cute for debugging; but that's it. It's gone in HTTP/2, and may be lost by intermediaries and/or software libraries; don't rely on it being preserved.
The format of the HTTP response is
Status-Line = HTTP-Version SP Status-Code SP Reason-Phrase CRLF
As per the spec,Here Reason Phrase can be custom. You can add your version of the error message.Please check here http://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html. It mentions that
The reason phrases listed here are only recommendations -- they MAY be replaced by local equivalents without affecting the protocol.
I am trying to invoke a WebService through SOAP using Erlang and YAWS (yaws_soap_lib module specifically). The examples published on http://yaws.hyber.org/soap_intro.yaws work for me.
However, when trying to invoke my own web service YAWS fails. The first problem were partner links in the WSDL that were put there because of BPEL is befind this service. I deleted them (for now).
Unfortunately, I've come across another problem: mentioned WSDL has an empty <types> tag. Now, I am not very familiar with WSDL specification and SOAP so my question is whether it is
Erlang SOAP library issue that cannot handle empty <types> tag or
badly generated WSDL?
Does anyone know any better Erlang library for handling SOAP? I have taken a look at erlsoap but it does not support WSDLs.
EDIT: error caused by mentioned WSDL:
::error:function_clause
in function erlsom_add:add_model/2
called as add_model({model,[{type,'_document',sequence,
[{el,[{alt,'soap:Envelope','soap:Envelope',...},
{alt,'soap:Header',...},
{alt,...},
{...}],
1,1,1}],
[],undefined,undefined,1,1,1,false,...},
{type,'soap:detail',sequence,
[{el,[{alt,'#any',...},{alt,...},{...}|...],0,unbound,1}],
[],
{anyAttr,"lax","##any",[...]},
undefined,2,1,1,...},
{type,'soap:Fault',sequence,
[{el,[{alt,...}],1,1,...},
{el,[{...}],1,...},
{el,[...],...},
{el,...}],
[],undefined,undefined,5,1,...},
{type,'soap:Body',sequence,
[{el,[{...}|...],0,...}],
[],
{anyAttr,[...],...},
undefined,2,...},
{type,'soap:Header',sequence,
[{el,[...],...}],
[],
{anyAttr,...},
undefined,...},
{type,'soap:Envelope',sequence,[{el,...},{...}|...],[],{...},...}],
[{ns,"http://schemas.xmlsoap.org/soap/envelope/","soap"},
{ns,"http://www.w3.org/2001/XMLSchema","xsd"}],
"http://schemas.xmlsoap.org/soap/envelope/",[]},undefined)
in call from yaws_soap_lib:initModel2/5
For those who are familiar with the source code: The problem is the Xsds array returned by getXsdsFromWsdl function is empty.
My XML schema fu is a bit rusty, but as far as I can see the schema permits empty <types/> elements. That would suggest the first alternative, though it's hard to be sure. What error message do you get?
I would guess given the function clause error that erlsom is not handling some particular function input as being undefined. But I assume you've already validated your WSDL to make sure it's OK? Also, any chance of posting the WSDL somewhere so we can see it?
The issue has been resolved in latest YAWS version. In order to construct mentioned WSDL model following command has to be invoked:
yaws_soap_lib:initModel(WSDL_FILE_URL, [{include_fun, {erlsom_lib, find_xsd}}])
I'm using Jon Crosby's open source Objective-C OAuth library http://code.google.com/p/oauthconsumer/ for some basic http authentication that does not deal with tokens, only consumer key and consumer secret. My code works great for GET, GET with parameters in the URL, and POST. When I issue a POST request that has parameters in the URL, though, the request fails authorization. I'm trying to figure out why.
The server is using Apache Commons OAuth, so I'd like to compare my base string with that library. Here's a contrived example and the base string and signature produced by my library. Can anyone see what the problem is?
consumer key: abcdef
consumer secret: ghijkl
POST request: http://emptyrandomhost.com/a/uriwith/params?interesting=foo&prolific=bar
my base string: POST&http%3A%2F%2Femptyrandomhost.com%2Fa%2Furiwith%2Fparams&interesting%3Dfoo%26oauth_consumer_key%3Dabcdef%26oauth_nonce%3D1%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D2%26oauth_version%3D1.0%26prolific%3Dbar
This data produces the following OAuth header authorization:
Authorization: OAuth oauth_consumer_key="abcdef",
oauth_version="1.0",
oauth_signature_method="HMAC-SHA1",
oauth_timestamp="2",
oauth_nonce="1",
oauth_signature="Z0PVIz5Lo4eB7aZFT8FE3%2FFlbz0%3D"
And apparently my signature is wrong. The problem has to either be in the construction of the base string, in the way that the HMAC-SHA1 function is implemented (using Apple's CCHmac from CommonHMAC.h, so hopefully this isn't it), or with my Base64Transcoder, which is open source c. 2003 by Jonathan Wight/Toxic Software. I primarily suspect the base string, since the requests work for GET and POST and only fail with POST with URL parameters as above.
Can someone with lots of OAuth experience spot the problem above? Something else that would be very useful is the base string that is produced by Apache Commons OAuth in their authentication. Thanks.
As per RFC 5849 section 3.4.1.2, the OAuth base string URI does not include the query string or fragment. If either the client or the server does not remove the query parameters from the base string URI and add them to the normalized OAuth parameter list, the signatures won't match. Unfortunately, it's hard to tell which side is making this mistake. But it's easy to determine this is the problem: If it always works without query parameters but always fails with query parameters, you can be pretty sure that one side or the other is generating the wrong base string. (Be sure that it always happens though... intermittent errors would be something else. Similarly, if it never works with or without a query string, that would also be something else.) The other possibility is that normalization was done incorrectly — the parameter list must be sorted and percent encoded sequences must be upper-cased. If it's not normalized correctly on both sides, that will also cause a base string mismatch, and thus a signature mismatch.
you can build and check visually your request at this URL:
http://hueniverse.com/2008/10/beginners-guide-to-oauth-part-iv-signing-requests/
Open the boxes denoted by [+] signs and fill in your values, that way you may be able to see if the problem is at your code, or at the provider side.