I'm using FiddlerScript to modify the request as follows:
oSession.oRequest["Host"] = "www.example.com";
oSession["x-overridehost"] = "Dotted.Quad.IP.Address";
Now, when I inspect one of the modified sessions, I see this:
GET https://www.example.com/rest/of/url HTTP/1.1
Host: www.example.com
My question is whether the host name from the full URL is passed to the server, or if the server is only sent something like:
GET /rest/of/url HTTP/1.1
in the first line of the request. I don't have access to the server's encryption key, so I can't use something like Wireshark to examine the exact traffic that is going out over the network.
If it helps at all, I see the following when performing a GET to an application running on my local machine:
GET http://localhost:51425/ HTTP/1.1
Host: localhost:51425
.
.
.
GET should always include a fully qualified domain name.
Related
I need to explore the traffic from one program.
The program makes something like a connection through the WebSockets.
Fiddler displays this:
Request Headers: CONNECT 144.***:443 HTTP/1.0
Response: HTTP/1.0 200 Connection Established
End empty body.
But the HTTP analyzer displays full information after that response, and that information continues flowing. Very likely like WebSockets (one connection and receive more answers).
And fiddler display zero traffic.
How can I explore such traffic through the fiddler?
A CONNECT call is always the first command a client sends if it uses a Proxy. Translated CONNECT just means: Please start a connection to the following server and that port. Through that connection the real HTP calls are then transmitted. Therefore CONNECT is not a real HTTP
request.
Fiddler does not show the content of CONNECT requests/responses to port 443 endpoints because those connections are HTTPS/TLS protected (hence the shown data would be useless). You need to enable HTTPS decryption and install the Fiddler root CA certificate into the client app/OS to see the decrypted content of those connections.
These one seems odd. When I fetch the OpenID config via Postman or in the browser, I get a valid config response.
For example a GET via Postman or in the browser to
http://127.0.0.1:8080/auth/realms/myrealm/.well-known/openid-configuration
returns the endpoint including the port 8080 correctly:
{
snip
"jwks_uri": "http://127.0.0.1:8080/auth/realms/myrealm/protocol/openid-connect/certs"
snip
}
However, fetching the from the very same host, target, port, scheme (http) in my C++ application returns the confi endpoints all without a port (e.g. 8080 is missing)
{
snip
"jwks_uri":"http://127.0.0.1/auth/realms/myrealm/protocol/openid-connect/certs"
snip
}
I do not see any issue in my C++ client code, I'm not sure what's making the difference at all. For completeness, this is the C++ code I'm using next to actual values when sending the request, though that should not really be a matter of programming language used:
req_.version(version); // HTTP 1.1
req_.method(method); // GET
req_.target(target); // /auth/realms/myrealm/.well-known/openid-configuration
req_.set(http::field::host, host); // 127.0.0.1
static const std::string agent = app.myAgent();
req_.set(http::field::user_agent, agent);
req_.set(http::field::content_type, contentType); // application/json
I have two questions here:
1) What causes Keycloak not to add the port to the endpoints? And how to workaround it?
2) What's making the difference between the calls? This should be a vanilla GET request.
Figured it out. According to the HTTP spec the port must be included in the host name if it's not a default name:
https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.23
That resolved the issue. Though, my opinion is it should be fixed in Keycloak as well. I would expect static URLs and not something that changes depending on your request. For an example, you could set host to localhost during request and it will return "localhost" in the config, set it to 127.0.0.1 and it will return that. What will it return for an actual IP when queried locally, but the server has a public IP? I didn't try that but it seems odd enough.
Given a URL that ends with .svc and that is supposed to run a SOAP web service, how can I get some data from it?
I tried:
to access it via a web browser
to access it via the Python's library Zeep
to access it via the Microsoft utilitary svcutil.exe
In all cases, I get a timeout error:
Failed to establish a new connection: [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time.
Does it mean that the web service does not work, or that I do things the wrong way?
Importantly - there is a big distinction between "service not active" (and by that I mean no listener on port 80), and "port not open in firewall".
If the problem were simply that you didn't have a service listening on port 80, you would have gotten something like "connection reset" or "connection rejected" as an error.
Instead, you appear to have gotten a timeout, which implies that either the SYN from the client doesn't reach the server, or the SYN/ACK from the server doesn't reach the client. [ You could verify this by doing a packet capture for port 80 on both client and server ]
I would be tempted to check any firewall in front of the server to see that it's letting port 80 traffic through from your client.
Diagnosing Connectivity Issues
Without more details it is difficult to say, but given your timeout error:
Failed to establish a new connection: [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time.
This indicates a network connectivity error at the TCP level, so it is likely web service is not active on the port your are using (default of 80 for http, 443 for https).
In a comment you said you pinged the URL and it responded normally - I assume this means you pinged the hostname. If this is responding normally it means the server is active, but that doesn't tell you anything about the availability of the web service on that server.
telnet %hostname% %port%
where %port% is 80 for http or 443 for https, or something else if there is a port number in the URL you are using (e.g. http://somehost.somewhere.com:port/path.scv)
If ping works and telnet does not connect, then the service is not active.
I suspect this is the case. If the service was active and it was simply that you requesting the data incorrectly, I believe you'd get a different error message - e.g. a valid HTTP response with status code 500 or 404 or similar.
Getting Data from a Web Service
As to your original question as to how to get data from it - once you verify that the service is active, the method to get the data will depend on the specification of the service - i.e.:
which HTTP methods (GET, POST, etc.) does it support
what parameters it requires
what format it requires the parameters in
are the parameters in the query string or POST body.
To interact with a web service there are many command line tools that can be used, as well as the options you have tried, including:
POSTMan Google Chrome Plugin
curl
wget
In windows Powershell, the Invoke-WebRequest
Getting Data from a SOAP Web Service
As you have said it is a SOAP web service, if you have the URL for the wsdl, you can often interract with it using Powershell SOAP WebService Proxies.
The wsdl location varies, but is often at a URL that looks something like.
http://host/path.svc?wsdl
http://host/path.svc/?wsdl
http://host/path/?wsdl
Also if it's configured correctly, just loading the URL in a browser will present a page with a link to the wsdl.
The general idea is:
$URI="http://hostname/path.svc?wsdl"
$Proxy = New-WebserviceProxy $URI –Namespace X
$Proxy | get-member -MemberType Method
This will return a list of methods on the proxy that you can invoke as powershell methods. Any types defined in the wsdl that are needed for arguments, or returned from methods will be available within the namespace X. Invoking the methods will proxy the request to the service, taking care of serializing parameters and serializing results into powershell objects.
Breif Description of what I am trying to accomplish. So I am working with Crestrons Simpl+ software. My job is to create a module for a sound masking system called QT Pro. Now, QT Pro has an API where you can control it via HTTP. I need a way to establish a connection with the QT Pro via HTTP( I have everything I need, IP, Username, Password).
Whats the problem? I have just started working with this language. Unfortunately there isn't as much documentation as I would like, otherwise I wouldn't be here. I know I need to create a socket connection via TCP on port 80. I just don't know what I'm supposed to send through it.
Here is an example:
http://username:password#address/cmd.htm?cmd=setOneZoneData&ZN=Value&mD=Value
&mN=Value&auxA=Value&auxB=Value&autoR=Value
If I were to put this into the URL box, and fill it in correctly. then it would change the values that I specify. Am I supposed to send the entire thing? Or just after cmd.htm? Or is there some other way I'm supposed to send data? I'd like to stay away from the TCP/IP Module so I can keep this all within the same module.
Thanks.
You send
GET /cmd.htm?cmd=setOneZoneData&ZN=Value&mD=Value&mN=Value&auxA=Value&auxB=Value&autoR=Value HTTP/1.1
Host: address
Connection: close
(End with a couple of newlines.)
If you need to use HTTP basic authentication, then also include a header like
Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=
where the gibberish is the base64-encoded version of username:password.
But surely there is some mechanism for opening HTTP connections already there for you? Just blindly throwing out headers like this and hoping the response is what you expect is not robust, to say the least.
To see what is going on with your requests and responses, a great tool is netcat (or telnet, for that matter.)
Do nc address 80 to connect to server address on port 80, then paste your HTTP request:
GET /cmd.htm HTTP/1.1
Host: address
Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=
Connection: close
and see what comes back. SOMETHING should come back. (Remember to terminate with two newlines.)
To see what requests your browser is sending when you do something that works, you can listen like this: nc -l -p 8080.
Then direct your browser to localhost:8080 with the rest of the URL as before, and you'll see the request that was sent. (Then you can type back to see how the browser handles the response.)
I am new to snort so hope you can help me out. I am trying to create my own rules in snort via the local.rules file. I have snort installed on Windows 7 via Virtual box. My configuration seems to be working fine as I can test it with the string -T and it works fine. Also when I run the first three simple rules in the attached screenshot entitled snort rules they work fine, my problem is when I try to right a specific rule which in this case is to log when bit torrent is download the rule is triggered but I am getting all these http_inspect errors. Not sure what they are and if I am missing something in the snort.conf file. I have not touched the preprocessor settings in snort.conf. The errors I get are also attached in the screenshot Snort Output. The rule syntax I am running is snort -i 1 -c c:\snort\etc\snort.conf -A console. The rule is getting logged in the log folder and also I attach the wireshark output. Any help on this would be much appreciated.
Thanks
Garreth
The messages from http_inspect are not errors with your config or errors at all, they are messages from preprocessor rules that are triggering from the traffic. Specifically rule 120:3:1 (GID = 120, SID = 3, REV = 1). The GID 120 rules are specific to the http server inspection from the http preprocessor. This rule generates an event when the http_inspect preprocessor detects anomalous network traffic. The message for this rule is "NO CONTENT-LENGTH or TRANSFER-ENCODING". What this rule is looking at is the server response headers. It's likely alerting because there was no "Content-Length" header or there was no "Transfer-Encoding" header in the response header.
Here is an example of http server response headers:
accept-ranges:bytes
content-length:67023552
content-type:application/x-apple-diskimage
date:Thu, 10 Mar 2016 05:32:31 GMT
server:downloads
status:200
There should always be a content-length header in the server response because it tells the client how much data there is for this request. When the actual length of the data is different than what is presented in the content-length header the client should discard it and throw an error. If there is no content-length header the client has no idea how much data is about to be sent and there is no way to validate that it got all of the data for this request. If this header is missing from the server response snort will generate the rule you are seeing because this is anomalous traffic. It could also be missing the transfer-encoding header. In the example above you'll notice there is not transfer-encoding, so snort would generate this rule if the server response headers looked like the above. The transfer-encoding header is not in a lot of http responses, and this is normal. I believe that these rules are generated when you have the "extended_response_inspection" and the text rules are set to alert (do you have a preprocessor.rules file snort is picking up, or is this rule in any of your rule files?). This option is explain in the snort manual for the http server configuration options. If you don't care about these alerts you can remove the gid rules from your rules files. If you do not have these rules in your rules files then you can add the "no_alerts" option to the http server config. From the snort manual for this option:
"This option turns off all alerts that are generated by the HTTP Inspect preprocessor module. This has no effect on HTTP rules in the rule set. No argument is specified."
You can also remove the extended_response_inspection options from the http config if the rule is being generated by that option.
looks like your tcp config is missing the client/server/both
keyword between ports and 80. Consequently, 80 is being lost and
reassembly is set to both for 8080 only. Change to this in config file:
preprocessor stream5_tcp: policy first, ports 80 8080
This means that every HTTP port needs to be listed in the stream5_tcp preprocessor and that this is not
enforced.
For more info on setting up snort, please go through the following links:
https://www.talentcookie.com/2015/05/snort-how-does-it-work/
https://www.talentcookie.com/2015/05/snort-an-open-source-ids-in-freebsd-10-or-above/
https://www.talentcookie.com/2015/10/snort-performance-is-your-snort-working-fast-enough/