SHOUTcast fetch request denied every time with 404 - sockets

Code
function radiotest(host,port)
local rstr="Online"
local sock, err = socket.tcp()
if not sock then
return "Failed"
end
sock:settimeout(1)
local res, err = sock:connect(host, port)
if not res then
return "offline"
else
sock:settimeout(1)
sock:send("GET /index.html HTTP/1.0\r\n UserAgent: SHOUTcast Song Status \r\n Accept: */*\r\n\r\n")
sock:settimeout(3)
local data=sock:receive('*a')
sock:close()
print(data)
-- Further processing content here
end
end
print( radiotest( "10.*.*.*", 1234 ) )
The above socket connection returns me:
ICY 404 Resource Not Found
icy-notice1:<BR>SHOUTcast Distributed Network Audio Server/win32 v1.9.7<BR>
icy-notice2:The resource requested was not found<BR>
I think the problem is in my headers listing, but I'm unable to trace it.
The page opens fine in all browsers(Opera does need to be masked as another browser; otherwise it just keeps on downloading all songs).
I've tried using following strings inside sock:send()
GET /index.html HTTP/1.0\r\n UserAgent: SHOUTcast Song Status (Mozilla Compatible)\r\n\r\n
GET /index.html HTTP/1.0\r\n UserAgent: Opera/9.80 (Windows NT 6.1; Win64; x64) Presto/2.12.388 Version/12.12\r\n\r\n
GET /index.html HTTP/1.0\r\n UserAgent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.57 Safari/537.17\r\n\r\n
I'm totally stuck at this part. How do I fetch the page using socket.tcp()?

After looking at your packet capture, it seems that what is actually getting sent over the wire is wrong. Your user-agent string isn't making it:
> GET /index.html HTTP/1.0
< ICY 404 Resource Not Found
< icy-notice1:<BR>SHOUTcast Distributed Network Audio Server/win32 v1.9.7<BR>
< icy-notice2:The resource requested was not found<BR>
If you don't specify a user-agent that contains Mozilla, you will be unable to access the admin interface, or any part of it. Go back and check your code again on what you're sending.

#Brad Thanks. Your help with Wireshark was indeed practicable. The User-Agent header was not being passed to the server because of an extra space I was providing in the request.
sock:send("GET /index.html HTTP/1.0\r\n UserAgent: SHOUTcast Song Status \r\n Accept: */*\r\n\r\n")
The \r\n UserAgent: SHOUTcast Song Status should instead be:
\r\nUser-Agent: SHOUTcast Song Status
And it is working fine now.
Thanks for the help. :D
The results from the function after filtering out the HTML is like:
Online(Tonic - If You Could Only See)
Online(Tonic - If You Could Only See) Stream is up at 256 kbps with 0 of 32 listeners (0 unique)

Related

Rate Limiting using HAProxy with Large Post Requests

I am using HAProxy v2.0.13 in front of an API and have attempted to implement URL based rate limiting to try and limit connections to 5 within a 30 minute sliding window per source IP for the "/get_link" path:
frontend fe_dev
mode http
bind *:8081,[::]:8081
stick-table type ip size 100k expire 30m store http_req_rate(30m)
http-request track-sc0 src if METH_POST { path -i -m beg /get_link }
http-request deny deny_status 429 if { sc_http_req_rate(0) gt 5 }
default_backend be_dev
This API endpoint is called from a JavaScript function using an XMLHttpRequest() request and I am using Google Chrome v83.
var xHR = new XMLHttpRequest();
xHR.open("POST", "get_link", true);
xHR.onload = function() {
console.log('status code is ' + this.status);
};
xHR.onerror = function() {
console.log("onerror()");
};
var obj = {};
xHR.setRequestHeader("Content-Type", "application/json");
xHR.send(JSON.stringify(obj));
When the size of my POST request is small (i.e. a few hundred bytes) then everything works fine - after 5 requests I start getting HTTP 429 returned. I then tried with a large POST request (the content length was around 35500 bytes) and this is when Chrome started to trigger the onerror function.
I have done a tcpdump and it looks like HAProxy doesn't wait for the whole request before sending back a 429 (output trimmed for brevity):
POST /get_link HTTP/1.1
Host: server:8081
Connection: keep-alive
Content-Length: 35687
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36
Content-Type: application/json
Accept: */*
Origin: http://server:8081
Referer: http://server:8081/index.html
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
{"req1":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXHTTP/1.1 429 Too Many Requests
content-length: 117
cache-control: no-cache
content-type: text/html
connection: close
<html><body><h1>429 Too Many Requests</h1>
You have sent too many requests in a given amount of time.
</body></html>
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
From looking at tcpdump I can also see that HAProxy sends a TCP RST as soon as it has sent back the 429 even though Chrome is still sending POST data. How do I get HAProxy to play nicely and wait until it has received the whole request before rejecting it?
The answer that no one came up with is enabling “option http-buffer-request”.

Jsoup redirect has not the same behaviour than browser redirect

I'm facing a server through Jsoup (latest v 1.10.2) to extract some data from a page.
This server is opened to anonymous users but it uses a redirect chain to grant a session ID to each user.
This is the sequence, I got by browser:
First request to http://SERVER_HOST/page
resp: 302 Redirect to Location http://SSO_SERVER
Follow redirect, opening http://SSO_SERVER
resp: 302 Redirect to Location http://SERVER_HOST/page?sessionID=123456
Follow redirect, opening http://SERVER_HOST/page?sessionID=123456
resp: 200 :)
Unexpectedly with Jsoup, the redirect chain fails. Look at the difference into the step 2:
First request to http://SERVER_HOST/page (without cookies)
resp: 302 Redirect to Location http://SSO_SERVER
Follow redirect, opening http://SSO_SERVER
resp: 302 Redirect to Location /shared/SSO/http%3a%2f%2SERVER_HOST/page%3dsessionID=123456
Follow redirect, opening http://SSO_SERVER/shared/SSO/http%3a%2f%2SERVER_HOST/page%3dsessionID=123456
resp: 400 not found :(
At the step 2, redirect location in server response start with "/" not with "http://", so at the step 3 it connects to the wrong host.
Why at the step 2, I got a different location in server response according the request belongs to browser or to JSoup?
I set JSoup request the same headers of browser request:
Response response = Jsoup.connect(link)
.userAgent("Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36")
.header("Accept","text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8")
.header("Accept-Encoding", "gzip, deflate, sdch")
.header("Accept-Language", "it,en-US;q=0.8,en;q=0.6")
.header("Upgrade-Insecure-Requests", "1")
.method(Method.GET)
.followRedirects(true).execute();
There was a bug in Jsoup.Connect() in how a query string in a redirect header was handled.
That's fixed now in this commit. You can build off HEAD to get access to the fix, and it will be available in the next release (1.10.3).

Failing to use eZ Platform JS REST API client with cross domain

Context and troubles
I'm currently building a web application on top of ezPlatform & Symfony.
My goal is now to request this app from an external website using a JS client (using the JS REST client of ezplatform : CAPI.js)
I tested my script localy (on the app itself = same domain) and everything is fine : I can GET and POST data.
But testing this script on an external webiste (CORS requests) didn't work. I am stuck with 2 differents problems :
Server side : The response headers do not contains the Access-Control-Allowed-Methods
Client side : No session cookies are given within a request
Details
Problem 1 : No header "allow_methods"
On chrome I always have this error :
XMLHttpRequest cannot load http://api.ezplatform.lan/api/ezp/v2/user/sessions. Response for preflight has invalid HTTP status code 405
Note that, on the server side, nelmio_cors bundle is used to configures the headers. The configs :
nelmio_cors:
paths:
'^/api/ezp/v2/':
max_age: 3600
allow_credentials: true
allow_origin: ['*']
allow_methods: ['POST', 'PUT', 'GET', 'DELETE', 'OPTIONS']
expose_headers: []
And now, here are the details of a failing preflight request :
GENERAL
Request URL:http://api.ezplatform.lan/api/ezp/v2/user/sessions
Request Method:OPTIONS
Status Code:405 Method Not Allowed
Remote Address:192.168.1.82:80
REPONSE HEADERS
Access-Control-Allow-Credentials:true
Access-Control-Allow-Headers:authorization, accept, content-type, x-csrf-token, destination, x-siteaccess
Access-Control-Allow-Origin:http://www.externalsite.lan
Access-Control-Max-Age:3600
Cache-Control:private
Connection:Keep-Alive
Content-Length:0
Content-Type:text/html; charset=UTF-8
Date:Tue, 13 Dec 2016 15:24:44 GMT
Keep-Alive:timeout=5, max=99
Server:Apache/2.4.23 (Ubuntu)
Vary:X-User-Hash
REQUEST HEADERS
Accept:*/*
Accept-Encoding:gzip, deflate, sdch
Accept-Language:fr-FR,fr;q=0.8,en-US;q=0.6,en;q=0.4
Access-Control-Request-Headers:content-type
Access-Control-Request-Method:POST
Cache-Control:no-cache
Connection:keep-alive
Host:api.ezplatform.lan
Origin:http://www.externalsite.lan
Pragma:no-cache
Referer:http://www.externalsite.lan/
User-Agent:Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36
In the "response headers", there is no Access-Control-Allow-Methods despite of the nelmio_cors configs.
Digging inside the nelmio_cors code I realized that the "allow_methods" config is retrieved but is then overwritten by something else, and here it becomes obscur to me.
This old ezpublish bug maybe give me a clue about the situation : the ezPublishRestBundle does not seems to find any "allowed method", and somehow overwrite the nelmio_cors config.
In the Resprovider.php file, if I force the methods getAllowedMethods to return this :
return ["POST", "PUT", "GET", "DELETE", "OPTIONS"]; then I have no more 405 error but a very different problem (explained right after this)
Problem 2 : no session cookies allowed
With my previous wonderful hack, I can now get a little bit further : my request are allowed but some of them are still failing.
I noticed that no session cookie is passed inside the requests (which is the normal behaviour testing on the same domain).
This times it seems to come from the CAPI.js file : the XmlHttpRequest object never has the property withCredentials to true.
If I add this code XHR.withCredentials = true; in CAPI.js before the request is send, then it seems to be fine.
Conclusion
I really wonder if the ez Platform rest client has been designed to be used for cross domain, but it would be very surprising if not.
So I must do something wrong, and if someone can explain me what, I would be extemely grateful :)

Read HTTP request from a SOCKET in perl

I have a problem with reading a socket in PERL.
The story goes like this :
1.1.1.1 is my server
2.2.2.2 is my other server that it is opened as ssh tunnel on 1.1.1.1
3.3.3.3 is my ip from Mozilla Firefox
I have a ssh tunnel opened on port 12345 and a tcp.pl "server" that is oppenning on port 2000 and it is forwarding all raw traffic to port 12345 then gets it back again.
I have oppened Mozilla Firefox and put at SOCKS5 1.1.1.1:2000 (the tcp.pl server), and when i surf the web, i surf with 2.2.2.2's ip, witch is good.
I wrote someware in my code to print all sockets that tcp.pl is getting with this command :
`print $buffer;`
The problem is that i can read HTTP HEADERS and i see stuff like
GET / HTTP/1.1
Host: site.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Cookie: _ga=GA1.2.1235518067.1410367631
Connection: keep-alive
witch is ok, but i don't quite get the HTTP CONTENT , i mean i don't get the HTML code. I sometimes get a part of the HTML CODE LIKE THIS
:G�� ��� <-- some wired characters
<html>...</html>
���lOP� <-- some wired characters
and sometimes i get
PuTTyPuTTyPuTTyPuTTyPuTTyPuTTyPuTTyPuTTyPuTTyPuTTyPuTTy
and a beep sound.
the code (tcp.pl)
an infinite loop that reads reads and reads ....
while (1)
{
for my $socket ($ioset->can_read)
{
if($socket == $server)
{
new_connection($server);
}
else
{
next unless exists $socket_map{$socket};
my $remote = $socket_map{$socket};
my $buffer;
# get data from main port
my $read = $socket->sysread($buffer, 4096);
if ($read)
{
print $read; # gives a number like 43243 5436346456 34654643464
print $buffer; # allways gives HTTP headers, and token is frequently distorted, showing characters like ":G�� ������lOP�" and some chunks of HTML CODE from time to time
# sometimes i get output like "PuTTyPuTTyPuTTyPuTTyPuTTyPuTTyPuTTyPuTTyPuTTyPuTTyPuTTy" and hearing some beeping sounds
# forward data to the other port (in browser you see no differance)
$remote->syswrite($buffer);
}
else {
close_connection($socket);
}
}
}
}
I believe this is because the fact that i don't order the packets properly, i mean i have to read SOCKETS and order by ACK and SEQ flags, then read it.
Now my main quession, how do i read the HTML content as it is?
Thank you.

How to post data with REST service from remote server

I am new to REST so bear with me if I'm missing something obvious.
Any pointer would be much appreciated as I am a bit lost.
Scenario
I needed to post some data to the following REST service: https://api.dotmailer.com/ from my web application https://myapp.com/.
During testing, I was able to post the data from my local pc.
However, as soon as I published the updated application to https://myapp.com/ on a remote server, I was no longer able to post any data.
What I've tried so far
Added rule to the remote server firewall to allow outgoing traffic to use https. Didn't solve the problem.
Disabled the url rewriting rule that change http to https for myapp.com. Didn't solve the problem.
Pasted the URL I use to post my data (https://api.dotmailer.com/v2/address-books/12345/contacts) in a browser on the remote server, entered the correct credentials, but couldn't access it.
the error message said "Unable to open this internet site. The requested site is either unavailable or cannot be found." If I do the same on my local PC I can access the URL.
Monitored the two calls with Fiddler2.
I include the results of the monitoring process below:
CALLS MADE FROM REMOTE SERVER
----------
POST /bla.aspx HTTP/1.1
Host: myapp.com
Connection: keep-alive
Content-Length: 10660
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: https://myapp.com
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
Content-Type: application/x-www-form-urlencoded
DNT: 1
Referer: https://myapp.com/bla.aspx
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8,it;q=0.6
Cookie: ASP.NET_SessionId=xxx; Myapp=xxx; GUID=xxx
CALLS MADE FROM LOCAL PC
----------
POST /bla.aspx HTTP/1.1
Host: localhost:xxx
Connection: keep-alive
Content-Length: 10656
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: http://localhost:60675
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
Content-Type: application/x-www-form-urlencoded
DNT: 1
Referer: http://localhost:xxx/bla.aspx
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8,it;q=0.6
Cookie: __eqtUser=xxx; ASP.NET_SessionId=xxx; Myapp=xxx; GUID=xxx
Question
I believe point 3 shows that the cause is some setting on the remote server.
Does anyone know what it could be? Or am I completely off-track?
Update
I spoke with the developer on the receiving end of my calls who can monitor incoming traffic.
He could see my local calls but not the ones submitted from https://myapp.com.
In response to gmlime reply, I've added the following to myapp.com web.config file but didn't help.
<system.webServer>
<httpProtocol>
<customHeaders>
<add name="Access-Control-Allow-Origin" value="*" />
</customHeaders>
</httpProtocol>
</system.webServer>
Should I put it at a higher level in the hierarchy?
Make sure that this gets added to the response:
YourAddHeaderMethod("Access-Control-Allow-Origin", "*");
Many servers deny posting from other domains and can terminate the connection. You can learn more about it from the w3 docs for Access-Conrol-Allow-Origin and Mozzilla covers some scenarios. You may have to check with the server administrator to rule out cross domain problems also.