I have a problem with reading a socket in PERL.
The story goes like this :
1.1.1.1 is my server
2.2.2.2 is my other server that it is opened as ssh tunnel on 1.1.1.1
3.3.3.3 is my ip from Mozilla Firefox
I have a ssh tunnel opened on port 12345 and a tcp.pl "server" that is oppenning on port 2000 and it is forwarding all raw traffic to port 12345 then gets it back again.
I have oppened Mozilla Firefox and put at SOCKS5 1.1.1.1:2000 (the tcp.pl server), and when i surf the web, i surf with 2.2.2.2's ip, witch is good.
I wrote someware in my code to print all sockets that tcp.pl is getting with this command :
`print $buffer;`
The problem is that i can read HTTP HEADERS and i see stuff like
GET / HTTP/1.1
Host: site.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Cookie: _ga=GA1.2.1235518067.1410367631
Connection: keep-alive
witch is ok, but i don't quite get the HTTP CONTENT , i mean i don't get the HTML code. I sometimes get a part of the HTML CODE LIKE THIS
:G�� ��� <-- some wired characters
<html>...</html>
���lOP� <-- some wired characters
and sometimes i get
PuTTyPuTTyPuTTyPuTTyPuTTyPuTTyPuTTyPuTTyPuTTyPuTTyPuTTy
and a beep sound.
the code (tcp.pl)
an infinite loop that reads reads and reads ....
while (1)
{
for my $socket ($ioset->can_read)
{
if($socket == $server)
{
new_connection($server);
}
else
{
next unless exists $socket_map{$socket};
my $remote = $socket_map{$socket};
my $buffer;
# get data from main port
my $read = $socket->sysread($buffer, 4096);
if ($read)
{
print $read; # gives a number like 43243 5436346456 34654643464
print $buffer; # allways gives HTTP headers, and token is frequently distorted, showing characters like ":G�� ������lOP�" and some chunks of HTML CODE from time to time
# sometimes i get output like "PuTTyPuTTyPuTTyPuTTyPuTTyPuTTyPuTTyPuTTyPuTTyPuTTyPuTTy" and hearing some beeping sounds
# forward data to the other port (in browser you see no differance)
$remote->syswrite($buffer);
}
else {
close_connection($socket);
}
}
}
}
I believe this is because the fact that i don't order the packets properly, i mean i have to read SOCKETS and order by ACK and SEQ flags, then read it.
Now my main quession, how do i read the HTML content as it is?
Thank you.
Related
My interface (an MKR Wifi 1010 Arduino) runs a very simple REST API, but when testing it with Mulesoft's Advanced Rest Client, I get this error:
The requested URL can't be reached
The service might be temporarily down or it may have moved permanently to a new web address.
The response status "0" is not allowed. See HTTP spec for more details: https://tools.ietf.org/html/rfc2616#section-6.1.1
When I check it with telnet though, it looks fine:
[bf#localhost ~]$ telnet 192.168.178.185 80
Trying 192.168.178.185...
Connected to 192.168.178.185.
Escape character is '^]'.
GET /api/gps HTTP/1.1
Host: 192.168.178.185
HTTP/1.1 200 OK
Connection: close
Content-Length: 9
Content-Type: application/json
"Success"
Connection closed by foreign host.
My question now is, is the rest client broken, or am I missing something in my reply? Of course I want any REST client to be able to process my interface correctly.
I am using HAProxy v2.0.13 in front of an API and have attempted to implement URL based rate limiting to try and limit connections to 5 within a 30 minute sliding window per source IP for the "/get_link" path:
frontend fe_dev
mode http
bind *:8081,[::]:8081
stick-table type ip size 100k expire 30m store http_req_rate(30m)
http-request track-sc0 src if METH_POST { path -i -m beg /get_link }
http-request deny deny_status 429 if { sc_http_req_rate(0) gt 5 }
default_backend be_dev
This API endpoint is called from a JavaScript function using an XMLHttpRequest() request and I am using Google Chrome v83.
var xHR = new XMLHttpRequest();
xHR.open("POST", "get_link", true);
xHR.onload = function() {
console.log('status code is ' + this.status);
};
xHR.onerror = function() {
console.log("onerror()");
};
var obj = {};
xHR.setRequestHeader("Content-Type", "application/json");
xHR.send(JSON.stringify(obj));
When the size of my POST request is small (i.e. a few hundred bytes) then everything works fine - after 5 requests I start getting HTTP 429 returned. I then tried with a large POST request (the content length was around 35500 bytes) and this is when Chrome started to trigger the onerror function.
I have done a tcpdump and it looks like HAProxy doesn't wait for the whole request before sending back a 429 (output trimmed for brevity):
POST /get_link HTTP/1.1
Host: server:8081
Connection: keep-alive
Content-Length: 35687
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36
Content-Type: application/json
Accept: */*
Origin: http://server:8081
Referer: http://server:8081/index.html
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
{"req1":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXHTTP/1.1 429 Too Many Requests
content-length: 117
cache-control: no-cache
content-type: text/html
connection: close
<html><body><h1>429 Too Many Requests</h1>
You have sent too many requests in a given amount of time.
</body></html>
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
From looking at tcpdump I can also see that HAProxy sends a TCP RST as soon as it has sent back the 429 even though Chrome is still sending POST data. How do I get HAProxy to play nicely and wait until it has received the whole request before rejecting it?
The answer that no one came up with is enabling “option http-buffer-request”.
I have been reading multiple articles on how to sniff and subsequently use the data obtained to interact with closed source apis recently.
I am concentrated on the tinder api since it seemed to me ample research had been done already on it, hence it would be easy to learn from.
http://ttcubicle.blogspot.com/2015/03/reverse-engineering-tinders-api.html
http://ec2-52-42-144-243.us-west-2.compute.amazonaws.com/tinder/
Althrough I managed to sniff the authentification between the tinder app on my phone and the server through fiddler, I am not able to actually simulate that login using curl on the command line
----- Below is the request send from my phone to the server --- for obvious reasons I changed my actual data ----
POST https://api.gotinder.com/auth HTTP/1.1
platform: android
User-Agent: Tinder Android Version 6.5.1
os-version: 23
Facebook-ID: 10151935000326599
Accept-Language: en
app-version: 1955
Content-Type: application/json; charset=utf-8
Content-Length: 257
Host: api.gotinder.com
Connection: Keep-Alive
Accept-Encoding: gzip
X-Auth-Token: ccXX9a-4a99c-4e32-8154-9b21asf5eec
{"facebook_token":"EAfasfasfasfN6solZAh8M3kwxsP1JzF6OBDocdNUEyxd8tsVCN6kWZA6fArZB0T5dZArmdVvKAXUuQZCOtoVZBPasfzUMz9RfFoSpEifEVm7bAIspEerbLKRgW3DCpHHuxVyZApr1koAHhIjCGtxUZAAZAtDvTTbayrkF","facebook_id":"111111119","locale":"en"}
My knowledge regarding POST / Headers and all of these things is still a bit shacky (thats why I am trying to re-enact) but from what I understand that next step should be to send a POST request with curl that sends the X-Auth-Token in the header and facebook_token and so on in the data part.
something like this:
curl -H "Content-Type: application/json" -H "X-Auth-Token: cc5555a-499c-4e32-8154-9b25555ec" -d '{"facebook_token":"EAAGasdpsBAEzbJDJdcHXLjKpDjN6solZAh8M3kwxsP1JzF6OBDocdNUEyxd8tsVCN6kWZA6fArZB0T5dZArmdVvKAXUuQZCOtoVZBPZBMTUJzUMz9RfFoSpEifEVm7bAIspEerbLKRgW3DCpHHuxVyZApr1koAHhIjCGtxUZAAZA555TTbayrkF","facebook_id":"101519555326599","locale":"en"}' https://api.gotinder.com/auth
However, no matter how I change the parameters around, I always get Errorcodes 500 or 401 thrown back at me. The maximum I can get is the server telling me that it excpects a facebook_token (which is obviously send in the data section)
Does anyone has experience with this sort of problem ?
Thank you
I just discovered phyton and... holyshit i am in love!
payloadauth = {"facebook_token":"EAA xxxxxx
header1 = { 'platform': 'android','User-Agent': 'Tinder, 'X-Auth-Token':'cblabla
with requests.Session() as c:
response = c.post('https://api.gotinder.com/auth',data=payloadauth)
response = c.get('https://api.gotinder.com/recs/core?locale=en', headers=header1)
print(response.json())
four lines of code...
I am new to REST so bear with me if I'm missing something obvious.
Any pointer would be much appreciated as I am a bit lost.
Scenario
I needed to post some data to the following REST service: https://api.dotmailer.com/ from my web application https://myapp.com/.
During testing, I was able to post the data from my local pc.
However, as soon as I published the updated application to https://myapp.com/ on a remote server, I was no longer able to post any data.
What I've tried so far
Added rule to the remote server firewall to allow outgoing traffic to use https. Didn't solve the problem.
Disabled the url rewriting rule that change http to https for myapp.com. Didn't solve the problem.
Pasted the URL I use to post my data (https://api.dotmailer.com/v2/address-books/12345/contacts) in a browser on the remote server, entered the correct credentials, but couldn't access it.
the error message said "Unable to open this internet site. The requested site is either unavailable or cannot be found." If I do the same on my local PC I can access the URL.
Monitored the two calls with Fiddler2.
I include the results of the monitoring process below:
CALLS MADE FROM REMOTE SERVER
----------
POST /bla.aspx HTTP/1.1
Host: myapp.com
Connection: keep-alive
Content-Length: 10660
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: https://myapp.com
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
Content-Type: application/x-www-form-urlencoded
DNT: 1
Referer: https://myapp.com/bla.aspx
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8,it;q=0.6
Cookie: ASP.NET_SessionId=xxx; Myapp=xxx; GUID=xxx
CALLS MADE FROM LOCAL PC
----------
POST /bla.aspx HTTP/1.1
Host: localhost:xxx
Connection: keep-alive
Content-Length: 10656
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: http://localhost:60675
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
Content-Type: application/x-www-form-urlencoded
DNT: 1
Referer: http://localhost:xxx/bla.aspx
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8,it;q=0.6
Cookie: __eqtUser=xxx; ASP.NET_SessionId=xxx; Myapp=xxx; GUID=xxx
Question
I believe point 3 shows that the cause is some setting on the remote server.
Does anyone know what it could be? Or am I completely off-track?
Update
I spoke with the developer on the receiving end of my calls who can monitor incoming traffic.
He could see my local calls but not the ones submitted from https://myapp.com.
In response to gmlime reply, I've added the following to myapp.com web.config file but didn't help.
<system.webServer>
<httpProtocol>
<customHeaders>
<add name="Access-Control-Allow-Origin" value="*" />
</customHeaders>
</httpProtocol>
</system.webServer>
Should I put it at a higher level in the hierarchy?
Make sure that this gets added to the response:
YourAddHeaderMethod("Access-Control-Allow-Origin", "*");
Many servers deny posting from other domains and can terminate the connection. You can learn more about it from the w3 docs for Access-Conrol-Allow-Origin and Mozzilla covers some scenarios. You may have to check with the server administrator to rule out cross domain problems also.
Code
function radiotest(host,port)
local rstr="Online"
local sock, err = socket.tcp()
if not sock then
return "Failed"
end
sock:settimeout(1)
local res, err = sock:connect(host, port)
if not res then
return "offline"
else
sock:settimeout(1)
sock:send("GET /index.html HTTP/1.0\r\n UserAgent: SHOUTcast Song Status \r\n Accept: */*\r\n\r\n")
sock:settimeout(3)
local data=sock:receive('*a')
sock:close()
print(data)
-- Further processing content here
end
end
print( radiotest( "10.*.*.*", 1234 ) )
The above socket connection returns me:
ICY 404 Resource Not Found
icy-notice1:<BR>SHOUTcast Distributed Network Audio Server/win32 v1.9.7<BR>
icy-notice2:The resource requested was not found<BR>
I think the problem is in my headers listing, but I'm unable to trace it.
The page opens fine in all browsers(Opera does need to be masked as another browser; otherwise it just keeps on downloading all songs).
I've tried using following strings inside sock:send()
GET /index.html HTTP/1.0\r\n UserAgent: SHOUTcast Song Status (Mozilla Compatible)\r\n\r\n
GET /index.html HTTP/1.0\r\n UserAgent: Opera/9.80 (Windows NT 6.1; Win64; x64) Presto/2.12.388 Version/12.12\r\n\r\n
GET /index.html HTTP/1.0\r\n UserAgent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.57 Safari/537.17\r\n\r\n
I'm totally stuck at this part. How do I fetch the page using socket.tcp()?
After looking at your packet capture, it seems that what is actually getting sent over the wire is wrong. Your user-agent string isn't making it:
> GET /index.html HTTP/1.0
< ICY 404 Resource Not Found
< icy-notice1:<BR>SHOUTcast Distributed Network Audio Server/win32 v1.9.7<BR>
< icy-notice2:The resource requested was not found<BR>
If you don't specify a user-agent that contains Mozilla, you will be unable to access the admin interface, or any part of it. Go back and check your code again on what you're sending.
#Brad Thanks. Your help with Wireshark was indeed practicable. The User-Agent header was not being passed to the server because of an extra space I was providing in the request.
sock:send("GET /index.html HTTP/1.0\r\n UserAgent: SHOUTcast Song Status \r\n Accept: */*\r\n\r\n")
The \r\n UserAgent: SHOUTcast Song Status should instead be:
\r\nUser-Agent: SHOUTcast Song Status
And it is working fine now.
Thanks for the help. :D
The results from the function after filtering out the HTML is like:
Online(Tonic - If You Could Only See)
Online(Tonic - If You Could Only See) Stream is up at 256 kbps with 0 of 32 listeners (0 unique)