HTTP CONNECT + GET return wrong status - sockets

I'm sending GET requests via proxy that only supports HTTP (no HTTPS support). When I use that proxy (or any other http-only proxy) to request HTTPS it returns 403 when I use curl (and it seems to be the correct status). But if I just use CONNECT and GET I get 200.
Curl - 403 Forbidden:
curl -x proxyhost:proxyport -I https://example.com -vvv
* Trying PROXYHOST:8080...
* TCP_NODELAY set
* Connected to PROXYHOST (PROXYHOST) port 8080 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to www.example.com:443
> CONNECT www.example.com:443 HTTP/1.1
> Host: www.example.com:443
> User-Agent: curl/7.68.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 403 Forbidden
HTTP/1.1 403 Forbidden
< Date: Fri, 15 Oct 2021 15:37:31 GMT
Date: Fri, 15 Oct 2021 15:37:31 GMT
< Server: Apache
Server: Apache
< Content-Length: 202
Content-Length: 202
< Content-Type: text/html; charset=iso-8859-1
Content-Type: text/html; charset=iso-8859-1
<
* Received HTTP code 403 from proxy after CONNECT
* CONNECT phase completed!
* Closing connection 0
curl: (56) Received HTTP code 403 from proxy after CONNECT
Pure HTTP - 200 OK:
CONNECT PROXYHOST:PROXYPORT HTTP/1.0
GET https://www.example.com:443 HTTP/1.0
HTTP/1.0 200 OK
Why do I get 200 with CONNECT + GET?
Additional info:
PROXYHOST is just a random host without any proxy setting or software. It just happens so that when you use it as proxy for http GET requests it returns correct status (200 if requested page exists, 404 if it does not, etc) and its own html instead of requested body. At the same time it always returns 403 if you try to use it as proxy for requesting https via anything but CONNECT + GET.
I also tried python requests and got this result (with verbose logs):
proxy = {'https': 'http://PROXYHOST:8080', 'http': 'http://PROXYHOST:8080'}
requests.get('https://example.com', proxies=proxy)
# logs
send: b'CONNECT example.com:443 HTTP/1.0\r\n'
send: b'\r\n'
# exception
File "/usr/lib/python3.8/http/client.py", line 276, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response / (Caused by ProxyError('Cannot connect to proxy.', RemoteDisconnected('Remote end closed connection without response')))
requests.get('http://example.com', proxies=proxy)
# logs
send: b'GET http://example.com/ HTTP/1.1\r\nHost: example.com\r\nUser-Agent: python-requests/2.25.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Date: Fri, 15 Oct 2021 17:10:00 GMT
header: ...
<Response [200]>

Related

Facebook Object Debugger Error 502 and Error parsing input URL

Well, this website uses AWS EC2, and right now when i shared the site on Facebook, facebook don't show nothing, but yesterday it work well.
Right now the Facebook lint(object debugger) says
Error parsing input URL, no data was cached, or no data was scraped.
And
Curl Error : RECV_ERROR Received HTTP code 502 from proxy after
CONNECT
But when i use curl via terminal on my Mac (like curl -v -X TRACE www.cuinzy.com) this return:
* Rebuilt URL to: www.cuinzy.com/
* Hostname was NOT found in DNS cache
* Trying 54.207.48.99...
* Connected to www.cuinzy.com (127.0.0.1) port 80 (#0)
> TRACE / HTTP/1.1
> User-Agent: curl/7.37.1
> Host: www.cuinzy.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Mon, 17 Aug 2015 10:14:36 GMT
* Server Apache is not blacklisted
< Server: Apache
< Connection: close
< Transfer-Encoding: chunked
< Content-Type: message/http
<
TRACE / HTTP/1.1
User-Agent: curl/7.37.1
Host: www.cuinzy.com
Accept: */*
* Closing connection 0
Curl return response code 200 on www.cuinzy.com, and if try with url https://www.cuinzy.com/itm/572-aumento-natural-del-busto-pueraria-mirifica-100-x-500mg.html, this return:
* Hostname was NOT found in DNS cache
* Trying 54.207.48.99...
* Connected to www.cuinzy.com (127.0.0.1) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
* Server certificate: *.cuinzy.com
* Server certificate: COMODO RSA Domain Validation Secure Server CA
* Server certificate: COMODO RSA Certification Authority
* Server certificate: AddTrust External CA Root
> TRACE /itm/572-aumento-natural-del-busto-pueraria-mirifica-100-x-500mg.html HTTP/1.1
> User-Agent: curl/7.37.1
> Host: www.cuinzy.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Mon, 17 Aug 2015 10:16:25 GMT
* Server Apache is not blacklisted
< Server: Apache
< Connection: close
< Transfer-Encoding: chunked
< Content-Type: message/http
<
TRACE /itm/572-aumento-natural-del-busto-pueraria-mirifica-100-x-500mg.html HTTP/1.1
User-Agent: curl/7.37.1
Host: www.cuinzy.com
Accept: */*
* Closing connection 0
And look how the use of SSL does not represent a change, additionally, notice that at all times the http response code is 200.
And, if i click on See exactly what our scraper sees for your URL link of Facebook Lint, this only return:
Document returned no data
Ah, additionally, i try this tools(and all return http response code 200 and work fine):
curl via terminal on mac
http://httpstatus.io/
http://iframely.com/debug?uri=https%3A%2F%2Fwww.cuinzy.com%2Fitm%2F572-aumento-natural-del-busto-pueraria-mirifica-100-x-500mg.html
http://opengraphcheck.com/result.php?url=https%3A%2F%2Fwww.cuinzy.com%2Fitm%2F572-aumento-natural-del-busto-pueraria-mirifica-100-x-500mg.html#.VdG3hVN_Oko
https://coveloping.com/tools/open-graph-tag-tester
Please, if you know whats happend tell me, i dont understand...
P.S: sorry for my bad english.

Context Provider Response is not recognized

Using the latest development version:
I have registered a context provider for an attribute. When the attribute is asked the context provider is interrogated through a POST request which returns
curl -v --request POST http://130.206.83.68/ngsi10/queryContext
* About to connect() to 130.206.83.68 port 80 (#0)
* Trying 130.206.83.68... connected
> POST /ngsi10/queryContext HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: 130.206.83.68
> Accept: */*
>
< HTTP/1.1 200 OK
< X-Powered-By: Express
< Content-Type: application/json; charset=utf-8
< Content-Length: 206
< Date: Fri, 27 Mar 2015 11:09:05 GMT
< Connection: keep-alive
<
* Connection #0 to host 130.206.83.68 left intact
* Closing connection #0
{"contextResponses":[{"contextElement":
{"isPattern":false,"id":"Madrid","type":"Ciudad",
"attributes":[{"name":"population7","value":"6000","type":"string"}]},
"statusCode": {"code":200,"reasonPhrase":"OK"}}]}
However Orion does not recognize the response and the following trace appears:
INFO#10:59:36 clientSocketHttp.cpp[154]: Starting transaction to 130.206.83.68:80/ngsi10/queryContext
INFO#10:59:36 clientSocketHttp.cpp[364]: Notification Successfully Sent to 130.206.83.68:80/ngsi10/queryContext
INFO#10:59:36 clientSocketHttp.cpp[375]: Transaction ended
WARNING#10:59:36 postQueryContext.cpp[211]: Other Error (context provider response to QueryContext is empty)
INFO#10:59:36 rest.cpp[383]: Transaction ended
This problem is probably due to current Orion version (0.19.0) assumes always that the context providers implement XML rendering. Thus, if your context provider responses in JSON, Orion will not understand the response.
The solution is to implement XML in your context provider (or wait until Orion implement the proper mechanism).

Questions on proper REST api design specifically on the PUT action when updating a resource

I'm creating a REST interface (aren't we all), and I want to UPDATE a resource.
So, I think to use a PUT.
So, i read this.
My take away is that i PUT to a URL like this
/hc/api/v1/organizer/event/762d36c2-afc5-4c51-84eb-9b5b0ef2990c
with a payload, then a permanent redirect to the URL that it can GET an updated version of the resource.
In this case it happens to be the same URL, different action.
So my questions are:
Is my understanding of updating a resource correct in using a PUT, and is my understanding of the use of the PUT correct.
When a client gets a redirect does it do the same action on the redirected URL as it did on the original URL? If its "depends" is there a standard most clients follow?
I ask the 2nd question, because POSTMAN and my JQuery AJAX calls are choking. JQuery because of net::ERR_TOO_MANY_REDIRECTS. So is it redirecting and trying the PUT again, which it will get another REDIRECT?
curl blows up too but even though it says if it gets a 301 it will switch to a GET, it doesn't really seem to do that when i look at the output (below).
When curl follows a redirect and the request is not a plain GET (for example POST or PUT), it will do the following request with a GET if the HTTP response was 301, 302, or 303. If the response code was any other 3xx code, curl will re-send the following request using the same unmodified method.
CURL OUTPUT (edited for brevity) (also note how it says its going to switch to a GET [incorrectly from a POST], but then it seems to do a PUT anyway):
curl -X PUT -H "Authorization: Basic AUTHZ==" -H "Content-Type: application/json" -H "Cache-Control: no-cache" -H "Postman-Token: e80657f0-a8f5-af77-1d9d-d7bc22ed0b30" -d '{ JSONDATA"}' http://localhost:8080/hc/api/v1/organizer/event/762d36c2-afc5-4c51-84eb-9b5b0ef2990c -v -L
* Hostname was NOT found in DNS cache
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> PUT /hc/api/v1/organizer/event/762d36c2-afc5-4c51-84eb-9b5b0ef2990c HTTP/1.1
> User-Agent: curl/7.37.1
> Host: localhost:8080
> Accept: */*
> Authorization: Basic AUTHZ==
> Content-Type: application/json
> Cache-Control: no-cache
> Postman-Token: e80657f0-a8f5-af77-1d9d-d7bc22ed0b30
> Content-Length: 203
>
* upload completely sent off: 203 out of 203 bytes
< HTTP/1.1 301 Moved Permanently
< Connection: keep-alive
< X-Powered-By: Undertow/1
< Set-Cookie: rememberMe=deleteMe; Path=/hc; Max-Age=0; Expires=Fri, 20-Feb-2015 03:53:28 GMT
< Set-Cookie: JSESSIONID=uwI3_41LAa7vlvapTsrZdw10.macbook-air; path=/hc
* Server WildFly/8 is not blacklisted
< Server: WildFly/8
< Location: /hc/api/v1/organizer/event/762d36c2-afc5-4c51-84eb-9b5b0ef2990c
< Content-Length: 0
< Date: Sat, 21 Feb 2015 03:53:28 GMT
<
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://localhost:8080/hc/api/v1/organizer/event/762d36c2-afc5-4c51-84eb-9b5b0ef2990c'
* Switch from POST to GET
* Found bundle for host localhost: 0x7f9e4b415430
* Re-using existing connection! (#0) with host localhost
* Connected to localhost (127.0.0.1) port 8080 (#0)
> PUT /hc/api/v1/organizer/event/762d36c2-afc5-4c51-84eb-9b5b0ef2990c HTTP/1.1
> User-Agent: curl/7.37.1
> Host: localhost:8080
> Accept: */*
> Authorization: Basic dGVzdHVzZXIxOlBhc3N3b3JkMQ==
> Content-Type: application/json
> Cache-Control: no-cache
> Postman-Token: e80657f0-a8f5-af77-1d9d-d7bc22ed0b30
>
< HTTP/1.1 500 Internal Server Error
< Connection: keep-alive
< Set-Cookie: JSESSIONID=fDXxlH2xI-0-DEaC6Dj5EhD9.macbook-air; path=/hc
< Content-Type: text/html; charset=UTF-8
< Content-Length: 8593
< Date: Sat, 21 Feb 2015 03:53:28 GMT
<
...failure ensues... It actually does a PUT
thanks in advance.
I think you're reading too much into the 301 redirect section.
If you want to update a resource using PUT, return:
201: if the resource was created
200: with the updated resource
The 301 in question only applies if there actually is a redirect in question - like, if something can be identified by name, and you need to redirect it to a url that has the id or something. (Maybe you refactor and people are still consuming the old endpoint).
So, do you really need to redirect your PUT requests? Because you should be sending back the updated resource within the same loop using 200, like stated above, instead of "redirecting to GET".
EDIT: Fix some spelling.

Is Paypal Sandbox offline?

Sorry for asking here, but this is a bit of an emergency and I know from past experience, it could take more than 7 days to get a response from Paypal.
I am working on a project that, about 10 days ago, was working fine. 2 Days ago I returned to the PP portion and all of a sudden, I am getting no response from Paypal.
This is the URL I am calling:
https://api.sandbox.paypal.com/v1/payments/payment
And absolutely no response is being returned. I'm certain that before it at least returned an error if no parameters are included. But now, nothing at all with or without parameters.
Paypal does have a status page for the live site. Not sure if it still extends to the sandbox, tho. There was a major outage on the live end yesterday.
Which API are you using?
Do you still have issues with this? It's working fine for me. I don't believe we had an outage either.
$ curl -v https://api.sandbox.paypal.com/v1/payments/payment
* About to connect() to api.sandbox.paypal.com port 443 (#0)
* Trying 23.43.75.39...
* connected
* Connected to api.sandbox.paypal.com (23.43.75.39) port 443 (#0)
*a lot of SSL stuff*
> GET /v1/payments/payment HTTP/1.1
> User-Agent: curl/7.28.1
> Host: api.sandbox.paypal.com
> Accept: */*
>
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 401 Unauthorized
< Server: Apache-Coyote/1.1
< Paypal-Debug-Id: 7177ce39c4142
< Content-Type: application/json
< Content-Length: 0
< DC: origin1-api.sandbox.paypal.com
< Date: Sun, 03 Nov 2013 12:07:31 GMT
< Connection: keep-alive
< Set-Cookie: DC=origin1-api.sandbox.paypal.com; secure
<
* Connection #0 to host api.sandbox.paypal.com left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
Note the HTTP 401 response (because I didn't specify a valid Authorization header with an access token for the request).

access forbidden error when listing caldav directory

I am trying to list the users directory of apple calendar server on my localhost. i am getting access forbidden error , its the same for groups as well. my operating system is ubuntu 12.04 LTS. and the package is from the repository.
here is the log of the runshell.py command
/calendars/users > ls
<-------- BEGIN HTTP CONNECTION -------->
Server: localhost
<-------- BEGIN HTTP REQUEST -------->
PROPFIND /calendars/users/ HTTP/1.1
Host: localhost:8008
Authorization: Digest username="test", realm="Test Realm", nonce="17913381079262023151194175611", uri="/calendars/users/", response="df3db481efdc68df9c39733a957f072a", algorithm="md5"
Content-Length: 145
Content-Type: text/xml; charset=utf-8
Depth: 1
Brief: t
<?xml version='1.0' encoding='utf-8'?>
<ns0:propfind xmlns:ns0="DAV:">
<ns0:prop>
<ns0:resourcetype />
</ns0:prop>
</ns0:propfind>
<-------- BEGIN HTTP RESPONSE -------->
HTTP/1.1 403 Forbidden
Date: Mon, 03 Jun 2013 06:48:12 GMT
DAV: 1, access-control
Content-Type: text/html;charset=utf-8
Content-Length: 139
Server: Twisted/8.2.0 TwistedWeb/8.2.0
<html><head><title>403 Forbidden</title></head><body><h1>Forbidden</h1>You don't have permission to access /calendars/users/.</body></html>
<-------- END HTTP RESPONSE -------->
<-------- END HTTP CONNECTION -------->
Ignoring error: 403
First of all, have you verified that the request uri that you are using corresponds to the DAV:principal-collection-set property ? See https://www.rfc-editor.org/rfc/rfc3744#section-5.8
Then, the principals namespace is typically not queried through PROPFIND but rather through a DAV:principal-property-search REPORT query. See https://www.rfc-editor.org/rfc/rfc3744#section-9.4
Now, if you want to retrieve all the users on the calendar server, I'm not sure that the server will actually let you do that, especially if you have a large number of users.