I have a multimodule Jersey project, where at some point in serving a request a Gateway module will talk to another Gateway module (this is the "service discovery" part of the communication) on another PC. This communication works well, but when the code execution gets to the next request where these 2 modules talk again (this is still for serving that first request, this is the "negotiation" part of the communication between the gateway modules), the request times out fast with a
Caused by: java.net.SocketException: Unexpected end of file from server
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:790)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:787)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1534)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1439)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:394)
at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:285)
... 38 more
I use the -Djavax.net.debug=SSL VM argument to see what's happening, and both gateways print the same log:
Keep-Alive-Timer, called close()
Keep-Alive-Timer, called closeInternal(true)
Keep-Alive-Timer, SEND TLSv1.2 ALERT: warning, description = close_notify
Keep-Alive-Timer, WRITE: TLSv1.2 Alert, length = 64
Keep-Alive-Timer, called closeSocket(true)
grizzly-nio-kernel(4) SelectorRunner, READ: TLSv1.2 Alert, length = 64
grizzly-nio-kernel(4) SelectorRunner, RECV TLSv1.2 ALERT: warning, close_notify
grizzly-nio-kernel(4) SelectorRunner, closeInboundInternal()
grizzly-nio-kernel(4) SelectorRunner, closeOutboundInternal()
grizzly-nio-kernel(4) SelectorRunner, SEND TLSv1.2 ALERT: warning, description = close_notify
grizzly-nio-kernel(4) SelectorRunner, WRITE: TLSv1.2 Alert, length = 64
This time the code execution does not reach my code on the receiver gateway. Each HTTP request I send in these modules, a new javax.ws.rs.client.Client is built and used. Both gateway requests have the exact same javax.ws.rs.client.Client configuration (same connect/read timeout, same SSL context, same hostnameverifier), only the path and payload is different, I'm not trying to keep connections open at all, since this is a REST application.
I cant find anything useful on the net for Keep-Alive-Timer, called close(), but I assume this is the TLS keep alive function, since this communication works well without TLS, on normal HTTP. What could be the problem? I can provide more detail if needed.
Related
I went to the documentation and found out these
# The time after which an idle connection will be automatically closed.
# Set to infinite to completely disable idle connection timeouts.
idle-timeout = 10 s
# Defines the default time period within which the application has to
# produce an HttpResponse for any given HttpRequest it received.
# The timeout begins to run when the *end* of the request has been
# received, so even potentially long uploads can have a short timeout.
# Set to `infinite` to completely disable request timeout checking.
#
# If this setting is not `infinite` the HTTP server layer attaches a
# `Timeout-Access` header to the request, which enables programmatic
# customization of the timeout period and timeout response for each
# request individually.
request-timeout = 20 s
I have a scenario where my server takes more than 10 seconds to process a response but before sending the HTTPResponse the TCP connection between the client and server is timed out because of idle timeout.
Although the connection is idle at the moment but the request is still processing.
I thought this was the responsibility of response timeout?
Can anyone in this context explain me the difference between idle-timeout and response-timeout?
Documentation is a bit confusing, I have run the experiments based on that :
idle-timeout: It is the max time a connection can sit idle. It behaves the same as the Request timeout. Example :
idle-timeout = 1 s
Application has sent a request to 3rd party API and connection established, But 3rd party is not responding back. Then you will get a Timeout exception.
"akka.stream.scaladsl.TcpIdleTimeoutException"
connecting-timeout: 500 ms. It indicates the maximum time (500 ms) in which an Http connection must be established.
From the documentation and more details here
# The idle timeout for an open connection after which it will be closed
# Set to null or "infinite" to disable the timeout, but notice that this
# is not encouraged since timeout are important mechanisms to protect your
# servers from malicious attacks or programming mistakes.
idleTimeout = 75 seconds
If looks like you are setting the idle timeout lower than the request timeout, so it takes precedence. Your idle timeout settings should be longer than the request timeout, so for each request, the request timeout decides when to close the connection.
I need to issue a series of parallelized web requests from the same server to a specific domain, but control what IP address these requests actually go to. Originally, I came up with a scheme where I would request the IP I wanted specifically, and then manually set the Host: www.example.com header on the request, and use a series of handlers to make sure that redirects issued followed the same pattern.
This seemed to work for some time, but lately I've been having trouble with redirects to HTTPS. The handshake will fail, and the request in turn. I have tried disabling SSL verification in a variety of ways, including:
local $ENV{ PERL_LWP_SSL_VERIFY_HOSTNAME } = 0;
local $ENV{ HTTPS_DEBUG } = 1;
$ua->ssl_opts(
SSL_ca_file => Mozilla::CA::SSL_ca_file(),
verify_hostname => 0,
SSL_verify_mode => 0x00,
);
IO::Socket::SSL::set_ctx_defaults(
SSL_verifycn_scheme => 'www',
SSL_verify_mode => 0,
);
I have also tried using LWP::UserAgent::DNS::Hosts to solve the problem, but it persists.
<edit>I should note that the reason why turning off peer validation for SSL is not solving the problem is likely because for some reason requesting this way is actually causing the handshake to fail, not failing on a validation point.</edit>
One thing that works is making an entry in /etc/hosts to point the domain at the appropriate IP, however this is not practical, because I may need to run tens, or hundreds, of tests, in parallel, on the same domain.
Is there a way to emulate the functionality of adding an entry to /etc/hosts that does not involve requesting the IP specifically and overriding the Host: ... HTTP header?
EDIT: SSL Debug Info
DEBUG: .../IO/Socket/SSL.pm:1914: new ctx 140288835318480
DEBUG: .../IO/Socket/SSL.pm:402: socket not yet connected
DEBUG: .../IO/Socket/SSL.pm:404: socket connected
DEBUG: .../IO/Socket/SSL.pm:422: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:455: not using SNI because hostname is unknown
DEBUG: .../IO/Socket/SSL.pm:478: set socket to non-blocking to enforce timeout=180
DEBUG: .../IO/Socket/SSL.pm:491: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:501: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:511: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:531: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:491: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:1388: SSL connect attempt failed with unknown error
DEBUG: .../IO/Socket/SSL.pm:497: fatal SSL error: SSL connect attempt failed with unknown error error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
DEBUG: .../IO/Socket/SSL.pm:1948: free ctx 140288835318480 open=140288835318480
DEBUG: .../IO/Socket/SSL.pm:1953: free ctx 140288835318480 callback
DEBUG: .../IO/Socket/SSL.pm:1956: OK free ctx 140288835318480
And in the response I get:
Can't connect to redacted.org:443
SSL connect attempt failed with unknown error error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure at /System/Library/Perl/Extras/5.18/LWP/Protocol/http.pm line 51.
It fails equally well on our server (using an older legacy version of Perl, which I will not disclose here as it seems irrelevant).
The server initially responds to a non-HTTPS request with a 301 redirect to the HTTPS site. Then the failure occurs. I will post reproducing code with the specific details of my request removed, but any site which redirects non-HTTPS traffic to HTTPS should suffice.
use IO::Socket::SSL qw/ debug4 /;
use LWP::UserAgent;
use LWP::UserAgent::DNS::Hosts;
use HTTP::Request;
use Mozilla::CA;
use Data::Dumper;
LWP::UserAgent::DNS::Hosts->register_hosts(
'recacted.org' => '127.0.0.1', # no I am not redirecting to loopback in reality, this is anonymized
'www.redacted.org' => '127.0.0.1',
);
LWP::UserAgent::DNS::Hosts->enable_override;
my $ua = LWP::UserAgent->new;
$ua->ssl_opts( SSL_ca_file => Mozilla::CA::SSL_ca_file() );
my $request = HTTP::Request->new(GET => 'http://redacted.org/');
my $response = $ua->request($request);
print $response->content; #Dumper ( $response->is_success ? $response->headers : $response );
Again, that is not the production code, just enough code to reproduce the issue. It doesn't seem to have anything to do with SSL verification, but moreover an inability to negotiate the request, presumably because LWP::UserAgent::DNS::Hosts is doing exactly what I was doing: changing the request target to the desired IP, and then writing the Host: ... header manually. Why this causes the SSL handshake to fail, I do not know.
On my local machine debugging
openssl version -a: 1.0.2j 26 Sep 2016
IO::Socket::SSL->VERSION == 1.966
Net::SSLeay->VERSION == 1.72
On a server of ours
openssl version -a: 1.0.1t 3 May 2016
IO::Socket::SSL->VERSION == 1.76
Net::SSLeay->VERSION == 1.48
Given that it works with an explicit /etc/hosts file but not with just replacing PeerAddr or using LWP::UserAgent::DNS::Hosts this looks like a problem with the SNI extension. This TLS extension is used to provide the TLS server with the requested hostname (similar to the HTTP Host header) so that it can choose the appropriate certificate. If this SNI extension is missing some servers return a default certificate while others throw an error, like in this case.
The fix is to provide the hostname using SSL_hostname in ssl_opts. Such fix could probably also help with LWP::UserAgent::DNS::Hosts, i.e in LWP/Protocol/https/hosts.pm:
12 if (my $peer_addr = LWP::UserAgent::DNS::Hosts->_registered_peer_addr($host)) {
13 push #opts, (
14 PeerAddr => $peer_addr,
15 Host => $host,
16 SSL_verifycn_name => $host,
NEW SSL_hostname => $host, # for SNI
17 );
18 }
I can see here how to set the socket adapter for Zend_Http_Client
http://framework.zend.com/manual/1.12/en/zend.http.client.adapters.html
The examples they give are tls or sslv2.
Does anyone know what the setting is for tls1.2?
I've tried a few but I'm just guessing. I get errors along the lines of:
Unable to find the socket transport "tls1.2" - did you forget to enable it when you configured PHP?'
If I try tls on it's own I get:
Unable to Connect to tls://www.sandbox.paypal.com:443
(For others Googling this is to fix our IPN verification with PayPal which gives the following error on our SSL connection:
Error in cURL request: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
)
You were close! Set ssltransport to tlsv1.2.
$config = array(
'adapter' => 'Zend_Http_Client_Adapter_Socket',
'ssltransport' => 'tlsv1.2'
);
$client = new Zend_Http_Client('https://www.sandbox.paypal.com', $config);
$response = $client->request();
echo $response->getStatus();
Figured it out by first checking what Zend_Http_Client_Adapter_Socket uses to send HTTP requests, which turned out to be stream_socket_client(). You can run the stream_get_transports() on your system to view the list of available socket transports.
See SSL/TLS version selection in the OpenSSL changes in PHP 5.6.x migration guide for more examples of how to select specific SSL/TLS versions.
Tested with PHP 5.6 on Ubuntu 14.04 Trusty, which supports TLSv1.2 out of the box.
Fiddler almost not work for me. Seems the problem only with https.
For example to open https google.com I need to wait around 40 seconds
Screenshots:
immediately after request
after ~40 seconds
Fiddler log:
18:02:46:3326 Fiddler Running...
18:02:46:3922 Windows 8+ AppContainer isolation feature detected.
18:03:09:5427 Assembly 'C:\Program Files (x86)\Fiddler2\CertMaker.dll' was not found. Using default Certificate Generator.
18:03:09:5467 /Fiddler.CertMaker> Using Fiddler.DefaultCertificateProvider+CertEnrollEngine for certificate generation; UseWildcards=False.
18:03:11:3745 HTTPSLint> Warning: ClientHello record was 508 bytes long. Some servers have problems with ClientHello's greater than 255 bytes. githubcom /ssllabs/research/wiki/Long-Handshake-Intolerance
18:03:11:3855 HTTPSLint> Warning: ClientHello record was 508 bytes long. Some servers have problems with ClientHello's greater than 255 bytes. githubcom /ssllabs/research/wiki/Long-Handshake-Intolerance
18:03:11:3895 HTTPSLint> Warning: ClientHello record was 508 bytes long. Some servers have problems with ClientHello's greater than 255 bytes. githubcom /ssllabs/research/wiki/Long-Handshake-Intolerance
18:03:11:3915 HTTPSLint> Warning: ClientHello record was 508 bytes long. Some servers have problems with ClientHello's greater than 255 bytes. githubcom /ssllabs/research/wiki/Long-Handshake-Intolerance
18:03:11:3945 HTTPSLint> Warning: ClientHello record was 508 bytes long. Some servers have problems with ClientHello's greater than 255 bytes. githubcom /ssllabs/research/wiki/Long-Handshake-Intolerance
18:03:20:2192 [Fiddler] No HTTPS request was received from (chrome:10428) new client socket, port 6091.
18:03:20:3110 [Fiddler] No HTTP request was received from (chrome:10428) new client socket, port 6134.
18:03:20:3120 [Fiddler] No HTTP request was received from (chrome:10428) new client socket, port 6130.
18:03:28:8160 HTTPSLint> Warning: ClientHello record was 508 bytes long. Some servers have problems with ClientHello's greater than 255 bytes. githubcom /ssllabs/research/wiki/Long-Handshake-Intolerance
18:03:30:2198 [Fiddler] No HTTPS request was received from (chrome:10428) new client socket, port 6095.
18:03:30:2198 [Fiddler] No HTTPS request was received from (chrome:10428) new client socket, port 6097.
18:03:30:2198 [Fiddler] No HTTPS request was received from (chrome:10428) new client socket, port 6099.
18:03:30:2198 [Fiddler] No HTTPS request was received from (chrome:10428) new client socket, port 6101.
18:03:50:2219 [Fiddler] No HTTPS request was received from (chrome:10428) new client socket, port 6163.
18:03:50:2219 [Fiddler] No HTTPS request was received from (chrome:10428) new client socket, port 6141.
18:03:50:2219 [Fiddler] No HTTPS request was received from (chrome:10428) new client socket, port 6167.
18:04:10:2230 [Fiddler] No HTTPS request was received from (chrome:10428) new client socket, port 6176.
18:04:10:2230 [Fiddler] No HTTPS request was received from (chrome:10428) new client socket, port 6179.
Many times in chrome I see: Waiting for proxy tunnel...and site shows This webpage is not available (ERR_TIMED_OUT)
In EDGE I even can't open http site for all sub requests I see blue up arrow which means fiddler trying to load it (after ~40 seconds I get loaded all that requests)
I tried to reset Internet Properties-> Advanced tab-> Restore advanced settings - it doesn't help me.
Also I tried to restarted my system, also I restarted fiddler after any changes I made.
Fiddler settings:
Certificates generated by CertEnroll engine. I tried to change it to MakeCert. Few times I reset All certificates, also manually removes certificates.
Browsers: Chrome/Firefox
Gateway info in fiddler: No upstream gateway proxy is configured.
Recently I made clear installation of Windows 10.
I do not have any Antivirus.
Windows 10 Pro x64
Fiddler v4.6.2.0
I need fiddler for my work. Please help me
UPDATED:
This is can be issue with Protocols. Currently in fiddler I have next protocols:
fiddler.network.https> HTTPS handshake to www.bing.com (for #4) failed. System.IO.IOException Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. < An existing connection was forcibly closed by the remote host
and
fiddler.network.https> HTTPS handshake to www.google.com.ua (for #23) failed. System.ComponentModel.Win32Exception The client and server cannot communicate, because they do not possess a common algorithm
As I thought the issue was with protocols enabled in Internet Options in Windows and Fiddler protocols.
I ticked Use SSL 3.0 and use TSL 1.0 in Interent properties (all other should be unticked)
in Fiddler protocols I typed: ;ssl3;tls1.0
And after this changes everything works perfectly
Yes, Use SSL 3.0 and use TSL 1.0 in Internet properties for it to work. Previously I also did the same mistake but now working fine. I also checked on fiddler for the same.
I use boost socket with SSL by using the following source code:
ssl::context ctx(ssl::context::sslv23);
ctx.set_verify_mode(ssl::verify_peer);
ctx.load_verify_file("ca.pem");
I would like to know if ssl::context::sslv23 also activates TLS ?
If I want to force TLS connection (no SSL connection), Does it work:
ctx.set_options( boost::asio::ssl::context::no_sslv2 | boost::asio::ssl::context::no_sslv3 );
Same question with SSL connection only:
ctx.set_options( boost::asio::ssl::context::no_tlsv1 );
Thanks
I believe you need to refer OpenSSL documentation, since asio::ssl::context is a thin wrapper for SSL_CTX. Thus ssl::context constructor calls SSL_CTX_new() function with an appropriate method. As well as the ssl::context::set_options() function calls SSL_CTX_set_options() function.
In particular, for ssl::context::sslv23 method it would be:
SSLv23_method(void), SSLv23_server_method(void),
SSLv23_client_method(void) A TLS/SSL connection established with these
methods may understand the SSLv2, SSLv3, TLSv1, TLSv1.1 and TLSv1.2
protocols.
If the cipher list does not contain any SSLv2 ciphersuites (the
default cipher list does not) or extensions are required (for example
server name) a client will send out TLSv1 client hello messages
including extensions and will indicate that it also understands
TLSv1.1, TLSv1.2 and permits a fallback to SSLv3. A server will
support SSLv3, TLSv1, TLSv1.1 and TLSv1.2 protocols. This is the best
choice when compatibility is a concern.
If any SSLv2 ciphersuites are included in the cipher list and no
extensions are required then SSLv2 compatible client hellos will be
used by clients and SSLv2 will be accepted by servers. This is not
recommended due to the insecurity of SSLv2 and the limited nature of
the SSLv2 client hello prohibiting the use of extensions.
The list of protocols available can later be limited using the
SSL_OP_NO_SSLv2, SSL_OP_NO_SSLv3, SSL_OP_NO_TLSv1, SSL_OP_NO_TLSv1_1
and SSL_OP_NO_TLSv1_2 options of the SSL_CTX_set_options() or
SSL_set_options() functions. Using these options it is possible to
choose e.g. SSLv23_server_method() and be able to negotiate with all
possible clients, but to only allow newer protocols like TLSv1,
TLSv1.1 or TLS v1.2.
Applications which never want to support SSLv2 (even is the cipher
string is configured to use SSLv2 ciphersuites) can set
SSL_OP_NO_SSLv2.