Web based application intermittently throwing '500 Internal Server Error'. This application is deployed on WAS 8.0 server with IHS and WAS servers hosted on different machines. Please find below the TRACE level plugin logs at IHS end.
DEBUG: ws_common: websphereExecute: Wrote the request; reading the response (timeout 900)
DETAIL: lib_htresponse: htresponseRead: Reading the response: 64040a70 TRACE: lib_rio: Blocking for read, waiting 900
DEBUG: lib_rio.c line 896 : Read failed, rc=104
DEBUG: lib_htresponse: htresponseSetError: Setting the error to: |READ_FAILED|(1, Line: 713)
DEBUG: ws_common: websphereExecute: Failed to read from an old stream; probably Keep-Alive timeout fired
DEBUG: ws_transport: transportStreamDequeue: Checking for existing stream from the queue
DEBUG: lib_stream: destroyStream: Destroying the stream
DEBUG: lib_rio: rclose: socket 13 closed
STATS: ws_server: serverSetFailoverStatus: Server xxxxx : pendingRequests 0 failedRequests 7 affinityRequests 57 totalRequests 51.
ERROR: ws_common: websphereHandleRequest: Failed to execute the transaction to 'xxxxx' on host 'yyyyy'; will try another one
ERROR: ws_common: websphereWriteRequestReadResponse: Failed to find an app server to handle this request
ERROR: ESI: getResponse: failed to get response: rc = 10 DEBUG: ESI: esiHandleRequest: failed to get response
DEBUG: ESI: esiRequestUrlStackDestroy
DEBUG: ESI: cacheURL: '/site/index'
DETAIL: ESI: esiRequestPopUrl: '/site/index'
DEBUG: ESI: esiUrlDestroy: '/site/index'
ERROR: [://site/index] ws_common: websphereHandleRequest: Failed to handle request rc=10
DEBUG: ws_common: websphereCloseConnection
DETAIL: ws_common: websphereEndRequest: Ending the request
Please help me to get the issue resolved. Thanks in advance
rc=104 is the linux return code indicating that the socket was reset by the partner address so you need to look at the application server for errors.
Related
I do request to https://bank.gov.ua
my $ua = Mojo::UserAgent->new;
$ua->get("https://bank.gov.ua/NBUStatService/v1/statdirectory/exchange?valcode=EUR&date=$date_now&json");
And get error:
DEBUG: .../IO/Socket/SSL.pm:3010: new ctx 146452496
DEBUG: .../IO/Socket/SSL.pm:1638: don't start handshake: IO::Socket::SSL=GLOB(0xc955978)
DEBUG: .../IO/Socket/SSL.pm:787: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:829: using SNI with hostname bank.gov.ua
DEBUG: .../IO/Socket/SSL.pm:864: request OCSP stapling
DEBUG: .../IO/Socket/SSL.pm:894: call Net::SSLeay::connect
DEBUG: .../IO/Socket/SSL.pm:897: done Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:907: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:894: call Net::SSLeay::connect
DEBUG: .../IO/Socket/SSL.pm:897: done Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:900: local error: SSL connect attempt failed
DEBUG: .../IO/Socket/SSL.pm:903: fatal SSL error: SSL connect attempt failed
DEBUG: .../IO/Socket/SSL.pm:3059: free ctx 146452496 open=
DEBUG: .../IO/Socket/SSL.pm:3063: free ctx 146452496 callback
DEBUG: .../IO/Socket/SSL.pm:3070: OK free ctx 146452496
I can do request with curl from this host to give url with no problem.
Does any know what problem is when I do this via IO::Socket::SSL (Mojo::UserAgent)?
This server is pretty strange:
the first request with openssl s_client -connect bank.gov.ua:443 fails with the server simply closing the connection: "SSL handshake has read 0 bytes and written 303 bytes"
the Mojo::UserAgent code fails too
doing a request with an explicit TLS 1.2 succeeds: openssl s_client -connect bank.gov.ua:443 -tls1_2
trying the first request again suddenly succeeds too
an the Mojo::UserAgent code now succeeds too
My only explanation is some firewall or load balancer which temporarily white lists an IP address in case it has seen a valid TLS ClientHello - and which considers TLS 1.3 not a valid one.
I have recently installed FreeIPA on RHEL7. This seems to be running well for few hours and then calls to ipa starts to fail with the following error.
ipa: ERROR: No valid Negotiate header in server response
==================================================
[root ~]# ipa -v user-find --all
ipa: INFO: trying https://xxx.xxx.xxx.xxx/ipa/json
ipa: INFO: [try 1]: Forwarding 'user_find/1' to json server 'https://xxx.xxx.xxx.xxx/ipa/json'
ipa: ERROR: No valid Negotiate header in server response
-=================================================
[I have masked the hostnames with 'xxx']
In /var/log/httpd/error_log - I see the following error.
[Thu Dec 14 15:50:23.413286 2017] [auth_gssapi:error] [pid 10694] [client xxx.xxx.xxx.xxx:50198] GSS ERROR In Negotiate Auth: gss_accept_sec_context() failed: [Unspecified GSS failure. Minor code may provide more information ( Request ticket server HTTP/xxx.xxxx.xxxx.xxx#EC2.INTERNAL kvno 2 not found in keytab; keytab is likely out of date)], referer: https://xxx.xxx.xxx.xxx/ipa/xml
What is the possible cause? Looks like some misconfiguration.
We upgraded perl on our windows server 2012 to latest stable version. Ever since we did that we are getting intermittent Cannot connect to the server errors throwing 500 error responses.
But it is so intermittent we cannot identify the problem. Here is the debug log for some idea:
DEBUG: .../IO/Socket/SSL.pm:763: done Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:773: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:783: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:803: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:759: call Net::SSLeay::connect
DEBUG: .../IO/Socket/SSL.pm:763: done Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:766: local error: SSL connect attempt failed
Windows server is running IIS 7.5 and We have a valid Certificate issued by COMODO.
Any insight would be much appreciated. Please let me know if you need any further information.
So updating windows server 2012 (my OS), which has not been updated in over a year fixed the problem. There must have been some windows patch for new tls support.
I'm trying to send email with Perl. My code works fine with Yahoo's SMTP server but when I try to use another SMTP server (with a self-signed SSL certificate), I get this error:
Couldn't start TLS: SSL connect attempt failed because of handshake problems error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
at mail2.pl line 2.
My code is:
use Net::SMTP::TLS;
my $mailer = new Net::SMTP::TLS(
'mail.SomeHost.com',
Hello => 'mail.SomeHost.com',
Port => 587,
User => 'info#SomeHost.com',
Password=> '123456789'
);
$mailer->mail('info#SomeHost.com');
$mailer->to('info#SomeHost.com');
$mailer->data();
$mailer->datasend("From: info#SomeHost.com\n");
$mailer->datasend("To: info#SomeHost.com\n");
$mailer->datasend("Subject: test\n");
$mailer->datasend("Content-type: text/html\n\n");
$mailer->datasend("\n");
$mailer->datasend("<html><body><p>hi text</p><br></body></html>\n");
$mailer->dataend();
$mailer->quit;
How can I fix this?
Result with -MIO::Socket::SSL=debug4:
DEBUG: .../IO/Socket/SSL.pm:2537: new ctx 17003088
DEBUG: .../IO/Socket/SSL.pm:1343: start handshake
DEBUG: .../IO/Socket/SSL.pm:553: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:589: not using SNI because hostname is unknown
DEBUG: .../IO/Socket/SSL.pm:621: request OCSP stapling
DEBUG: .../IO/Socket/SSL.pm:640: set socket to non-blocking to enforce timeout=5
DEBUG: .../IO/Socket/SSL.pm:653: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:663: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:673: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:693: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:653: Net::SSLeay::connect -> 0
DEBUG: .../IO/Socket/SSL.pm:701: connection failed - connect returned 0
DEBUG: .../IO/Socket/SSL.pm:1769: SSL connect attempt failed because of handshake problems
DEBUG: .../IO/Socket/SSL.pm:1774: SSL connect attempt failed because of handshake problems
error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
DEBUG: .../IO/Socket/SSL.pm:2570: free ctx 17003088 open=17003088
DEBUG: .../IO/Socket/SSL.pm:2575: free ctx 17003088 callback
DEBUG: .../IO/Socket/SSL.pm:2582: OK free ctx 17003088 Couldn't start TLS:
SSL connect attempt failed because of handshake problems
error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake
failure at mail2.pl line 2.
I am running Debian Jessie. I have a script that was working perfectly a couple of weeks ago on a virtual machine but has now stopped working suddenly. I can't think of any perl module updates I made that would have broken this. And if I run the script from another Jessie server, it works fine.
Here is the relevant code:
use LWP::UserAgent;
my $browser = LWP::UserAgent->new;
my $url = 'https://api.vimeo.com/oauth/authorize/client';
my $response = $browser->post($url, { grant_type => 'client_credentials' }, 'Authorization' => ' basic ' . encode_base64('CLIENT_ID:CLIENT_SECRET') );
if ($response->code ne '200') {
logf("There was a problem with the server response from Vimeo while requesting an access token.\nServer response: " . $response->decoded_content);
}
The error thrown is coming from /usr/share/perl5/LWP/Protocol/http.pm line 49 and reads: Can't connect to api.vimeo.com:443
When I run the script with use IO::Socket::SSL qw(debug3); here's what I get for output:
DEBUG: .../IO/Socket/SSL.pm:2537: new ctx 56137248
DEBUG: .../IO/Socket/SSL.pm:529: socket not yet connected
DEBUG: .../IO/Socket/SSL.pm:531: socket connected
DEBUG: .../IO/Socket/SSL.pm:553: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:586: using SNI with hostname api.vimeo.com
DEBUG: .../IO/Socket/SSL.pm:621: request OCSP stapling
DEBUG: .../IO/Socket/SSL.pm:640: set socket to non-blocking to enforce timeout=180
DEBUG: .../IO/Socket/SSL.pm:653: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:663: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:673: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:693: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:2393: ok=1 cert=58104432
DEBUG: .../IO/Socket/SSL.pm:2393: ok=1 cert=59040416
DEBUG: .../IO/Socket/SSL.pm:2393: ok=1 cert=58950080
DEBUG: .../IO/Socket/SSL.pm:1548: scheme=www cert=58950080
DEBUG: .../IO/Socket/SSL.pm:1558: identity=api.vimeo.com cn=*.vimeo.com alt=2 *.vimeo.com 2 vimeo.com
DEBUG: .../IO/Socket/SSL.pm:2443: got stapled OCSP response
run basic verify at /usr/share/perl5/IO/Socket/SSL.pm line 2450.
found issuer in chain at /usr/share/perl5/IO/Socket/SSL.pm line 2457.
got issuer at /usr/share/perl5/IO/Socket/SSL.pm line 2457.
DEBUG: .../IO/Socket/SSL.pm:653: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:1769: SSL connect attempt failed
DEBUG: .../IO/Socket/SSL.pm:1774: SSL connect attempt failed error:2707307E:OCSP routines:OCSP_check_validity:status not yet valid
DEBUG: .../IO/Socket/SSL.pm:659: fatal SSL error: SSL connect attempt failed error:2707307E:OCSP routines:OCSP_check_validity:status not yet valid
DEBUG: .../IO/Socket/SSL.pm:529: socket not yet connected
DEBUG: .../IO/Socket/SSL.pm:531: socket connected
DEBUG: .../IO/Socket/SSL.pm:553: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:586: using SNI with hostname api.vimeo.com
DEBUG: .../IO/Socket/SSL.pm:621: request OCSP stapling
DEBUG: .../IO/Socket/SSL.pm:640: set socket to non-blocking to enforce timeout=180
DEBUG: .../IO/Socket/SSL.pm:653: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:663: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:673: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:693: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:2393: ok=1 cert=58104432
DEBUG: .../IO/Socket/SSL.pm:2393: ok=1 cert=61018848
DEBUG: .../IO/Socket/SSL.pm:2393: ok=1 cert=61025968
DEBUG: .../IO/Socket/SSL.pm:1548: scheme=www cert=61025968
DEBUG: .../IO/Socket/SSL.pm:1558: identity=api.vimeo.com cn=*.vimeo.com alt=2 *.vimeo.com 2 vimeo.com
DEBUG: .../IO/Socket/SSL.pm:2443: got stapled OCSP response
run basic verify at /usr/share/perl5/IO/Socket/SSL.pm line 2450.
found issuer in chain at /usr/share/perl5/IO/Socket/SSL.pm line 2457.
got issuer at /usr/share/perl5/IO/Socket/SSL.pm line 2457.
DEBUG: .../IO/Socket/SSL.pm:653: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:1769: SSL connect attempt failed
DEBUG: .../IO/Socket/SSL.pm:1774: SSL connect attempt failed error:2707307E:OCSP routines:OCSP_check_validity:status not yet valid
DEBUG: .../IO/Socket/SSL.pm:659: fatal SSL error: SSL connect attempt failed error:2707307E:OCSP routines:OCSP_check_validity:status not yet valid
DEBUG: .../IO/Socket/SSL.pm:529: socket not yet connected
DEBUG: .../IO/Socket/SSL.pm:531: socket connected
DEBUG: .../IO/Socket/SSL.pm:553: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:586: using SNI with hostname api.vimeo.com
DEBUG: .../IO/Socket/SSL.pm:621: request OCSP stapling
DEBUG: .../IO/Socket/SSL.pm:640: set socket to non-blocking to enforce timeout=180
DEBUG: .../IO/Socket/SSL.pm:653: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:663: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:673: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:693: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:653: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:663: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:673: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:693: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:2393: ok=1 cert=58104432
DEBUG: .../IO/Socket/SSL.pm:2393: ok=1 cert=59035200
DEBUG: .../IO/Socket/SSL.pm:2393: ok=1 cert=61025968
DEBUG: .../IO/Socket/SSL.pm:1548: scheme=www cert=61025968
DEBUG: .../IO/Socket/SSL.pm:1558: identity=api.vimeo.com cn=*.vimeo.com alt=2 *.vimeo.com 2 vimeo.com
DEBUG: .../IO/Socket/SSL.pm:2443: got stapled OCSP response
run basic verify at /usr/share/perl5/IO/Socket/SSL.pm line 2450.
found issuer in chain at /usr/share/perl5/IO/Socket/SSL.pm line 2457.
got issuer at /usr/share/perl5/IO/Socket/SSL.pm line 2457.
DEBUG: .../IO/Socket/SSL.pm:653: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:1769: SSL connect attempt failed
DEBUG: .../IO/Socket/SSL.pm:1774: SSL connect attempt failed error:2707307E:OCSP routines:OCSP_check_validity:status not yet valid
DEBUG: .../IO/Socket/SSL.pm:659: fatal SSL error: SSL connect attempt failed error:2707307E:OCSP routines:OCSP_check_validity:status not yet valid
DEBUG: .../IO/Socket/SSL.pm:529: socket not yet connected
DEBUG: .../IO/Socket/SSL.pm:531: socket connected
DEBUG: .../IO/Socket/SSL.pm:553: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:586: using SNI with hostname api.vimeo.com
DEBUG: .../IO/Socket/SSL.pm:621: request OCSP stapling
DEBUG: .../IO/Socket/SSL.pm:640: set socket to non-blocking to enforce timeout=180
DEBUG: .../IO/Socket/SSL.pm:653: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:663: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:673: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:693: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:653: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:663: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:673: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:693: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:2393: ok=1 cert=58104432
DEBUG: .../IO/Socket/SSL.pm:2393: ok=1 cert=59044736
DEBUG: .../IO/Socket/SSL.pm:2393: ok=1 cert=61025968
DEBUG: .../IO/Socket/SSL.pm:1548: scheme=www cert=61025968
DEBUG: .../IO/Socket/SSL.pm:1558: identity=api.vimeo.com cn=*.vimeo.com alt=2 *.vimeo.com 2 vimeo.com
DEBUG: .../IO/Socket/SSL.pm:653: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:663: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:673: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:693: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:2443: got stapled OCSP response
run basic verify at /usr/share/perl5/IO/Socket/SSL.pm line 2450.
found issuer in chain at /usr/share/perl5/IO/Socket/SSL.pm line 2457.
got issuer at /usr/share/perl5/IO/Socket/SSL.pm line 2457.
DEBUG: .../IO/Socket/SSL.pm:653: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:1769: SSL connect attempt failed
DEBUG: .../IO/Socket/SSL.pm:1774: SSL connect attempt failed error:2707307E:OCSP routines:OCSP_check_validity:status not yet valid
DEBUG: .../IO/Socket/SSL.pm:659: fatal SSL error: SSL connect attempt failed error:2707307E:OCSP routines:OCSP_check_validity:status not yet valid
DEBUG: .../IO/Socket/SSL.pm:1758: IO::Socket::IP configuration failed
DEBUG: .../IO/Socket/SSL.pm:2570: free ctx 56137248 open=56137248
DEBUG: .../IO/Socket/SSL.pm:2575: free ctx 56137248 callback
DEBUG: .../IO/Socket/SSL.pm:2582: OK free ctx 56137248
The the $browser user agent is able to successfully execute a get request https://facebook.com without issue. It just doesn't seem to work for Vimeo.
fatal SSL error: ... :OCSP_check_validity:status not yet valid
Vimeo uses OCSP stapling to ease revocation checks but attaches a OCSP response which your system considers as not yet valid. It might be that the time on your system is wrong so that it thinks that the timestamp in the response is in the future, whereas it is in the present.
And if I run the script from another Jessie server, it works fine.
If both servers run the same software then I guess that the problematic server has the wrong time.