I installed Openstack on RedHad using RDO. Then I get the following error when I execute the glance commands.
glance image-list
Request returned failure status.
Invalid OpenStack Identity credentials.
Below is the backend error of api.log
2014-06-23 20:09:31.330 1262 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 127.0.0.1
2014-06-23 20:09:31.408 1262 DEBUG urllib3.connectionpool [-] "POST /v2.0/tokens HTTP/1.1" 200 9128 _make_request /usr/lib/python2.6/site-packages/urllib3/connectionpool.py:295
2014-06-23 20:09:31.418 1262 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 127.0.0.1
2014-06-23 20:09:31.435 1262 DEBUG urllib3.connectionpool [-] "GET /v2.0/tokens/revoked HTTP/1.1" 200 686 _make_request /usr/lib/python2.6/site-packages/urllib3/connectionpool.py:295
2014-06-23 20:09:31.443 1262 WARNING keystoneclient.middleware.auth_token [-] Verify error: Command 'openssl' returned non-zero exit status 4
2014-06-23 20:09:31.443 1262 DEBUG keystoneclient.middleware.auth_token [-] Token validation failure. _validate_user_token /usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py:943
2014-06-23 20:09:31.443 1262 TRACE keystoneclient.middleware.auth_token Traceback (most recent call last):
2014-06-23 20:09:31.443 1262 TRACE keystoneclient.middleware.auth_token File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 930, in _validate_user_token
2014-06-23 20:09:31.443 1262 TRACE keystoneclient.middleware.auth_token verified = self.verify_signed_token(user_token, token_ids)
2014-06-23 20:09:31.443 1262 TRACE keystoneclient.middleware.auth_token File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 1347, in verify_signed_token
2014-06-23 20:09:31.443 1262 TRACE keystoneclient.middleware.auth_token if self.is_signed_token_revoked(token_ids):
2014-06-23 20:09:31.443 1262 TRACE keystoneclient.middleware.auth_token File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 1299, in is_signed_token_revoked
2014-06-23 20:09:31.443 1262 TRACE keystoneclient.middleware.auth_token if self._is_token_id_in_revoked_list(token_id):
2014-06-23 20:09:31.443 1262 TRACE keystoneclient.middleware.auth_token File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 1306, in _is_token_id_in_revoked_list
2014-06-23 20:09:31.443 1262 TRACE keystoneclient.middleware.auth_token revocation_list = self.token_revocation_list
2014-06-23 20:09:31.443 1262 TRACE keystoneclient.middleware.auth_token File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 1413, in token_revocation_list
2014-06-23 20:09:31.443 1262 TRACE keystoneclient.middleware.auth_token self.token_revocation_list = self.fetch_revocation_list()
2014-06-23 20:09:31.443 1262 TRACE keystoneclient.middleware.auth_token File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 1459, in fetch_revocation_list
2014-06-23 20:09:31.443 1262 TRACE keystoneclient.middleware.auth_token return self.cms_verify(data['signed'])
2014-06-23 20:09:31.443 1262 TRACE keystoneclient.middleware.auth_token File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 1330, in cms_verify
2014-06-23 20:09:31.443 1262 TRACE keystoneclient.middleware.auth_token return verify()
2014-06-23 20:09:31.443 1262 TRACE keystoneclient.middleware.auth_token File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 1324, in verify
2014-06-23 20:09:31.443 1262 TRACE keystoneclient.middleware.auth_token inform=inform).decode('utf-8')
2014-06-23 20:09:31.443 1262 TRACE keystoneclient.middleware.auth_token File "/usr/lib/python2.6/site-packages/keystoneclient/common/cms.py", line 167, in cms_verify
2014-06-23 20:09:31.443 1262 TRACE keystoneclient.middleware.auth_token raise e
2014-06-23 20:09:31.443 1262 TRACE keystoneclient.middleware.auth_token CalledProcessError: Command 'openssl' returned non-zero exit status 4
2014-06-23 20:09:31.443 1262 TRACE keystoneclient.middleware.auth_token
2014-06-23 20:09:31.444 1262 DEBUG keystoneclient.middleware.auth_token [-] Marking token as unauthorized in cache _cache_store_invalid /usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py:1239
2014-06-23 20:09:31.444 1262 WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token
2014-06-23 20:09:31.444 1262 INFO keystoneclient.middleware.auth_
I don't think the source keystonerc_admin will solve this issue.
Try to change the token provider instead by doing this in "keystone.conf" :
[signing]
token_format=UUID
[token]
provider=keystone.token.providers.uuid.Provider
[ssl]
enable = False
You can always go there:
https://bugs.launchpad.net/python-keystoneclient/+bug/1176190
And not delete my replies ... :)
First, you need to source the file keystone_admin file located in /root
source keystone_admin
Then you can run any command like glance, nova, neutron, etc...
If you already did that, then change the following settings in keystone.conf and restart:
[signing]
token_format = UUID
I think disabling ssl is a bad idea, unless you are in a development environment.
Here you are great documentation: http://docs.openstack.org/admin-guide-cloud/content/ch-identity-mgmt-config.html
Related
I'm working on deploying my flask application on Nginx with Gunicorn and as of yesterday I was able to access my website on a remote device and run queries that affected the database but for some reason today it has abruptly stopped and I get the following error in the log file.
Connection._handle_dbapi_exception_noconnection(
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/lib/python3.8/site-
packages/sqlalchemy/engine/base.py", line 2117, in
_handle_dbapi_exception_noconnection
util.raise_(
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/lib/python3.8/site-
packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/lib/python3.8/site-
packages/sqlalchemy/engine/base.py", line 3280, in _wrap_pool_connect
return fn()
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/lib/python3.8/site-
packages/sqlalchemy/pool/base.py", line 310, in connect
return _ConnectionFairy._checkout(self)
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/lib/python3.8/site-
packages/sqlalchemy/pool/base.py", line 868, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/lib/python3.8/site-
packages/sqlalchemy/pool/base.py", line 476, in checkout
rec = pool._do_get()
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/lib/python3.8/site-
packages/sqlalchemy/pool/impl.py", line 146, in _do_get
self._dec_overflow()
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/lib/python3.8/site-
packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
compat.raise_(
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/lib/python3.8/site-
packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/lib/python3.8/site-
packages/sqlalchemy/pool/impl.py", line 143, in _do_get
return self._create_connection()
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/lib/python3.8/site-
packages/sqlalchemy/pool/base.py", line 256, in _create_connection
return _ConnectionRecord(self)
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/lib/python3.8/site-
packages/sqlalchemy/pool/base.py", line 371, in __init__
self.__connect()
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/lib/python3.8/site-
packages/sqlalchemy/pool/base.py", line 666, in __connect
pool.logger.debug("Error on connect(): %s", e)
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/lib/python3.8/site-
packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
compat.raise_(
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/lib/python3.8/site-
packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/lib/python3.8/site-
packages/sqlalchemy/pool/base.py", line 661, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/lib/python3.8/site-
packages/sqlalchemy/engine/create.py", line 590, in connect
return dialect.connect(*cargs, **cparams)
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/lib/python3.8/site-
packages/sqlalchemy/engine/default.py", line 597, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/lib/python3.8/site-
packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) connection to server at
"localhost" (127.0.0.1), port 5432 failed: FATAL: the database system is starting up
(Background on this error at: https://sqlalche.me/e/14/e3q8)
[2022-09-29 15:20:41 +0300] [29652] [INFO] Worker exiting (pid: 29652)
Traceback (most recent call last):
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/lib/python3.8/site-
packages/gunicorn/arbiter.py", line 209, in run
self.sleep()
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/lib/python3.8/site-
packages/gunicorn/arbiter.py", line 357, in sleep
ready = select.select([self.PIPE[0]], [], [], 1.0)
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/lib/python3.8/site-
packages/gunicorn/arbiter.py", line 242, in handle_chld
self.reap_workers()
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/lib/python3.8/site-
packages/gunicorn/arbiter.py", line 525, in reap_workers
raise HaltServer(reason, self.WORKER_BOOT_ERROR)
gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/bin/gunicorn", line 8,
in <module>
sys.exit(run())
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/lib/python3.8/site-
packages/gunicorn/app/wsgiapp.py", line 67, in run
WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/lib/python3.8/site-
packages/gunicorn/app/base.py", line 231, in run
super().run()
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/lib/python3.8/site-
packages/gunicorn/app/base.py", line 72, in run
Arbiter(self).run()
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/lib/python3.8/site-
packages/gunicorn/arbiter.py", line 229, in run
self.halt(reason=inst.reason, exit_status=inst.exit_status)
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/lib/python3.8/site-
packages/gunicorn/arbiter.py", line 342, in halt
self.stop()
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/lib/python3.8/site-
packages/gunicorn/arbiter.py", line 393, in stop
time.sleep(0.1)
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/lib/python3.8/site-
packages/gunicorn/arbiter.py", line 242, in handle_chld
self.reap_workers()
File "/home/dancungerald/Documents/Python/SCHEYE/scheye_venv/lib/python3.8/site-
packages/gunicorn/arbiter.py", line 525, in reap_workers
raise HaltServer(reason, self.WORKER_BOOT_ERROR)
gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
[2022-09-29 15:20:42 +0300] [29673] [INFO] Starting gunicorn 20.1.0
[2022-09-29 15:20:42 +0300] [29673] [INFO] Listening at: http://127.0.0.1:8000
(29673)
[2022-09-29 15:20:42 +0300] [29673] [INFO] Using worker: sync
[2022-09-29 15:20:42 +0300] [29675] [INFO] Booting worker with pid: 29675
[2022-09-29 15:20:42 +0300] [29676] [INFO] Booting worker with pid: 29676
[2022-09-29 15:20:42 +0300] [29677] [INFO] Booting worker with pid: 29677
[2022-09-29 15:20:42 +0300] [29678] [INFO] Booting worker with pid: 29678
[2022-09-29 15:20:42 +0300] [29679] [INFO] Booting worker with pid: 29679
[2022-09-29 15:20:42 +0300] [29680] [INFO] Booting worker with pid: 29680
[2022-09-29 15:20:42 +0300] [29681] [INFO] Booting worker with pid: 29681
[2022-09-29 15:20:43 +0300] [29682] [INFO] Booting worker with pid: 29682
[2022-09-29 15:20:43 +0300] [29683] [INFO] Booting worker with pid: 29683
[2022-09-29 15:46:12 +0300] [29682] [INFO] Worker exiting (pid: 29682)
[2022-09-29 15:46:12 +0300] [29681] [INFO] Worker exiting (pid: 29681)
[2022-09-29 15:46:12 +0300] [29683] [INFO] Worker exiting (pid: 29683)
[2022-09-29 15:46:12 +0300] [29679] [INFO] Worker exiting (pid: 29679)
[2022-09-29 15:46:12 +0300] [29678] [INFO] Worker exiting (pid: 29678)
[2022-09-29 15:46:12 +0300] [29677] [INFO] Worker exiting (pid: 29677)
[2022-09-29 15:46:12 +0300] [29680] [INFO] Worker exiting (pid: 29680)
[2022-09-29 15:46:12 +0300] [29675] [INFO] Worker exiting (pid: 29675)
[2022-09-29 15:46:12 +0300] [29676] [INFO] Worker exiting (pid: 29676)
[2022-09-29 15:46:12 +0300] [29673] [INFO] Handling signal: term
[2022-09-29 15:46:12 +0300] [29673] [WARNING] Worker with pid 29675 was terminated
due to signal 15
[2022-09-29 15:46:12 +0300] [29673] [WARNING] Worker with pid 29678 was terminated
due to signal 15
[2022-09-29 15:46:12 +0300] [29673] [WARNING] Worker with pid 29681 was terminated
due to signal 15
[2022-09-29 15:46:12 +0300] [29673] [WARNING] Worker with pid 29682 was terminated
due to signal 15
[2022-09-29 15:46:12 +0300] [29673] [WARNING] Worker with pid 29677 was terminated
due to signal 15
[2022-09-29 15:46:12 +0300] [29673] [WARNING] Worker with pid 29679 was terminated
due to signal 15
[2022-09-29 15:46:12 +0300] [29673] [WARNING] Worker with pid 29680 was terminated
due to signal 15
[2022-09-29 15:46:12 +0300] [29673] [WARNING] Worker with pid 29676 was terminated
due to signal 15
[2022-09-29 15:46:12 +0300] [29673] [WARNING] Worker with pid 29683 was terminated
due to signal 15
[2022-09-29 15:46:12 +0300] [29673] [INFO] Shutting down: Master
[2022-09-29 15:46:14 +0300] [30879] [INFO] Starting gunicorn 20.1.0
[2022-09-29 15:46:14 +0300] [30879] [INFO] Listening at: http://127.0.0.1:8000
(30879)
[2022-09-29 15:46:14 +0300] [30879] [INFO] Using worker: sync
[2022-09-29 15:46:14 +0300] [30881] [INFO] Booting worker with pid: 30881
[2022-09-29 15:46:14 +0300] [30882] [INFO] Booting worker with pid: 30882
[2022-09-29 15:46:14 +0300] [30883] [INFO] Booting worker with pid: 30883
[2022-09-29 15:46:14 +0300] [30884] [INFO] Booting worker with pid: 30884
[2022-09-29 15:46:14 +0300] [30885] [INFO] Booting worker with pid: 30885
[2022-09-29 15:46:14 +0300] [30886] [INFO] Booting worker with pid: 30886
[2022-09-29 15:46:14 +0300] [30887] [INFO] Booting worker with pid: 30887
[2022-09-29 15:46:14 +0300] [30888] [INFO] Booting worker with pid: 30888
[2022-09-29 15:46:14 +0300] [30889] [INFO] Booting worker with pid: 30889
It proposes that postgres is not running but I started the server and when I run netstat -pln |grep 5432 I see that postgres is running on port 5432 so I doubt it's anything to do with the database connection.
If it helps, when it worked I was operating on my home network/wifi but it failed when I ran it today on an institutional network/wifi.
I have no idea what transpired for the app to behave this way. Any help would be highly appreciated.
I enabled postgres to start up on boot with supervisord as per this website. Here's what my supervisor configuration file looked like:
[program:appName]
directory=/Project/File/Directory/appName
command=/Project/File/Location/appName/app_venv/bin/gunicorn -w 9 run:app
user=username
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
stderr_logfile=/var/log/appName/app.err.log
stdout_logfile=/var/log/appName/app.out.log
[program:postgresql]
user=postgres
command=/usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data start
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
redirect_stderr=true
I'm trying to use Perl and Net::FTPSSL on an ubuntu server to upload a local file to an externally hosted webserver. The login works fine, using TLS, but it fails when trying to ls the files on the remote server. The error is "Unexpected EOF on Command Channel [0] (0, 1) () at line…".
I get the same error if I use put or get to try to send/fetch a file. But if I try interactively using the normal unix FTP client, it works fine.
Any idea how I might fix it?
Update 21-May-2020 - I have updated the code and the output file providing more debug logging below
The code is very simple. Like this:
#!/usr/bin/perl
use strict;
use IO::Socket::SSL;
use Net::FTPSSL;
$IO::Socket::SSL::DEBUG = 3;
$Net::SSLeay::trace = 4;
my $file=$ARGV[0];
my $dir=$ARGV[1];
#----------------------------------------------------------------------
#1st try
if (1) {
#print "$server $username $password\n";
my $server="hubble.krystal.co.uk";
my $fields = `cat nickdata`;
chomp($fields);
my ($username, $password) = split /\t/, $fields;
my $rc;
open (STDERR, ">FTPLog.txt");
my $ftp = Net::FTPSSL->new($server,
Encryption => EXP_CRYPT,
Debug => 1,
OverrideHELP => 0,
Croak => 1);
$ftp->trapWarn();
$ftp->login($username, $password) || die "Cannot login\n";
$ftp->pwd;
$ftp->feat();
$ftp->noop();
$ftp->supported("MLSD");
$ftp->supported("NLST");
$ftp->quot("OPTS", "UTF8 ON");
$ftp->quot("TYPE", "I");
$ftp->quot("SYST");
#these fail
#$ftp->nlst();
$ftp->list();
exit;
}
The logfile now including SSL debug logging is as follows:
Net-FTPSSL Version: 0.42
IO-Socket-SSL Version: 2.068
Net-SSLeay Version: 1.84
IO-Socket-INET Version: 1.41
IO-Socket-INET6 might not be installed.
IO-Socket-IP Version: 0.39
IO Version: 1.42
Socket Version: 2.029
IPv6 support available via IO::Socket::IP (0.39) Key: (Family)
Perl: 5.026001 [5.26.1], OS: linux
***** IPv6 not yet supported in Net::FTPSSL! *****
Server (port): hubble.krystal.co.uk (21)
Keys: (OverrideHELP), (Encryption), (Debug), (Croak)
Values: (0), (E), (1), (1)
SKT <<< 220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
SKT <<< 220-You are user number 1 of 50 allowed.
SKT <<< 220-Local time is now 19:54. Server port: 21.
SKT <<< 220-This is a private system - No anonymous login
SKT <<< 220-IPv6 connections are also welcome on this server.
SKT <<< 220 You will be disconnected after 15 minutes of inactivity.
SKT >>> AUTH TLS
SKT <<< 234 AUTH TLS OK.
DEBUG: .../IO/Socket/SSL.pm:3010: new ctx 94714584143456
DEBUG: .../IO/Socket/SSL.pm:1620: start handshake
DEBUG: .../IO/Socket/SSL.pm:1177: global error: Undefined SSL object
DEBUG: .../IO/Socket/SSL.pm:787: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:829: using SNI with hostname hubble.krystal.co.uk
DEBUG: .../IO/Socket/SSL.pm:880: set socket to non-blocking to enforce timeout=120
DEBUG: .../IO/Socket/SSL.pm:894: call Net::SSLeay::connect
DEBUG: .../IO/Socket/SSL.pm:897: done Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:907: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:917: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:937: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:894: call Net::SSLeay::connect
DEBUG: .../IO/Socket/SSL.pm:897: done Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:907: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:917: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:937: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:894: call Net::SSLeay::connect
DEBUG: .../IO/Socket/SSL.pm:897: done Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:907: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:917: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:937: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:894: call Net::SSLeay::connect
DEBUG: .../IO/Socket/SSL.pm:897: done Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:907: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:917: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:937: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:894: call Net::SSLeay::connect
DEBUG: .../IO/Socket/SSL.pm:897: done Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:907: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:917: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:937: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:894: call Net::SSLeay::connect
DEBUG: .../IO/Socket/SSL.pm:897: done Net::SSLeay::connect -> 1
DEBUG: .../IO/Socket/SSL.pm:952: ssl handshake done
Object HASH Details ... (SSL:arguments - E)
SSL_hostname ==> hubble.krystal.co.uk
SSL_verify_mode ==> 0
SSL_version ==> TLSv12
Timeout ==> 120
Object Net::FTPSSL Details ... (hubble.krystal.co.uk:21 - E)
_FTPSSL_arguments ==> HASH(0x562475877a30)
-- Croak ===> 1
-- Crypt ===> E
-- FixGetTs ===> 0
-- FixPutTs ===> 0
-- Host ===> hubble.krystal.co.uk
-- OverrideHELP ===> 0
-- Pret ===> 0
-- Timeout ===> 120
-- buf_size ===> 10240
-- data_prot ===> P
-- dcsc_mode ===> 1
-- debug ===> 1
-- debug_extra ===> 0
-- debug_no_help ===> 0
-- help_cmds_found ===> HASH(0x5624758b7740)
-- help_cmds_msg ===> 214 HELP Command Overridden by request.
-- help_cmds_no_syntax_available ===> 1
-- last_ftp_msg ===> 234 AUTH TLS OK.
-- myContext ===> HASH(0x5624758b7b18)
-- SSL_reuse_ctx ----> IO::Socket::SSL::SSL_Context=HASH(0x562475b19828)
-- context ++++> 94714584143456
-- ocsp_cache ++++> (undef)
-- ocsp_mode ++++> 0
-- verify_mode ++++> 0
-- mySocketOpts ===> HASH(0x562475384650)
-- PeerAddr ----> hubble.krystal.co.uk
-- PeerPort ----> 21
-- Proto ----> tcp
-- Timeout ----> 120
-- start_SSL_opts ===> HASH(0x5624758775e0)
-- SSL_hostname ----> hubble.krystal.co.uk
-- SSL_verify_mode ----> 0
-- SSL_version ----> TLSv12
-- Timeout ----> 120
-- trace ===> 0
-- type ===> A
_SSL_arguments ==> HASH(0x562475b198b8)
-- PeerAddr ===> 77.72.1.20
-- PeerPort ===> 21
-- Proto ===> tcp
-- SSL_check_crl ===> 0
-- SSL_cipher_list ===> ECDHE-ECDSA-AES128-GCM-SHA256 ECDHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES128-GCM-SHA256 ECDHE-ECDSA-CHACHA20-POLY1305 ECDHE-RSA-CHACHA20-POLY1305 ECDHE-ECDSA-AES256-SHA ECDHE-RSA-AES256-SHA DHE-RSA-AES256-SHA ECDHE-ECDSA-AES128-SHA ECDHE-RSA-AES128-SHA DHE-RSA-AES128-SHA AES128-GCM-SHA256 AES256-SHA AES128-SHA DES-CBC3-SHA ECDHE-RSA-AES256-GCM-SHA384 ECDHE-ECDSA-AES256-GCM-SHA384 AES256-SHA256 DHE-DSS-AES256-SHA256 DHE-DSS-AES256-SHA DHE-DSS-AES128-SHA EDH-DSS-DES-CBC3-SHA !EXP !MEDIUM !LOW !eNULL !aNULL !RC4 !DES !MD5 !PSK !SRP
-- SSL_hostname ===> hubble.krystal.co.uk
-- SSL_server ===> 0
-- SSL_use_cert ===> 0
-- SSL_verify_mode ===> 0
-- SSL_version ===> TLSv12
_SSL_ctx ==> IO::Socket::SSL::SSL_Context=HASH(0x562475b19828)
-- context ===> 94714584143456
-- ocsp_cache ===> (undef)
-- ocsp_mode ===> 0
-- verify_mode ===> 0
_SSL_fileno ==> 3
_SSL_ioclass_upgraded ==> IO::Socket::INET
_SSL_last_err ==> ARRAY(0x562475afdca0)
[SSL wants a read first, 4]
_SSL_object ==> 94714593482576
_SSL_opened ==> 1
io_socket_domain ==> 2
io_socket_proto ==> 6
io_socket_timeout ==> 120
io_socket_type ==> 1
************************************************************
>>> USER +++++++
<<< 331 User <++++++> OK. Password required
>>> PASS *******
<<< 230 OK. Current restricted directory is /
<<+ 503 Unsupported command HELP.
>>> PWD
<<< 257 "/" is your current location
>>> FEAT
<<< 211-Extensions supported:
<<< EPRT
<<< IDLE
<<< MDTM
<<< SIZE
<<< MFMT
<<< REST STREAM
<<< MLST type*;size*;sizd*;modify*;UNIX.mode*;UNIX.uid*;UNIX.gid*;unique*;
<<< MLSD
<<< AUTH TLS
<<< PBSZ
<<< PROT
<<< UTF8
<<< TVFS
<<< ESTA
<<< PASV
<<< EPSV
<<< SPSV
<<< ESTP
<<< 211 End.
<<+ 111 Auto-adding OPTS Command!
>>> NOOP
<<< 200 Zzz...
<<+ 502 Unknown command MLSD.
<<+ 502 Unknown command NLST.
>>> OPTS UTF8 ON
<<< 200 OK, UTF-8 enabled
>>> TYPE I
<<< 200 TYPE is now 8-bit binary
>>> SYST
<<< 215 UNIX Type: L8
>>> PBSZ 0
<<< 200 PBSZ=0
>>> PROT P
<<< 200 Data protection level set to "private"
>>> PASV
<<< 227 Entering Passive Mode (77,72,1,20,213,32)
--- Host (77.72.1.20) Port (54560)
>>> LIST
<<< 150 Accepted data connection
DEBUG: .../IO/Socket/SSL.pm:1620: start handshake
DEBUG: .../IO/Socket/SSL.pm:1177: global error: Undefined SSL object
DEBUG: .../IO/Socket/SSL.pm:787: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:829: using SNI with hostname 77.72.1.20
DEBUG: .../IO/Socket/SSL.pm:894: call Net::SSLeay::connect
DEBUG: .../IO/Socket/SSL.pm:897: done Net::SSLeay::connect -> 1
DEBUG: .../IO/Socket/SSL.pm:952: ssl handshake done
Unexpected EOF on Command Channel [0] (0, 1) ()
at ./nickFTPTest.pl line 45.
DEBUG: .../IO/Socket/SSL.pm:3059: free ctx 94714584143456 open=94714584143456
DEBUG: .../IO/Socket/SSL.pm:3070: OK free ctx 94714584143456
Update 2, 21-May-2020: I have added a copy of the FileZilla trace attaching to the same FTP server. FileZilla succeeds, but when I try to replicate the same commands using FTPSSL, it fails.
Trace: CControlSocket::SendNextCommand()
Trace: CFtpLogonOpData::Send() in state 0
Status: Resolving address of <server>
Status: Connecting to 77.72.1.20:21...
Status: Connection established, waiting for welcome message...
Trace: CFtpControlSocket::OnReceive()
Response: 220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
Response: 220-You are user number 2 of 50 allowed.
Response: 220-Local time is now 20:09. Server port: 21.
Response: 220-This is a private system - No anonymous login
Response: 220-IPv6 connections are also welcome on this server.
Response: 220 You will be disconnected after 15 minutes of inactivity.
Trace: CFtpLogonOpData::ParseResponse() in state 1
Trace: CControlSocket::SendNextCommand()
Trace: CFtpLogonOpData::Send() in state 2
Command: AUTH TLS
Trace: CFtpControlSocket::OnReceive()
Response: 234 AUTH TLS OK.
Trace: CFtpLogonOpData::ParseResponse() in state 2
Status: Initializing TLS...
Trace: tls_layer_impl::client_handshake()
Trace: tls_layer_impl::continue_handshake()
Trace: TLS handshake: About to send CLIENT HELLO
Trace: TLS handshake: Sent CLIENT HELLO
Trace: tls_layer_impl::on_send()
Trace: tls_layer_impl::continue_handshake()
Trace: tls_layer_impl::on_read()
Trace: tls_layer_impl::continue_handshake()
Trace: tls_layer_impl::on_read()
Trace: tls_layer_impl::continue_handshake()
Trace: TLS handshake: Received SERVER HELLO
Trace: TLS handshake: Processed SERVER HELLO
Trace: tls_layer_impl::on_read()
Trace: tls_layer_impl::continue_handshake()
Trace: tls_layer_impl::on_read()
Trace: tls_layer_impl::continue_handshake()
Trace: tls_layer_impl::on_read()
Trace: tls_layer_impl::continue_handshake()
Trace: TLS handshake: Received CERTIFICATE
Trace: TLS handshake: Processed CERTIFICATE
Trace: TLS handshake: Received SERVER KEY EXCHANGE
Trace: TLS handshake: Processed SERVER KEY EXCHANGE
Trace: TLS handshake: Received SERVER HELLO DONE
Trace: TLS handshake: Processed SERVER HELLO DONE
Trace: TLS handshake: About to send CLIENT KEY EXCHANGE
Trace: TLS handshake: Sent CLIENT KEY EXCHANGE
Trace: TLS handshake: About to send FINISHED
Trace: TLS handshake: Sent FINISHED
Trace: tls_layer_impl::on_read()
Trace: tls_layer_impl::continue_handshake()
Trace: TLS handshake: Received NEW SESSION TICKET
Trace: TLS handshake: Processed NEW SESSION TICKET
Trace: TLS handshake: Received FINISHED
Trace: TLS handshake: Processed FINISHED
Trace: TLS Handshake successful
Trace: Protocol: TLS1.2, Key exchange: ECDHE-SECP256R1-RSA-SHA512, Cipher: AES-256-GCM, MAC: AEAD
Trace: tls_layer_impl::verify_certificate()
Trace: Hostname does not match certificate SANs
Status: Verifying certificate...
Trace: CFtpControlSocket::SetAsyncRequestReply
Status: TLS connection established.
Trace: CControlSocket::SendNextCommand()
Trace: CFtpLogonOpData::Send() in state 6
Command: USER <user>
Trace: CFtpControlSocket::OnReceive()
Trace: tls_layer_impl::on_read()
Trace: CFtpControlSocket::OnReceive()
Response: 331 User <user> OK. Password required
Trace: CFtpLogonOpData::ParseResponse() in state 6
Trace: CControlSocket::SendNextCommand()
Trace: CFtpLogonOpData::Send() in state 6
Command: PASS ********************
Trace: tls_layer_impl::on_read()
Trace: CFtpControlSocket::OnReceive()
Response: 230 OK. Current restricted directory is /
Trace: CFtpLogonOpData::ParseResponse() in state 6
Trace: CControlSocket::SendNextCommand()
Trace: CFtpLogonOpData::Send() in state 7
Command: SYST
Trace: tls_layer_impl::on_read()
Trace: CFtpControlSocket::OnReceive()
Response: 215 UNIX Type: L8
Trace: CFtpLogonOpData::ParseResponse() in state 7
Trace: CControlSocket::SendNextCommand()
Trace: CFtpLogonOpData::Send() in state 8
Command: FEAT
Trace: tls_layer_impl::on_read()
Trace: CFtpControlSocket::OnReceive()
Response: 211-Extensions supported:
Response: EPRT
Response: IDLE
Response: MDTM
Response: SIZE
Response: MFMT
Response: REST STREAM
Response: MLST type*;size*;sizd*;modify*;UNIX.mode*;UNIX.uid*;UNIX.gid*;unique*;
Response: MLSD
Response: AUTH TLS
Response: PBSZ
Response: PROT
Response: UTF8
Response: TVFS
Response: ESTA
Response: PASV
Response: EPSV
Response: SPSV
Response: ESTP
Response: 211 End.
Trace: CFtpLogonOpData::ParseResponse() in state 8
Trace: CControlSocket::SendNextCommand()
Trace: CFtpLogonOpData::Send() in state 10
Command: OPTS UTF8 ON
Trace: tls_layer_impl::on_read()
Trace: CFtpControlSocket::OnReceive()
Response: 200 OK, UTF-8 enabled
Trace: CFtpLogonOpData::ParseResponse() in state 10
Trace: CControlSocket::SendNextCommand()
Trace: CFtpLogonOpData::Send() in state 11
Command: PBSZ 0
Trace: tls_layer_impl::on_read()
Trace: CFtpControlSocket::OnReceive()
Response: 200 PBSZ=0
Trace: CFtpLogonOpData::ParseResponse() in state 11
Trace: CControlSocket::SendNextCommand()
Trace: CFtpLogonOpData::Send() in state 12
Command: PROT P
Trace: tls_layer_impl::on_read()
Trace: CFtpControlSocket::OnReceive()
Response: 200 Data protection level set to "private"
Trace: CFtpLogonOpData::ParseResponse() in state 12
Status: Logged in
Trace: Measured latency of 17 ms
Trace: CFtpControlSocket::ResetOperation(0)
Trace: CControlSocket::ResetOperation(0)
Trace: CFtpLogonOpData::Reset(0) in state 15
Trace: CFileZillaEnginePrivate::ResetOperation(0)
Trace: CControlSocket::SendNextCommand()
Trace: CFtpListOpData::Send() in state 0
Status: Retrieving directory listing...
Trace: CFtpChangeDirOpData::Send() in state 0
Trace: CFtpChangeDirOpData::Send() in state 1
Command: PWD
Trace: tls_layer_impl::on_read()
Trace: CFtpControlSocket::OnReceive()
Response: 257 "/" is your current location
Trace: CFtpChangeDirOpData::ParseResponse() in state 1
Trace: CFtpControlSocket::ResetOperation(0)
Trace: CControlSocket::ResetOperation(0)
Trace: CFtpChangeDirOpData::Reset(0) in state 1
Trace: CFtpListOpData::SubcommandResult(0) in state 1
Trace: CControlSocket::SendNextCommand()
Trace: CFtpListOpData::Send() in state 2
Trace: CFtpRawTransferOpData::Send() in state 0
Trace: CFtpRawTransferOpData::Send() in state 1
Command: TYPE I
Trace: tls_layer_impl::on_read()
Trace: CFtpControlSocket::OnReceive()
Response: 200 TYPE is now 8-bit binary
Trace: CFtpRawTransferOpData::ParseResponse() in state 1
Trace: CControlSocket::SendNextCommand()
Trace: CFtpRawTransferOpData::Send() in state 2
Command: PASV
Trace: tls_layer_impl::on_read()
Trace: CFtpControlSocket::OnReceive()
Response: 227 Entering Passive Mode (77,72,1,20,207,20)
Trace: CFtpRawTransferOpData::ParseResponse() in state 2
Trace: CControlSocket::SendNextCommand()
Trace: CFtpRawTransferOpData::Send() in state 4
Trace: Binding data connection source IP to control connection source IP 192.168.1.34
Trace: tls_layer_impl::client_handshake()
Trace: Trying to resume existing TLS session.
Command: MLSD
Trace: tls_layer_impl::on_send()
Trace: tls_layer_impl::continue_handshake()
Trace: TLS handshake: About to send CLIENT HELLO
Trace: TLS handshake: Sent CLIENT HELLO
Trace: tls_layer_impl::on_read()
Trace: CFtpControlSocket::OnReceive()
Response: 150 Accepted data connection
Trace: CFtpRawTransferOpData::ParseResponse() in state 4
Trace: CControlSocket::SendNextCommand()
Trace: CFtpRawTransferOpData::Send() in state 5
Trace: tls_layer_impl::on_read()
Trace: tls_layer_impl::continue_handshake()
Trace: TLS handshake: Received SERVER HELLO
Trace: TLS handshake: Processed SERVER HELLO
Trace: TLS handshake: Received FINISHED
Trace: TLS handshake: Processed FINISHED
Trace: TLS handshake: About to send FINISHED
Trace: TLS handshake: Sent FINISHED
Trace: TLS Handshake successful
Trace: TLS Session resumed
Trace: Protocol: TLS1.2, Key exchange: ECDHE-SECP256R1, Cipher: AES-256-GCM, MAC: AEAD
Trace: tls_layer_impl::verify_certificate()
Trace: CTransferSocket::OnConnect
Trace: CTransferSocket::OnReceive(), m_transferMode=0
Trace: tls_layer_impl::on_read()
Trace: CTransferSocket::OnReceive(), m_transferMode=0
Trace: tls_layer_impl::on_read()
Trace: CTransferSocket::OnReceive(), m_transferMode=0
Trace: tls_layer_impl::on_read()
Trace: CTransferSocket::OnReceive(), m_transferMode=0
Trace: CTransferSocket::TransferEnd(1)
Trace: tls_layer_impl::shutdown()
Trace: tls_layer_impl::continue_shutdown()
Trace: CFtpControlSocket::TransferEnd()
Trace: tls_layer_impl::on_read()
Trace: CFtpControlSocket::OnReceive()
Response: 226-Options: -a -l
Trace: tls_layer_impl::on_read()
Trace: CFtpControlSocket::OnReceive()
Response: 226 46 matches total
Trace: CFtpRawTransferOpData::ParseResponse() in state 7
Trace: CFtpControlSocket::ResetOperation(0)
Trace: CControlSocket::ResetOperation(0)
Trace: CFtpRawTransferOpData::Reset(0) in state 7
Trace: CFtpListOpData::SubcommandResult(0) in state 3
Trace: CFtpControlSocket::ResetOperation(0)
Trace: CControlSocket::ResetOperation(0)
Trace: CFtpListOpData::Reset(0) in state 3
Status: Directory listing of "/" successful
Trace: CFileZillaEnginePrivate::ResetOperation(0)
Thanks to SSL debugging one can now better see what's going on. This is from the initial FTP control connection:
DEBUG: .../IO/Socket/SSL.pm:829: using SNI with hostname hubble.krystal.co.uk
But this is from the data connection:
DEBUG: .../IO/Socket/SSL.pm:829: using SNI with hostname 77.72.1.20
This looks like two bugs for me in Net::FTPSSL which also seem to be in the current version 0.42: it should never use an IP address for SNI (which Net::FTPSSL explicitly does, IO::Socket::SSL by its own would not) and it should use the same SNI in control and data connection (i.e. the hostname in both).
Given that I have no access to the site (no anonymous login) I don't know if this is really the cause of the problem. But Net::FTP does not have this problem so it would be worth a more closer look (i.e. with SSL debugging enabled too if it does not work).
My approach so far, which is more avoiding the problem rather than solving it, has been to install lftp and pipe commands into lftp. This achieves my immediate aim of doing an upload using FTP over TLS.
Issue with Ambassador deployment starting in Kubernetes:
Kubernetes - v1.13
Ambassador - image: quay.io/datawire/ambassador:0.50.3
Container Runtime - Docker
Cluster networking done used Flannel
The entire set up is done on two Oracle VMs on a single Windows 10 machine.
the network applied is Host-Only with master - 192.168.99.110 and a node - 192.168.99.101.
I am deploying ambassador using kubectl apply -f https://getambassador.io/yaml/ambassador/ambassador-rbac.yaml. After 30 secs when the kubernetes pods starting the Kube watch it goes into a 'CrashLoopBackOff' state. I inspected the logs of the pod and it says below - which at the last sentence states that 10.96.0.1(API Server cluster IP) is unreachable :
" []# kubectl logs ambassador-76f644ddfb-vnj4d
2019-03-06 17:12:13 kubewatch [23 TMainThread] 0.50.3 INFO: kubewatch starting: mode 'cluster-id' ambassador_config_dir '/no/such/path' envoy_config_file '/dev/null' debug 'False' delay '1.0' pid 'None'
2019-03-06 17:12:13 kubewatch [23 TMainThread] 0.50.3 INFO: namespace default, watching all namespaces
2019-03-06 17:12:14,131 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f739506d748>: Failed to establish a new connection: [Errno 113] Host is unreachable',)': /api/v1/namespaces/default
2019-03-06 17:12:14 kubewatch [23 TMainThread] 0.50.3 WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f739506d748>: Failed to establish a new connection: [Errno 113] Host is unreachable',)': /api/v1/namespaces/default
2019-03-06 17:12:15,136 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f739506d7f0>: Failed to establish a new connection: [Errno 113] Host is unreachable',)': /api/v1/namespaces/default
2019-03-06 17:12:15 kubewatch [23 TMainThread] 0.50.3 WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f739506d7f0>: Failed to establish a new connection: [Errno 113] Host is unreachable',)': /api/v1/namespaces/default
2019-03-06 17:12:16,140 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f739506d898>: Failed to establish a new connection: [Errno 113] Host is unreachable',)': /api/v1/namespaces/default
2019-03-06 17:12:16 kubewatch [23 TMainThread] 0.50.3 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f739506d898>: Failed to establish a new connection: [Errno 113] Host is unreachable',)': /api/v1/namespaces/default
2019-03-06 17:12:17 kubewatch [23 TMainThread] 0.50.3 WARNING: kubewatch failed!
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 159, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/usr/lib/python3.6/site-packages/urllib3/util/connection.py", line 80, in create_connection
raise err
File "/usr/lib/python3.6/site-packages/urllib3/util/connection.py", line 70, in create_connection
sock.connect(sa)
OSError: [Errno 113] Host is unreachable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
self._validate_conn(conn)
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
conn.connect()
File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 301, in connect
conn = self._new_conn()
File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 168, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f739506d9b0>: Failed to establish a new connection: [Errno 113] Host is unreachable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/ambassador/kubewatch.py", line 527, in <module>
main()
File "/usr/lib/python3.6/site-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/usr/lib/python3.6/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python3.6/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/ambassador/kubewatch.py", line 518, in main
watcher.run(id_only=True)
File "/ambassador/kubewatch.py", line 342, in run
self.get_cluster_id(v1)
File "/ambassador/kubewatch.py", line 407, in get_cluster_id
ret = v1.read_namespace(wanted)
File "/usr/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 17572, in read_namespace
(data) = self.read_namespace_with_http_info(name, **kwargs)
File "/usr/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 17657, in read_namespace_with_http_info
collection_formats=collection_formats)
File "/usr/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 321, in call_api
_return_http_data_only, collection_formats, _preload_content, _request_timeout)
File "/usr/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 155, in __call_api
_request_timeout=_request_timeout)
File "/usr/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 342, in request
headers=headers)
File "/usr/lib/python3.6/site-packages/kubernetes/client/rest.py", line 231, in GET
query_params=query_params)
File "/usr/lib/python3.6/site-packages/kubernetes/client/rest.py", line 205, in request
headers=headers)
File "/usr/lib/python3.6/site-packages/urllib3/request.py", line 68, in request
**urlopen_kw)
File "/usr/lib/python3.6/site-packages/urllib3/request.py", line 89, in request_encode_url
return self.urlopen(method, url, **extra_kw)
File "/usr/lib/python3.6/site-packages/urllib3/poolmanager.py", line 323, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 667, in urlopen
**response_kw)
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 667, in urlopen
**response_kw)
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 667, in urlopen
**response_kw)
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='10.96.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/default (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f739506d9b0>: Failed to establish a new connection: [Errno 113] Host is unreachable',))
AMBASSADOR: kubewatch cluster-id exited with status 1
AMBASSADOR: shutting down (1)"
An update. I have disabled firewalld service in both master and node and the issue is solved now. I think the root cause was due to PODs in the node were not able to access the kube-dns on the master due to firewall restrictions.
I am currently attempting to use mongo-connector to automatically feed db updates to Solr. It's working fine through the use of the following command -
mongo-connector -m localhost:27017 -t http://localhost:8983/solr -d mongo_connector/doc_managers/solr_doc_manager.py
However, it is indexing every collection in my mongodb. I have tried the use of the option -n through the following -
mongo-connector -m localhost:27017 -t http://localhost:8983/solr -n feed_scraper_development.articles -d mongo_connector/doc_managers/solr_doc_manager.py
This fails with the following error -
2014-07-24 22:23:23,053 - INFO - Beginning Mongo Connector
2014-07-24 22:23:23,104 - INFO - Starting new HTTP connection (1): localhost
2014-07-24 22:23:23,110 - INFO - Finished 'http://localhost:8983/solr/admin/luke?show=schema&wt=json' (get) with body '' in 0.018 seconds.
2014-07-24 22:23:23,115 - INFO - OplogThread: Initializing oplog thread
2014-07-24 22:23:23,116 - INFO - MongoConnector: Starting connection thread MongoClient('localhost', 27017)
2014-07-24 22:23:23,126 - INFO - Finished 'http://localhost:8983/solr/update/?commit=true' (post) with body 'u'<commit ' in 0.006 seconds.
2014-07-24 22:23:23,129 - INFO - Finished 'http://localhost:8983/solr/select/?q=%2A%3A%2A&sort=_ts+desc&rows=1&wt=json' (get) with body '' in 0.003 seconds.
2014-07-24 22:23:23,337 - INFO - Finished 'http://localhost:8983/solr/select/?q=_ts%3A+%5B6038164010275176560+TO+6038164010275176560%5D&rows=100000000&wt=json' (get) with body '' in 0.207 seconds.
Exception in thread Thread-2:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 808, in __bootstrap_inner
self.run()
File "build/bdist.macosx-10.9-intel/egg/mongo_connector/oplog_manager.py", line 141, in run
cursor = self.init_cursor()
File "build/bdist.macosx-10.9-intel/egg/mongo_connector/oplog_manager.py", line 582, in init_cursor
cursor = self.get_oplog_cursor(timestamp)
File "build/bdist.macosx-10.9-intel/egg/mongo_connector/oplog_manager.py", line 361, in get_oplog_cursor
timestamp = self.rollback()
File "build/bdist.macosx-10.9-intel/egg/mongo_connector/oplog_manager.py", line 664, in rollback
if doc['ns'] in rollback_set:
KeyError: 'ns'
Any help or clues would be greatly appreciated!
Extra information: Solr 4.9.0 | MongoDB 2.6.3 | mongo-connector 1.2.1
It works as advertised after deleting all the indexes in the data folder, restarting solr and re-running the command with the -n option.
mongo-connector --unique-key=id --auto-commit-interval=0 -m localhost:27017 -t http://localhost:8983/solr -d /Library/Python/2.7/site-packages/mongo_connector/doc_managers/solr_doc_manager.py --admin-username admin --password bypass
I'm using the following to connect between MongoDB and Apache Solr but I'm getting the following error at the end:
2014-05-17 12:38:20,607 - INFO - Beginning Mongo Connector
2014-05-17 12:38:22,200 - INFO - Starting new HTTP connection (1): localhost
2014-05-17 12:38:22,439 - INFO - Finished 'http://localhost:8983/solr/admin/luke?show=schema&wt=json' (get) with body '' in 0.404 seconds.
2014-05-17 12:38:22,527 - INFO - OplogThread: Initializing oplog thread
2014-05-17 12:38:22,580 - INFO - MongoConnector: Starting connection thread MongoClient('localhost', 27017)
Exception in thread Thread-2:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 808, in __bootstrap_inner
self.run()
File "/Library/Python/2.7/site-packages/mongo_connector/oplog_manager.py", line 181, in run
dm.remove(entry)
File "/Library/Python/2.7/site-packages/mongo_connector/doc_managers/solr_doc_manager.py", line 192, in remove
self.solr.delete(id=str(doc[self.unique_key]),
KeyError: 'id'
Please help me.
Did you explicitly create a unique field called "id"? That is not the default field added by MongoDB, the default unique field is "_id".