From MongoDB to Elastic Search(5.6.5), I sync the database with Mongo-Connector using Elastic2DocManager:
mongo-connector -m localhost:27017 -t localhost:9200 -d elastic2_doc_manager
After seeing some update on docs.deleted of mongodb_meta on Elastic Search:
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open mongodb_meta 3wd6OjTT6tD3f6ZGezZw 5 1 1337173 8372 192.9mb 192.9mb
mongo-connector stops working with below error:
2018-07-11 07:16:41,977 [WARNING] elasticsearch:97 - POST http://localhost:9200/_bulk [status:N/A request:10.003s]
Traceback (most recent call last):
File "c:\programdata\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 387, in _make_request
six.raise_from(e, None)
File "<string>", line 2, in raise_from
File "c:\programdata\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 383, in _make_request
httplib_response = conn.getresponse()
File "c:\programdata\anaconda3\lib\http\client.py", line 1331, in getresponse
response.begin()
File "c:\programdata\anaconda3\lib\http\client.py", line 297, in begin
version, status, reason = self._read_status()
File "c:\programdata\anaconda3\lib\http\client.py", line 258, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "c:\programdata\anaconda3\lib\socket.py", line 586, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
...
Exception in thread Thread-1:
Traceback (most recent call last):
File "c:\programdata\anaconda3\lib\threading.py", line 916, in _bootstrap_inner
self.run()
File "c:\programdata\anaconda3\lib\site-packages\mongo_connector\doc_managers\elastic2_doc_manager.py", line 150, in run
self._docman.send_buffered_operations()
File "c:\programdata\anaconda3\lib\site-packages\mongo_connector\doc_managers\elastic2_doc_manager.py", line 482, in send_buffered_operations
action_buffer = self.BulkBuffer.get_buffer()
File "c:\programdata\anaconda3\lib\site-packages\mongo_connector\doc_managers\elastic2_doc_manager.py", line 696, in get_buffer
self.update_sources()
File "c:\programdata\anaconda3\lib\site-packages\mongo_connector\util.py", line 35, in wrapped
return f(*args, **kwargs)
File "c:\programdata\anaconda3\lib\site-packages\mongo_connector\doc_managers\elastic2_doc_manager.py", line 628, in update_sources
if ES_doc['found']:
KeyError: 'found'
What is the reason for this error?
Make sure the versions are compatible.
The required python packages are installed, this is my requirements.txt look like.
astroid==1.6.5
autopep8==1.3.5
certifi==2018.8.24
elasticsearch==6.3.1
elasticsearch-dsl==6.2.1
isort==4.3.4
lazy-object-proxy==1.3.1
mccabe==0.6.1
pycodestyle==2.4.0
pylint==1.9.2
pymongo==3.7.1
rope==0.11.0
six==1.11.0
urllib3==1.23
wrapt==1.10.11
Related
I'm running a pymongo client and repeatedly getting this error. what does this mean and how can i prevent this?
Traceback (most recent call last):
File "/home/secondaryvm/.local/lib/python3.6/site-packages/pymongo/pool.py", line 1278, in _get_socket
sock_info = self.sockets.popleft()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/secondaryvm/.local/lib/python3.6/site-packages/pymongo/pool.py", line 694, in command
exhaust_allowed=exhaust_allowed)
File "/home/secondaryvm/.local/lib/python3.6/site-packages/pymongo/network.py", line 150, in command
reply = receive_message(sock_info, request_id)
File "/home/secondaryvm/.local/lib/python3.6/site-packages/pymongo/network.py", line 195, in receive_message
_receive_data_on_socket(sock_info, 16, deadline))
File "/home/secondaryvm/.local/lib/python3.6/site-packages/pymongo/network.py", line 286, in _receive_data_on_socket
chunk_length = sock_info.sock.recv_into(mv[bytes_read:])
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "mgclient.py", line 18, in <module>
result=db.reviews.insert_one(business)
File "/home/secondaryvm/.local/lib/python3.6/site-packages/pymongo/collection.py", line 701, in insert_one
session=session),
File "/home/secondaryvm/.local/lib/python3.6/site-packages/pymongo/collection.py", line 615, in _insert
bypass_doc_val, session)
File "/home/secondaryvm/.local/lib/python3.6/site-packages/pymongo/collection.py", line 603, in _insert_one
acknowledged, _insert_command, session)
File "/home/secondaryvm/.local/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1498, in _retryable_write
return self._retry_with_session(retryable, func, s, None)
File "/home/secondaryvm/.local/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1384, in _retry_with_session
return self._retry_internal(retryable, func, session, bulk)
File "/home/secondaryvm/.local/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1408, in _retry_internal
with self._get_socket(server, session) as sock_info:
File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/home/secondaryvm/.local/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1247, in _get_socket
self.__all_credentials, checkout=exhaust) as sock_info:
File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/home/secondaryvm/.local/lib/python3.6/site-packages/pymongo/pool.py", line 1231, in get_socket
sock_info = self._get_socket(all_credentials)
File "/home/secondaryvm/.local/lib/python3.6/site-packages/pymongo/pool.py", line 1281, in _get_socket
sock_info = self.connect(all_credentials)
File "/home/secondaryvm/.local/lib/python3.6/site-packages/pymongo/pool.py", line 1193, in connect
sock_info.ismaster(all_credentials)
File "/home/secondaryvm/.local/lib/python3.6/site-packages/pymongo/pool.py", line 546, in ismaster
return self._ismaster(None, None, None, all_credentials)
File "/home/secondaryvm/.local/lib/python3.6/site-packages/pymongo/pool.py", line 580, in _ismaster
exhaust_allowed=awaitable)
File "/home/secondaryvm/.local/lib/python3.6/site-packages/pymongo/pool.py", line 699, in command
self._raise_connection_failure(error)
File "/home/secondaryvm/.local/lib/python3.6/site-packages/pymongo/pool.py", line 891, in _raise_connection_failure
_raise_connection_failure(self.address, error)
File "/home/secondaryvm/.local/lib/python3.6/site-packages/pymongo/pool.py", line 286, in _raise_connection_failure
raise AutoReconnect(msg)
pymongo.errors.AutoReconnect: 192.168.122.50:27017: [Errno 104] Connection reset by peer
I'm testing a setup where the responses from the server are delayed for long. But i want that client shouldn't get any type of timeout error.
what changes to be made to prevent these either to server or client?
I am Failing to install pandas in cmd using pip-install pandas due to the error below that pops up when installing. I am using python version 3.7
Traceback (most recent call last):
File "c:\users\chiya\appdata\local\programs\python\python37-32\lib\site-packages\pip\_vendor\urllib3\response.py", line 360, in _error_catcher
yield
File "c:\users\chiya\appdata\local\programs\python\python37-32\lib\site-packages\pip\_vendor\urllib3\response.py", line 442, in read
data = self._fp.read(amt)
File "c:\users\chiya\appdata\local\programs\python\python37-32\lib\site-packages\pip\_vendor\cachecontrol\filewrapper.py", line 62, in read
data = self.__fp.read(amt)
File "c:\users\chiya\appdata\local\programs\python\python37-32\lib\http\client.py", line 447, in read
n = self.readinto(b)
File "c:\users\chiya\appdata\local\programs\python\python37-32\lib\http\client.py", line 491, in readinto
n = self.fp.readinto(b)
File "c:\users\chiya\appdata\local\programs\python\python37-32\lib\socket.py", line 589, in readinto
return self._sock.recv_into(b)
File "c:\users\chiya\appdata\local\programs\python\python37-32\lib\ssl.py", line 1052, in recv_into
return self.read(nbytes, buffer)
File "c:\users\chiya\appdata\local\programs\python\python37-32\lib\ssl.py", line 911, in read
return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\users\chiya\appdata\local\programs\python\python37-32\lib\site-packages\pip\_internal\cli\base_command.py", line 179, in main
status = self.run(options, args)
File "c:\users\chiya\appdata\local\programs\python\python37-32\lib\site-packages\pip\_internal\commands\install.py", line 315, in run
resolver.resolve(requirement_set)
File "c:\users\chiya\appdata\local\programs\python\python37-32\lib\site-packages\pip\_internal\resolve.py", line 131, in resolve
self._resolve_one(requirement_set, req)
File "c:\users\chiya\appdata\local\programs\python\python37-32\lib\site-packages\pip\_internal\resolve.py", line 294, in _resolve_one
abstract_dist = self._get_abstract_dist_for(req_to_install)
File "c:\users\chiya\appdata\local\programs\python\python37-32\lib\site-packages\pip\_internal\resolve.py", line 242, in _get_abstract_dist_for
self.require_hashes
File "c:\users\chiya\appdata\local\programs\python\python37-32\lib\site-packages\pip\_internal\operations\prepare.py", line 334, in prepare_linked_requirement
progress_bar=self.progress_bar
File "c:\users\chiya\appdata\local\programs\python\python37-32\lib\site-packages\pip\_internal\download.py", line 878, in unpack_url
progress_bar=progress_bar
File "c:\users\chiya\appdata\local\programs\python\python37-32\lib\site-packages\pip\_internal\download.py", line 702, in unpack_http_url
progress_bar)
File "c:\users\chiya\appdata\local\programs\python\python37-32\lib\site-packages\pip\_internal\download.py", line 946, in _download_http_url
_download_url(resp, link, content_file, hashes, progress_bar)
File "c:\users\chiya\appdata\local\programs\python\python37-32\lib\site-packages\pip\_internal\download.py", line 639, in _download_url
hashes.check_against_chunks(downloaded_chunks)
File "c:\users\chiya\appdata\local\programs\python\python37-32\lib\site-packages\pip\_internal\utils\hashes.py", line 62, in check_against_chunks
for chunk in chunks:
File "c:\users\chiya\appdata\local\programs\python\python37-32\lib\site-packages\pip\_internal\download.py", line 607, in written_chunks
for chunk in chunks:
File "c:\users\chiya\appdata\local\programs\python\python37-32\lib\site-packages\pip\_internal\utils\ui.py", line 159, in iter
for x in it:
File "c:\users\chiya\appdata\local\programs\python\python37-32\lib\site-packages\pip\_internal\download.py", line 596, in resp_read
decode_content=False):
File "c:\users\chiya\appdata\local\programs\python\python37-32\lib\site-packages\pip\_vendor\urllib3\response.py", line 494, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "c:\users\chiya\appdata\local\programs\python\python37-32\lib\site-packages\pip\_vendor\urllib3\response.py", line 459, in read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
File "c:\users\chiya\appdata\local\programs\python\python37-32\lib\contextlib.py", line 130, in __exit__
self.gen.throw(type, value, traceback)
File "c:\users\chiya\appdata\local\programs\python\python37-32\lib\site-packages\pip\_vendor\urllib3\response.py", line 365, in _error_catcher
raise ReadTimeoutError(self._pool, None, 'Read timed out.')
pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.
I tried updating pip with the command pip install -U pip setuptools
but it showed that pip is up to date.
I tried again pip install fbchat which downloaded successfully but after trying to install pandas with the same "pip install pandas" the error popped up again.**
I suspect the code below ran out connection capacity, etc. Is there any interface I can send batch requests with? Or sleep a few ms?
def archive_pending_blobs(bucket: Bucket, blobs: typing.List[Blob], pending_prefix: str,
loaded_prefix: str) -> None:
"""Archive pending blobs to loaded prefix."""
try:
for b in blobs:
bucket.copy_blob(b, bucket, b.name.replace(pending_prefix, loaded_prefix))
bucket.delete_blobs(blobs)
except Exception as e:
print('gcs achieving error for path: {} err: {}'.format(pending_prefix, e))
raise e
Traceback (most recent call last): File "/env/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen chunked=chunked) File "/env/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 384, in _make_request six.raise_from(e, None) File "", line 2, in raise_from File "/env/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 380, in _make_request httplib_response = conn.getresponse()
File "/opt/python3.7/lib/python3.7/http/client.py", line 1321, in getresponse response.begin() File "/opt/python3.7/lib/python3.7/http/client.py", line 296, in begin version, status, reason = self._read_status() File "/opt/python3.7/lib/python3.7/http/client.py", line 257, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "/opt/python3.7/lib/python3.7/socket.py", line 589, in readinto return self._sock.recv_into(b) File "/opt/python3.7/lib/python3.7/ssl.py", line 1049, in recv_into return self.read(nbytes, buffer) File "/opt/python3.7/lib/python3.7/ssl.py", line 908, in read return self._sslobj.read(len, buffer) ConnectionResetError: [Errno 104] Connection reset by peer During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/env/local/lib/python3.7/site-packages/requests/adapters.py", line 445, in send timeout=timeout File "/env/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen _stacktrace=sys.exc_info()[2]) File "/env/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 367, in increment raise six.reraise(type(error), error, _stacktrace) File "/env/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 685, in reraise raise value.with_traceback(tb) File "/env/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen chunked=chunked) File "/env/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 384, in _make_request six.raise_from(e, None) File "", line 2, in raise_from File "/env/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 380, in _make_request httplib_response = conn.getresponse() File "/opt/python3.7/lib/python3.7/http/client.py", line 1321, in getresponse response.begin() File "/opt/python3.7/lib/python3.7/http/client.py", line 296, in begin version, status, reason = self._read_status() File "/opt/python3.7/lib/python3.7/http/client.py", line 257, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "/opt/python3.7/lib/python3.7/socket.py", line 589, in readinto return self._sock.recv_into(b) File "/opt/python3.7/lib/python3.7/ssl.py", line 1049, in recv_into return self.read(nbytes, buffer) File "/opt/python3.7/lib/python3.7/ssl.py", line 908, in read return self._sslobj.read(len, buffer) urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/user_code/main.py", line 230, in bq_merge archive_pending_blobs(bucket, blobs[min_idx:max_idx], pending_prefix, loaded_prefix) File "/user_code/main.py", line 44, in archive_pending_blobs raise e File "/user_code/main.py", line 40, in archive_pending_blobs bucket.copy_blob(b, bucket, b.name.replace(pending_prefix, loaded_prefix)) File "/env/local/lib/python3.7/site-packages/google/cloud/storage/bucket.py", line 711, in copy_blob _target_object=new_blob, File "/env/local/lib/python3.7/site-packages/google/cloud/_http.py", line 290, in api_request headers=headers, target_object=_target_object) File "/env/local/lib/python3.7/site-packages/google/cloud/_http.py", line 183, in _make_request return self._do_request(method, url, headers, data, target_object) File "/env/local/lib/python3.7/site-packages/google/cloud/_http.py", line 212, in _do_request url=url, method=method, headers=headers, data=data) File "/env/local/lib/python3.7/site-packages/google/auth/transport/requests.py", line 201, in request method, url, data=data, headers=request_headers, **kwargs) File "/env/local/lib/python3.7/site-packages/requests/sessions.py", line 512, in request resp = self.send(prep, **send_kwargs) File "/env/local/lib/python3.7/site-packages/requests/sessions.py", line 622, in send r = adapter.send(request, **kwargs) File "/env/local/lib/python3.7/site-packages/requests/adapters.py", line 495, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/env/local/lib/python3.7/site-packages/google/cloud/functions_v1beta2/worker.py", li
Per this StackOverflow answer about Connection reset by peer, seems like this is a fatal error were the remote server sends a RST packet to inmediatly drop the connection.
This other SO answer tackles how to solve it. The solution given is using time.sleep but, as we discussed on the comments, didn't work in your case. That's why I'm suggesting a different approach by using truncated exponential backoff:
Truncated exponential backoff is a standard error handling strategy for network applications in which a client periodically retries a failed request with increasing delays between requests.
[...]
Accessing Cloud Storage through a client library. Note that some client libraries, such as the Cloud Storage Client Library for Node.js, have built-in exponential backoff.
There's no built-in exponential backoff for Python, but there's an example of how handle retries in Python with this method.
I am trying to use Google Assistant SDK for the Raspberry Pi 3. I managed to install everything correctly, and the date is perfectly correct. However, when I try to Authorize the Google Assistant SDK sample, I get the error described in the title. It can't verify the certificate.
When I use date, the result is Thu Jun 29 17:16:42 CST 2017, which is very much correct at the time I use it. I am located in Costa Rica.
This is the complete error output.
Traceback (most recent call last):
File "/home/pi/env/lib/python3.4/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/home/pi/env/lib/python3.4/site-packages/urllib3/connectionpool.py", line 345, in _make_request
self._validate_conn(conn)
File "/home/pi/env/lib/python3.4/site-packages/urllib3/connectionpool.py", line 844, in _validate_conn
conn.connect()
File "/home/pi/env/lib/python3.4/site-packages/urllib3/connection.py", line 326, in connect
ssl_context=context)
File "/home/pi/env/lib/python3.4/site-packages/urllib3/util/ssl_.py", line 325, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python3.4/ssl.py", line 364, in wrap_socket
_context=self)
File "/usr/lib/python3.4/ssl.py", line 577, in __init__
self.do_handshake()
File "/usr/lib/python3.4/ssl.py", line 804, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/pi/env/lib/python3.4/site-packages/requests/adapters.py", line 440, in send
timeout=timeout
File "/home/pi/env/lib/python3.4/site-packages/urllib3/connectionpool.py", line 630, in urlopen
raise SSLError(e)
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/pi/env/bin/google-oauthlib-tool", line 11, in <module>
sys.exit(main())
File "/home/pi/env/lib/python3.4/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/home/pi/env/lib/python3.4/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/home/pi/env/lib/python3.4/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/pi/env/lib/python3.4/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/home/pi/env/lib/python3.4/site-packages/google_auth_oauthlib/tool/__main__.py", line 106, in main
creds = flow.run_console()
File "/home/pi/env/lib/python3.4/site-packages/google_auth_oauthlib/flow.py", line 358, in run_console
self.fetch_token(code=code)
File "/home/pi/env/lib/python3.4/site-packages/google_auth_oauthlib/flow.py", line 235, in fetch_token
**kwargs)
File "/home/pi/env/lib/python3.4/site-packages/requests_oauthlib/oauth2_session.py", line 221, in fetch_token
verify=verify, proxies=proxies)
File "/home/pi/env/lib/python3.4/site-packages/requests/sessions.py", line 549, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/home/pi/env/lib/python3.4/site-packages/requests_oauthlib/oauth2_session.py", line 360, in request
headers=headers, data=data, **kwargs)
File "/home/pi/env/lib/python3.4/site-packages/requests/sessions.py", line 502, in request
resp = self.send(prep, **send_kwargs)
File "/home/pi/env/lib/python3.4/site-packages/requests/sessions.py", line 612, in send
r = adapter.send(request, **kwargs)
File "/home/pi/env/lib/python3.4/site-packages/requests/adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)
I had the same problem. I found a guide saying the following dependencies are required. This fixed it for me.
sudo apt-get install portaudio19-dev libffi-dev libssl-dev
I am using django-pipeline, On running "sudo python manage.py collectstatic"
Getting this error:
Traceback (most recent call last):
File "manage.py", line 10, in <module> execute_from_command_line(sys.argv)
File "/Users/office/lib/python2.7/site-packages/django/core/management/__init__.py", line 338, in execute_from_command_line utility.execute()
File "/Users/office/lib/python2.7/site-packages/django/core/management/__init__.py", line 330, in execute self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/office/lib/python2.7/site-packages/django/core/management/base.py", line 390, in run_from_argv self.execute(*args, **cmd_options)
File "/Users/office/lib/python2.7/site-packages/django/core/management/base.py", line 441, in execute output = self.handle(*args, **options)
File "/Users/office/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 168, in handle collected = self.collect()
File "/Users/office/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 120, in collect raise processed ValueError: The file 'bower_components/eonasdan-bootstrap-datetimepicker/build/fonts/glyphicons-halflings-regular.eot' could not be found with <pipeline.storage.PipelineCachedStorage object at 0x10d274e10>.
Any idea how to fix this ?
Using django-pipeline-forgiving(https://pypi.python.org/pypi/django-pipeline-forgiving) solved the issue.
pip install django-pipeline-forgiving
Set in your settings.py: STATICFILES_STORAGE = 'django_pipeline_forgiving.storages.PipelineForgivingStorage'