I using MongoDB to store data scraped from web using Scrapy as scraper . The problem is, when I start to run a long process of scraping using multiple spiders the Mongo crashes and the spiders start to receive the following message:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/twisted/internet/defer.py", line 654, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "/home/ubuntu/search/decapod/updater/updater/pipelines.py", line 90, in process_item
self.db[self.collection_name].insert_one(dict(item))
File "/usr/local/lib/python3.5/dist-packages/pymongo/collection.py", line 693, in insert_one
session=session),
File "/usr/local/lib/python3.5/dist-packages/pymongo/collection.py", line 607, in _insert
bypass_doc_val, session)
File "/usr/local/lib/python3.5/dist-packages/pymongo/collection.py", line 595, in _insert_one
acknowledged, _insert_command, session)
File "/usr/local/lib/python3.5/dist-packages/pymongo/mongo_client.py", line 1242, in _retryable_write
with self._tmp_session(session) as s:
File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.5/dist-packages/pymongo/mongo_client.py", line 1571, in _tmp_session
s = self._ensure_session(session)
File "/usr/local/lib/python3.5/dist-packages/pymongo/mongo_client.py", line 1558, in _ensure_session
return self.__start_session(True, causal_consistency=False)
File "/usr/local/lib/python3.5/dist-packages/pymongo/mongo_client.py", line 1511, in __start_session
server_session = self._get_server_session()
File "/usr/local/lib/python3.5/dist-packages/pymongo/mongo_client.py", line 1544, in _get_server_session
return self._topology.get_server_session()
File "/usr/local/lib/python3.5/dist-packages/pymongo/topology.py", line 427, in get_server_session
None)
File "/usr/local/lib/python3.5/dist-packages/pymongo/topology.py", line 199, in _select_servers_loop
self._error_message(selector))
pymongo.errors.ServerSelectionTimeoutError: mongodb.getmore.com.br:27017: timed out
How can I automatically restart Mongo when it crashes or prevent this to happening?
I'm currently running mongo on a EC2 instance t2.small.
My suspicion is that your are not having keepAlive in your settings for the connection.
Can you try setting your keepAlive: 1 in your connection options:
options: {
server: {
socketOptions: {
keepAlive: 1,
connectTimeoutMS: 30000
}
}
}
Here is more on the optional parameters
Related
when using find/find_one pymongo methods when on an apache2 server i get the following error :
No <class 'cryptography.x509.extensions.TLSFeature'> extension was found
when using pymongo on an apache2 server to connect and insert or delete in my mongodb collections it works just fine, but when i try to pull something out of the database (using find or find_one methods) i get an error, please note that this error only occurs while running the apache2 server, because when i connected and used find methods in a locally run python script it gave me the information without any errors.
if it is of any relevance, the web framework i am using is flask.
Edit : As for what is producing the error i dont know for sure, so here are the logs.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 2525, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1822, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1820, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1796, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "/var/www/html/srv21/SRVApp.py", line 17, in downloadpage
ThreadID = ThreadID.next()
File "/usr/local/lib/python3.10/dist-packages/pymongo/cursor.py", line 1248, in next
if len(self.__data) or self._refresh():
File "/usr/local/lib/python3.10/dist-packages/pymongo/cursor.py", line 1165, in _refresh
self.__send_message(q)
File "/usr/local/lib/python3.10/dist-packages/pymongo/cursor.py", line 1052, in __send_message
response = client._run_operation(
File "/usr/local/lib/python3.10/dist-packages/pymongo/_csot.py", line 105, in csot_wrapper
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/pymongo/mongo_client.py", line 1330, in _run_operation
return self._retryable_read(
File "/usr/local/lib/python3.10/dist-packages/pymongo/_csot.py", line 105, in csot_wrapper
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/pymongo/mongo_client.py", line 1442, in _retryable_read
with self._socket_from_server(read_pref, server, session) as (sock_info, read_pref):
File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.10/dist-packages/pymongo/mongo_client.py", line 1282, in _socket_from_server
with self._get_socket(server, session) as sock_info:
File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.10/dist-packages/pymongo/mongo_client.py", line 1217, in _get_socket
with server.get_socket(handler=err_handler) as sock_info:
File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.10/dist-packages/pymongo/pool.py", line 1407, in get_socket
sock_info = self._get_socket(handler=handler)
File "/usr/local/lib/python3.10/dist-packages/pymongo/pool.py", line 1520, in _get_socket
sock_info = self.connect(handler=handler)
File "/usr/local/lib/python3.10/dist-packages/pymongo/pool.py", line 1358, in connect
sock = _configured_socket(self.address, self.opts)
File "/usr/local/lib/python3.10/dist-packages/pymongo/pool.py", line 1061, in _configured_socket
sock = ssl_context.wrap_socket(sock, server_hostname=host)
File "/usr/local/lib/python3.10/dist-packages/pymongo/pyopenssl_context.py", line 369, in wrap_socket
ssl_conn.do_handshake()
File "/usr/local/lib/python3.10/dist-packages/pymongo/pyopenssl_context.py", line 125, in do_handshake
return self._call(super(_sslConn, self).do_handshake, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/pymongo/pyopenssl_context.py", line 108, in _call
return call(*args, **kwargs)
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1894, in do_handshake
self._raise_ssl_error(self._ssl, result)
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1603, in _raise_ssl_error
self._context._ocsp_helper.raise_if_problem()
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 319, in raise_if_problem
raise self._problems.pop(0)
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 539, in wrapper
valid = callback(conn, ocsp_data, data)
File "/usr/local/lib/python3.10/dist-packages/pymongo/ocsp_support.py", line 298, in _ocsp_callback
ext = _get_extension(cert, _TLSFeature)
File "/usr/local/lib/python3.10/dist-packages/pymongo/ocsp_support.py", line 114, in _get_extension
return cert.extensions.get_extension_for_class(klass)
File "/usr/local/lib/python3.10/dist-packages/cryptography/x509/extensions.py", line 135, in get_extension_for_class
raise ExtensionNotFound(
cryptography.x509.extensions.ExtensionNotFound: No <class 'cryptography.x509.extensions.TLSFeature'> extension was found
google.api_core.exceptions.ServiceUnavailable: 503 Deadline Exceeded
using python 3.7 ,google-cloud-pubsub ==1.1.0 publishing data on topic. In my local machine it's working perfectly fine and able to publish data on that topic and also able to pull data from that topic through subscriber.
but don't understand it's not working when i deploy the code on server and it's failing with INLINE ERROR however when i explicitly call the publisher method on server it's publishing fine over server box also.code which is failing at below line while publishing:
future = publisher.publish(topic_path, data=data)
**ERROR:2020-02-20 14:24:42,714 ERROR Failed to publish 1 messages.**
Trackback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
return callable_(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 826, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Deadline Exceeded"
debug_error_string = "{"created":"#1582208682.711481693","description":"Deadline Exceeded","file":"src/core/ext/filters/deadline/deadline_filter.cc","file_line":69,"grpc_status":14}"
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 184, in retry_target
return target()
File "/usr/local/lib/python3.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.ServiceUnavailable: 503 Deadline Exceeded
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/google/cloud/pubsub_v1/publisher/_batch/thread.py", line 219, in _commit
response = self._client.api.publish(self._topic, self._messages)
File "/usr/local/lib/python3.7/site-packages/google/cloud/pubsub_v1/gapic/publisher_client.py", line 498, in publish
request, retry=retry, timeout=timeout, metadata=metadata
File "/usr/local/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in call
return wrapped_func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
on_error=on_error,
File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 206, in retry_target
last_exc,
File "", line 3, in raise_from
google.api_core.exceptions.RetryError: Deadline of 60.0s exceeded while calling functools.partial(.error_remapped_callable at 0x7f67d064e950>
You should try to chunk your data in reasonable sized chunks (max_messages) and don't forget to add a done callback.
# Loop over json containing records/rows
for idx, row in enumerate(rows_json):
publish_json(row, idx, rowmax=len(rows_json), topic_name)
# Publish messages asynchronous
def publish_json(msg, rowcount, rowmax, topic_project_id, topic_name):
batch_settings = pubsub_v1.types.BatchSettings(max_messages=100)
publisher = pubsub_v1.PublisherClient(batch_settings)
topic_path = publisher.topic_path(topic_project_id, topic_name)
future = publisher.publish(
topic_path, bytes(json.dumps(msg).encode('utf-8')))
future.add_done_callback(
lambda x: logging.info(
'Published msg with ID {} ({}/{} rows).'.format(
future.result(), rowcount, rowmax))
)
I was working on odoo, precisely in Website builder module while I was creating a website via the host IP address 192.168.1.2:9012, suddenly it crashes and shown me an Internal server error.
I tried to have a look at the log file and got this traceback,
2018-12-06 18:14:31,432 1 INFO ? odoo.sql_db: Connection to the database failed
2018-12-06 18:14:31,436 1 INFO ? werkzeug: 192.168.1.9 - - [06/Dec/2018 18:14:31] "GET /favicon.ico HTTP/1.1" 500 -
2018-12-06 18:14:31,441 1 ERROR ? werkzeug: Error on request:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 205, in run_wsgi
execute(self.server.app)
File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 193, in execute
application_iter = app(environ, start_response)
File "/usr/lib/python3/dist-packages/odoo/service/server.py", line 250, in app
return self.app(e, s)
File "/usr/lib/python3/dist-packages/odoo/service/wsgi_server.py", line 166, in application
return application_unproxied(environ, start_response)
File "/usr/lib/python3/dist-packages/odoo/service/wsgi_server.py", line 154, in application_unproxied
result = handler(environ, start_response)
File "/usr/lib/python3/dist-packages/odoo/http.py", line 1318, in _call_
return self.dispatch(environ, start_response)
File "/usr/lib/python3/dist-packages/odoo/http.py", line 1292, in _call_
return self.app(environ, start_wrapped)
File "/usr/lib/python3/dist-packages/werkzeug/wsgi.py", line 599, in _call_
return self.app(environ, start_response)
File "/usr/lib/python3/dist-packages/odoo/http.py", line 1455, in dispatch
self.setup_db(httprequest)
File "/usr/lib/python3/dist-packages/odoo/http.py", line 1387, in setup_db
httprequest.session.db = db_monodb(httprequest)
File "/usr/lib/python3/dist-packages/odoo/http.py", line 1539, in db_monodb
dbs = db_list(True, httprequest)
File "/usr/lib/python3/dist-packages/odoo/http.py", line 1506, in db_list
dbs = odoo.service.db.list_dbs(force)
File "/usr/lib/python3/dist-packages/odoo/service/db.py", line 369, in list_dbs
with closing(db.cursor()) as cr:
File "/usr/lib/python3/dist-packages/odoo/sql_db.py", line 634, in cursor
return Cursor(self.__pool, self.dbname, self.dsn, serialized=serialized)
File "/usr/lib/python3/dist-packages/odoo/sql_db.py", line 178, in _init_
self._cnx = pool.borrow(dsn)
File "/usr/lib/python3/dist-packages/odoo/sql_db.py", line 517, in _locked
return fun(self, *args, **kwargs)
File "/usr/lib/python3/dist-packages/odoo/sql_db.py", line 585, in borrow
**connection_info)
File "/usr/lib/python3/dist-packages/psycopg2/__init__.py", line 164, in connect
conn = _connect(dsn, connection_factory=connection_factory, async=async)
psycopg2.OperationalError: could not connect to server: No route to host
Is the server running on host "172.17.0.4" and accepting
TCP/IP connections on port 5432 ?
I believe that's a network problem, I couldn't connect to the host server. do you have any idea about this please?
Thank you.
The error is from Odoo server because it cannot connect to the PostgreSQL database. First make sure your database service is running and available.
From your ip addresses I assume you are running database locally in some kind of virtualization environment like Docker. If you are not able to connect, please describe your environment in more detail. This will make it possible for stackoverflowers to help you.
I tried to run a very simple task with a linked task mentioned in the tutorial
add.apply_async((2, 2), link=add.s(16))
and got an exception in the worker process:
[2014-09-21 19:56:38,531: WARNING/Worker-1] C:\Python33\lib\site-packages\celery-3.1.15-
py3.3.egg\celery\app\trace.py:364: RuntimeWarning: Exception raised outside body: OSError(ConnectionRefusedError(10061, 'No connection could be made because the target machine actively refused it', None, 10061),):
Traceback (most recent call last):
File "C:\Python33\lib\site-packages\kombu-3.0.23-py3.3.egg\kombu\utils\__init__.py", line 420, in __call__
return self.__value__
AttributeError: 'ChannelPromise' object has no attribute '__value__'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Python33\lib\site-packages\kombu-3.0.23-py3.3.egg\kombu\connection.py", line 436, in _ensured
return fun(*args, **kwargs)
File "C:\Python33\lib\site-packages\kombu-3.0.23-py3.3.egg\kombu\messaging.py", line 173, in _publish
channel = self.channel
File "C:\Python33\lib\site-packages\kombu-3.0.23-py3.3.egg\kombu\messaging.py", line 190, in _get_channel
channel = self._channel = channel()
File "C:\Python33\lib\site-packages\kombu-3.0.23-py3.3.egg\kombu\utils\__init__.py", line 422, in __call__
value = self.__value__ = self.__contract__()
File "C:\Python33\lib\site-packages\kombu-3.0.23-py3.3.egg\kombu\messaging.py", line 205, in <lambda>
channel = ChannelPromise(lambda: connection.default_channel)
File "C:\Python33\lib\site-packages\kombu-3.0.23-py3.3.egg\kombu\connection.py", line 756, in default_channel
self.connection
File "C:\Python33\lib\site-packages\kombu-3.0.23-py3.3.egg\kombu\connection.py", line 741, in connection
self._connection = self._establish_connection()
File "C:\Python33\lib\site-packages\kombu-3.0.23-py3.3.egg\kombu\connection.py", line 696, in _establish_connection
conn = self.transport.establish_connection()
File "C:\Python33\lib\site-packages\kombu-3.0.23-py3.3.egg\kombu\transport\pyamqp.py", line 112, in establish_connection
conn = self.Connection(**opts)
File "C:\Python33\lib\site-packages\amqp-1.4.6-py3.3.egg\amqp\connection.py", line 165, in __init__
self.transport = self.Transport(host, connect_timeout, ssl)
File "C:\Python33\lib\site-packages\amqp-1.4.6-py3.3.egg\amqp\connection.py", line 186, in Transport
return create_transport(host, connect_timeout, ssl)
File "C:\Python33\lib\site-packages\amqp-1.4.6-py3.3.egg\amqp\transport.py", line 299, in create_transport
return TCPTransport(host, connect_timeout)
File "C:\Python33\lib\site-packages\amqp-1.4.6-py3.3.egg\amqp\transport.py", line 95, in __init__
raise socket.error(last_err)
OSError: [WinError 10061] No connection could be made because the target machine actively refused it
I did a brief debugging in transport.py and found the worker was trying to connect to port 5672 on localhost. It seems that the worker thinks the linked task needs to be executed via local RabbitMQ instance. This is weird because I specified a remote RabbitMQ broker in the configuration setting. Also the setting works if I simply run the async call without a linked task:
add.apply_async((2, 2))
Here is my setup:
Use RabbitMQ as broker and Redis as results back end on a remote Windows Server
Run my test client on another windows 7 machine
Can anyone shed some light? Thanks.
I'm facing following exception when celery is trying to cleanup back-end.
Most probably, this is happening due to MySQL disconnect issue and can be solved by using pool_recycle parameter and retrying the task later. But this is out of my hand - I guess celery needs to provide support for this?
Now my question is, what is backend cleanup task and how such a failed task may affect our system?
Log:
[2014-04-08 04:00:00,017: INFO/Beat] Scheduler: Sending due task celery.backend_cleanup (celery.backend_cleanup)
[2014-04-08 04:00:00,020: INFO/MainProcess] Received task: celery.backend_cleanup[b70acd50-e72d-43b1-a702-0bfa8e7e83a6] expires:[2014-04-08 16:00:00.018317+01:00]
[2014-04-08 04:00:00,036: ERROR/MainProcess] Task celery.backend_cleanup[b70acd50-e72d-43b1-a702-0bfa8e7e83a6] raised unexpected: OperationalError('(OperationalError) MySQL Connection not available.',)
Traceback (most recent call last):
File "/webapps/phoenix/lib/python3.3/site-packages/celery/app/trace.py", line 238, in trace_task
R = retval = fun(*args, **kwargs)
File "/webapps/phoenix/lib/python3.3/site-packages/celery/app/trace.py", line 416, in __protected_call__
return self.run(*args, **kwargs)
File "/webapps/phoenix/lib/python3.3/site-packages/celery/app/builtins.py", line 56, in backend_cleanup
app.backend.cleanup()
File "/webapps/phoenix/lib/python3.3/site-packages/celery/backends/database/__init__.py", line 180, in cleanup
Task.date_done < (now - expires)).delete()
File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/orm/query.py", line 2626, in delete
delete_op.exec_()
File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/orm/persistence.py", line 866, in exec_
self._do_exec()
File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/orm/persistence.py", line 991, in _do_exec
params=self.query._params)
File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/orm/session.py", line 978, in execute
clause, params or {})
File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/engine/base.py", line 664, in execute
return meth(self, multiparams, params)
File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/sql/elements.py", line 282, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/engine/base.py", line 761, in _execute_clauseelement
compiled_sql, distilled_params
File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/engine/base.py", line 828, in _execute_context
None, None)
File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/engine/base.py", line 1023, in _handle_dbapi_exception
exc_info
File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/util/compat.py", line 174, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=exc_value)
File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/util/compat.py", line 167, in reraise
raise value.with_traceback(tb)
File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/engine/base.py", line 824, in _execute_context
context = constructor(dialect, self, conn, *args)
File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/engine/default.py", line 507, in _init_compiled
self.cursor = self.create_cursor()
File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/engine/default.py", line 671, in create_cursor
return self._dbapi_connection.cursor()
File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/pool.py", line 548, in cursor
return self.connection.cursor(*args, **kwargs)
File "/webapps/phoenix/lib/python3.3/site-packages/mysql/connector/connection.py", line 1231, in cursor
raise errors.OperationalError("MySQL Connection not available.")
sqlalchemy.exc.OperationalError: (OperationalError) MySQL Connection not available. 'DELETE FROM celery_taskmeta WHERE celery_taskmeta.date_done < %(date_done_1)s' [{}]
PS I've checked this SO question but seems like it's due to a different exception: Celery log shows cleanup failed
The closest Celery has to retrying is short lived sessions.
The task is cleaning out un-read task results. If it's failing, you may see those results start to build up, but should be OK otherwise.
You're right that there's very little documentation about it!