Problem:
Trying to connect to Softlayer Swift Object Storage using apache libcloud and I can't get it to work. I have tried passing in different options to the provider but no matter what I pass in I get the same error. I would appreciate any and all pointers to help me resolve this issue.
Code:
from pprint import pprint
from libcloud.storage.types import Provider
from libcloud.storage.providers import get_driver
_swift = get_driver(Provider.OPENSTACK_SWIFT)
_authurl = "https://lon02.objectstorage.softlayer.net/auth/v1.0"
_user = "IBMOS<redacted>"
_key = "redacted"
driver = _swift(_user, _key,
region='lon02',
ex_force_auth_url=_authurl,
ex_force_auth_version='1.0',
ex_force_service_type='object-store',
ex_force_service_name='cloudFiles')
#pprint(driver.__dict__)
pprint(driver.list_containers())
Stack Trace:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 441, in wrap_socket
cnx.do_handshake()
File "/usr/local/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1806, in do_handshake
self._raise_ssl_error(self._ssl, result)
File "/usr/local/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1539, in _raise_ssl_error
raise SysCallError(-1, "Unexpected EOF")
OpenSSL.SSL.SysCallError: (-1, 'Unexpected EOF')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 326, in connect
ssl_context=context)
File "/usr/local/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 329, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/local/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 448, in wrap_socket
raise ssl.SSLError('bad handshake: %r' % e)
ssl.SSLError: ("bad handshake: SysCallError(-1, 'Unexpected EOF')",)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
timeout=timeout
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='lon02.objectstorage.softlayer.net', port=443): Max retries exceeded with url: /auth/v1.0 (Caused by SSLError(SSLError("bad handshake: SysCallError(-1, 'Unexpected EOF')",),))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "swift.py", line 19, in <module>
pprint(driver.list_containers())
File "/usr/local/lib/python3.6/site-packages/libcloud/storage/base.py", line 209, in list_containers
return list(self.iterate_containers())
File "/usr/local/lib/python3.6/site-packages/libcloud/storage/drivers/cloudfiles.py", line 275, in iterate_containers
response = self.connection.request('')
File "/usr/local/lib/python3.6/site-packages/libcloud/storage/drivers/cloudfiles.py", line 165, in request
raw=raw)
File "/usr/local/lib/python3.6/site-packages/libcloud/common/openstack.py", line 223, in request
raw=raw)
File "/usr/local/lib/python3.6/site-packages/libcloud/common/base.py", line 536, in request
action = self.morph_action_hook(action)
File "/usr/local/lib/python3.6/site-packages/libcloud/common/openstack.py", line 290, in morph_action_hook
self._populate_hosts_and_request_paths()
File "/usr/local/lib/python3.6/site-packages/libcloud/common/openstack.py", line 324, in _populate_hosts_and_request_paths
osa = osa.authenticate(**kwargs) # may throw InvalidCreds
File "/usr/local/lib/python3.6/site-packages/libcloud/common/openstack_identity.py", line 761, in authenticate
resp = self.request('/v1.0', headers=headers, method='GET')
File "/usr/local/lib/python3.6/site-packages/libcloud/common/base.py", line 603, in request
headers=headers, stream=stream)
File "/usr/local/lib/python3.6/site-packages/libcloud/http.py", line 213, in request
verify=self.verification
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 506, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='lon02.objectstorage.softlayer.net', port=443): Max retries exceeded with url: /auth/v1.0 (Caused by SSLError(SSLError("bad handshake: SysCallError(-1, 'Unexpected EOF')",),))
To answer my own question, the specific issue seen above was due to the system I was using. Once I switched to another system it proceeded even further however there is a bug I identified and filed in apache libcloud with a documented workaround inside the jira ticket here https://issues.apache.org/jira/browse/LIBCLOUD-972.
Related
So i was trying to use my raspberry as a REST webserver, and i decided to start with a simple GET request, so i connected it to my wifi and started the script (it's really basic so i don't think i need to share it)
This is the client that i run from my pc
import requests
r=requests.get(url='http://192.168.1.254:8080/hello',headers={'Content-type':'application/json'})
Out=r.json()
print(Out)
and this is the error message i receive when i run it
Traceback (most recent call last):
File "C:\Users\franc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\urllib3\connection.py", line 174, in _new_conn
conn = connection.create_connection(
File "C:\Users\franc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\urllib3\util\connection.py", line 95, in create_connection
raise err
File "C:\Users\franc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\urllib3\util\connection.py", line 85, in create_connection
sock.connect(sa)
ConnectionRefusedError: [WinError 10061] Impossibile stabilire la connessione. Rifiuto persistente del computer di destinazione
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\franc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\urllib3\connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "C:\Users\franc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\urllib3\connectionpool.py", line 398, in _make_request
conn.request(method, url, **httplib_request_kw)
File "C:\Users\franc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\urllib3\connection.py", line 239, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2288.0_x64__qbz5n2kfra8p0\lib\http\client.py", line 1282, in request
self._send_request(method, url, body, headers, encode_chunked)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2288.0_x64__qbz5n2kfra8p0\lib\http\client.py", line 1328, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2288.0_x64__qbz5n2kfra8p0\lib\http\client.py", line 1277, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2288.0_x64__qbz5n2kfra8p0\lib\http\client.py", line 1037, in _send_output
self.send(msg)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2288.0_x64__qbz5n2kfra8p0\lib\http\client.py", line 975, in send
self.connect()
File "C:\Users\franc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\urllib3\connection.py", line 205, in connect
conn = self._new_conn()
File "C:\Users\franc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\urllib3\connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x000001C15FA75CF0>: Failed to establish a new connection: [WinError 10061] Impossibile stabilire la connessione. Rifiuto persistente del computer di destinazione
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\franc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\requests\adapters.py", line 489, in send
resp = conn.urlopen(
File "C:\Users\franc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\urllib3\connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "C:\Users\franc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\urllib3\util\retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='192.168.1.254', port=8080): Max retries exceeded with url: /ciao (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x000001C15FA75CF0>: Failed to establish a new connection: [WinError 10061] Impossibile stabilire la connessione. Rifiuto persistente del computer di destinazione'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\franc\Desktop\rasp\client.py", line 3, in <module>
r=requests.get(url='http://192.168.1.254:8080/ciao',headers={'Content-type':'application/json'})
File "C:\Users\franc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\requests\api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "C:\Users\franc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\requests\api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\franc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\requests\sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\franc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\requests\sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "C:\Users\franc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\requests\adapters.py", line 565, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='192.168.1.254', port=8080): Max retries exceeded with url: /ciao (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x000001C15FA75CF0>: Failed to establish a new connection: [WinError 10061] Impossibile stabilire la connessione. Rifiuto persistente del computer di destinazione'))
Any idea what's going on? if you need to know it " Impossibile stabilire la connessione. Rifiuto persistente del computer di destinazione" from the error message means " unable to establish connection. Persistent denial from destination's computer"
When i run both scripts from my pc (so using localhost) i have no issues (again, i was just trying to do a test), so i don't think they're the problem.
I refer https://github.com/ssaru/You_Only_Look_Once
when I run main.py, raising my connection error.
Looks like a connection error, but I don't know what the code is trying to connect to.
How can I solve this?
Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/urllib3/connection.py", line 159, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw) File "/usr/local/lib/python3.7/dist-packages/urllib3/util/connection.py", line 80, in create_connection
raise err File "/usr/local/lib/python3.7/dist-packages/urllib3/util/connection.py", line 70, in create_connection
sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked) File "/usr/local/lib/python3.7/dist-packages/urllib3/connectionpool.py", line 354, in _make_request
conn.request(method, url, **httplib_request_kw) File "/usr/lib/python3.7/http/client.py", line 1281, in request
self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib/python3.7/http/client.py", line 1327, in _send_request
self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib/python3.7/http/client.py", line 1276, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.7/http/client.py", line 1036, in _send_output
self.send(msg) File "/usr/lib/python3.7/http/client.py", line 976, in send
self.connect() File "/usr/local/lib/python3.7/dist-packages/urllib3/connection.py", line 181, in connect
conn = self._new_conn() File "/usr/local/lib/python3.7/dist-packages/urllib3/connection.py", line 168, in _new_conn
self, "Failed to establish a new connection: %s" % e) urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5e0a35a5d0>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/requests/adapters.py", line 449, in send
timeout=timeout File "/usr/local/lib/python3.7/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2]) File "/usr/local/lib/python3.7/dist-packages/urllib3/util/retry.py", line 399, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8097): Max retries exceeded with url: /update (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5e0a35a5d0>: Failed to establish a new connection: [Errno 111] Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/visdom/__init__.py", line 711, in _send
data=json.dumps(msg), File "/usr/local/lib/python3.7/dist-packages/visdom/__init__.py", line 677, in _handle_post
r = self.session.post(url, data=data) File "/usr/local/lib/python3.7/dist-packages/requests/sessions.py", line 578, in post
return self.request('POST', url, data=data, json=json, **kwargs) File "/usr/local/lib/python3.7/dist-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.7/dist-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.7/dist-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8097): Max retries exceeded with url: /update (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5e0a35a5d0>: Failed to establish a new connection: [Errno 111] Connection refused')) Exception in user code:
I am trying to train a neural network in PyTorch following a git tutorial. When it comes to train the network I get a row of exceptions linked to a failed connection attempt (probably with some google server):
Traceback (most recent call last):
File "/home/***/.local/lib/python3.6/site-packages/urllib3/connection.py", line 157, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/home/***/.local/lib/python3.6/site-packages/urllib3/util/connection.py", line 61, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/home/***/.conda/envs/cv-nd/lib/python3.6/socket.py", line 745, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/***/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
chunked=chunked,
File "/home/***/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 387, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/home/***/.conda/envs/cv-nd/lib/python3.6/http/client.py", line 1262, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/home/***/.conda/envs/cv-nd/lib/python3.6/http/client.py", line 1308, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/home/***/.conda/envs/cv-nd/lib/python3.6/http/client.py", line 1257, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/home/***/.conda/envs/cv-nd/lib/python3.6/http/client.py", line 1036, in _send_output
self.send(msg)
File "/home/*.*/.conda/envs/cv-nd/lib/python3.6/http/client.py", line 974, in send
self.connect()
File "/home/*.*/.local/lib/python3.6/site-packages/urllib3/connection.py", line 184, in connect
conn = self._new_conn()
File "/home/*.*/.local/lib/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7ff4d4cc87f0>: Failed to establish a new connection: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/*.*/.local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/home/*.*/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/home/*.*/.local/lib/python3.6/site-packages/urllib3/util/retry.py", line 436, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/attributes/keep_alive_token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff4d4cc87f0>: Failed to establish a new connection: [Errno -2] Name or service not known',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "NB2.py", line 229, in <module>
with active_session():
File "/home/*.*/.conda/envs/cv-nd/lib/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/home/*.*/Code/faceNNtrain/Facial_Keypoint_Detection/workspace_utils.py", line 31, in active_session
token = requests.request("GET", TOKEN_URL, headers=TOKEN_HEADERS).text
File "/home/*.*/.local/lib/python3.6/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/home/*.*/.local/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/home/*.*/.local/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/home/*.*/.local/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/attributes/keep_alive_token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff4d4cc87f0>: Failed to establish a new connection: [Errno -2] Name or service not known',))
I am confused since there is no external call required, and it is nowhere defined in my code (including model definition) - it probably sits somewhere hidden in the libraries, but still, it fails, messes everything up and idk how to fix it. I checked my local firewall and all, but the only host from the errors I could check (metadata.google.internal) is also not reachable from other clients. Any ideas?
google.api_core.exceptions.ServiceUnavailable: 503 Deadline Exceeded
using python 3.7 ,google-cloud-pubsub ==1.1.0 publishing data on topic. In my local machine it's working perfectly fine and able to publish data on that topic and also able to pull data from that topic through subscriber.
but don't understand it's not working when i deploy the code on server and it's failing with INLINE ERROR however when i explicitly call the publisher method on server it's publishing fine over server box also.code which is failing at below line while publishing:
future = publisher.publish(topic_path, data=data)
**ERROR:2020-02-20 14:24:42,714 ERROR Failed to publish 1 messages.**
Trackback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
return callable_(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 826, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Deadline Exceeded"
debug_error_string = "{"created":"#1582208682.711481693","description":"Deadline Exceeded","file":"src/core/ext/filters/deadline/deadline_filter.cc","file_line":69,"grpc_status":14}"
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 184, in retry_target
return target()
File "/usr/local/lib/python3.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.ServiceUnavailable: 503 Deadline Exceeded
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/google/cloud/pubsub_v1/publisher/_batch/thread.py", line 219, in _commit
response = self._client.api.publish(self._topic, self._messages)
File "/usr/local/lib/python3.7/site-packages/google/cloud/pubsub_v1/gapic/publisher_client.py", line 498, in publish
request, retry=retry, timeout=timeout, metadata=metadata
File "/usr/local/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in call
return wrapped_func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
on_error=on_error,
File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 206, in retry_target
last_exc,
File "", line 3, in raise_from
google.api_core.exceptions.RetryError: Deadline of 60.0s exceeded while calling functools.partial(.error_remapped_callable at 0x7f67d064e950>
You should try to chunk your data in reasonable sized chunks (max_messages) and don't forget to add a done callback.
# Loop over json containing records/rows
for idx, row in enumerate(rows_json):
publish_json(row, idx, rowmax=len(rows_json), topic_name)
# Publish messages asynchronous
def publish_json(msg, rowcount, rowmax, topic_project_id, topic_name):
batch_settings = pubsub_v1.types.BatchSettings(max_messages=100)
publisher = pubsub_v1.PublisherClient(batch_settings)
topic_path = publisher.topic_path(topic_project_id, topic_name)
future = publisher.publish(
topic_path, bytes(json.dumps(msg).encode('utf-8')))
future.add_done_callback(
lambda x: logging.info(
'Published msg with ID {} ({}/{} rows).'.format(
future.result(), rowcount, rowmax))
)
I have a similar issue to that which is highlited at https://www.pythonanywhere.com/forums/topic/2688/
the only difference is that I am using flask, and the module I am using is pytube. I trust that you shall help, the error message displayed is:
Traceback (most recent call last):
File "/home/*username*/.virtualenvs/*project*/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/home/*username*/.virtualenvs/*project*/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/*username*/.virtualenvs/*project*/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/*username*/.virtualenvs/*project*/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/home/*username*/.virtualenvs/*project*/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functionsrule.endpoint
File "/home/*username*/<project>/app.py", line 49, in upload
ytube = YouTube(url)
File "/home/*username*/.virtualenvs/*project*/local/lib/python2.7/site-packages/pytube/api.py", line 66, in init
self.from_url(url)
File "/home/*username*/.virtualenvs/*project*/local/lib/python2.7/site-packages/pytube/api.py", line 158, in from_url
video_data = self.get_video_data()
File "/home/*username*/.virtualenvs/*project*/local/lib/python2.7/site-packages/pytube/api.py", line 254, in get_video_data
response = urlopen(self.url)
File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 404, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 422, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1222, in https_open
return self.do_open(httplib.HTTPSConnection, req)
File "/usr/lib/python2.7/urllib2.py", line 1184, in do_open
raise URLError(err) URLError: <urlopen error Tunnel connection failed: 403 Forbidden>
Youtube doesn't allow downloading videos through their API, so all of the scripts that download youtube videos scrape the www site, and we don't allow scraping from free accounts.
more info here: http://help.pythonanywhere.com/pages/403ForbiddenError