I was orchestrating two dataflow job with cloud composer and it was working fine for month. Suddenly the two jobs stopped working with the following error message:
in download_blob File
"/usr/local/lib/python3.6/site-packages/google/cloud/storage/client.py",
line 399, in get_bucket retry=retry, File
"/usr/local/lib/python3.6/site-packages/google/cloud/storage/bucket.py",
line 1002, in reload retry=retry, File
"/usr/local/lib/python3.6/site-packages/google/cloud/storage/_helpers.py",
line 225, in reload retry=retry, File
"/usr/local/lib/python3.6/site-packages/google/cloud/storage/_http.py",
line 63, in api_request return call() File
"/usr/local/lib/python3.6/site-packages/google/api_core/retry.py",
line 286, in retry_wrapped_func on_error=on_error, File
"/usr/local/lib/python3.6/site-packages/google/api_core/retry.py",
line 184, in retry_target return target() File
"/usr/local/lib/python3.6/site-packages/google/cloud/_http.py", line
479, in api_request timeout=timeout, File
"/usr/local/lib/python3.6/site-packages/google/cloud/_http.py", line
337, in _make_request method, url, headers, data, target_object,
timeout=timeout File
"/usr/local/lib/python3.6/site-packages/google/cloud/_http.py", line
374, in _do_request return self.http.request( File
"/usr/local/lib/python3.6/site-packages/google/cloud/_http.py", line
157, in http return self._client._http File
"/usr/local/lib/python3.6/site-packages/google/cloud/client.py", line
187, in _http
self._http_internal.configure_mtls_channel(self._client_cert_source)
AttributeError: 'AuthorizedSession' object has no attribute
'configure_mtls_channel'
In the jobs I download a file from google cloud storage with the storage client. I assumed it was because of some dependencies issues. In the composer environment I installed google-cloud-storage without specifying a version. I tried specifying different versions of the package but nothing seems to work.
Thanks!
This seems to be related to this issue.
Try pinning google-cloud-core to 1.5.0, then I highly recommend for you to Drain your jobs once you get them back to work (assuming they have streaming jobs).
Related
We get the following error when we try to push backup using wal-e:
2020-07-16T21:18:55Z <Greenlet at 0x7f2a59fadc48: <wal_e.worker.upload.PartitionUploader object at 0x7f2a59f96cc0>([ExtendedTarInfo(submitted_path='/var/lib/postgres)> failed with PermissionError
wal_e.operator.backup WARNING MSG: blocking on sending WAL segments
DETAIL: The backup was not completed successfully, but we have to wait anyway. See README: TODO about pg_cancel_backup
STRUCTURED: time=2020-07-16T21:18:55.651073-00 pid=19697
wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling
DETAIL: Traceback (most recent call last):
File "/var/lib/postgresql/virtualenvs/wal-e/lib/python3.5/site-packages/wal_e/operator/backup.py", line 197, in database_backup
**kwargs)
File "/var/lib/postgresql/virtualenvs/wal-e/lib/python3.5/site-packages/wal_e/operator/backup.py", line 500, in _upload_pg_cluster_dir
pool.put(tpart)
File "/var/lib/postgresql/virtualenvs/wal-e/lib/python3.5/site-packages/wal_e/worker/upload_pool.py", line 108, in put
self._wait()
File "/var/lib/postgresql/virtualenvs/wal-e/lib/python3.5/site-packages/wal_e/worker/upload_pool.py", line 65, in _wait
raise val
File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run
File "/var/lib/postgresql/virtualenvs/wal-e/lib/python3.5/site-packages/wal_e/worker/upload.py", line 96, in __call__
gpg_key=self.gpg_key) as pl:
File "/var/lib/postgresql/virtualenvs/wal-e/lib/python3.5/site-packages/wal_e/pipeline.py", line 92, in __enter__
self.stdin = pipebuf.NonBlockBufferedWriter(stdin)
File "/var/lib/postgresql/virtualenvs/wal-e/lib/python3.5/site-packages/wal_e/pipebuf.py", line 225, in __init__
_setup_fd(self._fd)
File "/var/lib/postgresql/virtualenvs/wal-e/lib/python3.5/site-packages/wal_e/pipebuf.py", line 62, in _setup_fd
set_buf_size(fd)
File "/var/lib/postgresql/virtualenvs/wal-e/lib/python3.5/site-packages/wal_e/pipebuf.py", line 53, in set_buf_size
fcntl.fcntl(fd, fcntl.F_SETPIPE_SZ, OS_PIPE_SZ)
PermissionError: [Errno 1] Operation not permitted
It's not clear why fcntl call may lead to PermissionError.
PostgreSQL version: 9.6, Python: 3.5, Wal-e : 1.1.1 (tried also 1.0.3 and 1.1.0).
It was working previously and stopped working at some point (without any noticeable changes).
Well, I'm late to the game. See https://github.com/wal-e/wal-e/issues/270
I've worked around it by patching wal-e to not set this.
I am trying to set up pytest on Python 2.6.6 RHEL 6.10.
As pointed out in official change log pytest 3.3.0 doesn't support python 2.6 and hence pytest < 3.3 should be considered. I could successfully install the pytest=3.2.5, but when I execute a simple test I get the following error :
Traceback (most recent call last):
File "/usr/lib64/python2.6/runpy.py", line 122, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib64/python2.6/runpy.py", line 34, in _run_code
exec code in run_globals
File "/usr/lib/python2.6/site-packages/pytest.py", line 17, in <module>
raise SystemExit(pytest.main())
File "/usr/lib/python2.6/site-packages/_pytest/config.py", line 57, in main
return config.hook.pytest_cmdline_main(config=config)
File "/usr/lib/python2.6/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
File "/usr/lib/python2.6/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File "/usr/lib/python2.6/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
_MultiCall(methods, kwargs, hook.spec_opts).execute()
File "/usr/lib/python2.6/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
res = hook_impl.function(*args)
File "/usr/lib/python2.6/site-packages/_pytest/main.py", line 127, in pytest_cmdline_main
return wrap_session(config, _main)
File "/usr/lib/python2.6/site-packages/_pytest/main.py", line 111, in wrap_session
config.notify_exception(excinfo, config.option)
File "/usr/lib/python2.6/site-packages/_pytest/config.py", line 936, in notify_exception
excinfo=excinfo)
File "/usr/lib/python2.6/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
File "/usr/lib/python2.6/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File "/usr/lib/python2.6/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
_MultiCall(methods, kwargs, hook.spec_opts).execute()
File "/usr/lib/python2.6/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
res = hook_impl.function(*args)
File "/usr/lib/python2.6/site-packages/_pytest/terminal.py", line 163, in pytest_internalerror
for line in py.builtin.text(excrepr).split("\n"):
File "/usr/lib/python2.6/site-packages/_pytest/_code/code.py", line 664, in __unicode__
self.toterminal(tw)
File "/usr/lib/python2.6/site-packages/_pytest/_code/code.py", line 709, in toterminal
self.reprtraceback.toterminal(tw)
File "/usr/lib/python2.6/site-packages/_pytest/_code/code.py", line 725, in toterminal
entry.toterminal(tw)
File "/usr/lib/python2.6/site-packages/_pytest/_code/code.py", line 748, in toterminal
tw.write("".join(self.lines))
File "/usr/lib/python2.6/site-packages/py/_io/terminalwriter.py", line 250, in write
self._update_chars_on_current_line(msg)
File "/usr/lib/python2.6/site-packages/py/_io/terminalwriter.py", line 262, in _update_chars_on_current_line
current_line = current_line.decode('utf-8', errors='replace')
TypeError: decode() takes no keyword arguments
A quick search suggested that decode() takes argument in python 2.7.
If this is the case of dependence then which version of pytest is recommended for Python 2.6.6 ?
pytest==3.2.5 is the version that still supports Python 2.6, however just because pytest==3.2.5 supports Python 2.6, does not necessarily mean that its dependencies as the library authors specified in their setup.py supports Python 2.6.
Most packages specifies their dependencies using greater-than constraint, e.g. libxxx>=1.2.3, to ensure you'll automatically get newer versions of dependencies when the depended libraries author released them but you'll never download an incompatible older version. In most cases, this is great as it reduces the workload for the library's authors, but this means that if that depended library released a newer version that drops support for a particular version that the dependant library still supports, you'll be pulling that in, as there's no less-than constraint to prevent that.
In this case, according to the traceback you've shown, the py.io library that got pulled doesn't support python 2.6. You'll need to downgrade and pin py.io to an older version that still supports 2.6.
According to py.io changelog, the last version that still supports Python 2.6 is py.io==1.4.34.
If it's important for you to support an old software, you should create a requirements.txt file to freeze the list of dependencies rather than relying on the automatic version resolution from the packages.
I am using the latest master branch of the git repo https://github.com/celery/librabbitmq and installing librabbitmq==2.0.0 for Python 3.6 by following the instructions in the readme
Using the development version
You can clone the repository by doing the following:
$ git clone git://github.com/celery/librabbitmq.git
Then install it by doing the following:
$ cd librabbitmq
$ make install # or make develop
This works fine (after installing certain binaries for c compliation in the OS), but when I then make a small a+b add task and call it with add.delay(2,2) it fails with the following error. I looked up and saw that Celery 4 uses json as serializer, so clearly it is not because if pickle serialization
Changing from librabbitmq to pyamqp broker works normally
Same exact situation in both MacOS and Ubuntu 16
[2018-04-30 23:40:02,956: CRITICAL/MainProcess] Unrecoverable error:
SystemError(' returned a result with an error set',) Traceback (most
recent call last): File
"/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/kombu/messaging.py",
line 624, in _receive_callback
return on_m(message) if on_m else self.receive(decoded, message) File
"/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/consumer/consumer.py",
line 570, in on_task_received
callbacks, File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/strategy.py",
line 145, in task_message_handler
handle(req) File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/worker.py",
line 221, in _process_task_sem
return self._quick_acquire(self._process_task, req) File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/kombu/async/semaphore.py",
line 62, in acquire
callback(*partial_args, **partial_kwargs) File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/worker.py",
line 226, in _process_task
req.execute_using_pool(self.pool) File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/request.py",
line 531, in execute_using_pool
correlation_id=task_id, File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/concurrency/base.py",
line 155, in apply_async
**options) File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/billiard/pool.py",
line 1486, in apply_async
self._quick_put((TASK, (result._job, None, func, args, kwds))) File
"/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/concurrency/asynpool.py",
line 813, in send_job
body = dumps(tup, protocol=protocol) TypeError: can't pickle memoryview objects
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File
"/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/worker.py",
line 203, in start
self.blueprint.start(self) File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/bootsteps.py",
line 119, in start
step.start(parent) File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/bootsteps.py",
line 370, in start
return self.obj.start() File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/consumer/consumer.py",
line 320, in start
blueprint.start(self) File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/bootsteps.py",
line 119, in start
step.start(parent) File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/consumer/consumer.py",
line 596, in start
c.loop(*c.loop_args()) File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/loops.py",
line 88, in asynloop
next(loop) File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/kombu/async/hub.py",
line 354, in create_loop
cb(*cbargs) File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/kombu/transport/base.py",
line 236, in on_readable
reader(loop) File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/kombu/transport/base.py",
line 218, in _read
drain_events(timeout=0) File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/librabbitmq-2.0.0-py3.6-macosx-10.6-intel.egg/librabbitmq/init.py",
line 227, in drain_events
self._basic_recv(timeout) SystemError: returned a result with an error set
This library is not recommended to use as rabbitmq broker with celery. Instead please try py-amqp. this is more maintained and less buggy.
today I've updated my parseconsole.exe and I can't deploy my code when yesterday I could.
I've try to do another develop but the same error happens, this is the trace:
> parse new WebContent
Creating directory C:\Users\NOTEBOOK\Documents\docs\Parse\Works_pars
e_luna\Prueba_node\WebContent\config
Creating config file C:\Users\NOTEBOOK\Documents\docs\Parse\Works_pa
rse_luna\Prueba_node\WebContent\config\global.json
Creating directory C:\Users\NOTEBOOK\Documents\docs\Parse\Works_pars
e_luna\Prueba_node\WebContent\cloud
Writing out sample file C:\Users\NOTEBOOK\Documents\docs\Parse\Works
_parse_luna\Prueba_node\WebContent\cloud\main.js
Creating directory C:\Users\NOTEBOOK\Documents\docs\Parse\Works_pars
e_luna\Prueba_node\WebContent\public
Writing out sample file C:\Users\NOTEBOOK\Documents\docs\Parse\Works
_parse_luna\Prueba_node\WebContent\public\index.html
Email: xxxxxx
Password:
1: prueba
2: prueba_cloud
Select an App: 2
PS C:\Users\NOTEBOOK\Documents\docs\Parse\Works_parse_luna\Prueba_no
de> cd WebContent
PS C:\Users\NOTEBOOK\Documents\docs\Parse\Works_parse_luna\Prueba_no
de\WebContent> parse deploy
Uploading source files
Traceback (most recent call last):
File "<string>", line 6, in <module>
File ".\build\pyi.win32\parse\out00-PYZ.pyz\main", line 695, in main
File ".\build\pyi.win32\parse\out00-PYZ.pyz\main", line 206, in handle_deploy
File ".\build\pyi.win32\parse\out00-PYZ.pyz\parse", line 332, in deploy
File ".\build\pyi.win32\parse\out00-PYZ.pyz\parse", line 231, in upload_source
_files
File ".\build\pyi.win32\parse\out00-PYZ.pyz\parse", line 39, in post
File ".\build\pyi.win32\parse\out00-PYZ.pyz\urllib2", line 126, in urlopen
File ".\build\pyi.win32\parse\out00-PYZ.pyz\urllib2", line 391, in open
File ".\build\pyi.win32\parse\out00-PYZ.pyz\urllib2", line 409, in _open
File ".\build\pyi.win32\parse\out00-PYZ.pyz\urllib2", line 369, in _call_chain
File ".\build\pyi.win32\parse\out00-PYZ.pyz\urllib2", line 1178, in https_open
File ".\build\pyi.win32\parse\out00-PYZ.pyz\urllib2", line 1142, in do_open
File ".\build\pyi.win32\parse\out00-PYZ.pyz\httplib", line 914, in request
File ".\build\pyi.win32\parse\out00-PYZ.pyz\httplib", line 954, in _send_request
File ".\build\pyi.win32\parse\out00-PYZ.pyz\httplib", line 759, in send
File ".\build\pyi.win32\parse\out00-PYZ.pyz\ssl", line 203, in sendall
File ".\build\pyi.win32\parse\out00-PYZ.pyz\ssl", line 174, in send
TypeError: write() argument 1 must be string or read-only buffer, not bytearray
thaks for your help
After reading https://stackoverflow.com/a/26982677/2211980, it seems that Parse has released a new version. I've just downloaded it from: https://www.parse.com/downloads/windows/console/parse.zip.
There is a deprecated warning but after seeing exactly the same errors as yours yesterday, I can now deploy, thankfully.
I know this thread is old but it may help newcomers in the future,
Parse announed that they made a change that require everyone to manually download the newest version of their CLI as of today (7.2.2015) the current version is 2.1.2 you can download it directly from the Parse.com
or if you like I made a repo containing the files for it, CLI Repo
Hope it helps.
Now there is a new version that is working.
Parse CLI v3.0.2
https://github.com/ParsePlatform/parse-cli/releases/
(Cross post from Pinax Users Google group -- I figured it would be useful to have this question on here as well.)
I'm a bit stuck. My projects work fine when I run them on the original machine I create them on, but I get an error when I move them to another machine. I'm trying to use hosted version control to manage a project between two computers, but it only works on the machine I create it on. I'm just trying to get the project base "zero" to run on both machines. Here's the console output I'm getting when I run manage.py runserver:
Traceback (most recent call last):
File "manage.py", line 27, in <module>
execute_from_command_line()
File "/home/loren/pinax-env/lib/python2.6/site-packages/django/core/management/__init__.py", line 352, in execute_from_command_line
utility.execute()
File "/home/loren/pinax-env/lib/python2.6/site-packages/django/core/management/__init__.py", line 306, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/loren/pinax-env/lib/python2.6/site-packages/django/core/management/base.py", line 192, in run_from_argv
self.execute(*args, **options.__dict__)
File "/home/loren/pinax-env/lib/python2.6/site-packages/django/core/management/base.py", line 210, in execute
translation.activate('en-us')
File "/home/loren/pinax-env/lib/python2.6/site-packages/django/utils/translation/__init__.py", line 73, in activate
return real_activate(language)
File "/home/loren/pinax-env/lib/python2.6/site-packages/django/utils/translation/__init__.py", line 43, in delayed_loader
return g['real_%s' % caller](*args, **kwargs)
File "/home/loren/pinax-env/lib/python2.6/site-packages/django/utils/translation/trans_real.py", line 209, in activate
_active[currentThread()] = translation(language)
File "/home/loren/pinax-env/lib/python2.6/site-packages/django/utils/translation/trans_real.py", line 198, in translation
default_translation = _fetch(settings.LANGUAGE_CODE)
File "/home/loren/pinax-env/lib/python2.6/site-packages/django/utils/translation/trans_real.py", line 181, in _fetch
app = getattr(__import__(appname[:p], {}, {}, [appname[p+1:]]), appname[p+1:])
AttributeError: 'module' object has no attribute 'messages'
I expect that this has something to do with my pythonpath, but I'm just too new to this to know where to look. I dug up this thread, but I couldn't find a solution in it:
Django failing to find apps
Help would be great!