Struggling with Tornado - Celery integration - celery

I am in the process of building a Tornado Web server which requires to perform some blocking tasks such as zipping video files, etc which are time consuming. Ideally, I would like to hand off these tasks
to a subprocess utility like Celery and inform the Client accordingly. So, I took this code from GitHub & modified it to get better understanding. I am using Ubuntu 12.04 LTS and here's what I have done so far:
Installed Tornado and working great on its own.
Installed RabbitMQ (apt-get install rabbitmq-server).
Installed Tornado Celery (downloaded and unzipped the tar file).
Executed python -m pip install tornado-celery
Borrowed 2 example python files from above Github Repo and modified it.
My Tasks snippet code (myTornadoTasks.py) is:
import os
import time
from datetime import datetime
from celery import Celery
from celery import task
celery = Celery("myTornadoTasks", broker="amqp://")
celery.conf.CELERY_RESULT_BACKEND = os.environ.get('CELERY_RESULT_BACKEND', 'amqp')
#celery.task
def add(x, y):
return int(x) + int(y)
..... some more code
#celery.task
def error(msg):
raise Exception(msg)
if __name__ == "__main__":
celery.start()
I start the tasks using celery worker -A myTornadoTasks &. Here everything works fine and I see 24 threads running.
My Tornado Celery snippet code (myTornadoCelery.py) is:
from tornado import gen
from tornado import ioloop
from tornado.web import asynchronous, RequestHandler, Application
import myTornadoTasks
import tcelery tcelery.setup_nonblocking_producer()
class AsyncHandler(RequestHandler):
#asynchronous
def get(self):
myTornadoTasks.sleep.apply_async(args=[3], callback=self.on_result)
def on_result(self, response):
self.write(str(response.result))
self.finish()
... some more code
application = Application([
(r"/async-sleep", AsyncHandler),
(r"/gen-async-sleep", GenAsyncHandler),
(r"/gen-async-sleep-add", GenMultipleAsyncHandler),
])
if __name__ == "__main__":
application.listen(8887)
ioloop.IOLoop.instance().start()
I start Tornado using python -m tcelery --app=myTornadoCelery --address=0.0.0.0
I get an error AttributeError: 'module' object has no attribute 'celery'
Questions:
What does the above error mean? What am I missing?
Did I miss any steps here? Where is RabbitMQ invoked?
How do I increase/decrease the number of Celery workers from 24?
Thanks.
Updated with Error:
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/local/lib/python2.7/dist-packages/tcelery/__main__.py", line 62, in <module>
main()
File "/usr/local/lib/python2.7/dist-packages/tcelery/__main__.py", line 56, in main
cmd.execute_from_commandline()
File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 309, in execute_from_commandline
argv = self.setup_app_from_commandline(argv)
File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 469, in setup_app_from_commandline
self.app = self.find_app(app)
File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 489, in find_app
return find_app(app, symbol_by_name=self.symbol_by_name)
File "/usr/local/lib/python2.7/dist-packages/celery/app/utils.py", line 240, in find_app
found = sym.celery
AttributeError: 'module' object has no attribute 'celery'

Related

How to use the ibm_boto3 in python

Who has the same problem?
I want to store data in cos, but cannot use the ibm_boto3 on my machine.
To be sure to check with a sample, I used the code from the sample from this ibm-cos-sdk github.
Installed
pip3 freeze
backports.functools-lru-cache==1.5
botocore==1.12.28
docutils==0.14
futures==3.1.1
ibm-cos-sdk==2.3.2
ibm-cos-sdk-core==2.3.2
ibm-cos-sdk-s3transfer==2.3.2
-e git://github.com/boto/jmespath.git#1c9c35cf681b6605d8629e5ce8865221a4fd2a30#egg=jmespath
mock==1.3.0
nose==1.3.3
pbr==5.0.0
python-dateutil==2.7.3
s3transfer==0.1.13
six==1.11.0
urllib3==1.23
Here is my cli result and as you can see the ibm_boto3 is not found.
python3 test.py
Traceback (most recent call last):
File "test.py", line 1, in <module>
import ibm_boto3
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ibm_boto3/__init__.py", line 16, in <module>
from ibm_boto3.session import Session
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ibm_boto3/session.py", line 27, in <module>
import ibm_botocore.session
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ibm_botocore/session.py", line 37, in <module>
import ibm_botocore.credentials
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ibm_botocore/credentials.py", line 36, in <module>
import requests
ModuleNotFoundError: No module named 'requests'
Yeah, it looks like requests somehow fell out of the requirements file in the latest release. The team is patching it and will release an update soon.
In the meantime, you can manually install the package in your environment with pip3 install requests or by manually adding it to the requirements.txt file:
echo "requests==2.18.0" >> path/to/requirements.txt

Issue running psycopg2 inside AWS Lambda Function

I'm getting the following error when trying to run psycopg2 in a AWS Lambda:
/var/task/functions/../vendored/psycopg2/_psycopg.so: ELF file's phentsize not the expected size: ImportError
Traceback (most recent call last):
File "/var/task/functions/refresh_mv.py", line 64, in execute
session = SessionFactoryGraphQL.get_session(app=item['app'])
File "/var/task/lib/session_factory.py", line 22, in get_session
engine = create_engine(conn_string, poolclass=NullPool)
File "/var/task/functions/../vendored/sqlalchemy/engine/__init__.py", line 387, in create_engine
return strategy.create(*args, **kwargs)
File "/var/task/functions/../vendored/sqlalchemy/engine/strategies.py", line 80, in create
dbapi = dialect_cls.dbapi(**dbapi_args)
File "/var/task/functions/../vendored/sqlalchemy/dialects/postgresql/psycopg2.py", line 554, in dbapi
import psycopg2
File "/var/task/functions/../vendored/psycopg2/__init__.py", line 50, in <module>
from psycopg2._psycopg import ( # noqa
ImportError: /var/task/functions/../vendored/psycopg2/_psycopg.so: ELF file's phentsize not the expected size
The weird thing is: everything was working fine until yesterday (for more than 5 months), and suddenly stopped working. None of the libraries has been updated.
I tried to build from scratch, as in https://github.com/jkehler/awslambda-psycopg2, but still having the same error.
Can someone help me with it?
The problem is in the latest version of serverless framework. I assume that you are using serverless to deploy your lambda function.
serverless remove
npm install serverless#1.20.2 -g
This should work.

builtins.ImportError: cannot import name 'Empty'

While working the examples in a book on Parallel Programming in Python, I encountered the following error in code that uses the multiprocessing queue:
File "C:\pyDev\multiproc\queue-test.py", line 4, in <module>
queue = multiprocessing.Queue()
File "C:\Anaconda3\Lib\multiprocessing\context.py", line 100, in Queue
from .queues import Queue
File "C:\Anaconda3\Lib\multiprocessing\queues.py", line 20, in <module>
from queue import Empty, Full
builtins.ImportError: cannot import name 'Empty'
After some experimentation, I determined that all it takes to generate this error is the following code (which is the entirety of the queue-test.py file mentioned in the error message above).
import multiprocessing
if __name__ == "__main__":
queue = multiprocessing.Queue()
I'm running Python 3.5.1 with Anaconda 4.1.0 on a machine with Windows 7. I have ported the code above and the example from the book to another machine with Python 2.7.11 with Anaconda 2.5.0, with Windows 10 and it works fine on that machine.
Thinking that perhaps there was a corrupt file or other issue with my Python installation, I tried reinstalling Anaconda and it did not help. I performed a Google search and did not find this particular error. Other stackoverflow postings such as the one found here: ImportError: Cannot import name X don't appear to be relevent because this involves part of the Python standard library and not code and classes I wrote myself.
I encountered almost the same error in my code and finally figure out where went wrong. Hope it will help you somehow.
I named my python script as "queue.py", and then I run it, I got error info. below just like yours:
Traceback (most recent call last):
File "F:/02_Coding/01_Projects/PyHomeWork/Day23_Process/queue.py", line 19, in <module>
queue = multiprocessing.Queue()
File "E:\02_CodingSoftware\02_Installed\Anaconda3\lib\multiprocessing\context.py", line 101, in Queue
from .queues import Queue
File "E:\02_CodingSoftware\02_Installed\Anaconda3\lib\multiprocessing\queues.py", line 20, in <module>
from queue import Empty, Full
ImportError: cannot import name 'Empty'
I use the below method to create a Queue:
if __name__ =="__main__":
queue = multiprocessing.Queue()
Finally, I notice that I should not named the file in a name of "queue.py", it seems it will cause a misleading to python when it interpreter the script. And after I rename the script, the error is gone. What a stupid mistake, LoL.
So my suggestion is maybe you can check whether under your folder is there a script named as "queue.py" or any custom module would conflict with that in libraries.
Hope you can solve the issue. Best

unregistered task type import errors in celery

I'm having headaches with getting celery to work with my folder structure. Note I am using virtualenv but it should not matter.
cive /
celery_app.py
__init__.py
venv
framework /
tasks.py
__init__.py
civeAPI /
files tasks.py need
cive is my root project folder.
celery_app.py:
from __future__ import absolute_import
from celery import Celery
app = Celery('cive',
broker='amqp://',
backend='amqp://',
include=['cive.framework.tasks'])
# Optional configuration, see the application user guide.
app.conf.update(
CELERY_TASK_RESULT_EXPIRES=3600,
)
if __name__ == '__main__':
app.start()
tasks.py (simplified)
from __future__ import absolute_import
#import other things
#append syspaths
from cive.celery_app import app
#app.task(ignore_result=False)
def start(X):
# do things
def output(X):
# output files
def main():
for d in Ds:
m = []
m.append( start.delay(X) )
output( [n.get() for n in m] )
if __name__ == '__main__':
sys.exit(main(sys.argv[1:]))
I then start workers via (outside root cive dir)
celery -A cive worker --app=cive.celery_app:app -l info
which seems to work fine, loading the workers and showing
[tasks]
. cive.framework.tasks.start_sessions
But when I try to run my tasks.py via another terminal:
python tasks.py
I get the error:
Traceback (most recent call last):
File "tasks.py", line 29, in <module>
from cive.celery_app import app
ImportError: No module named cive.celery_app
If I rename the import to:
from celery_app import app #without the cive.celery_app
I can eventually start the script but celery returns error:
Received unregistered task of type 'cive.start_sessions'
I think there's something wrong with my imports or config but I can't say what.
So this was a python package problem, not particularly a celery issue. I found the solution by looking at How to fix "Attempted relative import in non-package" even with __init__.py .
I've never even thought about this before, but I wasn't running python in package mode. The solution is cd'ing out of your root project directory, then running python as a package (note there is no .py after tasks):
python -m cive.framework.tasks
Now when I run the celery task everything works.

pyql is not accessable after its installation from native for Ubuntu14.04 python2.7

After installing pyql from the package source(according to its own wizard described in this file) on Ubuntu 14.04 into /usr/local/lib/python2.7/dist-packages/ folder, all tests finished with success. This folder includes all additional installed packages, which are accessible, from the python using native import command.
But, this specific installation python don’t see, and I cannot import anything from it.
Have you any idea, what I need to define in addition?
Thanks,
Yigal B.
There are results of my sys.path:
import sys
print(sys.path)
['/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu',
'/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages',
'/usr/lib/python2.7/dist-packages',
'/usr/lib/python2.7/dist-packages/PILcompat',
'/usr/lib/python2.7/dist-packages/gst-0.10',
'/usr/lib/python2.7/dist-packages/gtk-2.0',
'/usr/lib/pymodules/python2.7',
'/usr/lib/python2.7/dist-packages/wx-2.8-gtk2-unicode']
python -c 'from quantlib.settings import __quantlib_version__; print __quantlib_version__'
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named quantlib.settings
Can you post the content of you sys.path? And the output of
python -c 'from quantlib.settings import __quantlib_version__; print __quantlib_version__'