using celery in airflow - celery

Question

I am new to airflow, for now I find out airflow is using celery to schedule its tasks. To run airflow, I need to run command 'airflow worker' which will start celery. However, there is always a bug here. Since I have searched in Internet, most problem happen to celery.py which write by user themselves. I use celery just by start airflow. So it is a little bit different.
Anyone could help me? Below is the screenshot of the bug.
airflow#linux-test:~$ airflow worker
[2018-06-22 07:29:04,068] {__init__.py:57} INFO - Using executor CeleryExecutor
[2018-06-22 07:29:04,125] {driver.py:124} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2018-06-22 07:29:04,146] {driver.py:124} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
-------------- celery#linux-test v4.2.0 (windowlicker)
---- **** -----
--- * *** * -- Linux-4.15.0-22-generic-x86_64-with-Ubuntu-18.04-bionic 2018-06-22 07:29:04
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: airflow.executors.celery_executor:0x7f2267122310
- ** ---------- .> transport: amqp://airflow:**#localhost:5672/airflow
- ** ---------- .> results: postgresql://airflow:**#localhost:5432/airflow
- *** --- * --- .> concurrency: 16 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> default exchange=default(direct) key=default
[2018-06-22 07:29:04,630] {__init__.py:57} INFO - Using executor CeleryExecutor
[2018-06-22 07:29:04,689] {driver.py:124} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2018-06-22 07:29:04,715] {driver.py:124} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
Starting flask
[2018-06-22 07:29:04,858] {_internal.py:88} INFO - * Running on http://0.0.0.0:8793/ (Press CTRL+C to quit)
[2018-06-22 07:29:06,122: ERROR/ForkPoolWorker-1] Pool process <celery.concurrency.asynpool.Worker object at 0x7f22648c8e10> error: TypeError("Required argument 'object' (pos 1) not found",)
Traceback (most recent call last):
File "/home/airflow/.local/lib/python2.7/site-packages/billiard/pool.py", line 289, in __call__
sys.exit(self.workloop(pid=pid))
File "/home/airflow/.local/lib/python2.7/site-packages/billiard/pool.py", line 347, in workloop
req = wait_for_job()
File "/home/airflow/.local/lib/python2.7/site-packages/billiard/pool.py", line 447, in receive
ready, req = _receive(1.0)
File "/home/airflow/.local/lib/python2.7/site-packages/billiard/pool.py", line 419, in _recv
return True, loads(get_payload())
File "/home/airflow/.local/lib/python2.7/site-packages/billiard/common.py", line 107, in pickle_loads
return load(BytesIO(s))
TypeError: Required argument 'object' (pos 1) not found
[2018-06-22 07:29:06,127: ERROR/MainProcess] Process 'ForkPoolWorker-1' pid:18839 exited with 'exitcode 1'

Answer 1

Uninstalling librabbitmq worked for me : pip uninstall librabbitmq. I didn't understand very well why, but apparently, there's some optimization on that library that made the thing fail. Here's the answer I found on some website (I had to translate the page, thus my inability to understand well the solution)
Hope it helps