Celery worker not reading pyrogram session file - celery

Question

I'm trying to execute a pyrogram function via a celery task (scheduling etc.)
The function works when ran via shell:
from app_name.users.tasks import establish_session
establish_session()
Sending it to celery via establish_session.delay() is where the problem arises.
The exact same function when executed via celery fails to read the session required session file.
I've confirmed that the session file is seen in both methods and have permissions for os.R_OK, os.W_OK, os.F_OK.
users.tasks
#shared_task
def establish_session():
from utils.telegram import get_new_session
user_bot = get_new_session()
print(user_bot)
utils.telegram
def get_new_session():
import os
cwd = os.getcwd()
print(cwd)
print(os.access('user.session', os.R_OK)) # Check for read access
print(os.access('user.session', os.W_OK)) # Check for write access
print(os.access('user.session', os.X_OK)) # Check for execution access
print(os.access('user.session', os.F_OK)) # Check for existence of file
user_bot = Client("user", api_id=ID, api_hash=HASH)
user_bot.start()
user_bot.stop()
return user_bot
Difference in outputs:
establish_session()
INFO 2021-01-23 18:07:21,379 connection Connecting...
INFO 2021-01-23 18:07:21,382 connection Connected! Production DC5 - IPv4 - TCPAbridgedO
INFO 2021-01-23 18:07:21,383 session NetworkTask started
INFO 2021-01-23 18:07:21,435 msg_id Time synced: 2021-01-23 10:07:21.439058 UTC
INFO 2021-01-23 18:07:21,439 session NextSaltTask started
INFO 2021-01-23 18:07:21,439 session Next salt in 33m 13s (at 2021-01-23 18:40:35)
INFO 2021-01-23 18:07:21,524 session Session initialized: Layer 122
INFO 2021-01-23 18:07:21,524 session Device: CPython 3.8.6 - Pyrogram 1.1.10
INFO 2021-01-23 18:07:21,524 session System: Linux 5.8.0-33-generic (EN)
INFO 2021-01-23 18:07:21,524 session Session started
INFO 2021-01-23 18:07:21,540 session PingTask started
INFO 2021-01-23 18:07:21,620 dispatcher Started 6 HandlerTasks
INFO 2021-01-23 18:07:21,632 syncer Synced "user" in 11.2832 ms
INFO 2021-01-23 18:07:21,639 syncer Synced "user" in 7.18236 ms
INFO 2021-01-23 18:07:21,640 dispatcher Stopped 6 HandlerTasks
INFO 2021-01-23 18:07:21,640 session PingTask stopped
INFO 2021-01-23 18:07:21,640 session NextSaltTask stopped
INFO 2021-01-23 18:07:21,640 connection Disconnected
INFO 2021-01-23 18:07:21,641 session NetworkTask stopped
INFO 2021-01-23 18:07:21,641 session Session stopped
vs
establish_session.delay()
[2021-01-23 18:07:35,832: INFO/ForkPoolWorker-2] Start creating a new auth key on DC2
[2021-01-23 18:07:35,832: INFO/ForkPoolWorker-2] Connecting...
[2021-01-23 18:07:36,105: INFO/ForkPoolWorker-2] Connected! Production DC2 - IPv4 - TCPAbridgedO
[2021-01-23 18:07:37,592: INFO/ForkPoolWorker-2] Done auth key exchange:
[2021-01-23 18:07:37,592: INFO/ForkPoolWorker-2] Disconnected
[2021-01-23 18:07:37,605: WARNING/ForkPoolWorker-2] Pyrogram v1.1.10, Copyright (C) 2017-2021 Dan <https://github.com/delivrance>
[2021-01-23 18:07:37,605: WARNING/ForkPoolWorker-2] Licensed under the terms of the GNU Lesser General Public License v3 or later (LGPLv3+)
[2021-01-23 18:07:37,605: INFO/ForkPoolWorker-2] Connecting...
[2021-01-23 18:07:37,875: INFO/ForkPoolWorker-2] Connected! Production DC2 - IPv4 - TCPAbridgedO
[2021-01-23 18:07:37,875: INFO/ForkPoolWorker-2] NetworkTask started
[2021-01-23 18:07:38,459: INFO/ForkPoolWorker-2] Time synced: 2021-01-23 10:07:38.353224 UTC
[2021-01-23 18:07:38,732: INFO/ForkPoolWorker-2] NextSaltTask started
[2021-01-23 18:07:38,732: INFO/ForkPoolWorker-2] Next salt in 44m 58s (at 2021-01-23 18:52:37)
[2021-01-23 18:07:39,096: INFO/ForkPoolWorker-2] Session initialized: Layer 122
[2021-01-23 18:07:39,096: INFO/ForkPoolWorker-2] Device: CPython 3.8.6 - Pyrogram 1.1.10
[2021-01-23 18:07:39,096: INFO/ForkPoolWorker-2] System: Linux 5.8.0-33-generic (EN)
[2021-01-23 18:07:39,096: INFO/ForkPoolWorker-2] Session started
[2021-01-23 18:07:39,099: WARNING/ForkPoolWorker-2] Enter phone number or bot token:
[2021-01-23 18:07:39,099: INFO/ForkPoolWorker-2] PingTask started
[2021-01-23 18:07:39,100: INFO/ForkPoolWorker-2] PingTask stopped
[2021-01-23 18:07:39,100: INFO/ForkPoolWorker-2] NextSaltTask stopped
[2021-01-23 18:07:39,100: INFO/ForkPoolWorker-2] Disconnected
[2021-01-23 18:07:39,101: INFO/ForkPoolWorker-2] NetworkTask stopped
[2021-01-23 18:07:39,101: INFO/ForkPoolWorker-2] Session stopped
Any assistance is greatly appreciated!

Answer 1

I did a lot of work to get pyrogram working under celery. It's not ideal but it works for my case. Maybe this could help you too.
I'm using the latest version of pyrogram(1.3.5) and celery(5.2.3)
# first need to create a client, save session file in memory
tg_client=Client(":memory:",APP_ID=123,APP_HASH="abc")
# create celery app
app = Celery('tasks', broker=BROKER)
#app.task
def some_task():
print(tg_client.get_me())
# define celery startup
def run_celery():
# pool must be threads
argv = [
"-A", "tasks", 'worker', '--loglevel=info',
"--pool=threads"]
app.worker_main(argv)
if __name__ == '__main__':
tg_client.start() # <-- I think you can also put it in the first line of `run_celery`
threading.Thread(target=run_celery, daemon=True).start()
idle()
celery_client.stop()
Key points are:
need to start celery worker in a different thread than main thread because pyrogram is async library, it relies on main thread while celery is blocking the main thread
celery pool must be threads or solo
Besides that, you can also use with in a task
#app.task
def some_task()
with tg_client():
print(tg_client.get_me())
some references:
https://github.com/pyrogram/pyrogram/issues/480
https://github.com/tgbot-collection/ytdlbot/blob/master/ytdlbot/tasks.py