How to send logs to Cloudwatch from a background process running in AWS Fargate containers? - amazon-ecs

Question

I'm using Fargate. My container is running two processes. Celery worker in the background and Django in the foreground. The foreground process emits logs to stdout, hence AWS takes care of sending Django logs to concerned Cloudwatch Log Group and Stream.
Since its running in the background, how do send the celery worker's logs to (a different Log Stream within same Log Group) Cloudwatch?

Answer 1

If there's no way to move the second process to the separate container and log it as usual, you may install awslogs package to the container and set it up to read background process' log files and send content to CloudWatch.
But I'd not recommend such approach.

Answer 2

Again this is not necessarily a Fargate based issue or question. For logging in celery check this out http://docs.celeryproject.org/en/latest/userguide/tasks.html#logging.
The worker won’t update the redirection if you create a logger instance somewhere in your task or task module.
If you want to redirect sys.stdout and sys.stderr to a custom logger you have to enable this manually, for example:
import sys
logger = get_task_logger(__name__)
#app.task(bind=True)
def add(self, x, y):
old_outs = sys.stdout, sys.stderr
rlevel = self.app.conf.worker_redirect_stdouts_level
try:
self.app.log.redirect_stdouts_to_logger(logger, rlevel)
print('Adding {0} + {1}'.format(x, y))
return x + y
finally:
sys.stdout, sys.stderr = old_outs
And for logging with Fargate, i would use the awslogs driver. Below is how you configure as documented here: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html
If using the console:
Or this if in cloudformation template:

Answer 3

Like #OK999 said, Celery is designed to swallow logs whether its on Fargate or not. We ended up using a Django LOGGING config like:
LOGGING = {
'version': 1,
# This only "disables" but the loggers don't propagate
# 'disable_existing_loggers': False,
...
'handlers': {
'console': {
'level': env.str('LOGGING_LEVEL'),
'class': 'logging.StreamHandler',
'formatter': 'verbose',
},
},
'loggers': {
...
# celery won't route logs to console without this
'celery': {
# filtered at the handler
'level': logging.DEBUG,
'handlers': ['console'],
},
...
We had to make this change long before transitioning to Fargate.