Apache Airflow - Scheduler Slowness - scheduler

Question

We are using Airflow 1.7.3 with CeleryExecutor. Airflow scheduler is setup as systemd service with --num-run set to 10 to stop it and restart every 10th run (as suggested here).
We noticed that at every 9th loop of scheduler takes considerably loger time (about 160 seconds or more) as compared to the regular loop of ~16 seconds. As per the logs, this is the loop when scheduler fills the DagBag by refreshing all the dags. This time gets increased as number of dags/tasks increase in our airflow installation.
Most of our tasks are very small and takes just about few seconds to run but they get stuck in the "undefined" state an does not get queued, while scheduler is busy "filling up dagbag". In the meantime celery workers are sitting idle. We have tried the following:
increased celeryd_concurrency (which gave us ability to send more tasks to workers)
increased non_pooled_task_slot_count (so that more tasks can get queued)
also increased parallelism and dag_concurrency
All these measures allows to launch more tasks, only if scheduler queues them, which it not effectively does when it goes to that refreshh stage. Here are the timings for each scheduler loop:
[2016-11-07 23:18:28,106] {jobs.py:680} INFO - Starting the scheduler
[2016-11-07 23:21:26,515] {jobs.py:744} INFO - Loop took: 16.422769 seconds
[2016-11-07 23:21:46,186] {jobs.py:744} INFO - Loop took: 16.058172 seconds
[2016-11-07 23:22:02,800] {jobs.py:744} INFO - Loop took: 14.410493 seconds
[2016-11-07 23:22:21,310] {jobs.py:744} INFO - Loop took: 16.275255 seconds
[2016-11-07 23:22:41,470] {jobs.py:744} INFO - Loop took: 17.93543 seconds
[2016-11-07 23:22:59,176] {jobs.py:744} INFO - Loop took: 15.484449 seconds
[2016-11-07 23:23:17,455] {jobs.py:744} INFO - Loop took: 16.130971 seconds
[2016-11-07 23:23:35,948] {jobs.py:744} INFO - Loop took: 16.311113 seconds
[2016-11-07 23:23:55,043] {jobs.py:744} INFO - Loop took: 16.830728 seconds
[2016-11-07 23:26:57,044] {jobs.py:744} INFO - Loop took: 179.613778 seconds
[2016-11-07 23:27:09,328] {jobs.py:680} INFO - Starting the scheduler
[2016-11-07 23:29:57,988] {jobs.py:744} INFO - Loop took: 16.881139 seconds
[2016-11-07 23:30:17,584] {jobs.py:744} INFO - Loop took: 17.021958 seconds
[2016-11-07 23:30:36,062] {jobs.py:744} INFO - Loop took: 16.148552 seconds
[2016-11-07 23:30:56,975] {jobs.py:744} INFO - Loop took: 18.532384 seconds
[2016-11-07 23:31:16,214] {jobs.py:744} INFO - Loop took: 16.907037 seconds
[2016-11-07 23:31:39,060] {jobs.py:744} INFO - Loop took: 15.637057 seconds
[2016-11-07 23:31:56,231] {jobs.py:744} INFO - Loop took: 15.003683 seconds
[2016-11-07 23:32:13,618] {jobs.py:744} INFO - Loop took: 15.215657 seconds
[2016-11-07 23:32:35,738] {jobs.py:744} INFO - Loop took: 19.938704 seconds
[2016-11-07 23:35:33,905] {jobs.py:744} INFO - Loop took: 176.030812 seconds
[2016-11-07 23:35:45,908] {jobs.py:680} INFO - Starting the scheduler
Questions:
does --num-run required anymore in 1.7.1.3 version (as mentioned in pitfalls: https://cwiki.apache.org/confluence/display/AIRFLOW/Common+Pitfalls)? do we still have to restart the scheduler after every n number of runs?
increasing the max_threads value (to launch multiple scheduler thread) would help? I think defualt is 2.
Thanks for any help.