Torque PBS jobs going to the debug queue - hpc

Question

On my new job, I administer a cluster that uses torque as a resource manager and maui as the scheduler.
Currently, I am facing this repeated problem where a specific users jobs are always sent to the debug queue. Here is a list of the active queues on the system:
Queue Memory CPU Time Walltime Node Run Que Lm State
---------------- ------ -------- -------- ---- --- --- -- -----
debug -- -- 00:20:00 -- 0 0 12 E R
intel -- -- -- -- 0 0 -- E R
medium -- -- 72:00:00 -- 0 0 12 E R
bighuge -- -- -- -- 0 0 -- E R
long -- -- -- -- 0 0 12 E R
----- -----
0 0
The Wall-time for jobs submitted by the user is in hours, so I am puzzled why its being sent to the debug queue.
Also, here is a output of the tracejob:
04/08/2016 15:46:48 S enqueuing into intel, state 1 hop 1
04/08/2016 15:46:48 S dequeuing from intel, state QUEUED
04/08/2016 15:46:48 S enqueuing into debug, state 1 hop 1
04/08/2016 15:46:48 S Job Queued at request of dawn#cm01, owner = dawn#cm01, job name = run01_submit.script, queue =
debug
04/08/2016 15:46:49 S Job Run at request of root#cm01
04/08/2016 15:46:49 S child reported success for job after 0 seconds (dest=n20), rc=0
04/08/2016 15:46:49 S preparing to send 'b' mail for job 15631.cm01 to dawn#cm01 (---)
04/08/2016 15:46:49 S Not sending email: User does not want mail of this type.
04/08/2016 15:46:49 S obit received - updating final job usage info
04/08/2016 15:46:49 S job exit status 1 handled
04/08/2016 15:46:49 S preparing to send 'e' mail for job 15631.cm01 to dawn#cm01 (Exit_status=1
04/08/2016 15:46:49 S Not sending email: User does not want mail of this type.
04/08/2016 15:46:49 S Exit_status=1 resources_used.cput=00:00:00 resources_used.mem=0kb resources_used.vmem=0kb
resources_used.walltime=00:00:00
04/08/2016 15:46:49 S on_job_exit task assigned to job
04/08/2016 15:46:49 S req_jobobit completed
04/08/2016 15:46:49 S JOB_SUBSTATE_EXITING
04/08/2016 15:46:49 S JOB_SUBSTATE_STAGEOUT
04/08/2016 15:46:49 S about to copy stdout/stderr/stageout files
04/08/2016 15:46:49 S JOB_SUBSTATE_STAGEOUT
04/08/2016 15:46:49 S JOB_SUBSTATE_STAGEDEL
04/08/2016 15:46:49 S JOB_SUBSTATE_EXITED
04/08/2016 15:46:49 S JOB_SUBSTATE_COMPLETE
04/08/2016 15:50:54 S Request invalid for state of job COMPLETE
04/08/2016 15:51:00 S Request invalid for state of job COMPLETE
04/08/2016 15:51:49 S dequeuing from debug, state COMPLETE
A workaround now is to manually change the assigned queue for the jobs using the qalter command.
Any ideas?

Answer 1

Because the job immediately jumps from the intel queue to debug, I'd suspect you have automatic routing configured, either in qmgr or in Maui. If the intel queue is configured as a routing queue, that would explain it.
Run qmgr -c "print queue intel" to check that.
If it's not a routing queue, you can probably increase loglevel to better see what's going on in the pbs_server logs.
When I create a routing queue that way, I get the same type of tracejob output when submitting a job:
05/20/2016 20:04:05.439 S enqueuing into route, state 1 hop 1
05/20/2016 20:04:05.440 S dequeuing from route, state QUEUED
05/20/2016 20:04:05.440 S enqueuing into test, state 1 hop 1
05/20/2016 20:04:05.737 S Job Run at request of root#testserver
Otherwise, examine the Maui config and logs for clues.