DiskSpace quota exception when running spark action in oozie - scala

Question

I am trying to run a spark action in oozie. My spark job fails with the below error :
The DiskSpace quota of /user/nidhin is exceeded: quota = 10737418240 B = 10 GB but diskspace consumed = 10973426088 B = 10.22 GB
I added the staging dir property in my oozie workflow and pointed to a HDFS directory other than home which has TBs of space , even then i get the same error.
<action name="CheckErrors" cred="hcat">
<spark xmlns="uri:oozie:spark-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<master>yarn</master>
<mode>cluster</mode>
<name>JobName</name>
<class>com.nidhin.util.CheckErrorsRaw
</class>
<jar>${processor_jar}</jar>
<spark-opts>--queue=${queue_name}
--num-executors 0
--conf spark.dynamicAllocation.enabled=true
--conf spark.shuffle.service.enabled=true
--conf spark.yarn.stagingDir=${hdfs_data_base_dir}
</spark-opts>
<arg>${load_dt}</arg>
</spark>
<ok to="End" />
<error to="Kill" />
</action>
${hdfs_data_base_dir} is /tenants/proj/ directory in HDFS and has TBs worth of space in it.
When i look into the spark jobtracker UI , the property is properly reflected.
spark.yarn.stagingDir hdfs://tenants/proj/
How do fix this error and point to the above mentioned stagingDir ?