Argo dag issue when submitting the workflow stuck in Container creating state - argo-workflows

Question

I have an argo dag set up where there is a dag based workflow submitting to argo.
The yaml structure is as follows:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: spark-argo-dag
namespace: argo
annotations:
workflows.argoproj.io/description: "Spark Argo Dag pipelines"
spec:
entrypoint: spark-argo-dag
arguments:
parameters:
- name: date
value: 20220404
- name: region
value: us-east
- name: asset_class
value: Equity
templates:
- name: spark-argo-dag
dag:
tasks:
- name: ingest-vendor-data
template: ingestStage
arguments:
parameters:
- name: region
value: '{{workflow.parameters.region}}'
- name: asset_class
value: '{{workflow.parameters.asset_class}}'
- name: transform-data
dependencies: [ingest-vendor-data]
template: transformData
arguments:
parameters:
- name: file_date
value: '{{workflow.parameters.date}}'
- name: save-data
dependencies: [transform-data]
template: saveData
- name: ingestStage
inputs:
parameters:
- name: region
- name: asset_class
container:
command:
args: [
"/bin/sh",
"-c",
"/opt/spark/bin/spark-submit \
--master k8s://https://kubernetes.default.svc \
--deploy-mode cluster \
--conf spark.kubernetes.container.image=test-spark-stage1:latest \
--conf spark.driver.extraJavaOptions='-Divy.cache.dir=/tmp -Divy.home=/tmp' \
--conf spark.app.name=test-spark-job \
--conf spark.jars.ivy=/tmp/.ivy \
--conf spark.kubernetes.driverEnv.HTTP2_DISABLE=true \
--conf spark.kubernetes.namespace=argo \
--conf spark.executor.instances=1 \
--packages org.postgresql:postgresql:42.1.4 \
--conf spark.kubernetes.driver.pod.name=custom-app \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=default \
--class IngestStage \
local:///opt/app/test-spark-stage1.jar local {{input.parameters.region}} {{input.parameters.asset_class}}"
]
image: srivastu/test-spark-stage1:latest
imagePullPolicy: IfNotPresent
volumeMounts:
- mountPath: /data
name: exepv
volumes:
- name: exepv
hostPath:
path: /home/docker/data
type: Directory
- name: transformData
inputs:
parameters:
- name: file_date
container:
command:
args: [
"/bin/sh",
"-c",
"/opt/spark/bin/spark-submit \
--master k8s://https://kubernetes.default.svc \
--deploy-mode cluster \
--conf spark.kubernetes.container.image=test-spark-stage2:latest \
--conf spark.driver.extraJavaOptions='-Divy.cache.dir=/tmp -Divy.home=/tmp' \
--conf spark.jars.ivy=/tmp/.ivy \
--conf spark.kubernetes.driverEnv.HTTP2_DISABLE=true \
--conf spark.kubernetes.namespace=argo \
--conf spark.executor.instances=1 \
--conf spark.kubernetes.executor.volumes.hostPath.exepv.mount.path='/data' \
--conf spark.kubernetes.executor.volumes.hostPath.exepv.options.path='/home/docker/data' \
--conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
--packages io.delta:delta-core_2.12:1.1.0 \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=default \
--class TransformDataStage \
local:///opt/app/test-spark-stage2.jar local {{inputs.parameters.file_date}}"
]
image: srivastu/test-spark-stage2:latest
imagePullPolicy: IfNotPresent
- name: saveData
container:
command:
args: [
"/bin/sh",
"-c",
"/opt/spark/bin/spark-submit \
--master k8s://https://kubernetes.default.svc \
--deploy-mode cluster \
--conf spark.kubernetes.container.image=test-spark-stage1:latest \
--conf spark.driver.extraJavaOptions='-Divy.cache.dir=/tmp -Divy.home=/tmp' \
--conf spark.jars.ivy=/tmp/.ivy \
--conf spark.kubernetes.driverEnv.HTTP2_DISABLE=true \
--conf spark.kubernetes.namespace=argo \
--conf spark.executor.instances=1 \
--conf spark.kubernetes.executor.volumes.hostPath.exepv.mount.path='/data' \
--conf spark.kubernetes.executor.volumes.hostPath.exepv.options.path='/home/docker/data' \
--conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
--packages io.delta:delta-core_2.12:1.1.0 \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=default \
--class SaveDataStage \
local:///opt/app/test-spark-stage1.jar local "
]
image: srivastu/test-spark-stage2:latest
imagePullPolicy: IfNotPresent
volumeMounts:
- mountPath: /data
name: exepv
volumes:
- name: exepv
hostPath:
path: /home/docker/data
type: Directory
Now when i submit the workflow using the command :
argo submit -n argo --watch test_spark_staged_argo.yaml
When i run this the container gets stuck in Container Creating state
Status: Running
Conditions:
PodRunning False
Created: Mon Apr 11 14:28:06 +0530 (22 minutes ago)
Started: Mon Apr 11 14:28:06 +0530 (22 minutes ago)
Duration: 22 minutes 35 seconds
Progress: 0/1
Parameters:
date: 20220404
region: us-east
asset_class: Equity
STEP TEMPLATE PODNAME DURATION MESSAGE
● spark-argo-dagsp2gk spark-argo-dag
└─◷ ingest-vendor-data ingestStage spark-argo-dagsp2gk-609318480 22m ContainerCreating
Since its stuck in Container creating state logs cannot be viewed in of container
When doing kubectl command to view the list of pods get the following output
[#a-uexsrqhp5gj1 spark_argo_test]$ kubectl get pods -n argo
NAME READY STATUS RESTARTS AGE
argo-server-78b4844f66-9ch7z 1/1 Running 0 3d20h
hello-world-zvjqz 0/2 Completed 0 3h29m
spark-argo-dagffpvf-1073011739 0/2 ContainerCreating 0 4h19m
spark-argo-dagsp2gk-609318480 0/2 ContainerCreating 0 111m
workflow-controller-756c8c87ff-w9l4z 1/1 Running 0 3d19h
[#a-uexsrqhp5gj1 spark_argo_test]$
Not able to figure out the exact reason why it went into Container Creating state