Use local script as source for Argo workflow - argo-workflows

Question

I have a python script that I'd like to execute on cloud using an Argo workflow.
Currently, I'm alternating between copying the source code to the workflow itself (using copy and paste), which is inconvenient and causes issues.
The second options is uploading my project directory to an s3 bucket, then downloading the source code to the Argo pod, then running the commands.
Both methods require some actions to sync the source code after I modify the script.
Is there a way to specify on the Argo workflow from where it should take the source code from?
Say, instead of creating a script template that takes the source from a string specified in the .yml file - take it from a local file by specifying a local path?
Prefer not to use Git for that
Also, if possible would prefer solutions with support for attaching additional dependencies source code files

Answer 1

If you have something more sophisticated than a simple script that you use in the .yml file, it might be worthwhile to use a docker image with a container template that you pre-build for your workflow.
The image will be named my-script and the entrypoint my-entrypoint
Assuming the script is in python called script.py you can have the following files:
workflow.yml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: workflow-name-
spec:
entrypoint: my-entrypoint
templates:
- name: my-entrypoint
container:
image: my-script
command: python3
args:
- script.py
script.py
import requests
response = requests.get('www.google.com')
print(response.status_code)
requirements.txt
requests
Dockerfile
FROM python:3.11.1-slim
COPY . .
RUN pip3 install -r requirements.txt
CMD python3 script.py
Assuming you can build your image to the cluster (in the case of minikube). You'd run:
docker build -t my-script .
This approach also makes your code testable, should you decide to have tests. For this it is not necessary to use Git, although I'd encourage you to use it for collaboration and versioning. Also the COPY command in the Dockerfile copies all files in your directory to the image, so you'd have other information readily available. I would discourage you to copy actual data in this way, but rather use argo parameters and artifacts.
Check out https://argoproj.github.io/argo-workflows/workflow-concepts/ for more info