My team runs into a recurring issue where if we mis-spell a parameter for our argo workflows, that parameter gets ignored without error. For example, say I run the following submission command, where the true (optional) parameter is validation_data_config:
argo submit --from workflowtemplate/training \
-p output=$( artifacts resolve-url $BUCKET $PROJECT $TAG) \
-p tag=${TAG} \
-p training_config="$( yq . training.yaml )" \
-p training_data_config="$( yq . train-data.yaml )" \
-p validation-data-config="$( yq . validation-data.yaml )" \
-p wandb-project="cyclegan_c48_to_c384" \
-p cpu=4 \
-p memory="15Gi" \
--name "${NAME}" \
--labels "project=${PROJECT},experiment=${EXPERIMENT},trial=${TRIAL}"
The validation configuration is ignored and the job is run without validation metrics because I used hyphens instead of underscores.
I also understand the parameters should use consistent hyphen/underscore naming, but we've also had this happen with e.g. the "memory" parameter.
Is there any way to detect this automatically, to have the submission error if a parameter is unused, or even to get a list of parameters for a given workflow template so I can write such detection myself?
Related
I want to submit my neural network model to google cloud via the following command as in the tutorial:
gcloud ai-platform jobs submit training ${JOB_NAME} \
--region=us-central1 \
--master-image-uri=gcr.io/cloud-ml-public/training/pytorch-gpu.1-10 \
--scale-tier=CUSTOM \
--master-machine-type=n1-standard-8 \
--master-accelerator=type=nvidia-tesla-p100,count=1 \
--job-dir=${JOB_DIR} \
--package-path=./trainer \
--module-name=trainer.task \
-- \
--train-files=gs://cloud-samples-data/ai-platform/chicago_taxi/training/small/taxi_trips_train.csv \
--eval-files=gs://cloud-samples-data/ai-platform/chicago_taxi/training/small/taxi_trips_eval.csv \
--num-epochs=10 \
--batch-size=100 \
--learning-rate=0.001
I was working with powershell and I have a problem with the master-accelerator argument which should be a dictionnary. I don't know how to pass such to gcloud. I have tried #{count=1; type=...}, but received a Bad syntax for dict arg: [#] error.
How can I pass a list of parameters in PowerShell such that the gcloud submit command accepts it?
I thank you in advance for your help.
EDIT
I have tried to use delimiters as ^^^^:^^^^, along this, but this still does not work (Invalid delimeter).
I'm currently using the following command to build an image with a tag:
buildctl build \
--frontend dockerfile.v0 --opt filename=${DOCKER_FILE} --local dockerfile=${DOCKER_ROOT} \
${BUILD_ARGS} --local context=${DOCKER_ROOT} \
--import-cache type=registry,ref=${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME} \
--output type=image,name="${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}:${IMAGE_TAG}",push=true
I want to add a new tag to the same image. I've tried what is suggested on Buildctl command to tag multiple images, but it fails.
buildctl build \
--frontend dockerfile.v0 --opt filename=${DOCKER_FILE} --local dockerfile=${DOCKER_ROOT} \
${BUILD_ARGS} --local context=${DOCKER_ROOT} \
--import-cache type=registry,ref=${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME} \
--output type=image,name="${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}:${IMAGE_TAG},${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}:latest",push=true
error: invalid value acme.com/namespace/image-name:latest
I also tried the following, but it only creates the image with the latest tag, not both
buildctl build \
--frontend dockerfile.v0 --opt filename=${DOCKER_FILE} --local dockerfile=${DOCKER_ROOT} \
${BUILD_ARGS} --local context=${DOCKER_ROOT} \
--import-cache type=registry,ref=${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME} \
--output type=image,name="${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}:${IMAGE_TAG},name=${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}:latest",push=true
What I want:
1 image with 2 tags: value of IMAGE_TAG and latest
Does anyone know how to do it?
There seems to be a difference between name="foo" and "name=foo". The correct one in this case is the latter.
I tracked the issue that enabled this funcitonality https://github.com/moby/buildkit/issues/797. The comments show how to do it (depending on your environment escaping might be necessary; although, not always).
Following the scenario from the question, the propper way to build and push a single image with multiple tags is as follows (notice how it's "name=)
buildctl build \
--frontend dockerfile.v0 --opt filename=${DOCKER_FILE} --local dockerfile=${DOCKER_ROOT} \
${BUILD_ARGS} --local context=${DOCKER_ROOT} \
--import-cache type=registry,ref=${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME} \
--output type=image,\"name=${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}:${IMAGE_TAG},${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}:latest\",push=true
I'm executing gcloud composer commands:
gcloud composer environments run airflow-composer \
--location europe-west1 --user-output-enabled=true \
backfill -- -s 20171201 -e 20171208 dags.my_dag_name \
kubeconfig entry generated for europe-west1-airflow-compos-007-gke.
It's a regular airflow backfill. The command above is printing the results at the end of the whole backfill range, is there any way to get the output in a streaming manner ? Each time a DAG gets backfilled it will be printed in the standard output, like in a regular airflow-cli.
I have custom provider created and deployed.
Now I goto user federation select the drop down and add my provider using UI and fine. Image using UI
Can some one please let me know how to add the same using CLI as I want to automate the manual process.
This worked for me:
kcadm.bat create user-federation/instances -r Test1 \
-s providerName=tatts-asg-authentication \
-s priority=0 \
-s config.debug=false
This is what works for Keycloak 3.4.3:
kcadm.bat create components -x -r MyRealm \
-s providerType=org.keycloak.storage.UserStorageProvider \
-s name=my-provider \
-s parentId=MyRealm \
-s providerId=my-provider \
-s 'config.path=["C:\\path\\to\\properties"]' \
-s 'config.priority=["0"]'
user-federation/instances has been replaced with components: issues.jboss.org/browse/KEYCLOAK-6583
The -x option is to output the stacktrace on error.
I'm trying to import this example into postman
curl -s --user 'api:YOUR_API_KEY' \
https://api.mailgun.net/v3/YOUR_DOMAIN_NAME/messages \
-F from='Excited User <mailgun#YOUR_DOMAIN_NAME>' \
-F to=YOU#YOUR_DOMAIN_NAME \
-F to=bar#example.com \
-F subject='Hello' \
-F text='Testing some Mailgun awesomness!'
Please help me understand the -s, --user, -F, what is that? And then when I try to import, I get this error: option-less arguments found. How can I fix this?
If you're trying to execute curl commands in Postman, select Import, then Paste Raw Text and then copy the command, but first remove all backslashes.