How to push a single image with multiple tags to a container registry? - ibm-cloud

I'm currently using the following command to build an image with a tag:
buildctl build \
--frontend dockerfile.v0 --opt filename=${DOCKER_FILE} --local dockerfile=${DOCKER_ROOT} \
${BUILD_ARGS} --local context=${DOCKER_ROOT} \
--import-cache type=registry,ref=${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME} \
--output type=image,name="${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}:${IMAGE_TAG}",push=true
I want to add a new tag to the same image. I've tried what is suggested on Buildctl command to tag multiple images, but it fails.
buildctl build \
--frontend dockerfile.v0 --opt filename=${DOCKER_FILE} --local dockerfile=${DOCKER_ROOT} \
${BUILD_ARGS} --local context=${DOCKER_ROOT} \
--import-cache type=registry,ref=${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME} \
--output type=image,name="${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}:${IMAGE_TAG},${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}:latest",push=true
error: invalid value acme.com/namespace/image-name:latest
I also tried the following, but it only creates the image with the latest tag, not both
buildctl build \
--frontend dockerfile.v0 --opt filename=${DOCKER_FILE} --local dockerfile=${DOCKER_ROOT} \
${BUILD_ARGS} --local context=${DOCKER_ROOT} \
--import-cache type=registry,ref=${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME} \
--output type=image,name="${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}:${IMAGE_TAG},name=${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}:latest",push=true
What I want:
1 image with 2 tags: value of IMAGE_TAG and latest
Does anyone know how to do it?

There seems to be a difference between name="foo" and "name=foo". The correct one in this case is the latter.
I tracked the issue that enabled this funcitonality https://github.com/moby/buildkit/issues/797. The comments show how to do it (depending on your environment escaping might be necessary; although, not always).
Following the scenario from the question, the propper way to build and push a single image with multiple tags is as follows (notice how it's "name=)
buildctl build \
--frontend dockerfile.v0 --opt filename=${DOCKER_FILE} --local dockerfile=${DOCKER_ROOT} \
${BUILD_ARGS} --local context=${DOCKER_ROOT} \
--import-cache type=registry,ref=${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME} \
--output type=image,\"name=${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}:${IMAGE_TAG},${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}:latest\",push=true

Related

Detect if argo workflow is given unused parameters

My team runs into a recurring issue where if we mis-spell a parameter for our argo workflows, that parameter gets ignored without error. For example, say I run the following submission command, where the true (optional) parameter is validation_data_config:
argo submit --from workflowtemplate/training \
-p output=$( artifacts resolve-url $BUCKET $PROJECT $TAG) \
-p tag=${TAG} \
-p training_config="$( yq . training.yaml )" \
-p training_data_config="$( yq . train-data.yaml )" \
-p validation-data-config="$( yq . validation-data.yaml )" \
-p wandb-project="cyclegan_c48_to_c384" \
-p cpu=4 \
-p memory="15Gi" \
--name "${NAME}" \
--labels "project=${PROJECT},experiment=${EXPERIMENT},trial=${TRIAL}"
The validation configuration is ignored and the job is run without validation metrics because I used hyphens instead of underscores.
I also understand the parameters should use consistent hyphen/underscore naming, but we've also had this happen with e.g. the "memory" parameter.
Is there any way to detect this automatically, to have the submission error if a parameter is unused, or even to get a list of parameters for a given workflow template so I can write such detection myself?

wget to mirror site make exception to `--no-parent`

I want to mirror a site using wget:
wget --mirror \
--convert-links \
--adjust-extension \
--page-requisites \
--no-parent \
--wait=2 \
--progress=bar \
--show-progress \
--output-file=$LOG_FILE \
--directory-prefix=$DIR_PATH \
$URL
Now, it has been working well but I have come accross a website where the main page from which I want to start is under https://www.website.org/unique_path/here.html but it contains references to files or links that are like: https://www2.website.org/unique_path/there.pdf. However, --no-parent prevents the download of the content under www2... URL. Is there a way to circumvent this? (Or some option that would explicitly work as --no-parent by specifying some wildcard expression that it is ok to go and download here and there?
Is there a way to circumvent this?
You are apparently looking for Spanning Hosts options, you must provide -H option and then you might deliver comma-separated list of acceptable domains via -D, using your example
wget <your current options here> -H -D www.website.org,www2.website.org <your URL here>

How can a Github workflow read a file on another private repository hosted on the same GitHub account?

I have a private git repository like:
Github.com/acct/repo1
I have a test.json file in another private repository in the same GitHub subscription and account like:
Github.com/acct/repo2/test.json
How can a workflow in repo1 have http get access to the url Github.com/acct/repo2/test.json?
All I need is a simple way to read the content of test.json in a workflow running on repo2.
It works the best for me to read the file using a curl get request.
First of all, you need to get an access token from GitHub. This is described here.
And add it to the secrets for the repository in which the Github Action will run. In your case, this is repo1. Read more about setting secrets.
After that, you can execute the request using curl.
To do this, run the following command:
curl -H 'Authorization: token ${{ secrets.GITHUB_TOKEN }}' \
-H 'Accept: application/vnd.github.v4.raw' \
-L https://api.github.com/repos/OWNER_HERE/REPO_NAME_HERE/contents/FOLDER_OR_FILE_HERE
After execution, the contents of the file will be displayed on the console.
If the contents of the file need to be saved to a variable, then use the following syntax:
VARIABLE_NAME="$(curl -H 'Authorization: token ${{ secrets.GITHUB_TOKEN }}' \
-H 'Accept: application/vnd.github.v4.raw' \
-L https://api.github.com/repos/OWNER_HERE/REPO_NAME_HERE/contents/FOLDER_OR_FILE_HERE)"
Add the -O flag to save the content to a file
curl -H 'Authorization: token ${{ secrets.GITHUB_TOKEN }}' \
-H 'Accept: application/vnd.github.v4.raw' \
-O \
-L https://api.github.com/repos/OWNER_HERE/REPO_NAME_HERE/contents/FOLDER_OR_FILE_HERE
In your case, the step in the .yml file to get the content would look like this:
#...
-name: Get file test.json
run: |
curl -H 'Authorization: token ${{ secrets.GITHUB_TOKEN }}' \
-H 'Accept: application/vnd.github.v4.raw' \
-O \
-L https://api.github.com/repos/acct/repo2/contents/test.json
#...
Please note that the path /contents/ must remain after REPO_NAME_HERE in the request, and after that you can specify the path to the FOLDER_OR_FILE_HERE file.

Build all packages for an image

Is it possible to build all of the packages for a specific image? I know I can build packages individually, but ideally would like to build all of them at once, through a single command.
Alternatively, is there a way to prevent the do_rootfs task from being executed for a particular image.
Cheers, Donal
First make an image that contains a packagegroup (or just list your dependencies there).
$ cat sources/meta-custom/recipes-custom/images/only-packages-image.bb
SUMMARY = "All dependencies no image"
LICENSE = "CLOSED"
version = "##DISTRO_VERSION##"
BB_SCHEDULER = "speed"
# option 1 - packagegroup, package list can be reused in real image
CORE_IMAGE_BASE_INSTALL += "\
packagegroup_all-depends \
"
# option 2 - list deps here, package list can not be reused in real image
CORE_IMAGE_BASE_INSTALL += "\
lshw \
systemd \
cronie \
glibc \
sqlite \
bash \
python3-dev \
python3-2to3 \
python3-misc \
python3-pyvenv \
python3-modules \
python3-pip \
wget \
apt \
pciutils \
file \
tree \
\
wpa-supplicant \
dhcpcd \
networkmanager \
curl-dev \
curl \
hostapd \
iw \
"
# remove the rootfs step
do_rootfs() {
}
Second make your packagegroup if you opted to reuse the list of packages
$ cat sources/meta-custom/recipes-custom/packagegroups/packagegroup-alldeps.bb
PACKAGE_ARCH = "${MACHINE_ARCH}"
inherit packagegroup
RDEPENDS_${PN} = " \
lshw \
systemd \
cronie \
glibc \
sqlite \
bash \
python3-dev \
python3-2to3 \
python3-misc \
python3-pyvenv \
python3-modules \
python3-pip \
wget \
apt \
pciutils \
file \
tree \
\
wpa-supplicant \
dhcpcd \
networkmanager \
curl-dev \
curl \
hostapd \
iw \
"
Finally build your new image placeholder
$ bitbake only-packages-image
In Yocto >=4.0 this is actually pretty easy to achieve. The packagegroup method did not work for me at all.
I don't know if this works in older versions though.
Create a new file in your custom layer, e.g. meta-custom/classes/norootfs.bbclass and put the following lines in there (as far as I noticed the order does not matter):
deltask do_deploy
deltask do_image
deltask do_rootfs
deltask do_image_complete
deltask do_image_setscene
then in your meta-custom/recipes-core/images/myimage.bb add norootfs to your other inherit commands
e.g. the most basic one
inherit core-image norootfs
You will notice your number of tasks decreasing by a fair amount (mine from ~4700 to ~3000) and there is no complete rootfs image anymore in build/tmp/deploy/images, except for bzImage and modules, just the plain ipk files in build/tmp/deploy/ipk.
I got this information by looking at https://docs.yoctoproject.org/ref-manual/tasks.html?highlight=do_image and .bbclass files in meta/classes where deltask is frequently used.

Add provider to User federation in RedHat SSO/keycloak using CLI

I have custom provider created and deployed.
Now I goto user federation select the drop down and add my provider using UI and fine. Image using UI
Can some one please let me know how to add the same using CLI as I want to automate the manual process.
This worked for me:
kcadm.bat create user-federation/instances -r Test1 \
-s providerName=tatts-asg-authentication \
-s priority=0 \
-s config.debug=false
This is what works for Keycloak 3.4.3:
kcadm.bat create components -x -r MyRealm \
-s providerType=org.keycloak.storage.UserStorageProvider \
-s name=my-provider \
-s parentId=MyRealm \
-s providerId=my-provider \
-s 'config.path=["C:\\path\\to\\properties"]' \
-s 'config.priority=["0"]'
user-federation/instances has been replaced with components: issues.jboss.org/browse/KEYCLOAK-6583
The -x option is to output the stacktrace on error.