What are good workflows for deploying podman/buildah created container images to minikube? - minikube

I am exploring and learning about containers and kubernetes using podman and minikube on a linux workstation. I use podman to build images on the workstation and would like to deploy these images in minikube also running on the workstation using the kvm2 virtual machine driver. I also start minikube using the CRI-O container runtime.
What are efficient workflows to deploy these images from the workstation to minikube in this scenario? Docker is not running on the minikube VM so the reusing the Docker daemon as described in the minikube documentation is not an option. Sharing the host file system with minikube also appears to not be viable at this time when using kvm2.
Is running a local registry that is visible to both the workstation and the minikube vm the best option? Answers to How to use local docker images with Minikube? and (Kubernetes + Minikube) can't get docker image from local registry appear to offer good solutions for configuring a local registry.
Would skopeo be a solution?
Edit: this is a nice post describing how to set up a registry using podman: https://computingforgeeks.com/create-docker-container-registry-with-podman-letsencrypt/
thank you
Brad

Minikube documentation provides the foundation for a potential workflow at https://minikube.sigs.k8s.io/docs/tasks/docker_registry/. In order to use podman in lieu of docker I did the following
Start minikube, as instructed, with the --insecure-registry flag. I specifically use
minikube start --network-plugin=cni --enable-default-cni --bootstrapper=kubeadm --container-runtime=cri-o --cpus 4 --memory 4g --insecure-registry "192.168.39.0/24"
Enable the minikube registry addon.
minikube addons enable registry
Configure podman to use the insecure minikube registry by adding the registry to the insecure registries section of /etc/containers/registries.conf. This section now looks like
[registries.insecure]
registries = ['192.168.39.175:5000']
where 192.168.39.175 is the minikube ip. This ip may change following minikube restarts.
Follow the build, push and run commands in https://minikube.sigs.k8s.io/docs/tasks/docker_registry/ substituting podman for docker. This assumes the test-img container file exists.
Build: podman build --tag $(minikube ip):5000/test-img .
Push: podman push $(minikube ip):5000/test-img
Run: kubectl run test-img --image=$(minikube ip):5000/test-img
This worked but suffers from a serious complication: there is no apparent way at this time to set the IP address for the minikube VM when using kvm2. The IP will always be in the 192.168.39.0/24 subnet but that is the only certainty. Each time minikube is started the IP address of the registry will change which has significant implications for podman and the workflow in general.
More to come an another solution.

Related

How to set minikube proxy when the driver is hyperkit or virtualbox?

I am trying to use Ingress in minikube by minikube addons enable ingress. However, currently Ingress cannot be used with minikube when the driver is docker on macOS based on this issue ticket.
So I turn to use hyperkit or virtualbox as driver. One image that need to be pulled when enabling Ingress is k8s.gcr.io/ingress-nginx/controller:v0.44.0. However, k8s.gcr.io is blocked in my current location.
So I try to use a VPN in global mode for my computer. However, I met this issue that hyperkit is unable to access k8s.gcr.io when the VPN is in use.
Then I found this document
https://minikube.sigs.k8s.io/docs/handbook/vpn_and_proxy/
My VPN is listening at 127.0.0.1:1087, I set
export HTTP_PROXY=http://127.0.0.1:1087
export HTTPS_PROXY=https://127.0.0.1:1087
export NO_PROXY=localhost,127.0.0.1,10.96.0.0/12,192.168.99.0/24,192.168.39.0/24
Then I tried all these methods to start minikube:
minikube start --driver=hyperkit
minikube start --driver=virtualbox
minikube start --driver=hyperkit --docker-env HTTP_PROXY=http://127.0.0.1:1087 --docker-env HTTPS_PROXY=https://127.0.0.1:1087 --docker-env NO_PROXY=localhost,127.0.0.1,10.96.0.0/12,192.168.99.0/24,192.168.39.0/24
But I saw these messages:
πŸ˜„ minikube v1.21.0 on Darwin 11.2.3
✨ Using the hyperkit driver based on user configuration
❗ Local proxy ignored: not passing HTTP_PROXY=http://127.0.0.1:1087 to docker env.
❗ Local proxy ignored: not passing HTTPS_PROXY=https://127.0.0.1:1087 to docker env.
πŸ‘ Starting control plane node minikube in cluster minikube
πŸ”₯ Creating hyperkit VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
❗ Local proxy ignored: not passing HTTP_PROXY=http://127.0.0.1:1087 to docker env.
❗ Local proxy ignored: not passing HTTPS_PROXY=https://127.0.0.1:1087 to docker env.
and
πŸ˜„ minikube v1.21.0 on Darwin 11.2.3
✨ Using the virtualbox driver based on existing profile
❗ Local proxy ignored: not passing HTTP_PROXY=http://127.0.0.1:1087 to docker env.
❗ Local proxy ignored: not passing HTTPS_PROXY=https://127.0.0.1:1087 to docker env.
Seems this "user configuration" overwrite my proxy config. But where is this "user configuration"?
What is the correct way to set proxy for minikube when the drive hyperkit or virtualbox? Thanks!
My guess is 127.0.0.1 conflicts with the VM's internal 127.0.0.1 address, and that's why it's ignored. You might need to configure your proxy to be your host's network IP instead of 127.0.0.1? You might not even need to configure a proxy? Also, the Virtualbox driver gives me problems with VPN. I have the best luck with the VMware driver, and can also get the HyperKit driver to work if I update the VM's DNS to my host's DNS.
minikube start --driver hyperkit
minikube ssh sudo resolvectl dns eth0 192.168.0.53
minikube ssh sudo resolvectl domain eth0 example.com
I also get the unable to access k8s.gcr.io error when creating the VM, but it doesn't seem to affect things.
Downloading this image using docker, exporting it to file, transfering it to minikube VM and importing it to local docker registry, like in this thread has solved the problem.
Your proxy is for circumventing the China Greate Firewall correct? Then I know why it is not working. It is not releated to hyperkit or virutalbox at all.
I checked the source code of minikube. "Local proxy ignored" actually means that your proxy url is set to localhost (127.0.*) and minikube thinks you set the proxy incorrectly so it will just ignore this setting.
The resolution is just to edit your host file (for Windows it is in C:\Windows\System32\drivers\etc\hosts), to give 127.0.0.1 a hostname. You can add the following line into the end of the host file.
127.0.0.1 localproxy
Then change environment variable http_proxy and https_proxy to http://localproxy:1235.
Reopen the CMD window to get the updated environment variable and restart the minikue. You should be able to find that the "Local proxy ignored" message is gone and finally you can download the image from gcr.io.

Location of Kubernetes config directory with Docker Desktop on Windows

I am running a local Kubernetes cluster through Docker Desktop on Windows. I'm attempting to modify my kube-apiserver config, and all of the information I've found has said to modify /etc/kubernetes/manifests/kube-apiserver.yaml on the master. I haven't been able to find this file, and am not sure what the proper way is to do this. Is there a different process because the cluster is through Docker Desktop?
Is there a different process because the cluster is through Docker Desktop?
You can get access to the kubeapi-server.yaml with a Kubernetes that is running on Docker Desktop but in a "hacky" way. I've included the explanation below.
For setups that require such reconfigurations, I encourage you to use different solution like for example minikube.
Minikube has a feature that allows you to pass the additional options for the Kubernetes components. You can read more about --extra-config ExtraOption by following this documentation:
Minikube.sigs.k8s.io: Docs: Commands: Start
As for the reconfiguration of kube-apiserver.yaml with Docker Desktop
You need to run following command:
docker run -it --privileged --pid=host debian nsenter -t 1 -m -u -n -i sh
Above command will allow you to run:
vi /etc/kubernetes/manifests/kube-apiserver.yaml
This lets you edit the API server configuration. The Pod running kubeapi-server will be restarted with new parameters.
You can check below StackOverflow answers for more reference:
Stackoverflow.com: Answer: Where are the Docker Desktop for Windows kubelet logs located?
Stackoverflow.com: Answer: How to change the default nodeport range on Mac (docker-desktop)?
I've used this answer without $ screen command and I was able to reconfigure kubeapi-server on Docker Desktop in Windows

Can I install minikube on ubuntu without virtualBox?

I want to start practicing with k8s for the CKAD exam. I run on ubuntu 18.04.
I noticed everywhere that I need to download Virtualbox for minikube. I believe that VB is needed in case I don't start my cluster with a driver but if I use the Docker driver when I start my cluster shouldn't that be enough? Is microk8s a better option?
It seems that the preferred way is use --driver=docker driver instead of --driver=none for minikube, although it is technically not baremetal as it is significantly easier to configure and does not require root access. The β€˜none’ driver is recommended for advanced users only. (info below from https://minikube.sigs.k8s.io/docs/drivers/docker/)
docker
Overview
The Docker driver allows you to install Kubernetes into an existing Docker install. On Linux, this does not require virtualization to be enabled.
Requirements
Install Docker 18.09 or higher
amd64 or arm64 system.
Usage
Start a cluster using the docker driver:
minikube start --driver=docker
To make docker the default driver:
minikube config set driver docker
Yes you can. Check here.
Minikube also supports a --driver=none option that runs the Kubernetes components on the host and not in a VM. Using this driver requires Docker and a Linux environment but not a hypervisor.
Jus run
$ minikube start
Caution: If you use the none driver, some Kubernetes components run as privileged containers that have side effects outside of the Minikube environment. Those side effects mean that the none driver is not recommended for personal workstations

How to set DOCKER_HOST with fixed ip address or system variable?

I am using eclipse docker tooling perspective to handle minikube docker repository on windows 10. The installation and configuration of minikube are successful.
> minikube docker-env
$Env:DOCKER_TLS_VERIFY = "1"
$Env:DOCKER_HOST = "tcp://172.17.63.134:2376"
$Env:DOCKER_CERT_PATH = "C:\Users\joseph\.minikube\certs"
$Env:MINIKUBE_ACTIVE_DOCKERD = "minikube"
# To point your shell to minikube's docker-daemon, run:
# & minikube -p minikube docker-env | Invoke-Expression
When managing docker images, I use docker explorer of docker tooling perspective. And below image is the the Docker Connection Dialog
[Test Connection] is successful and the connection throws no errors. But the problem is the DOCKER_HOST of minikube is variable. The DOCKER_HOST is not fixed and is changeable whenever minikube is started. I want to know how to fix the DOCKER_HOST ip address when minikube is started or the system variable of DOCKER_HOST with which I can set the value of docker tcp connection URI of eclipse IDE.
I've reproduced your problem using the following steps:
minikube start --vm-driver=xhyve
minikube ip (note the IP address)
minikube stop
minikube delete
minikube start --vm-driver=xhyve
minikube ip (IP address has changed)
After digging around I understood that at the moment there is no support for predictable IP's across restarts.
There is an open issue for this and you can follow it here.

minikube - How to know the name of the minikube VM being used?

I want to use minikube ssh [flags] command
There is Global Flag:
-p, --profile string The name of the minikube VM being used.
So, I need to know the name of the minikube VM. How to do this?
Profiles feature on some stacks are broken since 0.26.0, and still with version 0.28.2. #2818, #2574
Without setting a profile the default profile will create a VM called "minikube".
minikube profile default
minikube profile was successfully set to minikube
Set the profile to another name,
minikube profile experiment
then view the profile set with
minikube config view
and create a new Minikube VM with that profile name
minikube start --profile experiment
With the profile set to "experiment", the VM created in VirtualBox will (should) be named "experiment".
Minikube is a tool that makes it easy to run Kubernetes locally. Minikube runs a single-node Kubernetes cluster inside a VM on your laptop for users looking to try out Kubernetes or develop with it day-to-day.
Parameter "-p", that you mentioned, is used to name the VM where minikube creates a Kubernetes stack. Omitting this parameter presupposes default value "minikube".
For the system running minikube with the support of the VirtualBox:
VBoxManage list runningvms
First returned value is the name of minikube instance. I assume there is no other
VirtualBox session running.
Next, you can use this name to:
minikube ssh <name>
When you create minikube with -p parameter, to manage this setup you must use -p
with every minikube command call. This parameter is useful for multiple minikube instances running on the same host.