Currently, I deploy python scripts on Kubernetes using Codefresh. I'm looking to incorporate Kubeflow into the deployment plan to get all the Kubeflow goodies such as the UI and all but I'm a little clueless on how to start or where to look.
The docs for Kubeflow mainly only cover setting up with Google Cloud Platform only. Does anybody have any experience with this?
You can use these instructions to install kubeflow on any existing kubernetes cluster, regardless of whether you are running on GCP or any other platform.
These steps can be converted into a codefresh pipeline like any other set of commands. If you need help with that let me know, we use codefresh and I'm well versed with there pipeline files.
Related
Given a container in a Azure container registry and a kubernetes cluster setup via the portal. Are there any visual tools that I can use so that I don't have to use the command line commands ,for things like add/edit the yaml file and launching the cluster?
For example I found this tool https://k8syaml.com/, but this is only one part of the process and it is also not aware of the existing infrastructure.
What are the visual tools to manage kubernetes end-to-end?
One tool I always work with when dealing with Kubernetes is Lens. Here is a video showing you what it can do. Best of all, it just needs the kube config file and so it is agnostic to where the Kubernetes cluster is (On-Prem, GKE, AKS, EKS)
kubectx for switching between contexts (clusters) & K9s is widely used that is something hybrid between being a cli and visual tool.
Octant is another option - https://github.com/vmware-tanzu/octant, it is similar to lens
I encountered the same issue as in https://github.com/kubeflow/kubeflow/issues/6014 with my Kubeflow app. The fix is very simple (just a type casting), then I would like to fix it myself and redeploy Kubeflow.
The problem is that I am running a k3s cluster on my local machine where I have installed Kubeflow bundle via Juju. Then, I cannot change the source code.
How to modify Kubeflow source code before deploying it with Kubernetes?
Should I use the manifest installation https://github.com/kubeflow/manifests#installation ? or a totally different method?
Thank you.
The bug was fixed in the last version of the manifests, then I have finally installed kubeflow directly from the manifests.
But still I am in touch with one Kubeflow developer, I will post here the right way to do modify/deploy if interested.
You got to check out their Github repo. Make changes and use kustomize to install like explained in their wiki. If you check the example folder you can see that it points to all other component folders.
https://github.com/kubeflow/manifests#install-with-a-single-command
One another hack could be, just look for the controllers in Kubernetes eg., deployments created for kubeflow, then modify them; works only if your changes are only related to Kubernetes resource definitions. I suggest going with the first option above for a clean development experience, and hey, that way can you contribute back to the kubeflow project as well, if you changes will benefit others.
I am trying to use KubeFlow on GCP and I am following this codelab, but "click-to-deploy" is no longer supported so I followed the documentation of "kubectl and kpt". However, I keep getting this "You cannot perform this action because the Cloud SDK component manager is disabled for this installation." error and none of the solutions I found worked. I have 2 other friends told me they tried to make KubeFlow work since last year, it never worked, but I did see people post question about KubeFlow on Stackoverflow still, so I want to ask if it is still working, if so, where can I find a decent guide to follow?
Thanks!
I finally got it working. For that error message, it turned out that I just didn't install the Cloud SDK properly. There will be a lot of other issues too down the road, but at least the KubeFlow web UI is working for me now.
yes, as the kubectl and kpt says, the first step in getting prepared to install cluster is installing gcloud that is CLI that manages authentication, local configuration, developer workflow, interactions with Google Cloud APIs.
Without is you simply cant work with objects(in your case you need to enable kpt anthoscli beta) and perform tasks like
creating a Compute Engine VM instance, managing a Google Kubernetes
Engine cluster, and deploying an App Engine application, either from
the command line or in scripts and other automations..
I’m testing kubeflow pipeline and would like to use it on AWS/On-prem but I saw the below comment on the documentation. Should I wait using it with AWS/on-prem?
Due to kubeflow/pipelines#345 and kubeflow/pipelines#337, Kubeflow Pipelines depends on Google Cloud Platform (GCP) services and some of the functionality is currently not supported by non-GKE clusters.
I'd suggest to wait. See the status update on the issues and acitivity on kubeflow/pipelines#1131 to enable the support on AWS. Similar work is in progress for supporting on-prem as well.
On-prem is definitely possible.
KFP is not tied to GCP and can work even on Windows tables.
AWS support is mostly provided by the community, but most of the issues have been fixed. There are samples that show demonstrate KFP on AWS and there are multiple AWS and Sagemaker components created by people form AWS. https://github.com/kubeflow/pipelines/blob/091316b8bf3790e14e2418843ff67a3072cfadc0/samples/contrib/aws-samples/titanic-survival-prediction/titanic-survival-prediction.py
Let's say I have a flask app, a PostgreSQL, and a Redis app. what is the best practice way to develop those apps locally which then later be deployed to Kubernetes.
Because, I have tried to develop in minikube with ksync, but I get difficulties in getting detailed debug log information.
Any ideas?
What we do with our systems is that we develop and test them locally. I am not very knowledgeable with Flask and ksyncy, but say for example, you are using Lagom Microservices Framework in Java, you run you app locally using the SBT shell where you can view all your logs. We then automate the deployment using LightBend Orchestration.
When you then decide to test the app on Kubernetes, you can choose to use minikube, but you have to configure the logging properly. You can configure centralised logging for Kubernetes using the EFK stack. This will collect all the logs from the various components of your app and store them in Elastic Search. You can then view these logs using The Kibana Dashboard. You can do a lot with the dashboard, you can view logs for a given period, or search logs by k8s namespace, or by container.
There are multiple solutions for this (aka GitOps with Kubernetes):
Skaffold
Draft
Flux - IMO the most mature.
Ksonnet
GitKube
Argo - A bit more of a workflow engine.
Metaparticle - Deploy with actual code.
I think the solution is using skaffold