I have an HTTP application (Odoo). This app support install/updating modules(addons) dynamically.
I would like to run this app in a Kubernetes cluster. And I would like to dynamically install/update the modules.
I have 2 solutions for this problem. However, I was wondering if there are other solutions.
Solution 1:
Include the custom modules with the app in the Docker image
Every time I made a change in the custom module and push it to a git repository. Jinkins pull the changes and create a new image and then apply the new changes to the Kubernetes cluster.
Advantages: I can manage the docker image version and restart an image if something happens
Drawbacks: This solution is not bad for production however the list of all custom module repositories should all be included in the docker file. Suppose that I have two custom modules each in its repository a change to one of them will lead to a rebuild of the whole docker image.
Solution 2:
Have a persistent volume that contains only the custom modules.
If a change is made to a custom module it is updated in the persistent volume.
The changes need to apply to each pod running the app (I don't know maybe doing a restart)
Advantages: Small changes don't trigger image build. We don't need to recreate the pods each time.
Drawbacks: Controlling the versions of each update is difficult (I don't know if we have version control for persistent volume in Kubernetes).
Questions:
Is there another solution to solve this problem?
For both methods, there is a command that should be executed in order to take the module changes into consideration odoo --update "module_name". This command should include the module name. For solution 2, How to execute a command in each pod?
For solution 2 is it better to restart the app service(odoo) instead of restarting all the nodes? Meaning, if we can execute a command on each pod we can just restart the service of the app.
Thank you very much.
You will probably be better off with your first solution. Specially if you already have all the toolchain to rebuild and deploy images. It will be easier for you to rollback to previous versions and also to troubleshoot (since you know exactly which version is running in each pod).
There is an alternative solution that is sometime used to provision static assets on web servers: You can add an emptyDir volume and a sidecar container to the pod. The sidecar pull the changes from your plugins repositories into the emptyDir at fixed interval. Finally your app container, sharing the same emptyDir volume will have access to the plugins.
In any case running the command to update the plugin is going to be complicated. You could do it at fixed interval but your app might not like it.
Related
I have set up a Kubernetes cluster using Kubernetes Engine on GCP to work on some data preprocessing and modelling using Dask. I installed Dask using Helm following these instructions.
Right now, I see that there are two folders, work and examples
I was able to execute the contents of the notebooks in the example folder confirming that everything is working as expected.
My questions now are as follows
What are the suggested workflow to follow when working on a cluster? Should I just create a new notebook under work and begin prototyping my data preprocessing scripts?
How can I ensure that my work doesn't get erased whenever I upgrade my Helm deployment? Would you just manually move them to a bucket every time you upgrade (which seems tedious)? or would you create a simple vm instance, prototype there, then move everything to the cluster when running on the full dataset?
I'm new to working with data in a distributed environment in the cloud so any suggestions are welcome.
What are the suggested workflow to follow when working on a cluster?
There are many workflows that work well for different groups. There is no single blessed workflow.
Should I just create a new notebook under work and begin prototyping my data preprocessing scripts?
Sure, that would be fine.
How can I ensure that my work doesn't get erased whenever I upgrade my Helm deployment?
You might save your data to some more permanent store, like cloud storage, or a git repository hosted elsewhere.
Would you just manually move them to a bucket every time you upgrade (which seems tedious)?
Yes, that would work (and yes, it is)
or would you create a simple vm instance, prototype there, then move everything to the cluster when running on the full dataset?
Yes, that would also work.
In Summary
The Helm chart includes a Jupyter notebook server for convenience and easy testing, but it is no substitute for a full fledged long-term persistent productivity suite. For that you might consider a project like JupyterHub (which handles the problems you list above) or one of the many enterprise-targeted variants on the market today. It would be easy to use Dask alongside any of those.
I've read about partial upgrade, but it always requires to change some parts of the packages application. I'd like to know if there's a way to redeploy a package without version change. In a way, similar to what VS is doing when deploying to the dev cluster.
On your local dev cluster, VS simply deletes the application before it starts the re-deployment. You could do the same in your production cluster, however this results in downtime since the application is not accessible during that time.
What's the reason why you wouldn't want to use the regular monitored upgrade? It has many advantages, like automatic rollbacks and so on.
I'm wondering which options are there for docker container deployment in production. Given I have separate APP and DB server containers and data-only containers holding deployables and other holding database files.
I just have one server for now, which I would like to "docker enable", but what is the best way to deploy there(remotely will be the best option)
I just want to hit a button and some tool will take care of stopping, starting, exchanging all needed docker containers.
There is myriad of tools(Fleet, Flocker, Docker Compose etc.), I'm overwhelmed by the choices.
Only thing I'm clear is, I don't want to build images with codes from git repo. I would like to have docker images as wrappers for my releases. Have I grasped the docker ideas from wrong end?
My team recently built a Docker continuous deployment system and I thought I'd share it here since you seem to have the same questions we had. It pretty much does what you asked:
"hit a button and some tool will take care of stopping, starting, exchanging all needed docker containers"
We had the challenge that our Docker deployment scripts were getting too complex. Our containers depend on each other in various ways to make the full system so when we deployed, we'd often have dependency issues crop up.
We built a system called "Skopos" to resolve these issues. Skopos detects the current state of your running system and detects any changes being made and then automatically plans out and deploys the update into production. It creates deployment plans dynamically for each deployment based on a comparison of current state and desired state.
It can help you continuously deploy your application or service to production using tags in your repository to automatically roll out the right version to the right platform while removing the need for manual procedures or scripts.
It's free, check it out: http://datagridsys.com/getstarted/
You can import your system in 3 ways:
1. if you have a Docker Compose, we can suck that in and start working iwth it.
2. If your app is running, we can scan it and then start working with it.
3. If you have neither, you can create a quick descriptor file in YAML and then we can understand your current state.
I think most people start their container journey using tools from Docker Toolbox. Those tools provide a good start and work as promised, but you'll end up wanting more. With these tools, you are missing for example integrated overlay networking, DNS, load balancing, aggregated logging, VPN access and private image repository which are crucial for most container workloads.
To solve these problems we started to develop Kontena - Docker Container Orchestration Platform. While Kontena works great for all types of businesses and may be used to run containerized workloads at any scale, it's best suited for start-ups and small to medium sized business who require worry-free and simple to use platform to run containerized workloads.
Kontena is an open source project and you can view it on GitHub.
We are building a SaaS application. I don't have (for now - for this app) high demands on availability. It's mostly going to be used in a specific time zone and for business purposes only, so scheduled restarting at 3 in the morning shouldn't be a problem at all.
It is an ASP.NET application running in mono with the fastcgi server. Each customer will have - due to security reasons - his own application deployed. This is going to be done using docker containers, with an Nginx server in the front, to distribute the requests based on URL. The possible ways how to deploy it are for me:
Create a docker image with the fcgi server only and run the code from a mount point
Create a docker image with the fcgi server and the code
pros for 1. would seem
It's easier to update the code, since the docker containers can keep running
Configuration can be bundled with the code
I could easily (if I ever wanted to) add minor changes for specific clients
pros for 2. would seem
everything is in an image, no need to mess around with additional files, just pull it and run it
cons for 1.
a lot of folders for a lot of customers additionally to the running containers
cons for 2.
Configuration can't be in the image (or can it? - should i create specific images per customer with their configuration?) => still additional files for each customer
Updating a container is harder since I need to restart it - but not a big deal, as stated in the beginning
For now - the first year - the number of customers will be low and when the demand is low, any solution is good enough. I'm looking rather at - what is going to work with >100 customers.
Also for future I want to set up CI for this project, so we wouldn't need to update all customers instances manually. Docker images can have automated builds but not sure that will be enough.
My concerns are basically - which solution is less messier, maybe easier to automate?
I couldn't find any best practices with docker which cover a similar scenario.
It's likely that your application's dependencies are going to be dependent on the code, so you'll still have to sometimes rebuild the images and restart the containers (whenever you add a new dependency).
This means you would have two upgrade workflows:
One where you update just the code (when there are no dependency changes)
One where you update the images too, and restart the containers (when there are dependency changes)
This is most likely undesirable, because it's difficult to automate.
So, I would recommend bundling the code on the image.
You should definitely make sure that your application's configuration can be stored somewhere else, though (e.g. on a volume, or accessed through through environment variables).
Ultimately, Docker is a platform to package, deploy and run applications, so packaging the application (i.e. bundling the code on the image) seems to be the better way to use it.
I have realized that I have to make Image from EBS Volume everytime when I change my code
and following autoscaling configuration everytime (this is really bad).
I have heard that some people try to load their newest code from github or some similar sort of doing.
So that they can let server to have newest code automatically without making new image every single time.
I already have a private github.
Is it a only way to solve Auto-Scaling code management ?
If so, how can I configure this to work?
Use user-data scripts, which work on a lot of public images including Amazon's. You could have it download puppet manifests/templates/files and run directly. Search for master less puppet.
Yes, you can configure your AMI so that the instance loads the latest software and configuration on first boot before it is put into service in the auto scaling group.
How to set up a startup script may depend on the specific os and version you are running.