I will build the environment using the following Terraform.
https://github.com/scalar-labs/scalar-terraform
I want to fix the cassandra.yaml placed in a Cassandra instance with Terraform.
How do I change the Terraform?
You need to actually update the cassandra.yaml.j2 of the Cassandra module.
provision/ansible/playbooks/roles/cassandra/templates/cassandra.yaml.j2
It is not very recommended since you need to maintain the module itself if you do this.
Related
Given a container in a Azure container registry and a kubernetes cluster setup via the portal. Are there any visual tools that I can use so that I don't have to use the command line commands ,for things like add/edit the yaml file and launching the cluster?
For example I found this tool https://k8syaml.com/, but this is only one part of the process and it is also not aware of the existing infrastructure.
What are the visual tools to manage kubernetes end-to-end?
One tool I always work with when dealing with Kubernetes is Lens. Here is a video showing you what it can do. Best of all, it just needs the kube config file and so it is agnostic to where the Kubernetes cluster is (On-Prem, GKE, AKS, EKS)
kubectx for switching between contexts (clusters) & K9s is widely used that is something hybrid between being a cli and visual tool.
Octant is another option - https://github.com/vmware-tanzu/octant, it is similar to lens
I would like to use the Datadog Oracle Integration via the Helm Chart Datadog. Oracle Integration states To use the Oracle integration, either install the Oracle Instant Client libraries, or download the Oracle JDBC Driver.
I do not want to use a custom image to package the JDBC-driver, I want to use a standard image such as tag:7-jmx. Other options that come to mind (e.g. EFS volume with the driver inside) seem to be an overkill also.
Best option to me seems to be an init container that downloads the JDBC driver. But Datadog Helm Chart does not support custom init containers for the agents.
What's the best way to do this? To get an Datadog Agent with a JDBC driver via Helm?
Answer from Datadog Support to this:
Thanks again for reaching out to Datadog!
From looking further into this, there does not seem to be a way we can package the JDBC driver with the Datadog Agent. I understand that this is not desirable as you would prefer to use a standard image but I believe the best way to have these bundled together would be to have a custom image for your deployment.
Apologies for any inconveniences that this may cause.
I am interested in generating docker-compose.yaml files from Helm charts. Is there a good way or tool to do this?
I realize that this is in the opposite direction from what most people are doing. Why I want to do this:
Our production systems run Kubernetes via Helm charts. We've got a full blown k8s and Helm setup already; no need to use a tool like Kompose to get us there. The question is how to convert Helm to docker-compose, not the other way around.
We want our Helm charts to be the single authoritative source of container configuration. They are able to describe a superset of what docker-compose can.
Running a set of services using Helm on a development machine is more time and resource consuming than running the same set of services via docker-compose. We do not want to slow development down by having engineers run using Helm/k8s.
We do not want to maintain two sets of configurations.
Can anybody recommend how to do this, or suggest a different solution to the time/resources issue encountered on development machines?
I have set up a Kubernetes cluster using Kubernetes Engine on GCP to work on some data preprocessing and modelling using Dask. I installed Dask using Helm following these instructions.
Right now, I see that there are two folders, work and examples
I was able to execute the contents of the notebooks in the example folder confirming that everything is working as expected.
My questions now are as follows
What are the suggested workflow to follow when working on a cluster? Should I just create a new notebook under work and begin prototyping my data preprocessing scripts?
How can I ensure that my work doesn't get erased whenever I upgrade my Helm deployment? Would you just manually move them to a bucket every time you upgrade (which seems tedious)? or would you create a simple vm instance, prototype there, then move everything to the cluster when running on the full dataset?
I'm new to working with data in a distributed environment in the cloud so any suggestions are welcome.
What are the suggested workflow to follow when working on a cluster?
There are many workflows that work well for different groups. There is no single blessed workflow.
Should I just create a new notebook under work and begin prototyping my data preprocessing scripts?
Sure, that would be fine.
How can I ensure that my work doesn't get erased whenever I upgrade my Helm deployment?
You might save your data to some more permanent store, like cloud storage, or a git repository hosted elsewhere.
Would you just manually move them to a bucket every time you upgrade (which seems tedious)?
Yes, that would work (and yes, it is)
or would you create a simple vm instance, prototype there, then move everything to the cluster when running on the full dataset?
Yes, that would also work.
In Summary
The Helm chart includes a Jupyter notebook server for convenience and easy testing, but it is no substitute for a full fledged long-term persistent productivity suite. For that you might consider a project like JupyterHub (which handles the problems you list above) or one of the many enterprise-targeted variants on the market today. It would be easy to use Dask alongside any of those.
Is there any way to deploy a particular version of Kubernetes to ACS in Azure?
Using Azure resource manager, or az command.
It doesn't seem like template format for container service shows this info.
You can specify the version in ACS in selected regions. See the template example.
https://github.com/weinong/azure-quickstart-templates/tree/master/101-acs-kubernetes-with-version
We will be updating azure cli with this feature soon.
I suggest you to use acs-engine in this case. It is a tool for you to specify some custom definition like "orchestratorversion" for your case, and it can then generate an ARM template for deploying the k8s cluster.
You could download the acs-engine tool here https://github.com/Azure/acs-engine/releases (choose the version based on your need for the k8s version).
To achieve your goal, you have to provide a json file which the template you could find here https://github.com/Azure/acs-engine/blob/master/examples/kubernetes-releases/kubernetes1.7.json. You could alter the attribute "orchestratorversion" to either 1.5, 1.6 or 1.7 to suit your need. (or maybe 1.8 for the latest version)
When the json file is ready, you could turn it into the ARM template files by typing the following command
.\acs-engine.exe generate kubernetes.json
This will create a new directory called "_output" and you could find the azuredeploy.json and azuredeploy.parameters.json files there.
For more information about the attributes in the json file, take a look at https://github.com/Azure/acs-engine/blob/master/docs/clusterdefinition.md.
Another way you could also try is in the Deployment via Azure CLI specified here. https://github.com/Azure/ACS/tree/master/docs