How to connect to MongoDB replicaset on Kubernetes - mongodb

I have deployed bitnami's Mongodb helm chart in the replicaset mode on a K8s cluster, using this minimal config (for now):
architecture: replicaset
auth:
enabled: false
Now I have two replicas (one primary and one secondary, allegedly) and an arbiter (no data, just voting rights). Additionally, I got two services: appname-mongodb-headless, appname-mongodb-arbiter-headless. However, the MongoDB docs specify the connection string in case of replicaset installations as mongodb://mongodb0.example.com:27017,mongodb1.example.com:27017,mongodb2.example.com:27017/?replicaSet=myRepl.
I can use mongodb://appname-mongodb-headless:27017 to connect to the db, but is it the "real experience" (i.e. highly available, etc)?
Do I need to specify the replicaset name (as in the example) and how do I find out what it is?
If the current services are enough, do I need to include the arbiter in the connection string?
Or should I create separate services for each replica and build the recommended connection string?
externalAccess:
enabled: true
autoDiscovery:
enabled: true
I will eventually enable authentication after sorting out this part of the connection. What is this replicaSetKey ("Key used for authentication in the replicaset"), and where do I use it?
I found this similar question. It is for another chart, but it suggests me that some combination of the headless services should be enough. Nonetheless, it doesn't answer all aspects of my question.

The headless service serves as an entrypoint for you to connect to MongoDB, without the need to specify a pod address. Kubernetes will balance the load and serve it to a specific pod.
If you wish to connect to MongoDB, using externalAccess.enabled, you should take a look at the Helm deployment notes that are shown when you deploy the chart. You can read them post-deployment by executing the helm get notes MY-RELEASE command.
For instance, in my case, it shows this, though depending on the deployment options it would show something else:
To connect to your database, create a MongoDB® client container:
kubectl run --namespace default mbmongo-mongodb-client --rm --tty -i --restart='Never' --env="MONGODB_ROOT_PASSWORD=$MONGODB_ROOT_PASSWORD" --image docker.io/bitnami/mongodb:4.4.8-debian-10-r24 --command -- bash
Then, run the following command:
mongo admin --host "mbmongo-mongodb-0.mbmongo-mongodb-headless.default.svc.cluster.local:27017,mbmongo-mongodb-1.mbmongo-mongodb-headless.default.svc.cluster.local:27017,mbmongo-mongodb-2.mbmongo-mongodb-headless.default.svc.cluster.local:27017" --authenticationDatabase admin -u root -p $MONGODB_ROOT_PASSWORD
To connect to your database nodes from outside, you need to add both primary and secondary nodes hostnames/IPs to your Mongo client. To obtain them, follow the instructions below:
MongoDB® nodes domain: you can reach MongoDB® nodes on any of the K8s nodes external IPs.
kubectl get nodes -o wide
MongoDB® nodes port: You will have a different node port for each MongoDB® node. You can get the list of configured node ports using the command below:
echo "$(kubectl get svc --namespace default -l "app.kubernetes.io/name=mongodb,app.kubernetes.io/instance=mbmongo,app.kubernetes.io/component=mongodb,pod" -o jsonpath='{.items[*].spec.ports[0].nodePort}' | tr ' ' '\n')"
From the notes you can see this line:
To connect to your database nodes from outside, you need to add both primary and secondary nodes hostnames/IPs to your Mongo client
That means that you do need to specify all primary/secondary node names, but not arbiters.
As for replicaSetKey, it refers to the contents of the MongoDB keyFile to use for internal cluster replication, so it can happen in a secure manner. You can read more about it here.
If you continue finding issues, I suggest you create an issue in the bitnami/charts repository.

Adding to Marcos' answer, there is another URI format called DNS Seed List Connection Format from 3.6+ which you can use with a single host name of your headless service and still take advantage of the replicaset.
mongodb+srv://server.example.com/
It is described here: https://docs.mongodb.com/manual/reference/connection-string/#dns-seed-list-connection-format

Related

How to get Kubernetes cluster name from K8s API using client-go

How to get Kubernetes cluster name from K8s API mentions that
curl http://metadata/computeMetadata/v1/instance/attributes/cluster-name -H "Metadata-Flavor: Google"
(from within the cluster), or
kubectl run curl --rm --restart=Never -it --image=appropriate/curl -- -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/attributes/cluster-name
(from outside the cluster), can be used to retrieve the cluster name. That works.
Is there a way to perform the same programmatically using the k8s client-go library? Maybe using the RESTClient()? I've tried but kept getting the server could not find the requested resource.
UPDATE
What I'm trying to do is to get the cluster-name from an app that runs either in a local computer or within a k8s cluster. the k8s client-go allows to initialise the clientset via in cluster or out of cluster authentication.
With the two commands mentioned at the top that is achievable. I was wondering if there was a way from the client-go library to achieve the same, instead of having to do kubectl or curl depending on where the service is run from.
The data that you're looking for (name of the cluster) is available at GCP level. The name itself is a resource within GKE, not Kubernetes. This means that this specific information is not available using the client-go.
So in order to get this data, you can use the Google Cloud Client Libraries for Go, designed to interact with GCP.
As a starting point, you can consult this document.
First you have to download the container package:
➜ go get google.golang.org/api/container/v1
Before you will launch you code you will have authenticate to fetch the data:
Google has a very good document how to achieve that.
Basically you have generate a ServiceAccount key and pass it in GOOGLE_APPLICATION_CREDENTIALS environment:
➜ export GOOGLE_APPLICATION_CREDENTIALS=sakey.json
Regarding the information that you want, you can fetch the cluster information (including name) following this example.
Once you do do this you can launch your application like this:
➜ go run main.go -project <google_project_name> -zone us-central1-a
And the result would be information about your cluster:
Cluster "tom" (RUNNING) master_version: v1.14.10-gke.17 -> Pool "default-pool" (RUNNING) machineType=n1-standard-2 node_version=v1.14.10-gke.17 autoscaling=false%
Also it is worth mentioning that if you run this command:
curl http://metadata/computeMetadata/v1/instance/attributes/cluster-name -H "Metadata-Flavor: Google"
You are also interacting with the GCP APIs and can go unauthenticated as long as it's run within a GCE machine/GKE cluster. This provided automatic authentication.
You can read more about it under google`s Storing and retrieving instance metadata document.
Finally, one great advantage of doing this with the Cloud Client Libraries, is that it can be launched externally (as long as it's authenticated) or internally within pods in a deployment.
Let me know if it helps.
If you're running inside GKE, you can get the cluster name through the instance attributes: https://pkg.go.dev/cloud.google.com/go/compute/metadata#InstanceAttributeValue
More specifically, the following should give you the cluster name:
metadata.InstanceAttributeValue("cluster-name")
The example shared by Thomas lists all the clusters in your project, which may not be very helpful if you just want to query the name of the GKE cluster hosting your pod.

k8s, RabbitMQ, and Peer Discovery

We are trying to run an instance of the RabbitMQ chart with Helm from the helm/charts/stable/rabbit project. I had it running perfect but then I had to restart k8s for some maintenance. Now we are completely unable to launch the RabbitMQ chart in any way shape or form. I am not even trying to run the chart with any variables, i.e. just the default values.
Here is all I am doing:
helm install stable/rabbitmq
I have confirmed I can simply run the default right on my local k8s which I'm running with Docker for Desktop. When we run the rabbit chart on our shared k8s the exact same way as on desktop and what we did before the restart, the following error is thrown:
Failed to get nodes from k8s - 503
I have also posted an issue on the Helm charts repo as well. Click here to see the issue on Github.
We are suspecting the DNS but are unable to confirm anything yet. What is very frustrating is after the restart every single other chart we installed restarted perfectly except Rabbit which now will not start at all.
Anyone know what I could do to get Rabbits peer discovery to work? Anyone seen issue like this after restarting k8s?
So I actually got rabbit to run. Turns out my issue was the k8s peer discovery could not connect over the default port 443 and I had to use the external port 6443 because kubernetes.default.svc.cluster.local resolved to the public port and could not find the internal, so yeah our config is messed up too.
It took me a while to realize the variable below was not overriding when I overrode it with helm install . -f server-values.yaml.
rabbitmq:
configuration: |-
## Clustering
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s
cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
cluster_formation.k8s.port = 6443
cluster_formation.node_cleanup.interval = 10
cluster_formation.node_cleanup.only_log_warning = true
cluster_partition_handling = autoheal
# queue master locator
queue_master_locator=min-masters
# enable guest user
loopback_users.guest = false
I had to add cluster_formation.k8s.port = 6443 to the main values.yaml file instead of my own. Once the port was changed specifically in the values.yaml, rabbit started right up.
I'm wondering what is the reason of using rabbit_peer_discovery_k8s plugin, if values.yaml defaults to 1 replicas (your manifest file does not override this setting) ?
I was trying to reproduce your issue with given by you override values (dev-server.yaml), as per the details in your github issue #10811, but I somewhat failed. Here are my observations:
If to install RabbitMQ chart with your custom values, my rabbitmq-dev-default-0 pod gets stuck in CrashLoopBackOff state.
It`s quite hard to troubleshoot it further for me as bitnami`s rabbitmq image containers, used by this rabbitmq Helm chart, are shipped with non-root account.
On the other hand if rabbitmq chart is installed on my Kubernetes cluster (v1.13.2) in simplest form:
helm install stable/rabbitmq
I observe similar issue then. I mean rabbitmq server survives a simulated VM restart of all cluster nodes (including master), but I cannot connect to it from outside:
Post VM restart, I`m getting following error from my python mqclient:
socket.gaierror: [Errno -2] Name or service not known
Few remarks here:
Yes, I did port(s)-forward as per instructions on "helm status " command:
The readiness probe works fine:
curl -sS -f --user user:<my_pwd> 127.0.0.1:15672/api/healthchecks/node
{"status":"ok"}
rabbitmqctl to rabbitmq-server connectivity from inside the container works fine too:
kubectl exec rabbitmq-dev-default-0 -- rabbitmqctl list_queues
warning: the VM is running with native name encoding of latin1 which may cause Elixir to malfunction as it expects utf8. Please ensure your locale is set to UTF-8 (which can be verified by running "locale" in your shell)
Timeout: 60.0 seconds ...
Listing queues for vhost / ...
name messages
hello 11
From the moment I used kubectl port-forward to pod instead service, connectivity to rabbitmq server is restored:
kubectl port-forward --namespace default pod/rabbitmq-dev-default-0 5672:5672
$ python send.py
[x] Sent 'Hello World!'

oc cluster up command cannot find components such as metrics, logging, service-catalog

Thanks a lot for your help in advance. I am able to run all-in-one openshift origin v3.11 on my local virtualbox VM.
https://github.com/openshift/origin/blob/release-3.11/docs/cluster_up_down.md
Followed by this document, I learned that I am able to install additional components such as --metrics, --logging, --service-catalog, but I am trying to append this argument in oc cluster up, the usage said these options are not available, such as:
[root#master openshift]# oc cluster up --metrics
Error: unknown flag: --metrics
Usage:
oc cluster up [flags]
Examples:
# Start OpenShift using a specific public host name
oc cluster up --public-hostname=my.address.example.com
Options:
--base-dir='': Directory on Docker host for cluster up configuration
--enable=[*]: A list of components to enable. '*' enables all on-by-default components, 'foo' enables the component named 'foo', '-foo' disables the component named 'foo'.
All components: automation-service-broker, centos-imagestreams, persistent-volumes, registry, rhel-imagestreams, router, sample-templates, service-catalog, template-service-broker, web-console
Disabled-by-default components: automation-service-broker, rhel-imagestreams, service-catalog, template-service-broker
--forward-ports=false: Use Docker port-forwarding to communicate with origin container. Requires 'socat' locally.
--http-proxy='': HTTP proxy to use for master and builds
--https-proxy='': HTTPS proxy to use for master and builds
--image='openshift/origin-${component}:${version}': Specify the images to use for OpenShift
--no-proxy=[]: List of hosts or subnets for which a proxy should not be used
--public-hostname='': Public hostname for OpenShift cluster
--routing-suffix='': Default suffix for server routes
--server-loglevel=0: Log level for OpenShift server
--skip-registry-check=false: Skip Docker daemon registry check
--write-config=false: Write the configuration files into host config dir
Use "oc options" for a list of global command-line options (applies to all commands).
[root#master openshift]#
The same also fail when I append --logging, --service-catalog.
Note that, if I remove the argument, oc cluster up can startup successful. Should I install something that I missed?

Access to Mongodb in Kubernetes

I created a Mongodb service according to the Kubernetes tutorial.
Now my question is how do I gain access to the database itself, with a client like Robomongo or similar clients? Just for making backups or exploring what data have been entered.
The mongo-pod and service only have an internal endpoint, and a single mount.
Is there any way to safely access this instance with no public endpoint?
Internally URI is mongo:27***
You can use kubectl port-forward mypod 27017:27017 and then just connect your mongodb client to localhost:27017.
If you want to stop, just hit Ctrl+C on the same cmd window to stop the process.
The kubernetes cmd-line tool provides this functionality as #ainlolcat stated
kubectl get pods
Retrieves the pod names currently running and with:
kubectl exec -i mongo-controller-* bash
you get a basic bash, which lets you execute
mongo
to get into the database to create dumps, and so on. The bash is very basic and has no features like completion and so on. I have not found a solution for better shell but it does the job
when you create a service in kubernetes you give it a name, say for example "mymongo". After the service is created then
The DNS service of kubernetes (by default is on) will ensure that any pod can discover this servixe simply by its name. so you can set your uri like
uri: mongodb://**mymongo**:27017/mong
In addition the service IP and port will be set as environment variables at the running pod.
MYMONGO_SERVICE_HOST
MYMONGO_SERVICE_PORT
I have in fact wrote a blog that show a step by step example of an app with nodejs web server and mongo that can explain further
http://codefresh.io/blog/kubernetes-snowboarding-everything-intro-kubernetes/
feedback welcome!
Answer from #grchallenge is correct but it is deprecated as of in 2021
All new comers please use
kubectl exec mongo-pod-name -i -- bash

Get the nodes/pods information using cluster name or context name when we have multiple configs

I'm trying to fetch nodes list via ansible playbook using a context name. but its not working
my playbook:
getnodes.yaml
- name: "get nodes"
hosts: kubernetes
tasks:
- name: "nodes"
command: "kubectl get nodes --context='contextname'"
I do have multiple clusters in config file. I need to either specify cluster name or context name and get the nodes list or to perform any activity on a particular cluster
As far as I understand you when you run the command kubectl get nodes --context='contextname' directly on your master node, everything works fine, right ? And it fails only when you run it as a part of your ansible playbook against the master node ? What errors do you get ?
Yes that's correct. i'm able to execute from command line
"The connection to the server localhost:8080 was refused - did you
specify the right host or port?"
Are you sure it is available on the same host as you run your ansible playbook ? I mean your Kubernetes master node, on which you have kubectl binary installed ? My guess is that it is not and even if it is on the same host you'll not be able to connect to it using localhost:8080.
Look. You're not using here any particular Ansible module specific to manage Kubernetes cluster like this one, which you run directly against the API server and you need to provide its valid URL. Instead here you are just using simple command module which doesn't care what command you want to run as long as you provide a valid hostname with ssh access and Python installed.
In this case your Ansible simply tries to ssh to your Kubernetes master node and execute the shell command you passed to it:
kubectl get nodes --context='contextname'
I really doubt that your ssh server listens on port 8080.
If you run your ansible playbook on same host you can run your kubectl commands there are much easier solutions in Ansible for such cases like:
local_action or delegate_to: localhost statements in your task or more globally connection: local
More details on usage of all above mentioned statements in your Ansible plays you can find in Ansible docs and in this article.
I hope it will help you.