I need to do this weird scenario and I need to have a better way of doing it.
I have the following setup running as docker containers
A simple python web server
An nginx reverse proxy configured to route traffic to the above python server
I have setup docker volumes to mount the nginx logs to a host path
And everytime I access my simple web page, it produces the usual nginx access logs.
My requirement is:
access to the page should result in the creation of a new document in the MongoDB collection
My problem is, how can I get notified when nginx logs a new log stream so I can send the data to mongodb
My workaround: Since I have mounted the nginx logs to a host path, I can have some kind of a python code running and listening to the file in the host path where the nginx logs are being mounted and when every-time a modification happens, sends the modified content to mongodb. But this approach is really unreliable. Is there a better way of doing this at container level?
Thank you!
Related
I'm trying to setup Helm chart with some dependencies like MySQL, RabbitMQ and so on and when my actual microservice is starting, the moment when first connection is established to MySQL from the microservice, both instantly crash.
It works with docker-for-desktop but with minikube it doesn't work.
I tried manually to get inside the pod (of microservice and others also) and to try to login to MySQL server (MySQL pod) and it still crashes without any exception
BUT
strange thing is that if I try to login with wrong credentials for first the time it doesn't crash, it shows me an error that wrong credentials are in question and after it if I try with correct ones, it succeeds!
If I try to login from MySQL pod inside MySQL server it logins correctly.
Curl to MySQL port returns version so it works like it should, only the login to MySQL from external pod is the problem.
Does anyone of you have an idea what's going on here?
Maybe you can wait for your application to have dependencies ready before requesting them.
You could use readiness probe and init containers to delay your application from starting.
I have set up a Kubernetes cluster. The cluster contains, among other things, a cluster and deployment surfacing an API webservice (based on the subway-explorer-gmaps-proxy container).
I've deployed the service externally, using the LoadBalancer service type (this is on GCP):
$kubectl get svc subway-explorer-gmaps-proxy-service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
subway-explorer-gmaps-proxy-service LoadBalancer 10.35.252.232 35.224.78.225 9000:31396/TCP 19h
My understanding (and correct me if I'm wrong!) is that this service should now be queryable outside of the cluster, by visiting http://35.224.78.225 in the browser.
When running the Docker container locally, I can verify things are working correctly by navigating to the following URL:
http://localhost:49161/starting_x=-73.954527&starting_y=40.587243&ending_x=-73.977756&ending_y=40.687163
Looking at the kubectl get output, I expect visiting the following URL in the browser will serve me the content I'm looking for:
http://35.224.78.225:31396/starting_x=-73.954527&starting_y=40.587243&ending_x=-73.977756&ending_y=40.687163
But when I visit this URL, nothing gets served.
I suspect there is a non-fatal error in the deployment configuration. What is an effective way of debugging this effective way of debugging this problem? Are there access logs or a stdout stream somewhere I can check to see what's wrong?
You can try running through the official docs on debugging services: https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/
Beyond that, have you confirmed you're querying the load balancer on the right port? While I don't deploy on GCP, when launching a load balancer for a kubernetes service on AWS it'll accept traffic on port 80/443 and forward it to the NodePort of the service, which I'm guessing is 31396 for your case. What are the ports listed in kubectl get svc subway-explorer-gmaps-proxy-service -o yaml?
What I didn't realize is that Google Cloud has a separate firewall system, which is distinct from the connection settings managed by Kubernetes. In order to expose the application to the outside world (e.g. a web browser, for example), I need to also modify the Google Cloud Firewall rules (see for example this answer as to how).
To test that the application is working on the Kubernetes side, you need not modify cloud firewall rules. Instead, run wget, curl, or some similar data retrieval command from a different pod on the cluster, pointed at the internal IP address and port number of the pod of interest.
For example. The "hello world" pod used by the Kubernetes documentation is the busybox pod (defined here). By creating this pod in my cluster, and then running the following:
kubectl exec busybox -c busybox -- wget "10.35.249.23:9000"
I was able to confirm that the service is functioning correctly within Kubernetes. You can also use any other pod which defines a wget in the underlying OS, I just used busybox because all of my other pods use Google's Container Optimized OS, which doesn't include it.
Finally, for the purposes of debugging, I went ahead and added a /status endpoint to my API application service which serves {"status": "OK"} when the core service is working. I recommend following this pattern with other applications as well, as it gives a simple endpoint that you can test to make sure that, at a minimum, the webserver is responding to input. In my case, I discovered that the /status page is OK, but the API calls are failing, which allows me to narrow the issue down to unresolved Promises caused by a bad credentials secret.
I'm a complete newbie with Kubernetes, and have been trying to get secure CockroachDB running. I'm using the instructions and preconfigured .yaml files provided by Cockroach. https://www.cockroachlabs.com/docs/stable/orchestrate-cockroachdb-with-kubernetes.html
I'm using the Cloud Shell in my Google Cloud console to set everything up. Everything goes well, and I can do local SQL tests and load data. Monitoring the cluster by proxying to localhost, with the comnmand below starts off serving as expected
kubectl port-forward cockroachdb-0 8080
However, when using cloud shell web preview on port 8080 to attach to localhost, the browser session returns "too many redirects".
My next challenge will be to figure out how to expose the cluster on a public address, but for now I'm stuck on what seems to be a fairly basic problem. Any advice would be greatly appreciated.
Just to make sure this question has an answer, the problem was that the question asker was running port-forward from the Google Cloud Shell rather than from his local machine. This meant that the service was not accessible to his local machine's web browser (because the Cloud Shell is running on a VM in Google's datacenters).
The ideal solution is to run the kubectl port-forward command from his own computer.
Or, barring that, to expose the cockroachdb pod externally using the kubectl expose pod cockroachdb-0 --port=8080 --type=LoadBalancer as suggested in the comments.
I have a fully functioning Kubernetes cluster with one master and one worker, running on CoreOS.
Everything is working and my pods and services are running fine. Now I have no clue how to proceed in a webserver idea.
Before I go further: I have no configs at the moment about my idea I am going to explain. I just did a lot of research.
When setting up a pod (nginx) with a service. You get the default nginx page. After that you can setup a mount volume with a hostvolume (volume mapping from host to container).
But lets say I want to seperate every site (multiple sites separated with different pods), how can I let my users add files to their pod/nginx document root? Having FTP in the CoreOS node removes the Kubernetes way and adds security vulnerabilities.
If someone can help me shed some light on this issue, that would be great.
Thanks for your time.
I'm assuming that you want to have multiple nginx servers running. The content of each nginx server is managed by a different admin (you called them users).
TL;DR:
Option 1: Each admin needs to build their own nginx docker image every time the static files change and deploy that new image. This is if you consider these static files as a part of the source-code of the nginx application
Option 2: Use a persistent volume for nginx, the init-script for the nginx image should use something like s3 to sync all its files with s3 and then start nginx
Before you proceed with building an application with kubernetes. The most important thing is to separate your services into 2 conceptual categories, and give up your desire to touch the underlying nodes directly:
1) Stateless: These are services that are built by the developers and can be released. They can be stopped, started, moved from one node to another, their filesystem can be reset during restart and they will work perfectly fine. Majority of your web-services will fit this category.
2) Stateful: These services cannot be stopped and restarted willy nilly like the ones above. Primarily, their underlying filesystem must be persistent and remain the same across runs of the service. Databases, file-servers and similar services are in this category. These need special care and should use k8s persistent-volumes and now stateful-sets.
Typical application:
nginx: build the nginx.conf into the docker image, and deploy it as a stateless service
rails/nodejs/python service: build the source code into the docker image, configure with env-vars, deploy as a stateless service
database: mount a persistent volume, configure with env-vars, deploy as a stateful service.
Separate sites:
Typically, I think at a k8s deployment and a k8s service level. Each site can be one k8s deployment and k8s service set. You can then have separate ways to expose them (different external DNS/IPs)
Application users storing files:
This is firmly in the category of a stateful service. Use a persistent volume to mount to a /media kind of directory
Developers changing files:
Say developers or admins want to use FTP to change the files that nginx serves. The correct pattern is to build a docker image with the new files and then use that docker image. If there are too many files, and you don't consider those files to be a part of the 'source' of the nginx, then use something like s3 and a persistent volume. In your docker image init script, don't directly start nginx. Contact s3, sync all your files onto your persistent volume, then start nginx.
While the options and reasoning listed by iamnat are right, there's at least one more option to add to the list. You could consider using ConfigMap objects, maintain your file within the configmap and mount them to your containers.
A good example can be found in the official documentation - check the Real World Example configuring Redis section to get some actionable input.
I'm currently building a Kubernetes cluster. I plan on using Nginx containers as a server for static content, and to act as a web socket proxy. If you restart Nginx, you lose your web socket connection, so I do not want to restart the containers. But I will want to update the content within the container.
I do that same exact thing in my Kubernetes cluster. Our solution is for application to handle the web socket disconnect with consistent state kept intact.
However, other options you have are mount a volume to serve from the host; however, you cannot guarantee all nginx pods will have that volume on multi hosts, unless you use a kubernetes' persistent volume http://kubernetes.io/v1.1/docs/user-guide/persistent-volumes.html.
Another option you have is to have your static content on an object store like S3, Google Cloud Storage or Ceph, and then proxy the object store through nginx along with the websocket.