I'm running a storm (v1.2.1) container with the command:
storm,ui
on ECS. The UI container runs on the same task as the zookeeper and nimbus.
I've deployed the task on FARGATE as a service which has service discovery enabled.
The containers are all running fine with no errors in the logs.
I've configured the task definition to map port 8080 so that I can access the storm UI.
However, when I try, all I get is a jetty 404 page. This tells me that I'm hitting the container, but somehow the storm ui is not there. I suppose an alternative is that I'm hitting a different container, but I'm not sure how that's possible.
This is the error I see:
Why is storm UI giving me a 404?
I finally got access to the container logs using ECS ExecCommand and the error was that it couldn't bind to the port.
I modified the Storm UI container to include the following command flag:
-c ui.port=[not 8080]
and mapped the new port from the container to the host. The UI worked, then.
Related
I know you can debug java microservices with Telepresence. And Scala runs on JVM, so it shouldn't be a problem right? But for some reason I couldn't manage it.
I have followed this video on youtube for a tutorial of Telepresence.
I intercepted the service I wanted to debug. It gave some errors with Kafka and mariadb(mysql). So in IntelliJ I changed their configuration according to their properties on AKS like host = "kafka.default:9092", because its namespace is default, name is kafka and port on AKS is 9092. Both Kafka and Mysql seems to be connected and currently they are not giving any errors.
In IntelliJ I am calling a Boot.scala class which reads the config and builds the project as far as I understand. When I change this service's port to 80 in local application.conf (as it is in AKS) it gives me the following error. In any other port it just listens.
I|2022-07-13 15:41:00,893|c.a.libats.http.tracing.Tracing$|Request tracing disabled in config
E|2022-07-13 15:41:01,901|akka.io.TcpListener|Bind failed for TCP channel on endpoint [/0.0.0.0:80]
And a fetch process on the website is not finished when I intercept this service with Telepresence. I have breakpoints nearly everywhere, I always debug the project and it never hits any one of them. So, what exactly is going on? Also I am open to redirections on remote debugging AKS.
PS: This project is something I inherited and I don't have any previous scala experience. So I may be missing something easy.
PS2: Also nothing changes when I leave telepresence intercept and just run it locally. Same logs, same situation with 80 port.
Okay, it works. Just add telepresence it will intercept the pod and attach to your local application instead of the code running on the pod. So, you have to do your configuration accordingly, because it will be the code running inside the pod. Just don't use port 80 because it is reserved for HTTP even though when you "kubectl get svc -A" it says port as 80. Just use telepresence intercept nameoftheservice without specifying the port, it will do it instead of you and change the port accordingly in your local environment configuration.
Updated
I have some script that initializes our service.
The script fails when it runs in the container because of connection refused error in the first outbound request (to external service) in the script.
We tried to add a loop that makes curl and if it fails, re-try, if not - continuous the script.
Sometimes it succeeds for the first time, sometimes it fails 10-15 times in a row.
We recently started using istio
What may be a reason??
It is a famous istio bug https://github.com/istio/istio/issues/11130 ( App container unable to connect to network before Istio's sidecar is fully running) it seems the Istio proxy will not start in parallel , it is waiting for the app container to be ready. a sequential startup sequence as one blogger mentioned (https://medium.com/#marko.luksa/delaying-application-start-until-sidecar-is-ready-2ec2d21a7b74) quote: most Kubernetes users assume that after a pod’s init containers have finished, the pod’s regular containers are started in parallel. It turns out that’s not the case.
containers will start in order defined by the Deployment spec YAML.
so the biggest question is will the Istio proxy envoy will start while the first container is stuck in a curl-loop . (egg and chicken problem) .
App container script performs:
until curl --head localhost:15000 ; do echo "Waiting for Istio Proxy to start" ; sleep 3 ; done
as far as I saw: that script doesn't help a bit. proxy is up but connection to external hostname return "connection refused"
With istio 1.7 comes a new feature that configures the pod to start the sidecar first and hold every other container untill the sidecar is started.
Just set values.proxy.holdApplicationUntilProxyStarts to true.
Please note that the feature is still experimental.
I am trying to run 2 containers on Google Kubernetes Engine, both running ZeroMQ using the PUB/SUB model to relay messages from one container to the other. I am developing on Visual Studio Code and using the Cloud Code plugin to deploy the containers to my remote cluster using scaffold.
The GKE cluster consists of 3 nodes running e2-small instances
When I deploy the containers they get seem to get deployed correctly, however in the output section I never see any of the logs showing that the messages are getting sent correctly as the code suggests they should.
Please find the code here : https://github.com/alhennessey92/zeromq-kubernetes-test I think it is relevant to this issue as well as the final output logs from the terminal.
Final logs from terminal
Starting deploy...
- service/test-dev-pub created
- service/test-dev-sub created
- deployment.apps/test-dev-pub created
- deployment.apps/test-dev-sub created
Deploy complete in 969.953727ms
Waiting for Deployment 'test-dev-pub' to rollout...
Port forwarding service/test-dev-pub in namespace default, remote port 5556 -> local port 5556
Port forwarding service/test-dev-sub in namespace default, remote port 5556 -> local port 5557
Watching for changes...
Streaming logs from pod: test-dev-sub-84ccccbdc8-slnvr container: sub
Streaming logs from pod: test-dev-pub-6f9b99bccf-t4q56 container: pub
Waiting for Deployment 'test-dev-sub' to rollout...
No publicly exposed service endpoints found in the application.
No ingress endpoints found in the application.
I am not sure why this does not seem to be working as it should, I should be seeing the printed logs displaying from both the pub and sub containers, but am not.
I am trying to implement a service mesh to a service with Kubernetes using Istio and Envoy. I was able to set up the service and istio-proxy but I am not able to control the order in which the container and istio-proxy are started.
My container is the first started and tries to access an external resource via TCP but at that time, istio-proxy has not completely loaded and so does the ServiceEntry for the external resource
I tried adding a panic in my service and also tried with a sleep of 5 seconds before accessing the external resource.
Is there a way that I can control the order of these?
On istio version 1.7.X and above you can add configuration option values.global.proxy.holdApplicationUntilProxyStarts, which causes the sidecar injector to inject the sidecar at the start of the pod’s container list and configures it to block the start of all other containers until the proxy is ready. This option is disabled by default.
According to https://istio.io/latest/news/releases/1.7.x/announcing-1.7/change-notes/
I don't think you can control the order other than listing the containers in a particular order in your pod spec. So, I recommend you configure a Readiness Probe so that you are pod is not ready until your service can send some traffic to the outside.
Github issue here:
Support startup dependencies between containers on the same Pod
We're currently recommending that developers solve this problem
themselves by running a startup script on their application container
which delays application startup until Envoy has received its initial
configuration. However, this is a bit of a hack and requires changes
to every one of the developer's containers.
I've built a Container that leverages a CF app that's bound to a service, Cloudant to be specific.
When I run the container locally I can connect to my Cloudant service.
When I build and run my image in the Bluemix container service I can no longer connect to my Cloudant service. I did use --bind to bind my app to the container. I have verified that the VCAP_Services info is propagating to my container successfully.
To narrow the problem down further, I tried just doing an
ice -run --name NAME IMAGE_NAME ping CLOUDANT_HOST
and I found I was getting an unknown host.
So I then tried to just ping the IP, and got Network is unreachable.
If we can not resolve bluemix services over the network, how can we leverage them? Is there just a temporary problem, or perhaps I'm missing something?
Again, runs fine locally but fails when hosted in the container service.
It has been my experience that networking is not reliable in IBM Containers for about 5 seconds at startup. Try adding a "sleep 10" to your CMD or ENTRYPOINT. Or set it up to retry for X seconds before giving up.
Once the networking comes up it has been reliable for me. But the first few seconds of a containers life have had troubles with DNS, binding, and outgoing traffic.
looking at your problem it could be related to a network error on container when on Bluemix.
Try to access your container through shell when on Bluemix (using cf ic console or docker one) and check if the network has been rised correctly and then if its network interface(s) has an IP to use.