I am trying to get Grafana stood up in a ECS Fargate stack. The stack is comprise of the following:
An Application Load Balancer with an ACM cert applied listening on port 443
The ALB's target group is forwarding traffic port 3000 on my Grafana container. SSL is terminated at the ALB so the traffic coming into the container is non-SSL.
I have an A record created in Route53 that points to the ALB.
The container is using ephemeral storage that Fargate provides to the container.
What's wrong?
When I try to access Grafana from my A-record I get to the home page fine.
But when I'm prompted to enter the user name and password it fails with an unauthorized exception. On my very first attempt, when asked to change the password. I get a "password was changed successfully" followed by an "unauthorized" message at the same time.
However, if try to go directly to the the container's private IP address everything works fine.
Is this a CORS issue or is there a setting I need to pass into the container to tell is to accept traffic from the ALB as the origin (e.g. whitelisting).
Thanks!
Issue was resolved by turning on session stickiness in the target group property for the ALB.
Related
I have an issue with my kubernetes routing.
The issue is that one of the pods makes a GET request to auth.domain.com:443 but the internal routing is directing it to auth.domain.com:8443 which is the container port.
Because the host returning the SSL negotiation identifies itself as auth.domain.com:8443 instead of auth.domain.com:443 the connection times out.
[2023/01/16 18:03:45] [provider.go:55] Performing OIDC Discovery...
[2023/01/16 18:03:55] [main.go:60] ERROR: Failed to initialise OAuth2 Proxy: error intiailising provider: could not create provider data: error building OIDC ProviderVerifier: could not get verifier builder: error while discovery OIDC configuration: failed to discover OIDC configuration: error performing request: Get "https://auth.domain.com/realms/master/.well-known/openid-configuration": net/http: TLS handshake timeout
(If someone knows the root cause of why it is not identifying itself with the correct port 443 but instead the container port 8443, that would be extremely helpful as I could fix the root cause.)
To workaround this issue, I have the idea to force it to route out of the pod onto the internet and then back into the cluster.
I tested this by setting up the file I am trying to GET on a host external to the cluster, and in this case the SSL negoiation works fine and the GET request succeeds. However, I need to server the file from within the cluster, so this isn't a viable option.
However, if I can somehow force the pod to route through the internet, I believe it would work. I am having trouble with this though, because everytime the pod looks up auth.domain.com it sees that it is an internal kubernetes IP, and it rewrites the routing so that it is routed locally to the 10.0.0.0/24 address. After doing this, it seems to always return with auth.domain.com:8443 with the wrong port.
If I could force the pod to route through the full publicly routable IP, I believe it would work as it would come back with the external facing auth.domain.com:443 with the correct 443 port.
Anyone have any ideas on how I can achieve this or how to fix the server from identifying itself with the wrong auth.domain.com:8443 port instead of auth.domain.com:443 causing the SSL negotiation to fail?
SUMMARY
I have installed zabbix on OpenShift cluster. I am trying to monitor a host(vm) outside the cluster but the zabbix server is unable to connect to it. In the /etc/zabbix/zabbix_agentd.conf file I have mentioned the DNS name of the server zabbix-server but it looks like there server is trying to connect through a different public IP. I am not sure what this IP is.
OS / ENVIRONMENT / Used docker-compose files
I applied the kubernetes.yaml file present in this repo - https://github.com/zabbix/zabbix-docker/blob/6.2/kubernetes.yaml - on an OpenShift cluster.
CONFIGURATION
In the /etc/zabbix/zabbix_agentd.conf file Server=zabbix-server.
STEPS TO REPRODUCE
Apply the kubernetes.yaml file on Openshift cluster and try to monitor any external vm.
EXPECTED RESULTS
The zabbix server should be able to connect to the vm.
ACTUAL RESULTS
Zabbix server logs.
Defaulted container "zabbix-server" out of: zabbix-server, zabbix-snmptraps
\*\* Updating '/etc/zabbix/zabbix_server.conf' parameter "DBHost": 'mysql-server'...added
287:20230120:060843.131 Zabbix agent item "system.cpu.load\[all,avg5\]" on host "Host-C" failed: first network error, wait for 15 seconds
289:20230120:060858.592 Zabbix agent item "system.cpu.num" on host "Host-C" failed: another network error, wait for 15 seconds
289:20230120:060913.843 Zabbix agent item "system.sw.arch" on host "Host-C" failed: another network error, wait for 15 seconds
289:20230120:060929.095 temporarily disabling Zabbix agent checks on host "Host-C": interface unavailable
Logs from the agent installed on the vm.
350446:20230122:103232.230 failed to accept an incoming connection: connection from "9.x.x.219" rejected, allowed hosts: "zabbix-server"
350444:20230122:103332.525 failed to accept an incoming connection: connection from "9.x.x.219" rejected, allowed hosts: "zabbix-server"
350445:20230122:103432.819 failed to accept an incoming connection: connection from "9.x.x.210" rejected, allowed hosts: "zabbix-server"
350446:20230122:103533.114 failed to accept an incoming connection: connection from "9.x.x.217" rejected, allowed hosts: "zabbix-server"
If I add this IP in /etc/zabbix/zabbix_agentd.conf it will work. But what IP is this? Is this a service? Or any node/pod IP? It keeps on changing. Everytime I cannot change this id in the conf file. I need something more stable.
Kindly help me out with this issue.
So I don't know zabbix. So I have to make some educated guesses both in how the agent works and how the server works.
But, to summarize, unlike something like docker compose where you are running the zabbix server on a known server, in Openshift/Kubernetes you are deploying into a cluster of machines with their own networking. In other words, the whole point of OpenShift is that OpenShift will control where the application's pod gets deployed and will relocate/restart that pod as needed. With a different IP every time. (And the DNS name is meaningless since the two systems aren't sharing DNS anyway.) Most likely the IP's you are seeing are the pod's randomly assigned IPs.
So, what are you to do when you have a situation like yours where an external application requires a predicable IP? Well, option 1, is to remove that requirement. Using something like a certificate is obviously more secure and more reliable than depending on an IP anyway. But another option is to use an egress IP. This is a feature of OpenShift where you essentially use a proxy to provide an external application with a consistent IP.
When trying to add a Postgresql data source (running inside the service mesh/ with sidecar) to Grafana (running outside service mesh/ without sidecar), a connection cannot be established, however it is able to translate the fully qualified domain name ({app}.{namespace}.svc.cluster.local) into the correct IP address for the service.
I have attempted pinging the grafana pod from the Postgres pod by exec-ing in, which times out but correctly translates the FQDN to the IP address.
Postgres logs do not show any errors or connection attempts, grafana logs show a 500 error when trying to "Save & Test" the datasource in the grafana dashboard.
Expected behaviour is a successful connection to the datasource, would I have to configure a certain istio object to enable the out-of-service-mesh dashboard/pod to access the in-service-mesh datasource/pod.
You can use PERMISSIVE mTLS mode - see https://istio.io/docs/tasks/security/mtls-migration/.
I'm a noob when it comes to docker and aws so it's very likely that I'm missing something trivial but I've been stuck on this issue for quite some time.
So, I have three docker container hosted in aws - contA, contB and contC. contA and contB are part of the same virtual network. contC is a standalone container. contA is a running a koa server which has a bunch of api endpoints registered. contB is running an oauth service. contC is running a nodejs udp server. What I'm trying to do is make a post call to an endpoint inside contA from contC. So inside udp server I'm simply making a fetch post call to 'xxx.com/some/endpoint'. When trying to test if the communication from contC to contA works, I created containers off the images hosted in aws, mapped appropriate pubic and private ports and started the containers. But I got an error saying
'FetchError: request to xxx.com/some/endpoint failed, reason: getaddrinfo ENOTFOUND xxx.com xxx.com:443'
I don't want contC to be part of the same virtual network. I tried adding a --link from contC to contA when creating docker container. But that didn't help. Any insight on how to solve would be really helpful. Thanks!
Update
I have understood that the problem is with contC's host name resolution.I am able to access these endpoints on Postman. I am also able to ping other public endpoints. Is there a way to configure the container's /etc/resolv.conf file to resolve xxx.com to xxx.xx.xx.xxx ?
If I understood the problem right
You can not.
What you are looking for is, to contact a resource in VPC from outside of VPC. Which defeats the purpose of VPC in first place. Right!
In order to setup communication between an non VPC container and a VPC container, you can take help of API Gateway in between.
This might entertain you well
Amazon API Gateway Supports Endpoint Integrations with Private VPCs
You can now provide access to HTTP(S) resources within your Amazon
Virtual Private Cloud (VPC) without exposing them directly to the
public Internet.
Or else
Make those endpoints public, so that non VPC container can hit them.
Debug:
Can you hit those VPC secured endpoints from your laptop(public internet),
If yes, then your non VPC container should be able to do the same
If no, then how can a poor non VPC container can use those endpoints :(
Error: getaddrinfo ENOTFOUND
Also, have a look at this SO question, dealing with the same error
So either DNS is not able to resolve the URL (because, VPC shielding rocks 😎)
or
URL format is wrong
Inside a docker container, when trying to access a server inside another container one can add a record to the container's resolv.conf that points to some ip address. If that endpoint is public, you shouldn't have any trouble accessing the same public endpoint.
So when creating the container you include a --dns='some.ip.address.' option and that tells the container to resolve xxx.com to some.ip.address. The command would look something like this
docker container run -p 8080:80 --dns='192.62.0.7' image-name
I have a query which is basically a clarification regarding Routes in OpenShift Origin.
I managed to setup OpenShift Origin version 1.4.0-rc1 on a CentOS hosted in local VMWare installation. Am also able to pull and setup image for nginx and pod status shows Running. Able to access nginx on the service endpoint also. Now as per documentations if I want to access this nginx instance outside the hosted system I need to create a Route, which I also did.
Confusion is on the Create Route screen from OpenShift Web Console it generates a hostname or allows to enter a hostname. Both of the option i tried, generated hostname seems to be a a long subdomain kind of hostname and it doesn't work. What I mean is I'm not able to access this hostname from anywhere in the network including the hosting OS as well.
To summarize, service endpoints which looks like 172.x.x.x is working on the local machine which is hosting OpenShift. But the generated/entered hostname for the route doesn't work from anywhere.
Please clarify the idea behind this route concept and how could one access a service from outside the host machine (Part of same network)
As stated in documentation:
An OpenShift Origin route exposes a service at a host name, like
www.example.com, so that external clients can reach it by name. DNS
resolution for a host name is handled separately from routing; your
administrator may have configured a cloud domain that will always
correctly resolve to the OpenShift Origin router, or if using an
unrelated host name you may need to modify its DNS records
independently to resolve to the router.
It is important to notice the difference between "route" and "router". The Opensfhit router (that is mentioned above)listens to all requests to Openshift deployed applications, and has to be previoulsy deployed, in order for routes to work.
https://docs.openshift.org/latest/architecture/core_concepts/routes.html
So once you have the router deployed and working, all routes that you create in openshift should resolve where that Openshift router is listening. For example, configuring your DNS with a wildcard (this is dnsmaq wildcard example):
address=/.yourdomain.com/107.117.239.50
This way all your "routes" to services should be like this:
service1.yourdomain.com
service2.yourdomain.com
...
Hope this helps