Getting this error in instance log. I could not seek out any errors in nova or neutron log.
Checked with all configuration and everything is fine.
url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [50/120s]: request error [(, 'Connection to 169.254.169.254 timed out. (connect timeout=50.0)')]
Anyone could help whats the actual error and how to solve it?
Probable Reason1:-
I guess you are running instances with GUI installed on them. When you install GUI on an instance(ubuntu/centos or whatever), they bring in a bunch of different services. Specially, in ubuntu, a service named "avahi" gets added and started which adds a route of 169.254/16 on the instance. This starts causing the issue as now the instance thinks that it can reach 169.254.169.254 directly rather than sending the packets to the gateway.
More details why this happens and how you can stop this can be found on this blog.
https://rahulait.wordpress.com/2016/04/02/metadata-failure-with-ubuntu-desktop-on-openstack/
Probable Reason2:-
If you have a private-network and it is not connected to any "router", the gateway interface for that private-network will be down. For communicating to metadata service, the packets needs to be sent to gateway of network, which would not be reachable in this case and hence you would see these logs.
I hope it helps.
In my case this error raised because the L3 agent was down due to some corruption in the ini file. Check if there is an agent down in neutron:
openstack network agent list
Fix the issue (check the log at /var/log/neutron and restart the service)
service neutron-l3-agent restart
This happened to me on a node that was still running nova-network from a previous configuration.
The effect on the faulty node was this (bad):
# curl -v http://169.254.169.254/openstack
* Hostname was NOT found in DNS cache
* Trying 169.254.169.254...
* connect to 169.254.169.254 port 80 failed: Connection refused
* Failed to connect to 169.254.169.254 port 80: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 169.254.169.254 port 80: Connection refused
...instead of this (good):
# curl -v http://169.254.169.254/openstack
* Hostname was NOT found in DNS cache
* Trying 169.254.169.254...
* Immediate connect fail for 169.254.169.254: Network is unreachable
* Closing connection 0
curl: (7) Couldn't connect to server
If this is the case get rid of the legacy service on your node and enjoy.
Related
SUMMARY
I have installed zabbix on OpenShift cluster. I am trying to monitor a host(vm) outside the cluster but the zabbix server is unable to connect to it. In the /etc/zabbix/zabbix_agentd.conf file I have mentioned the DNS name of the server zabbix-server but it looks like there server is trying to connect through a different public IP. I am not sure what this IP is.
OS / ENVIRONMENT / Used docker-compose files
I applied the kubernetes.yaml file present in this repo - https://github.com/zabbix/zabbix-docker/blob/6.2/kubernetes.yaml - on an OpenShift cluster.
CONFIGURATION
In the /etc/zabbix/zabbix_agentd.conf file Server=zabbix-server.
STEPS TO REPRODUCE
Apply the kubernetes.yaml file on Openshift cluster and try to monitor any external vm.
EXPECTED RESULTS
The zabbix server should be able to connect to the vm.
ACTUAL RESULTS
Zabbix server logs.
Defaulted container "zabbix-server" out of: zabbix-server, zabbix-snmptraps
\*\* Updating '/etc/zabbix/zabbix_server.conf' parameter "DBHost": 'mysql-server'...added
287:20230120:060843.131 Zabbix agent item "system.cpu.load\[all,avg5\]" on host "Host-C" failed: first network error, wait for 15 seconds
289:20230120:060858.592 Zabbix agent item "system.cpu.num" on host "Host-C" failed: another network error, wait for 15 seconds
289:20230120:060913.843 Zabbix agent item "system.sw.arch" on host "Host-C" failed: another network error, wait for 15 seconds
289:20230120:060929.095 temporarily disabling Zabbix agent checks on host "Host-C": interface unavailable
Logs from the agent installed on the vm.
350446:20230122:103232.230 failed to accept an incoming connection: connection from "9.x.x.219" rejected, allowed hosts: "zabbix-server"
350444:20230122:103332.525 failed to accept an incoming connection: connection from "9.x.x.219" rejected, allowed hosts: "zabbix-server"
350445:20230122:103432.819 failed to accept an incoming connection: connection from "9.x.x.210" rejected, allowed hosts: "zabbix-server"
350446:20230122:103533.114 failed to accept an incoming connection: connection from "9.x.x.217" rejected, allowed hosts: "zabbix-server"
If I add this IP in /etc/zabbix/zabbix_agentd.conf it will work. But what IP is this? Is this a service? Or any node/pod IP? It keeps on changing. Everytime I cannot change this id in the conf file. I need something more stable.
Kindly help me out with this issue.
So I don't know zabbix. So I have to make some educated guesses both in how the agent works and how the server works.
But, to summarize, unlike something like docker compose where you are running the zabbix server on a known server, in Openshift/Kubernetes you are deploying into a cluster of machines with their own networking. In other words, the whole point of OpenShift is that OpenShift will control where the application's pod gets deployed and will relocate/restart that pod as needed. With a different IP every time. (And the DNS name is meaningless since the two systems aren't sharing DNS anyway.) Most likely the IP's you are seeing are the pod's randomly assigned IPs.
So, what are you to do when you have a situation like yours where an external application requires a predicable IP? Well, option 1, is to remove that requirement. Using something like a certificate is obviously more secure and more reliable than depending on an IP anyway. But another option is to use an egress IP. This is a feature of OpenShift where you essentially use a proxy to provide an external application with a consistent IP.
I am trying to setup github-runner on windows machine for CI/CD but when execute "config.cmd --url --token it is failing with error "A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (api.github.com:443)"
I have setup the proxy on server but still same error.
I'm trying to install SonarQube onto a bare-metal kubernetes cluster.
All is working except for when the software inside the pod tries to make a HTTPS request.
I've checked using wget (pod doesn't have curl & cannot use ping) & using the kubernetes DNS debugging guide, however whenever I make a call such as wget https://google.com I get the following error:
Connecting to google.com (192.168.1.179:443)
ssl_client: google.com: TLS connect failed
wget: error getting response: Connection reset by peer
command terminated with exit code 1
The IP address 192.168.1.179 is the address of another server on the host network.
The resolv.conf I have (ubuntu host) is:
nameserver 1.1.1.1
nameserver 1.0.0.1
I can't figure out why this is happening or how to fix it. DNS is working but not resolving HTTPS.
I'm using Calico, kubernetes dashboard, MetalLB, ingress-nginx & sonarqube
Edit:
After restarting the host the DNS servers successfully changed to 1.1.1.1.
However, now I'm presented with the following
Connecting to google.com (142.250.204.14:443)
ssl_client: google.com: TLS connect failed
wget: error getting response: Connection reset by peer
command terminated with exit code 1
This error went away when I:
Disabled my firewall (ufw) and;
Restarted the machine for the DNS changes to take affect.
Unable to communicate with server - I/O error on GET request for "https://api.ng.bluemix.net/info":Connect to 10.81.82.132:80 [/10.81.82.132] failed: Connection timed out: connect; nested exception is org.apache.http.conn.HttpHostConnectException: Connect to 10.81.82.132:80 [/10.81.82.132] failed: Connection timed out: connect
I am getting this error when trying to create a Bluemix server in Eclipse, specifically on click of Validate Account button. I have downloaded and installed the IBM Bluemix for Eclipse plug-in but still having connectivity issues. Any help is appreciated.
The connection error is depending from the wrong IP address your internet ISP/provider is returning for api.ng.bluemix.net
Check your DNS configuration, you could run the following command
nslookup api.ng.bluemix.net
to check which DNS server is returning the wrong IP address.
I can't connect to ContextBroker from another machine, even a machine in the same LAN.
Accessing by ssh without any problem
ssh geezar#192.168.1.115
and then
curl localhost:1026/statistics
the terminal shows the statistics, all right
<orion>
<xmlRequests>3</xmlRequests>
<jsonRequests>1</jsonRequests>
<updates>1</updates>
<versionRequests>1</versionRequests>
<statisticsRequests>2</statisticsRequests>
<uptime_in_secs>84973</uptime_in_secs>
<measuring_interval_in_secs>84973</measuring_interval_in_secs>
</orion>
But when I try without ssh connection...
curl 192.168.1.115:1026/statistics
curl: (7) Failed to connect to 192.168.1.115 port 1026: No route to host
Even, I routed the port 1026 to that machine (192.168.1.115) on the router configuration, and tried to access from my public IP, the result is the same, failed to connect
I think I am missing something, but.. what is it?
The most probable causes of this problem are:
Something in the host (e.g a firewall or security group) is blocking the incoming connection
Something in the client (e.g a firewall) is blocking the outcoming connection
There is some other network issue is causing the connection problem.
EDIT: in GNU/Linux system, iptables is usually used as firewall. It can be disabled typically running iptables -F.