I'm trying to setup metricbeat to monitor filebeat stats. But when I tried the beats module for doing so in my metricbeat config, I'm getting this error:
error message from metricbeat logs:
Error fetching data for metricset beat.stats: error making http request: Get http://filebeat_ip:5044/stats: dial tcp filebeat_ip:5044: connect: connection refused
metricbeat.yml file
metricbeat.modules:
- module: beat
metricsets:
- stats
- state
period: 10s
hosts: ["filebeat_ip:5044"]
where filebeat_ip is the ip where my filebeat is running, which's the same machine as my metricbeat.
Can someone please help me as to why I'm getting this error?
If it's the same machine I would just use localhost or 127.0.0.1.
PS: If not running on localhost, I'd double check if the port is actually reachable and not blocked by a firewall. Something like telnet <ip> 5044 should be a quick sanity check.
Related
I just tried to install Linkerd viz extension following official documentation but all the pods are in a crash loop.
linkerd viz install | kubectl apply -f -
Linkerd-getting-started
r proxy-admin
[ 29.797889s] INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}: linkerd_app_inbound::policy::authorize::http: Request denied server=proxy-admin tls=None(NoClientHello) client=50.50.55.177:47068
[ 29.797910s] INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:rescue{client.addr=50.50.55.177:47068}: linkerd_app_core::errors::respond: Request failed error=unauthorized connection on server proxy-admin
[ 29.817790s] INFO ThreadId(01) linkerd_proxy::signal: received SIGTERM, starting shutdown
The error appeared on Kubernetes cluster Server Version: v1.21.5-eks-bc4871b
The issue was the policy that come by default installation.
This authorizes unauthenticated requests from IPs in the clusterNetworks configuration. If the source IP (<public-ip-address-of-hel-k1>) is not in that list, these connections will be denied. To fix this, the authorization policy could be updated with the following:
spec:
client:
unauthenticated: true
networks:
- cidr: 0.0.0.0/0
The default policy missing the client part
networks:
- cidr: 0.0.0.0/0
To update the policy, get the server authorization
k get ServerAuthorization -n linkerd-viz
NAME SERVER
admin admin
grafana grafana
metrics-api metrics-api
proxy-admin proxy-admin
Now edit admin,grafana, proxy-admin and grafana and add the networks part.
k edit ServerAuthorization metrics-api
as after fixing this I was also getting errors for grafana which help me to fix by adding network part.
[ 32.278014s] INFO ThreadId(01) inbound:server{port=3000}:rescue{client.addr=50.50.53.140:44718}: linkerd_app_core::errors::respond: Request failed error=unauthorized connection on server grafana
[ 38.176927s] INFO ThreadId(01) inbound:server{port=3000}: linkerd_app_inbound::policy::authorize::http: Request denied server=grafana tls=None(NoClientHello) client=50.50.55.177:33170
All linkerd-viz pods in CrashLoopBackOff
getting error curl: (56) response reading failed while trying to send email via smtp using curl. checked the isto-proxy logs of sidecar but don't see any error logs related to this host. Tried the solution mentioned in How to access external SMTP server from within Kubernetes cluster with Istio Service Mesh as well but didn't work.
service entry
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: smtp
spec:
addresses:
- 192.168.8.45/32
hosts:
- smtp.example.com"
location: MESH_EXTERNAL
ports:
- name: tcp-smtp
number: 2255
protocol: TCP
Most probably port number is causing the error and if not, try deleting the mesh policies
Also please validate based on below points:
1.If you recently updated istio try downgrading it.
2.Look again in Sidecar logs for any conflicts or try disabling it.
3.When it comes to curl 56 error packet transmission; limit could be the problem.
The curl requests from primary container are routed via sidecar when istio is enabled, the response from smtp server is being masqueraded by sidecar and returned to primary container, which was quite misleading.
upon disabling Istio and trying to do curl on smtp port curl request failed with error Failed to connect to smtp.example.com port 2255: Operation timed out. which was because firewall from cluster to smtp server port was not open.
while istio was enabled the curl response didn't give timeout error but gave curl: (56) response reading failed which mislead me to think that the response was coming from smtp server.
I'm trying to setup a Cloud SQL Proxy Docker image for PostgreSQL as mentioned here.
I can get my app to connect to the proxy docker image but the proxy times out. I suspect it's my credentials or the port, so how do I debug and find out if it works?
This is what I have on my project
kubectl create secret generic cloudsql-instance-credentials --from-file=credentials.json=my-account-credentials.json
My deploy spec snippet:
spec:
containers:
- name: mara ...
- name: cloudsql-proxy
image: gcr.io/cloudsql-docker/gce-proxy:1.11
command: ["/cloud_sql_proxy",
"-instances=<MY INSTANCE NAME>=tcp:5432",
"-credential_file=/secrets/cloudsql/credentials.json"]
volumeMounts:
- name: cloudsql-instance-credentials
mountPath: /secrets/cloudsql
readOnly: true
volumes:
- name: cloudsql-instance-credentials
secret:
secretName: cloudsql-instance-credentials
The logs of my cloudsql-proxy show a timeout:
2019/05/13 15:08:25 using credential file for authentication; email=646092572393-compute#developer.gserviceaccount.com
2019/05/13 15:08:25 Listening on 127.0.0.1:5432 for <MY INSTANCE NAME>
2019/05/13 15:08:25 Ready for new connections
2019/05/13 15:10:48 New connection for "<MY INSTANCE NAME>"
2019/05/13 15:10:58 couldn't connect to <MY INSTANCE NAME>: dial tcp <MY PRIVATE IP>:3307: getsockopt: connection timed out
Questions:
I specify 5432 as my port, but as you can see in the logs above,it's hitting 3307. Is that normal and if not, how do I specify 5432?
How do I check if it is a problem with my credentials? My credentials file is from my service account 123-compute#developer.gserviceaccount.com
and the service account shown when I go to my cloud sql console is p123-<somenumber>#gcp-sa-cloud-sql.iam.gserviceaccount.com. They don't seem the same? Does that make a difference?
If I make the Cloud SQL instance available on a public IP, it works.
I specify 5432 as my port, but as you can see in the logs above,it's
hitting 3307
The proxy listens locally on the port you specified (in this case 5432), and connects to your Cloud SQL instance via port 3307. This is expected and normal.
How do I check if it is a problem with my credentials?
The proxy returns an authorization error if the Cloud SQL instance doesn't exist, or if the service account doesn't have access. The connection timeout error means it failed to reach the Cloud SQL instance.
My credentials file is from my service account 123-compute#developer.gserviceaccount.com and the service account shown when I go to my cloud sql console is p123-#gcp-sa-cloud-sql.iam.gserviceaccount.com. They don't seem the same?
One is just the name of the file, the other is the name of the service account itself. The name of the file doesn't have to match the name of the service account. You can check the name and IAM roles of a service account on the Service Account page.
2019/05/13 15:10:58 couldn't connect to : dial tcp :3307: getsockopt: connection timed out
This error means that the proxy failed to establish a network connection to the instance (usually because a path from the current location doesn't exist). There are two common causes for this:
First, make sure there isn't a firewall or something blocking outbound connections on port 3307.
Second, since you are using Private IP, you need to make sure the resource you are running the proxy on meets the networking requirements.
Proxy listen port 3307. This is mentioned on documentation
Port 3307 is used by the Cloud SQL Auth proxy to connect to the Cloud SQL Auth proxy server. -- https://cloud.google.com/sql/docs/postgres/connect-admin-proxy#troubleshooting
You may need to create a firewall like the following:
Direction: Egress
Action on match: Allow
Destination filters : IP ranges 0.0.0.0/0
Protocols and ports : tcp:3307 & tcp:5432
When i run
oc import-image centos:7 --confirm true
I am getting
The import completed with errors.
Name: centos
Namespace: pd-kube-ci
Created: Less than a second ago
Labels: <none>
Annotations: openshift.io/image.dockerRepositoryCheck=2018-12-27T21:00:26Z
Docker Pull Spec: docker-registry.default.svc:5000/pd-kube-ci/centos
Image Lookup: local=false
Unique Images: 0
Tags: 1
7
tagged from centos:7
! error: Import failed (InternalError): Internal error occurred: Get https://registry-1.docker.io/v2/: proxyconnect tcp: EOF
Less than a second ago
error: tag 7 failed: Internal error occurred: Get https://registry-1.docker.io/v2/: proxyconnect tcp: EOF
For the life of me, i cannot find the source of proxyconnect tcp: EOF. Its not found anywhere in the OpenShift/Kubernetes source. Google knows next to nothing about that.
I have also verified that i can docker pull centos from each node (including master and infra nodes). Its only when openshift tries to pull that image.
Any ideas?
Turns out it was a mis-configuration in our openshift_https_proxy ansible var. Specifically we had:
openshift_https_proxy=https://proxy.mycompany.com:8443
And we should have had
openshift_https_proxy=http://proxy.mycompany.com:8443
To fix this, we had to edit /etc/origin/master/master.env on the masters and /etc/sysconfig/docker on all nodes, then restart per the Working with HTTP Proxies documentation.
I have Alertmanager and Prometheus running in Openshift. Alertmanager receives and shows the alerts from Prometheus but when sending it with any smtp server (I'm using Gmail now but I tried others), I get the following error:
time="2017-05-30T08:47:21Z" level=error msg="Notify for 1 alerts failed: dial tcp 74.125.206.109:587: getsockopt: connection timed out" source="dispatch.go:261"
I have a config.yml which worked when I tried it locally with alertmanager and prometheus. I received the email alerts, so I dont get why it doesn't work when running in Openshift. I ran out of ideas.
My config file:
global:
smtp_smarthost: 'smtp.gmail.com:587'
smtp_from: 'emailtestingxxx#gmail.com'
smtp_auth_username: 'emailtestingxxx#gmail.com'
smtp_auth_password: 'ABCD1234'
templates:
- '/etc/alertmanager/template/*.tmpl'
group_by: ['alertname', 'cluster', 'service']
group_wait: 1m
group_interval: 1m
repeat_interval: 1m
receiver: team-X-mails
routes:
- match:
severity: page
receiver: team-X-mails
receivers:
- name: 'team-X-mails'
email_configs:
- to: 'myemail#myemail.com'
Solved. The problem was a firewall configuration which was blocking the outgoing requests.