send notification from Prometheus alertmanager to Google Chat room - kubernetes

we recently started using Google Chat in our organisation therefor we need to send alerts from Prometheus-Alertmanager to Google Chat room. below is my alertmanager config for Google chat
receiver:
- name: 'gchat-receiver'
webhook_configs:
- url: "https://chat.googleapis.com/v1/spaces/AAAArcv5Snw/messages?key=KEY&token=TOKEN"
send_resolved: false
and when we implementing above config we are getting below error in our alertmanager logs:
level=error ts=2021-02-23T06:23:43.931Z caller=dispatch.go:309 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="gchat-receiver/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 400: https://chat.googleapis.com/v1/spaces/AAAArcv5Snw/messages?key=KEY&token=TOKEN"
level=error ts=2021-02-23T06:23:43.971Z caller=dispatch.go:309 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="gchat-receiver/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 400: https://chat.googleapis.com/v1/spaces/AAAArcv5Snw/messages?key=KEY&token=TOKEN"
level=error ts=2021-02-23T06:23:43.976Z caller=dispatch.go:309 component=dispatcher msg="Notify for alerts failed" num_alerts=21 err="gchat-receiver/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 400: https://chat.googleapis.com/v1/spaces/AAAArcv5Snw/messages?key=KEY&token=TOKEN"
level=error ts=2021-02-23T06:23:43.994Z caller=dispatch.go:309 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="gchat-receiver/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 400: https://chat.googleapis.com/v1/spaces/AAAArcv5Snw/messages?key=KEY&token=TOKEN"
any help or guidance on this would be highly appreciated

That is because Google Chat expects a specific JSON payload schema to be posted to that webhook address and the default Alertmanager payload doesn't match it, so you get HTTP 400 response: bad request.
You might need to put a custom application in between to transform the JSON to Google Chat acceptable format. There are already some repos in Github that match that general criteria as well. (e.g calert)

Related

Prometheus --> Getting "Server returned HTTP status 400 Bad Request" attempting scraping

I have a Grafana Agent with Kube State Metrics in an EKS cluster and I am getting this error attempting to scrape a kubelet.
Any reason why such error can appear?
ts=2022-11-14T19:42:49.643418574Z caller=scrape.go:1302
level=debug agent=prometheus instance=6adb012c62a4c8a8139b7fbf04a08668 component="scrape manager"
scrape_pool=integrations/kubernetes/kubelet target=http://kubernetes.default.svc.cluster.local:443/api/v1/nodes/ip-192-168-0-8.ec2.internal/proxy/metrics
msg="Scrape failed"
err="server returned HTTP status 400 Bad Request"

Grafana8.3 webhook way to send test then get "Webhook response status 500 INTERNAL SERVER ERROR"

when i login the grafana server then found the logs covers some "error level" like:
"msg=Missing receiver logger=alerting.notifier.webhook"
"msg=Missing group labels logger=alerting.notifier.webhook"
Is there anyone had solved this kind of problems or have any ideas, in the grafana7 this way
works.

Failed to add global permission to user Rancher

I tried to add user global permissions to a cluster, but fail to add, received the following error message:
Internal error occurred: failed calling webhook "rancherauth.cattle.io": Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation?timeout=10s": dial tcp 10.X.X.X: connect: connection refused
I found nothing about this.
Thanks.

OpenMapTiles auth not working

When I try to log in via GitHub or google I get the following messages
, and OpenMapTiles refuses to let me login.
Content Security Policy: The page's settings blocked the loading of a resource at self ("default-src").
iframe.html:1
Content Security Policy: The page's settings blocked the loading of a resource at self ("default-src"). Source: .vimvixen-console-frame {
margin: 0;
Received message: !_{"h":"I0_1529357937694"} from https://openmaptiles.firebaseapp.com
client.js:181:7
Received message: trunc
!_{"s":"/I0_1529357937694::_g_restyleMe","f":"I0_1529357937694","r":"I0_1529357937694","t":"33540592","c":2,"a":[{"setHideOnLeave":false}],"g":false} from https://openmaptiles.firebaseapp.com
client.js:181:7
Received message: !_{"s":"__cb","f":"I0_1529357937694","r":"I0_1529357937694","t":"33540592","c":null,"a":[4,[null]],"g":false} from https://openmaptiles.firebaseapp.com
client.js:181:7
Received message: !_{"s":"__cb","f":"I0_1529357937694","r":"I0_1529357937694","t":"33540592","c":null,"a":[3,[true]],"g":false} from https://openmaptiles.firebaseapp.com
client.js:181:7
Received message: !_{"s":"/I0_1529357937694::authEvent","f":"I0_1529357937694","r":"I0_1529357937694","t":"33540592","c":3,"a":[{"type":"authEvent","authEvent":{"type":"signInViaRedirect","eventId":null,"urlResponse":null,"sessionId":null,"postBody":null,"error":{"code":"auth/network-request-failed","message":"A network error (such as timeout, interrupted connection or unreachable host) has occurred."}}}],"g":false} from https://openmaptiles.firebaseapp.com
client.js:181:7
Anyone got any ideas?
This looks like a problem unrelated to the OpenMapTiles project.
Based on the error, it seems to me some of your extensions is colliding with the login iframe.
There is something about vimvixen in the console. Try disabling it (or other extensions).
It's because the application is down - it's using : http://openmaptiles.firebaseapp.com/ which is not running.
I guess this project is dead, or sleeping.

Akka Http non-responsive

I had a responsive Akka Http app, did some changes, then got these messages:
14:55:56.128 INFO bootstrap.akka.HttpActor - Accepted new connection from /127.0.0.1:55192
14:56:23.938 INFO bootstrap.akka.HttpActor - Accepted new connection from /127.0.0.1:55193
14:56:26.684 ERROR akka.actor.ActorSystemImpl - Internal server error, sending 500 response
akka.stream.impl.io.TcpStreamActor$$anonfun$handleSubscriptionTimeout$1$$anon$1: Publisher was not attached to upstream within deadline (5000) ms
14:56:26.684 ERROR akka.actor.ActorSystemImpl - Internal server error, sending 500 response
akka.stream.impl.io.TcpStreamActor$$anonfun$handleSubscriptionTimeout$1$$anon$1: Publisher was not attached to upstream within deadline (5000) ms
I cannot figure out what they mean. They are not mentioned in the docs and no Google hits. Could anyone share some insight?
akka.version=2.3.12
akka.stream.version=1.0
scala.version=2.11.6
jdk.version=1.8.0_60