I have tried to install heapster with grafana and influxdb on my kubernetes cluster. I cannot manage to see the page of grafana, it only shows me alert.title.
I think that I did everything right, all the logs seems good, but this is the last problem: If someone will be kind enough to show me what's happening I would be grateful.
Here is a pick of my log for:
2016/06/23 13:31:23 [I] Completed 172.17.77.1 - "GET /favicon.ico HTTP/1.1" 404 Not Found 2929 bytes in 1224us
2016/06/23 13:31:30 [I] Completed 172.17.77.1 - "GET /grafana HTTP/1.1" 404 Not Found 2929 bytes in 1154us
2016/06/23 13:31:30 [I] Completed 172.17.77.1 - "GET /api/v1/proxy/namespaces/default/services/monitoring-grafana/public/app/app.ca0ab6f9.js HTTP/1.1" 404 Not Found 23 bytes in 545us
2016/06/23 13:31:30 [I] Completed 172.17.77.1 - "GET /api/v1/proxy/namespaces/default/services/monitoring-grafana/public/css/grafana.dark.min.a95b3754.css HTTP/1.1" 404 Not Found 23 bytes in 786us
2016/06/23 13:31:40 [I] Completed 172.17.77.1 - "GET /monitoring-grafana HTTP/1.1" 404 Not Found 2929 bytes in 1409us
2016/06/23 13:31:40 [I] Completed 172.17.77.1 - "GET /api/v1/proxy/namespaces/default/services/monitoring-grafana/public/app/app.ca0ab6f9.js HTTP/1.1" 404 Not Found 23 bytes in 879us
2016/06/23 13:31:40 [I] Completed 172.17.77.1 - "GET /api/v1/proxy/namespaces/default/services/monitoring-grafana/public/css/grafana.dark.min.a95b3754.css HTTP/1.1" 404 Not Found 23 bytes in 1349us
2016/06/23 13:31:46 [I] Completed 172.17.77.1 - "GET /api/v1/proxy/namespaces/default/services/monitoring-grafana/public/app/app.ca0ab6f9.js HTTP/1.1" 404 Not Found 23 bytes in 837us
2016/06/23 13:31:46 [I] Completed 172.17.77.1 - "GET /api/v1/proxy/namespaces/default/services/monitoring-grafana/public/css/grafana.dark.min.a95b3754.css HTTP/1.1" 404 Not Found 23 bytes in 1181us
Update :
Ok I found something in the influxdb-grafana-controller.yaml I changed value : /api/v1/proxy/namespaces/kube-system/services/monitoring-grafana/ to value: /
I dont know if it's a good solution but it's working.
Ok I found the solution, my cluster was flawed. I had to install flannel on the master too. With the option --iface=eth1 because of vagrant.
I followed this guide http://severalnines.com/blog/installing-kubernetes-cluster-minions-centos7-manage-pods-services but they didnt say to install flannel on the master.
You can remove NodePort from influxdb-grafana-controller.yaml and you can also put the value : api/v1/proxy/namespaces/kube-system/services/monitoring-grafana/ back.
Now everything is working.
Related
I use Go-micro(v2) to deploy services inside docker-compose
user-service:
build:
context: ./user-service
restart: always
ports:
- "8086:8086"
deploy:
mode: replicated
replicas: 1
environment:....
See the service configuration
srv = micro.NewService(
micro.Name("my.user"),
micro.Address("127.0.0.1:8086"))
when running docker-compose, the container logs show
2022-07-31 05:43:53 file=v2#v2.9.1/service.go:200 level=info Starting [service] my.user
2022-07-31 05:43:53 file=grpc/grpc.go:864 level=info Server [grpc] Listening on [::]:8086
2022-07-31 05:43:53 file=grpc/grpc.go:697 level=info Registry [mdns] Registering node: my.user-00ee4795-06df-47f1-a07a-cc362e135864
All looks good.
But when I want to query some handlers using curl or postman(for development purpose), It doesn't work,
see an exemple of failed request with postman
GET http://127.0.0.1:8086/my.user/Get
Error: Parse Error: Expected HTTP/
Request Headers
Content-Type: application/json
User-Agent: PostmanRuntime/7.29.2
Accept: */*
Postman-Token: b5ab718a-341b-40ff-81fa-37c66fd4d9f2
Host: 127.0.0.1:8086
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Request Body
GET http://127.0.0.1:8086/my.user/userService/Get // same error
with curl it is not better
curl --header "Content-Type:application/json" --http0.9 --output GET http://localhost:8086/my.user/Get
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 15 0 15 0 0 10638 0 --:--:-- --:--:-- --:--:-- 15000
curl --header "Content-Type:application/json" --http0.9 --output GET http://localhost:8086/my.user/userService/Get
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 15 0 15 0 0 13550 0 --:--:-- --:--:-- --:--:-- 15000
Any idea how to query locally some go-micro services ? Thank you.
ps: Note that the 'Get' handler is working
I'm new to go-micro, tried to have micro server running as another service in my docker-compose setup however my first micro-service is not getting show in Micro Web (the Go Micro dashboard). If I run micro server outside docker-compose I can see several go-micro-related micro-services in the dashboard, but inside docker-compose none is displayed.
I have put together an example code that reproduces what I experience, I generated the service using micro new and changed nothing in it, the docker-compose.yml reflects how I have it setup in my project where I began experiencing this:
https://github.com/shackra/go-micro-docker-compose-bug
What am I doing wrong and how do I get the Micro dashboard to work as expected inside docker-compose?
EDIT
I have added a network in docker compose without any luck, also made my example service compilable inside docker.
EDIT 2
if I restart the service micro complains about it:
micro_1 | 2020-07-02 03:25:36 file=auth/wrapper.go:84 level=error service=web none available
micro_1 | 172.21.0.1 - - [02/Jul/2020:03:25:36 +0000] "GET /services HTTP/1.1" 500 15 "http://localhost:8082/" "Mozilla/5.0 (X11; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0"
micro_1 | 2020-07-02 03:25:39 file=auth/wrapper.go:84 level=error service=web none available
micro_1 | 172.21.0.1 - - [02/Jul/2020:03:25:39 +0000] "GET /client HTTP/1.1" 500 15 "http://localhost:8082/" "Mozilla/5.0 (X11; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0"
micro_1 | 2020-07-02 03:28:34 file=auth/wrapper.go:84 level=error service=web none available
micro_1 | 172.21.0.1 - - [02/Jul/2020:03:28:34 +0000] "GET /client HTTP/1.1" 500 15 "http://localhost:8082/" "Mozilla/5.0 (X11; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0"
micro_1 | 2020-07-02 03:28:38 file=auth/wrapper.go:84 level=error service=web none available
micro_1 | 172.21.0.1 - - [02/Jul/2020:03:28:37 +0000] "GET /services HTTP/1.1" 500 15 "http://localhost:8082/" "Mozilla/5.0 (X11; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0"
example_1 | 2020-07-02 03:30:55 file=grpc/grpc.go:791 level=info Deregistering node: go.micro.service.example-58b408cc-eec2-41a3-8d7f-5ac1751ad795
example_1 | 2020-07-02 03:30:55 file=grpc/grpc.go:814 level=info Unsubscribing from topic: go.micro.service.example
example_1 | 2020-07-02 03:30:55 file=grpc/grpc.go:959 level=info Broker [http] Disconnected from 127.0.0.1:45863
micro_1 | 2020-07-02 03:30:55 file=registry/registry.go:102 level=error service=api unable to get go.micro.service.example service: service not found
go-micro-docker-compose-bug_example_1 exited with code 0
micro_1 | 2020-07-02 03:30:58 file=handler/handler.go:227 level=error service=debug Error calling go.micro.service.example#172.21.0.2:38625 ({"id":"go.micro.client","code":500,"detail":"connection error: desc = \"transport: Error while dialing dial tcp 172.21.0.2:38625: connect: connection refused\"","status":"Internal Server Error"})
micro_1 | 2020-07-02 03:31:08 file=handler/handler.go:227 level=error service=debug Error calling go.micro.service.example#172.21.0.2:38625 ({"id":"go.micro.client","code":500,"detail":"connection error: desc = \"transport: Error while dialing dial tcp 172.21.0.2:38625: connect: connection refused\"","status":"Internal Server Error"})
micro_1 | 2020-07-02 03:31:18 file=handler/handler.go:227 level=error service=debug Error calling go.micro.service.example#172.21.0.2:38625 ({"id":"go.micro.client","code":500,"detail":"connection error: desc = \"transport: Error while dialing dial tcp 172.21.0.2:38625: connect: connection refused\"","status":"Internal Server Error"})
I don't really get it, why would it report no services nor clients were found but still being able to connect with my service?
it seems that despite micro's dashboard showing nothing, if I issue micro list services my example service is listed
I then removed the network from the docker-compose.yml, stopped docker-compose and put it up again, issued micro list services again and my example service was included in the listing. So, this seems to be a bug with the dashboard itself.
I have one application which uses Spring Boot Admin Server library (v2.2.1) to monitor the other applications with integrated Spring Boot Client library (v2.2.1). It works very well; server tracks status of client applications with low performance impact.
Nevertheless, when I open page Insight -> Details (default page), CPU usage of client application grows to 90 – 100 % which causes that my application (running on 1CPU system) responds very slowly. Other pages of Spring Boot Server are OK.
According to my observation, high CPU usage is caused by frequent refresh (1 second) of information there, especially charts. In my case sends Spring Boot Admin about 16 requests in 1 second. Processing each takes about 400 ms (last number)
2020-03-11 08:31:45.290 127.0.0.1 - GET "/actuator/metrics/cache.gets?tag=name:ui-logbook,result:miss" 200 HTTP/1.0 308 421
2020-03-11 08:31:45.291 127.0.0.1 - GET "/actuator/metrics/cache.gets?tag=name:area-services,result:miss" 200 HTTP/1.0 312 438
2020-03-11 08:31:45.291 127.0.0.1 - GET "/actuator/metrics/cache.gets?tag=name:area-services,result:hit" 200 HTTP/1.0 286 437
2020-03-11 08:31:45.290 127.0.0.1 - GET "/actuator/metrics/cache.size?tag=name:area-services" 200 HTTP/1.0 314 420
2020-03-11 08:31:45.292 127.0.0.1 - GET "/actuator/metrics/cache.gets?tag=name:ui-logbook,result:hit" 200 HTTP/1.0 282 428
2020-03-11 08:31:45.426 127.0.0.1 - GET "/actuator/metrics/cache.size?tag=name:ui-logbook" 200 HTTP/1.0 310 100
2020-03-11 08:31:46.513 127.0.0.1 - GET "/actuator/metrics/jvm.threads.peak" 200 HTTP/1.0 219 436
2020-03-11 08:31:46.520 127.0.0.1 - GET "/actuator/metrics/jvm.threads.live" 200 HTTP/1.0 215 434
2020-03-11 08:31:46.520 127.0.0.1 - GET "/actuator/metrics/process.cpu.usage" 200 HTTP/1.0 207 434
2020-03-11 08:31:46.520 127.0.0.1 - GET "/actuator/metrics/system.cpu.usage" 200 HTTP/1.0 177 433
2020-03-11 08:31:46.521 127.0.0.1 - GET "/actuator/metrics/jvm.gc.pause" 200 HTTP/1.0 401 433
2020-03-11 08:31:46.945 127.0.0.1 - GET "/actuator/metrics/jvm.threads.daemon" 200 HTTP/1.0 179 398
2020-03-11 08:31:46.991 127.0.0.1 - GET "/actuator/metrics/jvm.memory.max?tag=area:heap" 200 HTTP/1.0 282 425
2020-03-11 08:31:46.998 127.0.0.1 - GET "/actuator/metrics/jvm.memory.max?tag=area:nonheap" 200 HTTP/1.0 369 420
2020-03-11 08:31:46.998 127.0.0.1 - GET "/actuator/metrics/jvm.memory.used?tag=area:nonheap" 200 HTTP/1.0 318 422
2020-03-11 08:31:46.999 127.0.0.1 - GET "/actuator/metrics/jvm.memory.used?tag=area:heap" 200 HTTP/1.0 233 420
Is there any way how to reduce refresh rate in order to reduce CPU load on client system?
So far I have found only this answer but suggested solution did not work.
Thanks for sharing ideas.
I'm observing Consumer instance not found error, at the time of consumer registration even if consumer instance is not timedout. Using Confluent API's.
Following are steps followed for this negative testing:
Running a script for consumer registration.
Kafka Topology : 3 ZK instance (where 2 ZK are dummy value) and 1 node cluster (single instance of rest-proxy & broker node).
When script is in progress for consumer registration, have cancelled the script & re-run it.
Have seen that for the last registered consumer, Instance not found error is returned. But, in logs after some ms, 200 OK is listed for that consumer registration request [Consumer name in shared below logs : CGStress_TEST111111111111111_6].
[2018-08-28 09:05:48,411] INFO http://localhost:8082/v1/consumer/CGStress_TEST111111111111111_5 Service: TransacationId:5 EntityId:ed_1 Authorized:true Allowed:true (io.confluent.kafkarest.resources.SecurityRestrictions)
{"X-Nssvc-serviceid":null,"Type":"API","X-Nssvc-customerid":null,"Client-IP":"127.0.0.1","Severity":"INFO","X-Cws-Transactionid":"5","message":{"request":{"content-length":81,"method":"POST","time":"2018-08-28 09:05:48.409","uri":"mr/v1/consumer/CGStress_TEST111111111111111_5","entity-id":"ed_1","user-agent":"python-requests/2.11.1"},"response":{"status_code":200,"time":"2018-08-28 09:05:48.412"}}}
[2018-08-28 09:05:48,412] INFO 127.0.0.1 - - [28/Aug/2018:09:05:48 +0000] "POST /mr/v1/consumer/CGStress_TEST111111111111111_4 HTTP/1.1" 200 205 19 (io.confluent.rest-utils.requests)
[2018-08-28 09:05:48,420] INFO http://localhost:8082/mr/v1/consumer/CGStress_TEST111111111111111_6 Service: TransacationId:6 EntityId:ed_1 Authorized:true Allowed:true (io.confluent.kafkarest.resources.SecurityRestrictions)
{"X-Nssvc-serviceid":null,"Type":"API","X-Nssvc-customerid":null,"Client-IP":"127.0.0.1","Severity":"INFO","X-Cws-Transactionid":"6","message":{"request":{"content-length":81,"method":"POST","time":"2018-08-28 09:05:48.419","uri":"mr/v1/consumer/CGStress_TEST111111111111111_6","entity-id":"ed_1","user-agent":"python-requests/2.11.1"},"response":{"status_code":404,"error_response":{"message":"Consumer instance not found.","error":40403},"time":"2018-08-28 09:05:48.421"}}}
[2018-08-28 09:05:48,423] INFO 127.0.0.1 - - [28/Aug/2018:09:05:48 +0000] "POST /mr/v1/consumer/CGStress_TEST111111111111111_5 HTTP/1.1" 200 205 15 (io.confluent.rest-utils.requests)
[2018-08-28 09:05:48,431] INFO 127.0.0.1 - - [28/Aug/2018:09:05:48 +0000] "POST /mr/v1/consumer/CGStress_TEST111111111111111_6 HTTP/1.1" 404 61 13 (io.confluent.rest-utils.requests)
[2018-08-28 09:05:49,299] WARN Client session timed out, have not heard from server in 1501ms for sessionid 0x0 (org.apache.zookeeper.ClientCnxn)
[2018-08-28 09:05:49,300] INFO Client session timed out, have not heard from server in 1501ms for sessionid 0x0, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2018-08-28 09:05:49,400] INFO Opening socket connection to server localhost/0:0:0:0:0:0:0:1:32181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2018-08-28 09:05:49,400] INFO Socket connection established to localhost/0:0:0:0:0:0:0:1:32181, initiating session (org.apache.zookeeper.ClientCnxn)
[2018-08-28 09:05:49,403] INFO Session establishment complete on server localhost/0:0:0:0:0:0:0:1:32181, sessionid = 0x1657f54045b00f3, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2018-08-28 09:05:49,403] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2018-08-28 09:05:49,404] INFO [CGStress_TEST111111111111111_6_UbuntuNTP-1535447146187-7b4d0350], starting auto committer every 60000 ms (kafka.consumer.ZookeeperConsumerConnector)
{"X-Nssvc-serviceid":null,"Type":"API","X-Nssvc-customerid":null,"Client-IP":"127.0.0.1","Severity":"INFO","X-Cws-Transactionid":"6","message":{"request":{"content-length":81,"method":"POST","time":"2018-08-28 09:05:46.172","uri":"mr/v1/consumer/CGStress_TEST111111111111111_6","entity-id":"ed_1","user-agent":"python-requests/2.11.1"},"response":{"status_code":200,"time":"2018-08-28 09:05:49.405"}}}
[2018-08-28 09:05:49,409] INFO 127.0.0.1 - - [28/Aug/2018:09:05:46 +0000] "POST /mr/v1/consumer/CGStress_TEST111111111111111_6 HTTP/1.1" 200 124 3239 (io.confluent.rest-utils.requests)
root#UbuntuNTP:~/CloudServices/MsgRelay#
Is this something related to dummy zk instance added?
I'm trying to deploy an application to the Swisscom App Cloud from the console. It reports progress until at the end, an 504 without further explanation is reported:
Updating app helloclass-fe-develop in org UCID-Bern Team / space HELLOCLASS-TEST as christian.cueni#iterativ.ch...
OK
Uploading helloclass-fe-develop...
FAILED
Error processing app files: Error uploading application.
Server error, status code: 504, error code: 0, message:
The log of the app reports that the app has been updated:
2017-01-03 09:37:39 [RTR/0] OUT helloclass-develop.scapp.io - [03/01/2017:08:37:39.584 +0000] "GET / HTTP/1.1" 200 0 594 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon" 66.249.93.201:50868 10.0.18.35:64341 x_forwarded_for:"83.76.152.96" x_forwarded_proto:"https" vcap_request_id:8a8adcc7-9e97-4bd9-4492-68e92883ee3d response_time:0.001739219 app_id:310166b4-f3a6-4168-a9ac-530e45dbfb10 app_index:0
2017-01-03 09:37:39 [APP/PROC/WEB/0] OUT 83.76.152.96, 66.249.93.201, 66.249.93.201 - - - [03/Jan/2017:08:37:39 +0000] "GET / HTTP/1.1" 200 606
2017-01-03 10:05:50 [API/2] OUT Updated app with guid 310166b4-f3a6-4168-a9ac-530e45dbfb10 ({"name"=>"helloclass-fe-develop"})
2017-01-03 10:57:15 [API/1] OUT Updated app with guid 310166b4-f3a6-4168-a9ac-530e45dbfb10 ({"state"=>"STOPPED"})
2017-01-03 10:57:15 [CELL/0] OUT Exit status 0
2017-01-03 10:57:15 [APP/PROC/WEB/0] OUT Exit status 0
2017-01-03 10:57:15 [CELL/0] OUT Destroying container
2017-01-03 10:57:15 [CELL/0] OUT Successfully destroyed container
2017-01-03 10:57:16 [API/1] OUT Updated app with guid 310166b4-f3a6-4168-a9ac-530e45dbfb10 ({"state"=>"STARTED"})
2017-01-03 10:57:16 [CELL/0] OUT Creating container
2017-01-03 10:57:16 [CELL/0] OUT Successfully created container
2017-01-03 10:57:17 [CELL/0] OUT Starting health monitoring of container
2017-01-03 10:57:19 [CELL/0] OUT Container became healthy
In spite of those messages which would indicate that the app has been updated, I still see the old version of the app being served.
EDIT
After running the command with the -v parameter, I see that the reason for the failure is a gateway timeout:
RESPONSE: [2017-01-03T13:32:39+01:00]
HTTP/1.1 504 Gateway Timeout
Connection: close
Content-Length: 176
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Content-Type: text/html
Date: Tue, 03 Jan 2017 12:32:39 GMT
Expires: 0
Pragma: no-cache
Strict-Transport-Security: max-age=15768000; includeSubDomains
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Vcap-Request-Id: 3ac831ef-e70b-4f4e-7c56-e308806f039e
X-Xss-Protection: 1; mode=block
<html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>
FAILED
Error processing app files: Error uploading application.
Server error, status code: 504, error code: 0, message:
Is this something cloudfoundry specific or rather related to Swisscom AppCloud? Are there cloudfoundry inherent timeout limits?
You can run cf push with -v or enable CF_TRACE to see more of the interaction of the CLI with your CF endpoint.
The error message looks similar to https://github.com/cloudfoundry/cli/issues/1042: the Cloud Controller could not complete a request in time and the router that routed the API request to Cloud Controller did not wait any longer and returned the 504 (Gateway timeout) to the CLI.
The trace should tell you which API call timed out.
The CLI aborted the operation there, while the Cloud Controller may have completed the operation successfully, eventually.
I would have thought the operations the CLI would perform here are:
send a list of files in your app and their checksums for resource matching (so it can skip uploading unmodified app bits that the CC cached from a previous push)
upload app files
(re)start app (which includes staging)
poll & wait until an app instance returns that it's running
From your CLI output I assume the first operation timed out, so not clear how your app was restarted.