So I have a service like as follow:
{
"kind": "Service",
"apiVersion": "v1",
"metadata": {
"name": "monitoring-grafana",
"namespace": "kube-system",
"selfLink": "/api/v1/namespaces/kube-system/services/monitoring-grafana",
"uid": "be0f72b2-c482-11e5-a22c-fa163ebc1085",
"resourceVersion": "143360",
"creationTimestamp": "2016-01-26T23:15:51Z",
"labels": {
"kubernetes.io/cluster-service": "true",
"kubernetes.io/name": "monitoring-grafana"
}
},
"spec": {
"ports": [
{
"protocol": "TCP",
"port": 80,
"targetPort": 3000,
"nodePort": 0
}
],
"selector": {
"name": "influxGrafana"
},
"clusterIP": "192.168.182.76",
"type": "ClusterIP",
"sessionAffinity": "None"
},
"status": {
"loadBalancer": {}
}
However, whenever I try to access it through the proxy API, it always fails with this response.
http://10.32.10.44:8080/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana/
Error: 'dial tcp 192.168.182.132:3000: getsockopt: no route to host'
Trying to reach: 'http://192.168.182.132:3000/'
It happens on all of my services also, not just the one posted.
What could be going wrong? Is something not installed?
Looking at the error you posted it seems like the traffic can not be routed from your master to the Docker subnet of your node. The easiest way to validate this is to open a shell on your master and perform a request on your podIP:daemonPort: curl -I http://192.168.182.132:3000
Each node in your cluster should be able to communicate with every other node, and every Docker subnet should be routable. For most deployments you will need to setup an extra network fabric to make this happen, like flannel or Weave.
Take a look at Getting started from Scratch >> Network
Something else is funny. The cluster IP used by your service (192.168.182.76) and the pod IP of the endpoint (192.168.182.132) seem to be in the same subnet. However you need 3 different subnets:
one for the hosts
one for the Docker bridges (--bip flag of Docker)
one for the service (--service-cluster-ip-range= of the API server)
In my case I didn't realize that I have active firewall that was simply preventing access to the ports needed by kubernetes. Quick and crude solution is to run systemctl stop firewalld on the master and all minion nodes and of course you can just open ports needed instead
Related
Having looked through the Azure DevOps REST API documentation, and a few failed attempts at guessing the endpoint, there doesn't appear to be any mention of how to view or create 'Approvals and checks' associated with a given service connection:
https://learn.microsoft.com/en-us/rest/api/azure/devops/serviceendpoint/endpoints?view=azure-devops-rest-6.1
Are there any ideas on how to do this, or where the Rest API documentation for approvals/checks for service connections are?
For background information, when creating a service connection via the REST API we are aiming to assign a check to the service connection so that it uses a given YAML template, as the service connections themselves are already being created as part of an automated flow.
You can use an unrecorded REST API:
POST https://dev.azure.com/{organization}/{project}/_apis/pipelines/checks/configurations?api-version=5.2-preview.1
Here is an example of its request body:
{
"type": {
"name": "ExtendsCheck"
},
"settings": {
"extendsChecks": [
{
"repositoryType": "git",
"repositoryName": "{project}/{repository}",
"repositoryRef": "refs/heads/master",
"templatePath": "templates.yml"
}
]
},
"resource": {
"type": "endpoint",
"id": "{service connection id}",
"name": "{service connection name}"
}
}
To get the service connection id, you can use the REST API Endpoints - Get Service Endpoints or Endpoints - Get Service Endpoints By Names.
I can list all the custom.metrics available, but I don't know how to query an individual value. For example I have tried:
curl http://localhost:8001/apis/custom.metrics.k8s.io/v1beta1/ | jq .
{
"kind": "APIResourceList",
"apiVersion": "v1",
"groupVersion": "custom.metrics.k8s.io/v1beta1",
"resources": [
{
"name": "deployments.apps/aws_sqs_approximate_number_of_messages_visible_average",
"singularName": "",
"namespaced": false,
"kind": "MetricValueList",
"verbs": [
"get"
]
}
]
}
But if I try this:
curl http://localhost:8001/apis/custom.metrics.k8s.io/v1beta1/deployments.apps/aws_sqs_approximate_number_of_messages_visible_average | jq .
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "the server could not find the requested resource",
"reason": "NotFound",
"details": {
},
"code": 404
}
I get a 404. I've seen this issue which shows how to get a namespaced metric, but mine does not have a namespace? Is there a definition for how to use this API?
Just like Resource Metrics, Custom Metrics are bound to Kubernetes objects too.
What you're missing in your URL is the resource you want the metric to relate to.
For example the Pod the custom metric is related to, but the same is true for Deployments.
Try to adjust this url to your needs:
kubectl get --raw \
'/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pod/podinfo-67c9fd95d-fqk4g/http_requests_per_second' \
| jq .
Here are the slides for the talk we gave at FOSDEM 2019 on the Prometheus Adapter: https://speakerdeck.com/metalmatze/kubernetes-metrics-api?slide=26
I'll update this answer, once the video is available too.
Since I'm using DirectXMan12/k8s-prometheus-adapter there are a few things to know:
I think it can only work with namespaced metrics.
If a query does not return a metric for a particular time period in prometheus k8s-prometheus-adapter will report it as non-existent.
This is my actual problem.
Using the custom metrics API is very simple:
kubectl proxy to open a proxy to your kubernetes API
curl http://localhost:8001/apis/custom.metrics.k8s.io/v1beta1/ to list all custom metrics available.
For example you may see:
{
"name": "deployments.extensions/kube_deployment_status_replicas_available",
"singularName": "",
"namespaced": true,
"kind": "MetricValueList",
"verbs": [
"get"
]
}
We know it is namespaced from namespaced: true and beneath the namespace we can select via deployment from the name field.
So we would build our query like so:
curl http://localhost:8001/apis/custom.metrics.k8s.io/v1beta1/namespace/$NAMESPACE/deployments.extensions/$DEPLOYMENT/kube_deployment_status_replicas_available
At least I think that's how it should work, although if you do the same query without deployments.extensions section it will show the value for the namespace:
curl http://localhost:8001/apis/custom.metrics.k8s.io/v1beta1/namespace/$NAMESPACE/kube_deployment_status_replicas_available
Perhaps this is due to how the query executes in prometheus.
I'm trying to setup via script a kubernetes cluster on GCE, which always worked for the past, but I created a new project on GCE and I suddenly get all these permissions errors:
Example:
Error from server (Forbidden): serviceaccounts is forbidden: User "client" cannot list serviceaccounts in the namespace "default": Unknown user "client"
Also when I kubectl proxy and open http://localhost:8001/ I get:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "forbidden: User \"client\" cannot get path \"/\": Unknown user \"client\"",
"reason": "Forbidden",
"details": {
},
"code": 403
}
Could somebody hint me please into the right direction? Thx!
Duplicate of what does Unknown user "client" mean?:
Found out there is some issue with gcloud config. This command solved it:
gcloud config unset container/use_client_certificate
I have difficulties with making the accumulation server to work. I started it, however it doesn't give any results if OCB receive for example new subscription. The process looks like this:
I start acc. server as told in tutorial from freshly cloned repo of OCB. As a result i get in console:
tmp#tmp-VirtualBox:~/fiware-orion/scripts$ ./accumulator-server.py --port 1028 --url /accumulate --host ::1 --pretty-print -v
verbose mode is on
port: 1028
host: ::1
server_url: /accumulate
pretty: True
https: False
Running on http://[::1]:1028/ (Press CTRL+C to
And after this nothing at all happens. If I make a subscription (the most basic one from the tutorial) I get response in the medium from which i made the request:
< HTTP/1.1 201 Created
< Connection: Keep-Alive
< Content-Length: 0
< Location: /v2/subscriptions/5ab5248e50bfc821d0a1b1e0
< Fiware-Correlator: 45df4ff6-2eb3-11e8-912c-0242ac110003
< Date: Fri, 23 Mar 2018 16:00:14 GMT
However, and that might be the culprit, status of subscription is set on failed (checked with asking for listing all subscriptions and in Orion Context Explorer). And cannot be changed to inactive for instance. Everything is running as intended (I guess). OCB is running as a container in docker which is installed on LUbuntu, and is working really well. It might be my error, cuz I'm using Insomnia to communicate with OCB and could mixed something, but the response from OCB is that everything is allright. Any help will be appreciated.
EDIT:
Acc. server is not working. I got:
* Trying 127.0.0.1...
* TCP_NODELAY set
* connect to 127.0.0.1 port 1028 failed: Connection refused
* Failed to connect to localhost port 1028: Connection refused
* Closing connection 0
curl: (7) Failed to connect to localhost port 1028
after running the check command (curl -vvvv localhost:1028/accumulate).
Regarding making subscription I POST this payload:
{
"description": "A subscription to get info about Room1",
"subject": {
"entities": [
{
"id": "Room1",
"type": "Room"
}
],
"condition": {
"attrs": [
"pressure"
]
}
},
"notification": {
"http": {
"url": "http://localhost:1028/accumulate"
},
"attrs": [
"temperature"
]
},
"expires": "2040-01-01T14:00:00.00Z",
"throttling": 5
}
to a localhost:1026/v2/subscriptions URL. Beforehand entities and their arguments and types are allright. After creating, I request get on all subscriptions and get:
[
{
"id": "5ab7d819209f52528cc2faf7",
"description": "A subscription to get info about Room1",
"expires": "2040-01-01T14:00:00.00Z",
"status": "failed",
"subject": {
"entities": [
{
"id": "Room1",
"type": "Room"
}
],
"condition": {
"attrs": [
"pressure"
]
}
},
"notification": {
"timesSent": 1,
"lastNotification": "2018-03-25T17:10:49.00Z",
"attrs": [
"temperature"
],
"attrsFormat": "normalized",
"http": {
"url": "http://localhost:1028/accumulate"
},
"lastFailure": "2018-03-25T17:10:49.00Z"
},
"throttling": 5
}
]
I guess he fails cuz did not send a notification, but I'm not sure.
I see two problems here.
First, accumulator is not working. Maybe is a weird networking problem which combines an IPv4 name lookup (i.e. curl localhost:1028/accumulate is solved as curl 127.0.0.1:1028/accumulate by the OS) with an accumulator listening only in the IPv6 interface (i.e. only in ::1 but not in 127.0.0.1). I understand you are running the curl commmand in the same host where accumulator is listening, isn't it?
My recomendation is to play with the --host accumualtor parameter (e.e. --host 127.0.0.1) and use a direct IP in the curl command in order to make it work.
The second problem is due to you are using localhost as notification endpoint:
"url": "http://localhost:1028/accumulate"
This means port 1028 inside the docker container where Orion is running. However, as far as I understand, your accumulator server runs outside the container, in the containers host. Thus, you should use an IP which allows you to reach the host from the container (and ensure no network traffic blocker is in place, e.g. firewall). So, your question here translates to "How to reach docker containers host from a docket container" (I'm not sure of the answer but there should be pretty much literature about the topic out there :)
The accumulation server needs to be run on available physical interface. To put it simply interactions using loopback interface with Orion Context Broker run as a Docker container are almost impossible. For sure as far as virtualization of host running host comes in place (as is in my situation).
Available interfaces can be checked in linux using
ip addr
After choosing one that is matching our requirements, we run accumulator as has been told before, however ip address for it is the one that we choose. Then we add subscription to OCB using address used while launching acc. server and are good to go, communication is alright.
I've been banging my head against the wall on this one for 2 days, and will post the resolution in the hopes it helps someone in the past. I created an Alexa skill to stream a radio station, and just couldn't get it to play, even using the sample code. Finally I just hardcoded the values sent down the wire, and it still didn't work. Here's what I was sending:
{
"version": "1.0",
"response": {
"shouldEndSession": 1,
"response": {
"outputSpeech": {
"text": "Playing Somgwriters island",
"type": "PlainText"
},
"directives": [
{
"playBehavior": "REPLACE_ALL",
"audioItem": {
"stream": {
"url": "http://la2-ssd.myautodj.com:8198/stream.mp3",
"token": "",
"offsetInMilliseconds": 0
}
},
"type": "AudioPlayer.Play"
}
],
"reprompt": {},
"card": {}
}
},
"sessionAttributes": {},
"statusCode": "200"
}
I finally found the cause in their documentation:
audioItem.stream.url:
Identifies the location of audio content at a remote HTTPS location.
The audio file must be hosted at an Internet-accessible HTTPS endpoint. HTTPS is required, and the domain hosting the files must present a valid, trusted SSL certificate. Self-signed certificates cannot be used. Many content hosting services provide this. For example, you could host your files at a service such as Amazon Simple Storage Service (Amazon S3) (an Amazon Web Services offering).