How to query Go-micro services(v2) inside docker with curl or postman - docker-compose

I use Go-micro(v2) to deploy services inside docker-compose
user-service:
build:
context: ./user-service
restart: always
ports:
- "8086:8086"
deploy:
mode: replicated
replicas: 1
environment:....
See the service configuration
srv = micro.NewService(
micro.Name("my.user"),
micro.Address("127.0.0.1:8086"))
when running docker-compose, the container logs show
2022-07-31 05:43:53 file=v2#v2.9.1/service.go:200 level=info Starting [service] my.user
2022-07-31 05:43:53 file=grpc/grpc.go:864 level=info Server [grpc] Listening on [::]:8086
2022-07-31 05:43:53 file=grpc/grpc.go:697 level=info Registry [mdns] Registering node: my.user-00ee4795-06df-47f1-a07a-cc362e135864
All looks good.
But when I want to query some handlers using curl or postman(for development purpose), It doesn't work,
see an exemple of failed request with postman
GET http://127.0.0.1:8086/my.user/Get
Error: Parse Error: Expected HTTP/
Request Headers
Content-Type: application/json
User-Agent: PostmanRuntime/7.29.2
Accept: */*
Postman-Token: b5ab718a-341b-40ff-81fa-37c66fd4d9f2
Host: 127.0.0.1:8086
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Request Body
GET http://127.0.0.1:8086/my.user/userService/Get // same error
with curl it is not better
curl --header "Content-Type:application/json" --http0.9 --output GET http://localhost:8086/my.user/Get
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 15 0 15 0 0 10638 0 --:--:-- --:--:-- --:--:-- 15000
curl --header "Content-Type:application/json" --http0.9 --output GET http://localhost:8086/my.user/userService/Get
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 15 0 15 0 0 13550 0 --:--:-- --:--:-- --:--:-- 15000
Any idea how to query locally some go-micro services ? Thank you.
ps: Note that the 'Get' handler is working

Related

postgress dvdrental sample database

Why do i get this error while trying to get the sample database on my Ubuntu?
postgres#vagrant:~$ curl -O https://sp.postgresqltutorial.com/wp-content/uploads/2019/05/dvdrental.zip
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0
curl: (51) SSL: no alternative certificate subject name matches target host name 'sp.postgresqltutorial.com'
I downloaded it manually but can't access it from psql shell

Kafka REST proxy: how to retrieve and deserialize Kafka data based on AVRO schema stored in schema-registry

I am new to Kafka. I run a docker based Kafka ecosystem on my local machine, including broker/zookeeper/schema-registry/rest-proxy. Also I have a external producer(temp-service), which sends AVRO schema serialized data to the topic temp-topic in broker.
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
411564d10c06 confluentinc/cp-kafka-rest:latest "/etc/confluent/dock…" 27 seconds ago Up 25 seconds 0.0.0.0:8082->8082/tcp kafka_kafka-rest_1
38c4e3ea008c confluentinc/cp-schema-registry:latest "/etc/confluent/dock…" 30 seconds ago Up 27 seconds 0.0.0.0:8081->8081/tcp kafka_schema-registry_1
7abe6cf9a7a0 confluentinc/cp-kafka:latest "/etc/confluent/dock…" 30 minutes ago Up 30 seconds 0.0.0.0:9092->9092/tcp kafka
bdffd9e03088 confluentinc/cp-zookeeper:latest "/etc/confluent/dock…" 30 minutes ago Up 30 seconds 2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp zookeeper
d1909c6877c5 temp-service:latest "node /home/tempserv…" 3 hours ago Up 2 hours (healthy) 0.0.0.0:8107->8107/tcp, 0.0.0.0:9107->9107/tcp, 0.0.0.0:9229->9229/tcp temp-service
I have also posted the AVRO schema of the kafka data of temp-service to the schema-registry, so that it is stored there as id 1.
I created a consumer group temp_consumers and a consumer instance temp_consumer_instance,
$ curl -X POST -H "Content-Type: application/vnd.kafka.v2+json" --data '{"name": "temp_consumer_instance", "format": "avro", "auto.offset.reset": "earliest"}' http://localhost:8082/consumers/temp_consumers
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 219 0 134 100 85 158 100 --:--:-- --:--:-- --:--:-- 260
{"instance_id":"temp_consumer_instance","base_uri":"http://kafka-rest:8082/consumers/temp_consumers/instances/temp_consumer_instance"}
checked topics in kafka:
$ curl -X GET http://localhost:8082/topics
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 29 0 29 0 0 1610 0 --:--:-- --:--:-- --:--:-- 1705
["temp-topic","_schemas"]
subscribed to the topic temp-topic.
$ curl -X POST -H "Content-Type: application/vnd.kafka.v2+json" --data '{"topics":["temp-topic"]}' http://localhost:8082/consumers/temp_consumers/instances/temp_consumer_instance/subscription
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 29 0 0 100 29 0 1046 --:--:-- --:--:-- --:--:-- 1115
tried to consume the records in the topic but failed:
$ curl -X GET -H "Accept: application/vnd.kafka.binary.v2+json" http://localhost:8082/consumers/temp_consumers/instances/temp_consumer_instance/records
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 126 0 126 0 0 10598 0 --:--:-- --:--:-- --:--:-- 12600
{"error_code":40601,"message":"The requested embedded data format does not match the deserializer for this consumer instance"}
I would like to know if there are any ways to deserialize the kafka data posted by the producer, based on the AVRO schema stored in schema registry?

Istio envoy upstream reset: reset reason connection failure

I have a GKE cluster (gke v1.13.6) and using istio (v1.1.7) with several services deployed and working successfully except one of them which always responds with HTTP 503 when calling through the gateway : upstream connect error or disconnect/reset before headers. reset reason: connection failure.
I've tried calling the pod directly from another pod with curl enabled and it ends up in 503 as well :
$ kubectl exec sleep-754684654f-4mccn -c sleep -- curl -v d-vine-machine-dev:8080/d-vine-machine/swagger-ui.html
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 10.3.254.3...
* TCP_NODELAY set
* Connected to d-vine-machine-dev (10.3.254.3) port 8080 (#0)
> GET /d-vine-machine/swagger-ui.html HTTP/1.1
> Host: d-vine-machine-dev:8080
> User-Agent: curl/7.60.0
> Accept: */*
>
upstream connect error or disconnect/reset before headers. reset reason: connection failure< HTTP/1.1 503 Service Unavailable
< content-length: 91
< content-type: text/plain
< date: Thu, 04 Jul 2019 08:13:52 GMT
< server: envoy
< x-envoy-upstream-service-time: 60
<
{ [91 bytes data]
100 91 100 91 0 0 1338 0 --:--:-- --:--:-- --:--:-- 1378
* Connection #0 to host d-vine-machine-dev left intact
Setting the log level to TRACE at the istio-proxy level :
$ kubectl exec -it -c istio-proxy d-vine-machine-dev-b8df755d6-bpjwl -- curl -X POST http://localhost:15000/logging?level=trace
I looked into the logs of the injected sidecar istio-proxy and found this :
[2019-07-04 07:30:41.353][24][debug][router] [external/envoy/source/common/router/router.cc:381] [C119][S9661729384515860777] router decoding headers:
':authority', 'api-dev.d-vine.tech'
':path', '/d-vine-machine/swagger-ui.html'
':method', 'GET'
':scheme', 'http'
'cache-control', 'max-age=0'
'upgrade-insecure-requests', '1'
'user-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36'
'accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3'
'accept-encoding', 'gzip, deflate'
'accept-language', 'fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7'
'x-forwarded-for', '10.0.0.1'
'x-forwarded-proto', 'http'
'x-request-id', 'e38a257a-1356-4545-984a-109500cb71c4'
'content-length', '0'
'x-envoy-internal', 'true'
'x-forwarded-client-cert', 'By=spiffe://cluster.local/ns/default/sa/default;Hash=8b6afba64efe1035daa23b004cc255e0772a8bd23c8d6ed49ebc8dabde05d8cf;Subject="O=";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account;DNS=istio-ingressgateway.istio-system'
'x-b3-traceid', 'f749afe8b0a76435192332bfe2f769df'
'x-b3-spanid', 'bfc4618c5cda978c'
'x-b3-parentspanid', '192332bfe2f769df'
'x-b3-sampled', '0'
[2019-07-04 07:30:41.353][24][debug][pool] [external/envoy/source/common/http/http1/conn_pool.cc:88] creating a new connection
[2019-07-04 07:30:41.353][24][debug][client] [external/envoy/source/common/http/codec_client.cc:26] [C121] connecting
[2019-07-04 07:30:41.353][24][debug][connection] [external/envoy/source/common/network/connection_impl.cc:644] [C121] connecting to 127.0.0.1:8080
[2019-07-04 07:30:41.353][24][debug][connection] [external/envoy/source/common/network/connection_impl.cc:653] [C121] connection in progress
[2019-07-04 07:30:41.353][24][debug][pool] [external/envoy/source/common/http/conn_pool_base.cc:20] queueing request due to no available connections
[2019-07-04 07:30:41.353][24][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:811] [C119][S9661729384515860777] decode headers called: filter=0x4f118b0 status=1
[2019-07-04 07:30:41.353][24][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:384] [C119] parsed 1272 bytes
[2019-07-04 07:30:41.353][24][trace][connection] [external/envoy/source/common/network/connection_impl.cc:282] [C119] readDisable: enabled=true disable=true
[2019-07-04 07:30:41.353][24][trace][connection] [external/envoy/source/common/network/connection_impl.cc:440] [C121] socket event: 3
[2019-07-04 07:30:41.353][24][trace][connection] [external/envoy/source/common/network/connection_impl.cc:508] [C121] write ready
[2019-07-04 07:30:41.353][24][debug][connection] [external/envoy/source/common/network/connection_impl.cc:526] [C121] delayed connection error: 111
[2019-07-04 07:30:41.353][24][debug][connection] [external/envoy/source/common/network/connection_impl.cc:183] [C121] closing socket: 0
[2019-07-04 07:30:41.353][24][debug][client] [external/envoy/source/common/http/codec_client.cc:82] [C121] disconnect. resetting 0 pending requests
[2019-07-04 07:30:41.353][24][debug][pool] [external/envoy/source/common/http/http1/conn_pool.cc:129] [C121] client disconnected, failure reason:
[2019-07-04 07:30:41.353][24][debug][pool] [external/envoy/source/common/http/http1/conn_pool.cc:164] [C121] purge pending, failure reason:
[2019-07-04 07:30:41.353][24][debug][router] [external/envoy/source/common/router/router.cc:644] [C119][S9661729384515860777] upstream reset: reset reason connection failure
[2019-07-04 07:30:41.353][24][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:1200] [C119][S9661729384515860777] encode headers called: filter=0x4f0e5f0 status=0
[2019-07-04 07:30:41.353][24][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:1200] [C119][S9661729384515860777] encode headers called: filter=0x4f0edc0 status=0
[2019-07-04 07:30:41.353][24][debug][filter] [src/envoy/http/mixer/filter.cc:133] Called Mixer::Filter : encodeHeaders 2
[2019-07-04 07:30:41.353][24][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:1200] [C119][S9661729384515860777] encode headers called: filter=0x4f0f0e0 status=0
[2019-07-04 07:30:41.353][24][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1305] [C119][S9661729384515860777] encoding headers via codec (end_stream=false):
':status', '503'
'content-length', '91'
'content-type', 'text/plain'
'date', 'Thu, 04 Jul 2019 07:30:41 GMT'
'server', 'istio-envoy'
Has anyone encountered such an issue ? If you need more info about the configuration, I can provide.
Thanks for your answer Manvar. There was no problem with the curl enabled pod but thanks for the insight. It was a misconfiguration of our tomcat port that was not matching the service/virtualService config.
When pod with an istio side car is started, the follwing things happen
an init container changes the iptables rules so that all the outgoing tcp traffic is routed to the sidecar istio-proxy on port 15001.
the containers of the pod are started in parallel (curl and istio-proxy)
If your curl container is executed before istio-proxy listens on port 15001, you get the error.

Cannot retrieve file list from Azure File Storage using REST API and curl

I'm trying to retrieve the list of files stored in an Azure File Storage account using the REST API and curl, I correctly computed headers according to the documentation by using the shared key , but curl request neither returns the files list nor any error message.
Here is my request and the response:
curl -v -H "Authorization: SharedKey myaccount:bAJKeY0xyOZLSJOLDoHfXXOqfA4kOGo1DVFP3BejhY8=" -H "x-ms-date:Mon, 13 Aug 2018 15:22:31 GMT" -H "x-ms-version:2017-07-29" --url https://myaccount.file.core.windows.net/myshare/mydir?restype=directory&comp=list
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 52.239.140.8...
* Connected to myaccount.file.core.windows.net (52.239.140.8) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 597 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
* server certificate verification OK
* server certificate status verification SKIPPED
* common name: *.file.core.windows.net (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #3
* subject: CN=*.file.core.windows.net
* start date: Thu, 09 Nov 2017 05:42:03 GMT
* expire date: Sat, 09 Nov 2019 05:42:03 GMT
* issuer: C=US,ST=Washington,L=Redmond,O=Microsoft Corporation,OU=Microsoft IT,CN=Microsoft IT TLS CA 5
* compression: NULL
* ALPN, server did not agree to a protocol
GET /myshare/mydir?restype=directory HTTP/1.1
Host: myaccount.file.core.windows.net
User-Agent: curl/7.47.0
Accept: */*
Authorization: SharedKey
myaccount:bAJKeY0xyOZLSJOLDoHfXXOqfA4kOGo1DVFP3BejhY8=
x-ms-date:Mon, 13 Aug 2018 15:22:31 GMT
x-ms-version:2017-07-29
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Last-Modified: Fri, 27 Apr 2018 16:11:14 GMT
ETag: "0x8D5AC597FF96B3D"
Server: Windows-Azure-File/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: 75d6d7c8-f01a-0011-5b19-33104d000000
x-ms-version: 2017-07-29
x-ms-server-encrypted: true
Date: Mon, 13 Aug 2018 15:22:29 GMT
{ [5 bytes data]
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
* Connection #0 to host myaccount.file.core.windows.net left intact
No XML with file list is returned.
I tried to retrieve the share list under myaccount and it works, as well as downloading a single file, but I cannot receive the list of files under a directory.
Two points:
See url in curl command
--url https://myaccount.file.core.windows.net/myshare/mydir?restype=directory&comp=list
You forget to put the url in "" so the parameter &comp=list is cut because & is a reserved sign. This is also proved by the output GET /myshare/mydir?restype=directory HTTP/1.1.
Commonly speaking, if the url misses the comp parameter, we should get error message AuthenticationFailed because comp is used in generating SharedKey. However you get HTTP/1.1 200 OK with the SharedKey.
Based on the response headers you get, I guess you also missed the comp when constructing the SharedKey, so the SharedKey and url is capable to get directory properties correctly.

Server-Sent Events with Play: response only received when process killed

I'm trying to get the sample webapp play-streaming-scala to run and in some circumstances I get a weird behavior.
I've got the app running directly on port 80 of some host and I'm checking the output with curl -iv --raw http://somehost/scala/eventSource/liveClock.
What I'm expecting is something like this:
* Hostname was NOT found in DNS cache
* Trying 195.176.3.71...
* Connected to somehost (0.0.0.0) port 80 (#0)
> GET /scala/eventSource/liveClock HTTP/1.1
> User-Agent: curl/7.39.0
> Host: somehost
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Transfer-Encoding: chunked
Transfer-Encoding: chunked
< Content-Type: text/event-stream; charset=utf-8
Content-Type: text/event-stream; charset=utf-8
< Date: Wed, 18 Jan 2017 13:24:55 GMT
Date: Wed, 18 Jan 2017 13:24:55 GMT
<
10
data: 14 24 56
10
data: 14 24 56
10
data: 14 24 56
etc., and clearly see the chunks appear one after the other as time goes by.
Now, on some machines, this works well. On some others on campus, this fails. curl only shows this and then stops:
* Trying 195.176.3.71...
* Connected to somehost (0.0.0.0) port 80 (#0)
> GET /scala/eventSource/liveClock HTTP/1.1
> Host: somehost
> User-Agent: curl/7.43.0
> Accept: */*
>
Now the interesting thing is: if I kill the webapp on the host, curl suddenly “catches up” and spits all the chunks together, closing the connection like this:
10
data: 14 35 20
* transfer closed with outstanding read data remaining
* Closing connection 0
curl: (18) transfer closed with outstanding read data remaining
What can be causing the behavior? What on earth is going on and intercepting these events? Is there any way I can “force flush” something from the Play response?
Turns out the local “hidden” proxy set up automatically by OS X's parental controls system is not forwarding chunked responses properly, thus making a system based on Server-Sent Events inoperable. A shame.