Spring Boot with server.contextPath set vs. URL to hystrix.stream via Eureka Server - spring-cloud

I have Eureka Server with Turbine instance running and a few discovery clients that are connected to it. Everything works fine, but if I register a discovery client that has server.contextPath set, it didn't get recognized by InstanceMonitor and Turbine stream is not able to combine its hystrix.stream.
This is how it looks in the logs of Eureka/Turbine server:
2015-02-12 06:56:23.265 INFO 1 --- [ Timer-0] c.n.t.discovery.InstanceObservable : Hosts up:3, hosts down: 0
2015-02-12 06:56:23.266 INFO 1 --- [ Timer-0] c.n.t.monitor.instance.InstanceMonitor : Url for host: http://user-service:8887/hystrix.stream default
2015-02-12 06:56:23.268 ERROR 1 --- [InstanceMonitor] c.n.t.monitor.instance.InstanceMonitor : Could not initiate connection to host, giving up: []
2015-02-12 06:56:23.269 WARN 1 --- [InstanceMonitor] c.n.t.monitor.instance.InstanceMonitor : Stopping InstanceMonitor for: user-service default
com.netflix.turbine.monitor.instance.InstanceMonitor$MisconfiguredHostException: []
at com.netflix.turbine.monitor.instance.InstanceMonitor.init(InstanceMonitor.java:318)
at com.netflix.turbine.monitor.instance.InstanceMonitor.access$100(InstanceMonitor.java:103)
at com.netflix.turbine.monitor.instance.InstanceMonitor$2.call(InstanceMonitor.java:235)
at com.netflix.turbine.monitor.instance.InstanceMonitor$2.call(InstanceMonitor.java:229)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
It tries to get hystrix stream from http://user-service:8887/hystrix.stream where the correct URL including sever.contextPath should be http://user-service:8887/uaa/hystrix.stream
The application.yml of that client contains:
server:
port: 8887
contextPath: /uaa
security:
ignored: /css/**,/js/**,/favicon.ico,/webjars/**
basic:
enabled: false
My question is: should I add some additional configuration options to this user-service discovery client to register proper hystrix.stream URL location?
I didn't dig into that yet, I will let you know if found something before getting answer to that question.
Current solution
There is one problem when it comes to using server.contextPath and management.context-path. When both are set, turbine stream is being served on ${HOST_URL}/${server.contextPath}/${management.context-path}/hystrix.stream. In that case I had to drop using server.contextPath (I replaced it with a prefix in controllers #RequestMapping).
Now, when you user management.context-path, then your hystrix.stream is being served from the URL that uses it as a prefix. In that case you have to follow Spencer's suggestion and set
turbine.instanceUrlSuffix=/{PUT_YOUR_MANAGEMENT_CONTEXT_PATH_HERE}/hystrix.stream
And of course this management.context-path must be set with the same value for all your Discovery Clients - it can be done easily with Spring Cloud Config http://cloud.spring.io/spring-cloud-config/spring-cloud-config.html

You can set turbine.instanceUrlSuffix.<CLUSTERNAME>=/uaa/hystrix.stream. Where <CLUSTERNAME> is the value set in turbine.aggregator.clusterConfig. All of the config options from the Turbine 1 wiki work. You don't need to add the port to the suffix as Spring Cloud Netflix Turbine adds the port from eureka.

Related

How to remove the unwanted characters from fluentd logs

Currently I am sending my Kubernetes logs to cloud watch using Fluentd, but when I check the logs in cloudwatch, the logs are having extra unicode characters. I tried different ways to and regexp to solve but no luck. Here is the sample how my log is in cloud watch
Log in Cloudwatch: "log": "\u001b[2m2021-10-13 20:07:10.351\u001b[0;39m \u001b[32m INFO\u001b[0;39m \u001b[35m1\u001b[0;39m \u001b[2m---\u001b[0;39m \u001b[2m[trap-executor-0]\u001b[0;39m \u001b[36mc.n.d.s.r.aws.ConfigClusterResolver \u001b[0;39m \u001b[2m:\u001b[0;39m Resolving eureka endpoints via configuration\n"
Actual log : 2021-10-13 20:07:10.351 INFO 1 --- [trap-executor-0] c.n.d.s.r.aws.ConfigClusterResolver : Resolving eureka endpoints via configuration

Consul agent on kubernetes, on node or pod?

I deployed an aws eks cluster via terraform. I also deployed Consul following hasicorp’s tutorial and I see the nodes in consul’s UI.
Now I’m wondering how al the consul agents will know about the pods I deploy? I deploy something and it’s not shown anywhere on consul.
I can’t find any documentation as to how to register pods (services) on consul via the node’s consul agent, do I need to configure that somewhere? Should I not use the node’s agent and register the service straight from the pod? Hashicorp discourages this since it may increase resource utilization depending on how many pods one deploy on a given node. But then how does the node’s agent know about my services deployed on that node?
Moreover, when I deploy a pod in a node and ssh into the node, and install consul, consul’s agent can’t find the consul server (as opposed from the node, which can find it)
EDIT:
Bottom line is I can't find WHERE to add the configuration. If I execute ON THE POD:
consul members
It works properly and I get:
Node Address Status Type Build Protocol DC Segment
consul-consul-server-0 10.0.103.23:8301 alive server 1.10.0 2 full <all>
consul-consul-server-1 10.0.101.151:8301 alive server 1.10.0 2 full <all>
consul-consul-server-2 10.0.102.112:8301 alive server 1.10.0 2 full <all>
ip-10-0-101-129.ec2.internal 10.0.101.70:8301 alive client 1.10.0 2 full <default>
ip-10-0-102-175.ec2.internal 10.0.102.244:8301 alive client 1.10.0 2 full <default>
ip-10-0-103-240.ec2.internal 10.0.103.245:8301 alive client 1.10.0 2 full <default>
ip-10-0-3-223.ec2.internal 10.0.3.249:8301 alive client 1.10.0 2 full <default>
But if i execute:
# consul agent -datacenter=voip-full -config-dir=/etc/consul.d/ -log-file=log-file -advertise=$(wget -q -O - http://169.254.169.254/latest/meta-data/local-ipv4)
I get the following error:
==> Starting Consul agent...
Version: '1.10.1'
Node ID: 'f10070e7-9910-06c7-0e12-6edb6cc4c9b9'
Node name: 'ip-10-0-3-223.ec2.internal'
Datacenter: 'voip-full' (Segment: '')
Server: false (Bootstrap: false)
Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, gRPC: -1, DNS: 8600)
Cluster Addr: 10.0.3.223 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
==> Log data will now stream in as it occurs:
2021-08-16T18:23:06.936Z [WARN] agent: skipping file /etc/consul.d/consul.env, extension must be .hcl or .json, or config format must be set
2021-08-16T18:23:06.936Z [WARN] agent: Node name "ip-10-0-3-223.ec2.internal" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.
2021-08-16T18:23:06.946Z [WARN] agent.auto_config: skipping file /etc/consul.d/consul.env, extension must be .hcl or .json, or config format must be set
2021-08-16T18:23:06.947Z [WARN] agent.auto_config: Node name "ip-10-0-3-223.ec2.internal" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.
2021-08-16T18:23:06.948Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: ip-10-0-3-223.ec2.internal 10.0.3.223
2021-08-16T18:23:06.948Z [INFO] agent.router: Initializing LAN area manager
2021-08-16T18:23:06.950Z [INFO] agent: Started DNS server: address=127.0.0.1:8600 network=udp
2021-08-16T18:23:06.950Z [WARN] agent.client.serf.lan: serf: Failed to re-join any previously known node
2021-08-16T18:23:06.950Z [INFO] agent: Started DNS server: address=127.0.0.1:8600 network=tcp
2021-08-16T18:23:06.951Z [INFO] agent: Starting server: address=127.0.0.1:8500 network=tcp protocol=http
2021-08-16T18:23:06.951Z [WARN] agent: DEPRECATED Backwards compatibility with pre-1.9 metrics enabled. These metrics will be removed in a future version of Consul. Set `telemetry { disable_compat_1.9 = true }` to disable them.
2021-08-16T18:23:06.953Z [INFO] agent: started state syncer
2021-08-16T18:23:06.953Z [INFO] agent: Consul agent running!
2021-08-16T18:23:06.953Z [WARN] agent.router.manager: No servers available
2021-08-16T18:23:06.954Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers"
2021-08-16T18:23:34.169Z [WARN] agent.router.manager: No servers available
2021-08-16T18:23:34.169Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers"
So where to add the config?
I also tried adding a service in k8s pointing to the pod, but the service doesn't come up on consul's UI...
What do you guys recommend?
Thanks
Consul knows where these services are located because each service
registers with its local Consul client. Operators can register
services manually, configuration management tools can register
services when they are deployed, or container orchestration platforms
can register services automatically via integrations.
if you planning to use manual option you have to register the service into the consul.
Something like
echo '{
"service": {
"name": "web",
"tags": [
"rails"
],
"port": 80
}
}' > ./consul.d/web.json
You can find the good example at : https://thenewstack.io/implementing-service-discovery-of-microservices-with-consul/
Also this is a very nice document for having detailed configuration of the health check and service discovery : https://cloud.spring.io/spring-cloud-consul/multi/multi_spring-cloud-consul-discovery.html
Official document : https://learn.hashicorp.com/tutorials/consul/get-started-service-discovery
BTW, I was finally able to figure out the issue.
consul-dns is not deployed by default, i had to manually deploy it, then forward all .consul requests from coredns to consul-dns.
All is working now. Thanks!

Failed to establish connection to Neo4j usign bolt scheme even after successfully enabling Bolt

I want to connect to Neo4j database using my creds. I am tunneling into a machine and once that is done, I open my broswer at the port: localhost:7474.
I tried with both neo4j and bolt scheme to connect at the url:
bolt://<node_ip>:7687 and neo4j://<node_ip>:7687 but the connection times out.
I tried checking the logs but only found that bolt scheme is enabled:
bash-4.2$ tail -f /logs/debug.log
2021-07-02 21:26:03.323+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.logging.impl.LogKitLogger` from plugin jar `/home/sandbox/neo/plugins/apoc-4.2.0.2-all.jar`: org/apache/log/Logger
2021-07-02 21:26:03.946+0000 INFO [c.n.m.g.GlobalMetricsExtension] Sending metrics to CSV file at /home/sandbox/neo/metrics
2021-07-02 21:26:03.973+0000 INFO [o.n.b.BoltServer] Bolt enabled on 0.0.0.0:7687.
2021-07-02 21:26:03.974+0000 INFO [o.n.b.BoltServer] Bolt (Routing) enabled on 0.0.0.0:7688.
2021-07-02 21:26:03.974+0000 INFO [o.n.s.AbstractNeoWebServer$ServerComponentsLifecycleAdapter] Starting web server
2021-07-02 21:26:04.001+0000 INFO [o.n.s.m.ThirdPartyJAXRSModule] Mounted unmanaged extension [n10s.endpoint] at [/rdf]
2021-07-02 21:26:05.341+0000 INFO [c.n.s.e.EnterpriseNeoWebServer] Remote interface available at http://<node_ip>:7474/
2021-07-02 21:26:05.341+0000 INFO [o.n.s.AbstractNeoWebServer$ServerComponentsLifecycleAdapter] Web server started.
2021-07-02 21:35:34.565+0000 INFO [c.n.c.c.c.l.s.Segments] [system/00000000] Pruning SegmentFile{path=raft.log.0, header=SegmentHeader{formatVersion=2, recordOffset=56, prevFileLastIndex=-1, segmentNumber=0, prevIndex=-1, prevTerm=-1}}
2021-07-02 21:35:46.079+0000 INFO [c.n.c.c.c.l.s.Segments] [neo4j/32f6599b] Pruning SegmentFile{path=raft.log.0, header=SegmentHeader{formatVersion=2, recordOffset=56, prevFileLastIndex=-1, segmentNumber=0, prevIndex=-1, prevTerm=-1}}
The query log is empty, as I could not execute any query:
bash-4.2$ tail -f query.log
2021-07-02 21:25:52.510+0000 INFO Query started: id:1 - 1009 ms: 0 B - embedded-session neo4j - - call db.clearQueryCaches() - {} - runtime=pipelined - {}
2021-07-02 21:25:52.580+0000 INFO id:1 - 1080 ms: 112 B - embedded-session neo4j - - call db.clearQueryCaches() - {} - runtime=pipelined - {}
The other articles or answers that I read were mostly about misconfiguration: wrong ports but I don't think that is the case with me since I checked from debug.log file that my ports are alright.
FWIW, I am using 3 replicas for my Neo4j and right now, connecting to just one pod.
I am tunnelling both the ports:
ssh -L 7687:$IP:7687 -L 7474:$IP:7474 domain_name.com -N```
Perhaps you've already checked this, but if not, can you ensure that port 7687 is also forwarded. When I tunnelled via browser, my expectation was that 7474 would be sufficient, but it turned out that forwarding 7687 is also necessary.
So, instead of providing localhost in the connection string, I made a silly mistake of writing down the actual IP and that was the reason for connection timeout.

why must eureka.client.serviceUrl.defaultZone be provided in `bootstrap.properties` when config the spring cloud config server in a discovery manner?

I was aiming to config the location of spring cloud config server by setting spring.applicaton.name and server.port and eureka.client.serviceUrl.defaultZone in application.properties together with spring.cloud.config.discovery.enabled=true and spring.cloud.config.discovery.service-id=cloud-config in bootstrap.properties, which turned out to be insufficient. The following error messages are shown in log:'
com.netflix.discovery.DiscoveryClient : DiscoveryClient_BOOTSTRAP/192.168.1.5:bootstrap - was unable to refresh its cache! status = Cannot execute request on any known server
No instances found of configserver (cloud-config)
According to the docs, I moved eureka.client.serviceUrl.defaultZone into bootstrap.properties and succeeded.
My question is, if spring.application.name and server.port are essential for a eureka client to register on the eureka server, why they can be unsettled in bootstrap.properties for the config client?
I suspect that the config client will first use eureka.client.serviceUrl.defaultZone alone to connect with the eureka server and fetch service registration informations but not register itself so as to locate the config server and pull something. After that, since the config client is also a eureka client, it uses relative parameters in application.properties to register on eureka server. As some evidence of my suspect, I found the following logs during the startup of the application:
2017-09-07 06:13:09.651 INFO [bootstrap,,,] 74104 --- [ restartedMain] com.netflix.discovery.DiscoveryClient : Getting all instance registry info from the eureka server
2017-09-07 06:13:09.817 INFO [bootstrap,,,] 74104 --- [ restartedMain] com.netflix.discovery.DiscoveryClient : The response status is 200
2017-09-07 06:13:09.821 INFO [bootstrap,,,] 74104 --- [ restartedMain] com.netflix.discovery.DiscoveryClient : Not registering with Eureka server per configuration
2017-09-07 06:13:37.427 INFO [-,,,] 74104 --- [ restartedMain] com.netflix.discovery.DiscoveryClient : Getting all instance registry info from the eureka server
Is it?

Spring Cloud - Registry Service port customization

I'd like to customize the Eureka port with Spring Cloud.
With the default port below, the services registry sees itself right (within the provided GUI)
spring:
application:
name: services-registry
server:
port: 8761
eureka:
instance:
hostname: localhost
nonSecurePort: ${server.port}
client:
register-with-eureka: true
fetch-registry: false
service-url:
default-zone: http://${eureka.instance.hostname}:${server.port}/eureka/
But if I just change server.port to 8787, no service can register itself, not even the services registry itself.
2017-01-09 16:18:21.584 WARN 17496 --- [nfoReplicator-0] c.n.d.s.t.d.RetryableEurekaHttpClient : Request execution failure
2017-01-09 16:18:21.584 WARN 17496 --- [nfoReplicator-0] com.netflix.discovery.DiscoveryClient : DiscoveryClient_SERVICES-REGISTRY/xxx.org:services-registry:8787 - registration failed Cannot execute request on any known server
com.netflix.discovery.shared.transport.TransportException: Cannot execute request on any known server
...
2017-01-09 16:13:33.299 WARN 17496 --- [nfoReplicator-0] c.n.discovery.InstanceInfoReplicator : There was a problem with the instance info replicator
com.netflix.discovery.shared.transport.TransportException: Cannot execute request on any known server
Can someone explain this issue and save my day? Thanks!
Ok, got it... the label after service-url property (which can be aliased as serviceUrl in YML) is a HashMap KEY, not a property label. So it has to be kept as a Camel Case tag in any ways!
eureka.client.service-url.defaultZone=http://[myIP#]:8787/eureka