Spring-Boot-Admin on Cloud Foundry: Showing UP applications as OFFLINE - spring-boot-admin

We are developing a Boot-Admin dashboard using the codecentric provided library of spring-boot-admin-server version 1.4.5
Some of the applications are registering itself with the server via Eureka, and some directly using the spring-boot-admin-starter-client version 1.4.5.
All components are deployed on a PCF environment and are communication over HTTPS. In either ways, the applications though are able to register themselves with the admin sever but are showing up as OFFLINE only. There are no errors reported in the logs for any of the components viz. admin-server, admin-client, eureka-server, eureka-client.
However the only application showing as Up is the admin server itself.
The application properties for spring-boot-admin-client app to run in PCF is:
spring:
application:
name: bootadmin-ms-charlie
boot:
admin:
url: https://bootadmin-dashboard.abc.intl.com
ssl:
trust_store:
path: classpath:ssl/sslcacert.jks
password: a-password
As the result is same for both methods of registration, I've skipped putting the config here for apps registering via Eureka path to keep it simple.
The same is working perfectly fine locally, where the admin dashboard shows all the applications as expected.
Is there any configuration that needs to be done in specific to Cloud Foundry?
Or any obvious mistake that I might have made?
Any suggestions are most welcome.
---EDIT---
Here are the logs from SBA server showing that the communication between server and client is working okay. If these logs give any indication of an error, please point out.
OUT 2017-01-23 05:15:15.139 DEBUG 10 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : DispatcherServlet with name 'dispatcherServlet' processing POST request for [/api/applications]
OUT 2017-01-23 05:15:15.151 DEBUG 10 --- [nio-8080-exec-1] m.m.a.RequestResponseBodyMethodProcessor : Read [class de.codecentric.boot.admin.model.Application] as "application/json;charset=UTF-8" with [org.springframework.http.converter.json.MappingJackson2HttpMessageConverter#7df33a9f]
OUT 2017-01-23 05:15:15.163 DEBUG 10 --- [nio-8080-exec-1] o.s.w.s.m.m.a.HttpEntityMethodProcessor : Written [Application [id=3805ee6a, name=bootadmin-ms-charlie, managementUrl=http://23fcf304-82d6-44cd-7fce-2a5027de9f21:8080, healthUrl=http://23fcf304-82d6-44cd-7fce-2a5027de9f21:8080/health, serviceUrl=http://23fcf304-82d6-44cd-7fce-2a5027de9f21:8080]] as "application/json" using [org.springframework.http.converter.json.MappingJackson2HttpMessageConverter#7df33a9f]
OUT 2017-01-23 05:15:15.166 DEBUG 10 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : Null ModelAndView returned to DispatcherServlet with name 'dispatcherServlet': assuming HandlerAdapter completed request handling
OUT 2017-01-23 05:15:15.166 DEBUG 10 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : Successfully completed request
OUT bootadmin-dashboard.abc-intl.com - [23/01/2017:05:15:15.140 +0000] "POST /api/applications HTTP/1.1" 201 302 308 "-" "Java/1.8.0_121" 60.16.25.20:43224 x_forwarded_for:"10.10.10.10" x_forwarded_proto:"https" vcap_request_id:a40159e4-543f-40e0-627e-e8f1e7688b99 response_time:0.034164523 app_id:adcc8a33-83f4-448d-9ae2-bf2a2b16ea72
OUT 2017-01-23 05:15:18.719 DEBUG 10 --- [ updateTask1] o.s.web.client.RestTemplate : Created GET request for "http://23fcf304-82d6-44cd-7fce-2a5027de9f21:8080/health"
OUT 2017-01-23 05:15:18.722 DEBUG 10 --- [ updateTask1] o.s.web.client.RestTemplate : Setting request Accept header to [application/json, application/*+json]
The logs from client are all clean. It throws warning of "Failed to refister" only when the server is down.

Based upon the discussion on https://github.com/codecentric/spring-boot-admin/issues/399 it has been found out that below properties are vital for SBA clients to work with Dashboard on Cloud Foundry or container based architecture:
spring:
boot:
admin:
client:
management-url: <complete management-url for the client>
health-url: <complete health endpoint url for the client>
service-url: <complete root/service url for the client>
This is due to the fact that, when a client registers itself with the
SBA server it uses the runC container ID to form its service url. Such url is not valid for the Cloud Foundry router. That results in failure of communication between SBA dashboard and the client later on, causing it to show as OFFLINE.
The other approach can be to go ahead with container IPs, using spring.boot.admin.client.prefer-ip=true. That would list all containers/CF instacnes on SBA but not give a clear picture of overall health of the complete app from a site/AZ. Also direct connection to containers is never encouraged in CF as per the principles of cloud-native app and 12-factors.

Related

Error - Failed to add 'SAP-Connectivity-Authentication' header for on-premise connectivity

I am connecting an On-premise S/4 HANA with SAP Cloud Platform trial account. I am using SAP Cloud SDK to fetch all Business Partners from S/4 HANA.
My Cloud Connector is set
My Destination at Sub-Account level is set and can ping to my on-premise system
My Service instances - XSUAA/Destination/Connectivity is set with the application
But I have the following error
Failed to add 'SAP-Connectivity-Authentication' header for on-premise connectivity: no JWT bearer found in the 'Authorization' header of the request. Continuing without a header. Connecting to on-premise systems may not be possible
The code which I am using is -
final List<BusinessPartner> businessPartners =
new DefaultBusinessPartnerService()
.getAllBusinessPartner()
.select(BusinessPartner.BUSINESS_PARTNER)
.execute(destination);
It seems AppRouter is the recommended for Authorization and Access and hence I tried implementing one- but my approuter shows - Not Found
Approuter App -Name - approuter-demo
Below is the xs-app.json
{
"routes": [
{
"source": "^/s4ext/(.*)",
"target": "/s4ext/$1",
"destination": "******"
}
]
}
The Manifest file is as below:
---
applications:
- name: approuter-demo
routes:
- route: approuter-demo-*****trial.cfapps.eu10.hana.ondemand.com
path: approuter
memory: 128M
env:
TENANT_HOST_PATTERN: 'approuter-demo-(.*).cfapps.eu10.hana.ondemand.com'
destinations: '[{"name":"******", "url" :"https://s4ext-***.cfapps.eu10.hana.ondemand.com", "forwardAuthToken": true }]'
services:
- xsuaa-demo
- connectivity-demo
- destination-demo
Kindly guide me. Thanks.
Your destination type might be wrong. The authorization header is set via the destination.
Try other types in sap cp -> connectivity.
Reading your question again I can identify two issues:
This error message in your log:
Failed to add 'SAP-Connectivity-Authentication' header for on-premise connectivity: no JWT bearer found in the 'Authorization' header of the request. Continuing without a header. Connecting to on-premise systems may not be possible
It may be that this error message is actually superfluous and hence indicating a problem which is actually none. In your case this header is possibly not necessary and the SAP Cloud SDK should not try to add it. But in any case, this will not influence the actual connection, so this error message is at most confusing, but not harmful in the sense of altering functionality.
Still, I am asking you to add the stack trace of this exception to your question to be very sure here.
Your app router shows "Not Found":
Here I am missing more information. When does what exactly show "Not Found"? Is it that your browser cannot find your app router, or can your app router not find the target URL of the application?

Multiple Authorization Server

I have a Zuul Proxy that host behind it all Micro services including UI and Authorization servers .
All are part of spring cloud.
I tried to start a second Authorization Servers but that lead to the below errors
2017-11-17 08:07:47.146 INFO 9652 --- [http-nio-9191-exec-2] o.s.s.o.p.token.store.JdbcTokenStore : Failed to find access token for token 08fa5f86-4bbd-4e5b-818b-a79730232f30
2017-11-17 08:07:47.146 DEBUG 9652 --- [http-nio-9191-exec-2] p.a.OAuth2AuthenticationProcessingFilter : Authentication request failed: error="invalid_token", error_description="Invalid access token: 08fa5f86-4bbd-4e5b-818b-a79730232f30"
2017-11-17 08:07:47.147 DEBUG 9652 --- [http-nio-9191-exec-2] s.s.o.p.e.DefaultOAuth2ExceptionRenderer : Written [error="invalid_token", error_description="Invalid access token: 08fa5f86-4bbd-4e5b-818b-a79730232f30"] as "application/json;charset=UTF-8" using [org.springframework.http.converter.json.MappingJackson2HttpMessageConverter#3c3df936]
also
2017-11-17 07:39:51.151 DEBUG 9652 --- [http-nio-9191-exec-3] o.s.s.o.provider.endpoint.TokenEndpoint : Clearing scope of incoming token request
2017-11-17 07:39:51.178 DEBUG 9652 --- [http-nio-9191-exec-3] .s.s.o.p.c.AuthorizationCodeTokenGranter : Getting access token for: acme
2017-11-17 07:39:51.204 INFO 9652 --- [http-nio-9191-exec-3] o.s.s.o.provider.endpoint.TokenEndpoint : Handling error: InvalidGrantException, Invalid authorization code: Pb3JKA
2017-11-17 07:40:04.797 DEBUG 9652 --- [http-nio-9191-exec-4] .s.o.p.e.FrameworkEndpointHandlerMapping : Looking up handler method for path /oauth/token
2017-11-17 07:40:04.797 DEBUG 9652 --- [http-nio-9191-exec-4] .s.o.p.e.FrameworkEndpointHandlerMapping : Returning handler method [public org.springframework.http.ResponseEntity<org.springframework.security.oauth2.common.OAuth2AccessToken> org.springframework.security.oauth2.provider.endpoint.TokenEndpoint.postAccessToken(java.security.Principal,java.util.Map<java.lang.String, java.lang.String>) throws org.springframework.web.HttpRequestMethodNotSupportedException]
2017-11-17 07:40:04.850 DEBUG 9652 --- [http-nio-9191-exec-4] o.s.s.o.provider.endpoint.TokenEndpoint : Clearing scope of incoming token request
My conclusion was I need to have a shared session management between both authorization servers like spring session even though I am using the same jdbctoken database back end.
I could not find any sample or any one talking about this so is this correct or it should work without having a shared session.
It turned out that yes , shared session is needed like spring session , my first attempt did not work because I had mixed configuration between both servers.
After that just enabling shared session management, it all worked perfectly

Dredd seems to hang with api calls to asp.net kestrel server

When I run dredd, it seems to just hang after the first api call. I'm not sure how to debug or diagnose. I'm running an asp.net 5 mvc 6 api service with the dnx web command. How can I see what's going on or what the issue might be?
I tried adding hooks for debugging and see that it calls the first api and then just never triggers the after event. I tested the api with postman and curl with correct results. Dredd also works with the same blueprint if I'm using an express node.js server.
The only combination that fails is Dredd with the asp.net kestrel server response. The server logs that the request is made and a response is sent so it does trigger the test, but just never finishes.
Is Dredd looking for some sort of after processing hook, because I see the server receiving the request with dredd regardless of .net or node.js. Even if I spin up the .net api process in another process it doesn't work, why would that matter? It should just be http requests for dredd in any case right?
Does it care that it's coming back from a kestrel server?
kestrel response
Content-Type → application/json; charset=utf-8
Date → Wed, 06 Jan 2016 17:14:10 GMT
Server → Kestrel
Transfer-Encoding → chunked
Body {"foo":"bar"}
dredd.yml
dry-run: null
hookfiles: null
language: nodejs
sandbox: false
server: dnx web
server-wait: 3
init: false
names: false
only: []
output: []
header: []
sorted: false
user: null
inline-errors: false
details: false
method: []
color: true
level: verbose
timestamp: false
silent: false
path: []
blueprint: test.apib
endpoint: 'http://localhost:5000'
console output of dredd command
Starting server with command: dnx web
Waiting 3 seconds for server command to start...
Hosting environment: Production
Now listening on: http://localhost:5000
Application started. Press Ctrl+C to shut down.
info: Beginning Dredd testing...
info: Found Hookfiles: hooks.js
hook: before all
hook: before each
info: Microsoft.AspNet.Hosting.Internal.HostingEngine[1]
Request starting HTTP/1.1 GET http://localhost:5000/message
info: Microsoft.AspNet.Mvc.Controllers.ControllerActionInvoker[1]
Executing action method Foo.Controllers.HelloController.Get with arguments () - ModelState is Valid'
info: Microsoft.AspNet.Mvc.Infrastructure.ObjectResultExecutor[1]
Executing ObjectResult, writing value Microsoft.AspNet.Mvc.ActionContext.
info: Microsoft.AspNet.Mvc.Infrastructure.MvcRouteHandler[2]
Executed action Foo.Controllers.HelloController.Get in 0.0165ms
info: Microsoft.AspNet.Hosting.Internal.HostingEngine[2]
Request finished in 0.041ms 200 application/json; charset=utf-8
test.apib
# GET /message
+ Response 200 (application/json; charset=utf-8)
{"foo":"bar"}
It is correctly passing on my machine (MacOSX with coreclr mono rc2)
The same goes with the nodejs version you provided in the repo.
The only thing I had to change in my application was the startup command:
dnx run --server Microsoft.AspNet.Server.Kestrel
but I do not really think that is the problem here.

Eureka based configuration server breaking actuator endpoints

I setup my services to use the spring cloud eureka based config server.
version info: spring cloud 1.0.1.RELEASE
When I set it up as a fixed endpoint, I can see that it gets the right configuration file and that I can access actuator endpoints like health, info etc. so a .../manage/info returns the correct information.
However when I set it up to use discovery, the same actuator endpoints timeout on trying got access them.
In each case the configuration file is retrieved and downloaded (included log file).
Is there an issue with how I setup config server and bookmark service (the service which uses the config server)?
My configuration server setting is as follows:
server:
port: 8888
contextPath: /configurationservice
eureka:
client:
registerWithEureka: true
fetchRegistry: true
serviceUrl:
defaultZone: http://localhost:8761/eureka/
instance:
leaseRenewalIntervalInSeconds: 10
statusPageUrlPath: /configurationservice/info
homePageUrlPath: /configurationservice/
healthCheckUrlPath: /configurationservice/health
preferIpAddress: true
spring:
cloud:
config:
server:
native:
searchLocations: file:/Users/larrymitchell/libertas/configserver/configfiles
The service bootstrap.yml settings are:
spring:
profiles:
default: development
active: development
application:
name: bookmarkservice
cloud:
config:
enabled: true # note this needs to be turned on if you wnat the config server to work
# uri: http://localhost:8888/configurationservice
label: 1.0.0
discovery:
enabled: true
serviceId: configurationservice
The application.yml settings are:
# general spring settings
spring:
application:
name: bookmarkservice
profiles:
default: development
active: development
# name of the service
service:
name: bookmarkservice
# embedded web server settings
# some of these are specific to tomcat
server:
port: 9001
# the context path is the part after http:/localhost:8080
contextPath: /bookmarkservice
tomcat:
basedir: target/tomcat
uri-encoding: UTF-8
management:
context-path: /manage
security:
enabled: false
eureka:
client:
registerWithEureka: true
fetchRegistry: true
serviceUrl:
defaultZone: http://localhost:8761/eureka/
instance:
statusPageUrlPath: /bookmarkservice/manage/info
homePageUrlPath: /bookmarkservice/manage
healthCheckUrlPath: /bookmarkservice/manage/health
preferIpAddress: true
The startup log for bookmark service is as follows:
2015-06-24 17:52:49.806 DEBUG 11234 --- [ main] o.s.web.client.RestTemplate : Created GET request for "http://10.132.1.56:8888/configurationservice/bookmarkservice/development/1.0.0"
2015-06-24 17:52:49.890 DEBUG 11234 --- [ main] o.s.web.client.RestTemplate : Setting request Accept header to [application/json, application/*+json]
2015-06-24 17:52:50.439 DEBUG 11234 --- [ main] o.s.web.client.RestTemplate : GET request for "http://10.132.1.56:8888/configurationservice/bookmarkservice/development/1.0.0" resulted in 200 (OK)
2015-06-24 17:52:50.441 DEBUG 11234 --- [ main] o.s.web.client.RestTemplate : Reading [class org.springframework.cloud.config.environment.Environment] as "application/json;charset=UTF-8" using [org.springframework.http.converter.json.MappingJackson2HttpMessageConverter#2b07e607]
2015-06-24 17:52:50.466 INFO 11234 --- [ main] b.c.PropertySourceBootstrapConfiguration : Located property source: CompositePropertySource [name='configService', propertySources=[MapPropertySource [name='file:/Users/larrymitchell/libertas/configserver/configfiles/1.0.0/bookmarkservice-development.yml']]]
2015-06-24 17:52:50.503 INFO 11234 --- [ main] ationConfigEmbeddedWebApplicationContext : Refreshing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext#5fa23965: startup date [Wed Jun 24 17:52:50 EDT 2015]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext#5cced717
2015-06-24 17:52:51.723 WARN 11234 --- [ main] .i.s.PathMatchingResourcePatternResolver : Skipping [/var/folders/kq/ykvl3t4n3l71p7s9ymywb4ym0000gn/T/spring-boot-libs/06f98804e83cf4a94380b46591b976b1d17c36b8-eureka-client-1.1.147.jar] because it does not denote a directory
2015-06-24 17:52:53.662 INFO 11234 --- [ main] o.s.b.f.config.PropertiesFactoryBean : Loading properties file from URL [jar:file:/Users/larrymitchell/libertas/vipaas/applicationservices/bookmarkservice/target/bookmarkservice.jar!/lib/spring-integration-core-4.1.2.RELEASE.jar!/META-INF/spring.integration.default.properties]
Ok, after talking it over with another coworker I figured out what the actual issue is.
Part of the confusion is that I am using the spring cloud (https://github.com/VanRoy/spring-cloud-dashboard) which is a great front end by the way. So when the service starts we see it for to discovery and retrieve the correct configuration file and load it. After I go to the spring cloud console and see a setting of UP which means that it is discovered and registered though discovery. There is a second status indicator which is when the spring cloud dashboard takes the registered endpoint and get health. In my issue the endpoint was showing up as UNKNOWN.
If I then use the endpoint that shows up in the console and try the info actuator endpoint then the request times out. This was the nature of my problem
Ok, so what was the issue?
Basically since I defined the in the application.yml and since when the service registers in bootstrap it does not know the port yet and then it picks a default of 8080 (my assumption since that is what it does). The server port is set in application.yml to 9001 but discovery sees a registration of 8080 so the spring cloud console cannot access the localhost:8080/bookmarkservice/manage/health since there is no service at that endpoint (which is at 9001 actually). Other services also cannot find the service.
By moving the server.port to bootstrap.yml then the correct endpoint of rate service is registered and the service properly accessible.

Jasig CAS - 404 code after successful service ticket validation

We are currently trying to deploy CAS 4.0.1 on a JBoss EAP 6.3.0 server.
The login webflow was customised in order to redirect to a specific login form depending on the service calling CAS for authentication. Depending on these forms, we use specific authentication handlers, and a specific Credential model. Besides that, the configuration is rather standard.
At the moment, we are experiencing the following issue: when a user attempts to access a service secured by CAS, he is correctly redirected to the portal, and the expected login view is rendered ; upon successful login, the Service Ticket is delivered to the authentication filter on the service side (standard j_spring_cas_security_check), which then validates it successfully against CAS' ticket registry. We see in the logs that CAS is rendering the cas2ServiceSuccessView ; however, instead of delivering the expected XML response, the user is redirected to the login form.
We then confirmed that we were in fact getting a 404 error after the cas2ServiceSuccessView... Any idea what could trigger such behaviour/what we could have done wrong ?
Note that we are getting the same error regardless of how we call CAS for the ST validation: whether it is manually through /serviceValidate?ticket=ST-YYY&service=XXX, or via the /j_spring_cas_security_check on the service side...
Edit: we have the same behaviour running CAS on Tomcat 7.
Thanks in advance.
Below the debug logs that we are getting:
08:54:10,806 DEBUG [org.springframework.web.servlet.DispatcherServlet] (http-/0.0.0.0:8080-7) Last-Modified value for [/cas/serviceValidate] is: -1
08:54:10,809 INFO [org.perf4j.TimingLogger] (http-/0.0.0.0:8080-7) start[1433314450807] time[2] tag[VALIDATE_SERVICE_TICKET]
08:54:10,810 INFO [com.github.inspektr.audit.support.Slf4jLoggingAuditTrailManager] (http-/0.0.0.0:8080-7) Audit trail record BEGIN
=============================================================
WHO: audit:unknown
WHAT: ST-3-uecoOwdbdIn4bc2WvXfe-cas-test
ACTION: SERVICE_TICKET_VALIDATED
APPLICATION: CAS
WHEN: Wed Jun 03 08:54:10 CEST 2015
CLIENT IP ADDRESS: 127.0.0.1
SERVER IP ADDRESS: 127.0.0.1
=============================================================
08:54:10,810 DEBUG [org.springframework.validation.DataBinder] (http-/0.0.0.0:8080-7) DataBinder requires binding of required fields [renew]
08:54:10,811 DEBUG [org.springframework.web.servlet.DispatcherServlet] (http-/0.0.0.0:8080-7) Rendering view [org.springframework.web.servlet.view.InternalResourceView: name 'cas2ServiceSuccessView'; URL [/WEB-INF/view/jsp/cas2ServiceSuccessView.jsp]] in DispatcherServlet with name 'cas'
08:54:10,811 DEBUG [org.springframework.web.servlet.view.InternalResourceView] (http-/0.0.0.0:8080-7) Added model object 'assertion' of type [org.jasig.cas.validation.ImmutableAssertion] to request in view with name 'cas2ServiceSuccessView'
08:54:10,811 DEBUG [org.springframework.web.servlet.view.InternalResourceView] (http-/0.0.0.0:8080-7) Removed model object 'pgtIou' from request in view with name 'cas2ServiceSuccessView'
08:54:10,811 DEBUG [org.springframework.web.servlet.view.InternalResourceView] (http-/0.0.0.0:8080-7) Forwarding to resource [/WEB-INF/view/jsp/cas2ServiceSuccessView.jsp] in InternalResourceView 'cas2ServiceSuccessView'
08:54:10,812 DEBUG [org.springframework.web.servlet.DispatcherServlet] (http-/0.0.0.0:8080-7) Successfully completed request
08:54:10,814 DEBUG [org.springframework.web.servlet.DispatcherServlet] (http-/0.0.0.0:8080-7) DispatcherServlet with name 'cas' processing GET request for [/cas/login]
08:54:10,814 DEBUG [org.springframework.webflow.mvc.servlet.FlowHandlerMapping] (http-/0.0.0.0:8080-7) Mapping request with URI '/cas/login' to flow with id 'login'
In SpringSecurity 4.x, CasAuthenticationFilter's defaultFilterProcessesUrl path is changed.
So Change '/j_spring_cas_security_check' to '/login/cas' in Configuration.
... and of course, the cause was rather silly: somehow (I have to look at our merge/git history), the viewResolver bean defined in cas-servlet.xml did not have a basenames property set.