Why Is My Spring Cloud Function Attempting to Open Local HTTP Connections? - spring-cloud

I'm deploying a rather simple Spring Cloud Function to AWS Lambda and am running into an issue with slow cold starts and occasional failures noted in calling the function once deployed.
First, here is my single class. (Eventually this function will do some domain record lookups against a database, so the name 'domain' is used here fairly liberally. I've also removed any of the actual data handling and am just returning strings.
<< imports >>
#SpringBootConfiguration
public class DomainApplication implements ApplicationContextInitializer<GenericApplicationContext> {
private static Log logger = LogFactory.getLog(DomainApplication.class);
public static void main(String[] args) throws Exception {
FunctionalSpringApplication.run(DomainApplication.class, args);
}
public Supplier<String> domains(){
return () -> {
logger.info("Return a List of Domains");
return "All Domains";
};
}
public Function<String, String> domain() {
return value -> {
logger.info("Return A Single Domains");
return "This Domain" + value;
};
}
#Override
public void initialize(GenericApplicationContext context) {
context.registerBean("domain", FunctionRegistration.class,
() -> new FunctionRegistration<Function<String, String>>(domain())
.type(FunctionType.from(String.class).to(String.class).getType()));
context.registerBean("domains", FunctionRegistration.class,
() -> new FunctionRegistration<Supplier<String>>(domains())
.type(FunctionType.from(String.class).to(String.class).getType()));
}
}
Here's the dependencies of the project:
...
set('springCloudVersion', '2.1.0.RELEASE')
...
implementation "org.springframework.cloud:spring-cloud-function-context:${springCloudVersion}"
implementation "org.springframework.cloud:spring-cloud-starter-function-webflux:${springCloudVersion}"
implementation "org.springframework.cloud:spring-cloud-function-adapter-aws:${springCloudVersion}"
implementation 'com.amazonaws:aws-lambda-java-core:1.2.0'
implementation 'com.amazonaws:aws-lambda-java-events:2.2.6'
testCompile("org.springframework.boot:spring-boot-starter-test:${springCloudVersion}")
Now, when I package and deploy a 'shadowJar' version of the app to AWS Lambda the startup logs show a connection refused failure:
2019-05-14 20:45:21.205 ERROR 1 --- [or-http-epoll-3] reactor.Flux.MonoRepeatPredicate.1 : onError(io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: localhost/127.0.0.1:80)
... is there a reason why the startup would be attempting to connect locally to port 80? (And as importantly - can I shut that off?)

Facing the same issue, already reported to spring cloud team with
https://github.com/spring-cloud/spring-cloud-function/issues/367

Related

Implement Retry mechanism using Mono.retry() within Spring WebFlux Reactive code

Am using Java 8 and Spring WebFlux to call an external REST based server which also has a backup server.
Currently, if there's a failure when calling the server, it hits the backup server with no retry mechanism:
public class ServiceImpl {
#Autowired
MyRestClient myRestClient;
#Autowired
MyRestClient myRestClientBackup;
public Mono<ResponseOutput> getResponseOutput(ResponseInput responseInput) {
return Mono.just(responseInput)
.flatMap(input -> {
Mono<ResponseOutput> mono = myRestClient.post(input)
.doOnSuccess(responseOutput -> {
log.info("Successfully got responseOutput={}", responseOutput);
});
return mono;
})
.onErrorResume(e -> {
log.warn("Call to server failed, falling back to backup server...");
Mono<ResponseOutput> mono = myRestClientBackup.post(responseInput)
.doOnSuccess(responseOutput ->
log.info("Successfully got backup responseOutput={}", responseOutput));
return mono;
});
}
}
Am trying to implement a retry mechanism where upon setting a numRetries property in application.yml set to a specific number, this should happen:
e.g. if the numRetries = 2
Use MyRestClient to hit the original server twice (since numRetries = 2) and if that fails to hit the backup server.
This has the numRetries configuration property set to a static value:
application.yml:
client:
numRetries: 2;
Class that loads config properties from application.yml file:
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Configuration;
#Configuration
public ClientConfig {
#Value("${client.numRetries:0}")
private int numRetries;
// Getters and setters omitted for brevity
}
ClientConfig is used by MyRestClient:
public class MyRestClient {
#Autowired
ClientConfig config;
// Getters and setters omitted for brevity
}
Upon obtaining the value for the numRetries from MyRestClient how can I change the original implementation inside ServiceImpl.getResponseOutput() to use numRetries to hit the original server before calling the backup server? As you can see that this is all originally done using Java 8 lambda and streams...
Found this annotation #Retryable(value = RestClientException.class) but don't know how to specify the else if numRetries is used to call the backup server (pseudocode)?
Is there a lambda function to denote the numRetries to use within the:
return Mono.just(responseInput)
.flatMap(input -> {
How to do use the Mono.retry() operator?
Also, the version of reactor in this code seems to have deprecated the Mono.retry().retryBackOff() method?
How could I use the value referencing numRetries (which would be an int) to dictate to keep retrying (upon failure) and then when numRetries is exceeded up .onErrorResume() to call the other backup server?
Use retry(int) operator from Mono as below:
Mono.just(responseInput)
.flatMap(input -> myRestClient.post(input)
.retry(getNumRetries()) // this is the important part
.doOnSuccess(responseOutput -> log.info("Successfully got responseOutput={}", responseOutput)))
.onErrorResume(e -> {
log.warn("Call to server failed, falling back to backup server...");
return myRestClientBackup.post(responseInput)
.doOnSuccess(responseOutput -> log.info("Successfully got backup responseOutput={}", responseOutput));
});

Add a self hosted SignalR server to a .Net Core Worker Service

I'm trying to extend a .NET Core Worker Service (<Project Sdk="Microsoft.NET.Sdk.Worker">) with SignalR (self hosted web app).
All the examples/tutorials/docs I have found are based on web applications, so they don't fit my case.
This is what I've done until now:
MyService Program.cs:
public class Program
{
public static void Main(string[] args)
{
CreateHostBuilder(args).Build().Run();
}
public static IHostBuilder CreateHostBuilder(string[] args) =>
Host.CreateDefaultBuilder(args)
.UseWindowsService()
.ConfigureServices((hostContext, services) =>
{
services.AddHostedService<Worker>();
DependencyBuilder.Build(hostContext, services); // inject the stuff I need in my service
// create a SignalR Web host
SignalRWebHostCreator.CreateHost(services, "http://localhost:8090", (endpoints) => {
endpoints.MapHub<MyHub>("/result");
});
});
}
and the class I want to use to "extend" the servie with a SignalR server application.
public class SignalRWebHostCreator
{
public static void CreateHost(IServiceCollection services, string serviceUrl, Action<IEndpointRouteBuilder> mapHubs)
{
services.AddSignalR(); // is it ok here ?
WebHost.CreateDefaultBuilder()
.UseUrls(serviceUrl)
.Configure((IApplicationBuilder app) => {
app.UseRouting();
app.Map("/check", config => { // just a test: it works!
config.Run(async context =>
{
context.Response.ContentType = "text/plain";
byte[] data = System.Text.Encoding.UTF8.GetBytes("OK");
await context.Response.Body.WriteAsync(data, 0, data.Length);
await context.Response.Body.FlushAsync();
});
});
app.UseEndpoints(endpoints =>
{
//endpoints.MapHub<ClockHub>("/hubs/clock"); // ERROR
//endpoints.MapHub<PingHub>("/ping"); // ERROR
//mapHubs(endpoints); // ERROR
});
})
.Build().Run();
}
}
(ClockHub is taken from MS example and PingHub is another simple Hub I tried to use instead of my "injected" Hubs)
It starts the web application properly and it responds properly to the url http://localhost:8090/check.
When I uncomment the calls to enpoint.MapHub() or my cusom Actions I have this error:
System.InvalidOperationException: 'Unable to find the required services. Please add all the required services by calling 'IServiceCollection.AddSignalR' inside the call to 'ConfigureServices(...)' in the application startup code.'
2nd try:
Seems like service.AddSignalR() is not doing its job, so I added this in SignalRWebHostCreator:
.Configure((IApplicationBuilder app) => {
app.ApplicationServices = services.BuildServiceProvider();
and now I have this error:
System.InvalidOperationException: 'Unable to resolve service for type 'System.Diagnostics.DiagnosticListener' while attempting to activate 'Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware'.'
that at least has a callstack:
_This exception was originally thrown at this call stack:
Microsoft.Extensions.Internal.ActivatorUtilities.ConstructorMatcher.CreateInstance(System.IServiceProvider)
Microsoft.Extensions.Internal.ActivatorUtilities.CreateInstance(System.IServiceProvider, System.Type, object[])
Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.UseMiddleware.AnonymousMethod__0(Microsoft.AspNetCore.Http.RequestDelegate)
Microsoft.AspNetCore.Builder.ApplicationBuilder.Build()
Microsoft.AspNetCore.Hosting.WebHost.BuildApplication()
Microsoft.AspNetCore.Hosting.WebHost.StartAsync(System.Threading.CancellationToken)
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
System.Runtime.CompilerServices.TaskAwaiter.GetResult()_
If I add services.AddSingleton(new System.Diagnostics.DiagnosticListener("diagnostic listener")); I can use endpoints.MapHub(..) without errors but now a call to http://8090/check returns a 500 internal error, so I don't think this is the right way to solve the issue.
I found some example using WebApp from Microsoft.Owin.hosting.
It requires Microsoft.Owin.4.1.0, Microsoft.Owin.Hosting.4.1.0 and Owin.1.0.0 and the last one require Net Framework 4.6.1, I don't want this.
I have included Microsoft.AspNetCore.Owin.3.1.2 (100% .NET Core) but that does not offer WebApp or something similar.
I started experiencing same error when I upgraded nuget EFCore package to new version.
I noticed, that in my bin directory, a System.Diagnostics.DiagnosticSource.dll appeared, while when I downgraded - it disappeared.
I suspect, that DiagnosticListener type from old assembly version is registered in DI container, while on activation newer version is expected. Or reverse - I didn't dig that deep.
My solution was to revert EFCore to match Product version of System.Diagnostics.DiagnosticSource.dll so it will not appear in bin folder.

Overridden RabbitSourceConfiguration (app starters) does not work with Spring Cloud Edgware

I'm testing an upgrade of my Spring Cloud DataFlow services from Spring Cloud Dalston.SR4/Spring Boot 1.5.9 to Spring Cloud Edgware/Spring Boot 1.5.9. Some of my services extend source (or sink) components from the app starters. I've found this does not work with Spring Cloud Edgware.
For example, I have overridden org.springframework.cloud.stream.app.rabbit.source.RabbitSourceConfiguration and bound my app to my overridden version. This has previously worked with Spring Cloud versions going back almost a year.
With Edgware, I get the following (whether the app is run standalone or within dataflow):
***************************
APPLICATION FAILED TO START
***************************
Description:
Field channels in org.springframework.cloud.stream.app.rabbit.source.RabbitSourceConfiguration required a bean of type 'org.springframework.cloud.stream.messaging.Source' that could not be found.
Action:
Consider defining a bean of type 'org.springframework.cloud.stream.messaging.Source' in your configuration.
I get the same behaviour with the 1.3.0.RELEASE and 1.2.0.RELEASE of spring-cloud-starter-stream-rabbit.
I override RabbitSourceConfiguration so I can set a header mapper on the AmqpInboundChannelAdapter, and also to perform a connectivity test prior to starting up the container.
My subclass is bound to the Spring Boot application with #EnableBinding(HeaderMapperRabbitSourceConfiguration.class). A cutdown version of my subclass is:
public class HeaderMapperRabbitSourceConfiguration extends RabbitSourceConfiguration {
public HeaderMapperRabbitSourceConfiguration(final MyHealthCheck healthCheck,
final MyAppConfig config) {
// ...
}
#Bean
#Override
public AmqpInboundChannelAdapter adapter() {
final AmqpInboundChannelAdapter adapter = super.adapter();
adapter.setHeaderMapper(new NotificationHeaderMapper(config));
return adapter;
}
#Bean
#Override
public SimpleMessageListenerContainer container() {
if (config.performConnectivityCheckOnStartup()) {
if (LOGGER.isInfoEnabled()) {
LOGGER.info("Attempting connectivity with ...");
}
final Health health = healthCheck.health();
if (health.getStatus() == Status.DOWN) {
LOGGER.error("Unable to connect .....");
throw new UnableToLoginException("Unable to connect ...");
} else if (LOGGER.isInfoEnabled()) {
LOGGER.info("Connectivity established with ...");
}
}
return super.container();
}
}
You really should never do stuff like healthCheck.health(); within a #Bean definition. The application context is not yet fully baked or started; it may, or may not, work depending on the order that beans are created.
If you want to prevent the app from starting, add a bean that implements SmartLifecycle, put the bean in a late phase (high value) so it's started after everything else. Then put your code in start(). autStartup must be true.
In this case, it's being run before the stream infrastructure has created the channel.
Some ordering might have changed from the earlier release but, in any case, performing activity like this in a #Bean definition is dangerous.
You just happened to be lucky before.
EDIT
I just noticed your #EnableBinding is wrong; it should be Source.class. I can't see how that would ever have worked - that's what creates the bean for the channels field of type Source.
This works fine for me after updating stream and the binder to 1.3.0.RELEASE...
#Configuration
public class MySource extends RabbitSourceConfiguration {
#Bean
#Override
public AmqpInboundChannelAdapter adapter() {
AmqpInboundChannelAdapter adapter = super.adapter();
adapter.setHeaderMapper(new MyMapper());
return adapter;
}
}
and
#SpringBootApplication
#EnableBinding(Source.class)
public class DemoApplication {
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
}
If that doesn't work, please edit the question to show your POM.

How to use regexmapper based routing in Zuul? PatternServiceRouteMapper not working?

What I try to achieve is a routing for example:
http://zuul-host:8080/v1/foo/hello to my service foo-v1, resource hello
I'm trying out the regexmapper example described at http://cloud.spring.io/spring-cloud-netflix/spring-cloud-netflix.html
My problem is that I see that a service called foo-v1 gets mapped to /v1/foo in the PatternServiceRouteMapper but then I'm not able to call that route. It's also no visible at /mappings. Do I have to activate that route somewhere?
Setup
Foo Service
application.properties
server.port=9092
spring.application.name=foo-v1
eureka.client.serviceUrl.defaultZone=http://localhost:8761/eureka/
eureka.instance.healthcheck.enable=true
Zuul
My configuration class Routings.java. I added some sysout log output for the service mapping and I get foo-v1 -> v1/foo in the log. Therefore this mapping should be active.
#Configuration
public class Routings {
#Bean
public PatternServiceRouteMapper serviceRouteMapper() {
return new PatternServiceRouteMapper(
"(?<name>^.+)-(?<version>v.+$)",
"${version}/${name}") {
#Override
public String apply(final String serviceId) {
String route = super.apply(serviceId);
System.out.println(serviceId + " -> " +route);
return route;
}
};
}
}
My ZuulApplication.java
#SpringBootApplication
#EnableZuulProxy
#ComponentScan
public class ZuulApplication {
public static void main(String[] args) {
SpringApplication.run(ZuulApplication.class, args);
}
#RefreshScope
#ConfigurationProperties("zuul")
public ZuulProperties zuulProperties() {
return new ZuulProperties();
}
}
Ok, found the solution.
Remove ignoredServices: '*' from the zuul config.
This happens if you work through the examples. They start with explicitly configured routes and ignore dynamic routings. It's in the documentation but made no sense to me at that point :-)
To skip having a service automatically added, set zuul.ignored-services to a list of service id patterns.
When using the regexmapper we start using services that get added automatically and that's the feature we disabled with ignoredServices: '*'

Is it possible to read EF Code First Connection String from Azure Role Environment

I have an Azure Worker Role which is using Entity Framework Code First (5.0) to talk to a SQL Azure database. Currently I have the connection string in the app.config of the worker role however I would like to move the connection string into the Woker Role's Role Environment settings in order to make connection string changes easier for my live services colleagues without requiring redeployment of the Azure package.
Currently I am initializing the context in the form:
protected BaseContext()
: base("name=DataStore")
{
try
{
((IObjectContextAdapter)this).ObjectContext.Connection.Open();
var storeConnection = (SqlConnection)currentDbConn;
new SqlCommand("declare #i int", storeConnection).ExecuteNonQuery();
}
catch (Exception ex)
{
currentDbConn.Close();
Trace.TraceError("Error occured while getting connection to context", ex.Message);
throw;
}
}
I haven't been able to override DbContext(string nameOrConnectionString) to be able to pull the connection string from RoleEnvironment and I've also tried creating a new SqlConnection and assigning that the Database.Connection property but there's no setter :/.
Has anyone any ideas or guidance on how this could be achieved?
Thanks in advance for taking time to look at this question.
I would suggest either create a ContextFactory class or a static Factory method on your context like this
public class BaseContext : DbContext
{
public BaseContext(string connectionString)
: base(connectionString)
{
}
public static BaseContext Create()
{
return new BaseContext(
RoleEnvironment.GetConfigurationSettingValue("connectionString"));
}
}