Container managed MongoDB Connection in Liberty + Spring Data - mongodb

We have developed an application in Spring Boot + spring data (backend) + MongoDB and used IBM Websphere Liberty as application Server. We were used "Application Managed DB Connection" in an yml file and enjoyed the benefit of Spring Boot autoconfiguration.
Due to policy changes, we would need to manage our DB Connection in Liberty Server(using mongo feature), in Server.xml. I spent whole day in finding out an good example to do this, but dont find any example in Spring with "Container Managed MongoDB Connection" in IBM Websphere Liberty Server.
Can someone please support here?

Check out this other stackoverflow solution. The following is an extension of how you would use that in your Spring Boot app.
You should be able to inject your datasource the same way. You could even inject it into your configuration and wrap it in a Spring DelegatingDataSource.
#Configuration
public class DataSourceConfiguration {
// This is the last code section from that link above
#Resource(lookup = "jdbc/oracle")
DataSource ds;
#Bean
public DataSource mySpringManagedDS() {
return new DelegatingDataSource(ds);
}
}
Then you should be able to inject the mySpringManagedDS DataSource into your Component, Service, etc.

In the past Liberty had a dedicated mongodb-2.0 feature for the server.xml, however this feature provided pretty minimal benefit, since you still needed to bring your own MongoDB libraries. Also, over time MongoDB made significant breaking changes to their API, including how MongoDB gets configured.
Since the MongoDB API is changing so drastically between releases, we found it better to not provide any new MongoDB features in Liberty and instead suggest that users simply use a CDI producer like this:
CDI producer (holds any configuration too):
#ApplicationScoped
public class MongoProducer {
#Produces
public MongoClient createMongo() {
return new MongoClient(new ServerAddress(), new MongoClientOptions.Builder().build());
}
#Produces
public MongoDatabase createDB(MongoClient client) {
return client.getDatabase("testdb");
}
public void close(#Disposes MongoClient toClose) {
toClose.close();
}
}
Example usage:
#Inject
MongoDatabase db;
#POST
#Path("/add")
#Consumes(MediaType.APPLICATION_JSON)
public void add(CrewMember crewMember) {
MongoCollection<Document> crew = db.getCollection("Crew");
Document newCrewMember = new Document();
newCrewMember.put("Name",crewMember.getName());
newCrewMember.put("Rank",crewMember.getRank());
newCrewMember.put("CrewID",crewMember.getCrewID());
crew.insertOne(newCrewMember);
}
This is just the basics, but the following blog post goes into much greater detail along with code examples:
https://openliberty.io/blog/2019/02/19/mongodb-with-open-liberty.html

Related

Spring Boot Micrometer metrics for MongoDB

I use spring boot 2.2.5 + micrometer 1.3.5 + starter-data-mongodb
Under "io.micrometer.core.instrument.binder.mongodb" I can see 2 classes CommandListener and ConnectionPoolListener. I would like to know what purpose these serve?
In actuator metrics endpoint, mongo metrics are not available.
How do I enable metrics for mongodb in actuator? For example, actuator automatically shows several metrics of RabbitMQ. I was expecting something similar in case of MongoDB as well.
Should I create my own metrics?
In order to enable Spring Boot applying its AutoConfiguration I suggest to use the customizer pattern:
Kotlin:
#Configuration
class MongoConfiguration {
#Bean
fun mongoClientSettingsBuilderCustomizer(meterRegistry: MeterRegistry) =
MongoClientSettingsBuilderCustomizer {
it.addCommandListener(MongoMetricCommandListener(meterRegistry))}
}
Java:
#Configuration
public class MongoConfiguration {
#Bean
public MongoClientSettingsBuilderCustomizer mongoClientSettingsBuilderCustomizer(MeterRegistry meterRegistry) {
return builder -> builder.addCommandListener(new MongoMetricsCommandListener(meterRegistry));
}
}
Please note you currenly will neither see a relation to the spring data repository nor to the mongo collection in the metrics. see open issue
EDIT (07/30/2021):
The issues has been fixed, so you likely get collection metrics in a current release.
Adding those listeners is not as straighforward as i thought and it's quite dependend on which properties you use for configuring Spring Data MongoDB.
The key for the integration is to customize the com.mongodb.MongoClientSettings instance which is used for creating the MongoClient. There are multiple possiblities to do so as documented in Connecting to MongoDB with Spring
The following is a working example (simplified from an application of ours) based on Spring Boot 2.3 which assumes you are using spring.data.mongodb.uri to specify the connection string in your application.properties.
package com.example.demo;
import com.mongodb.ConnectionString;
import com.mongodb.MongoClientSettings;
import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.binder.mongodb.MongoMetricsCommandListener;
import io.micrometer.core.instrument.binder.mongodb.MongoMetricsConnectionPoolListener;
import org.springframework.boot.autoconfigure.mongo.MongoProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.mongodb.core.MongoClientFactoryBean;
#Configuration
public class MongoConfiguration {
#Bean
public MongoClientFactoryBean mongoClientFactoryBean(MongoProperties properties, MeterRegistry meterRegistry) {
MongoClientFactoryBean mongoClientFactoryBean = new MongoClientFactoryBean();
mongoClientFactoryBean.setConnectionString(new ConnectionString(properties.getUri()));
MongoClientSettings settings = MongoClientSettings.builder()
.addCommandListener(new MongoMetricsCommandListener(meterRegistry))
.applyToConnectionPoolSettings(builder ->
builder.addConnectionPoolListener(new MongoMetricsConnectionPoolListener(meterRegistry)))
.build();
mongoClientFactoryBean.setMongoClientSettings(settings);
return mongoClientFactoryBean;
}
}
Unfortunately a lot has in configuring MongoDB from Spring Boot 2.2 to 2.3. If you cannot use Spring Boot 2.3 and you are stuck in backporting this to 2.2 please let me know.
Short answer
Create a MongoClientOptions bean with addCommandListener and you are good to go.
#Configuration
public class MongoConfiguration {
#Autowired
private MeterRegistry meterRegistry;
#Bean
public MongoClientOptions myMongoClientOptions() {
return MongoClientOptions.builder()
.addCommandListener(new MongoMetricsCommandListener(meterRegistry)).build();
}
}
#chargue's answer won't work for certain version of spring-data-mongodb. Because org.springframework.boot.autoconfigure.mongo.MongoAutoConfiguration initializes MongoClient with MongoClientOptions, not MongoClientSettings. It might work in newer version of spring-data-mongodb cuz MongoClientSettings is the recommended way according to mongodb's documents.
#Bean
#ConditionalOnMissingBean(type = { "com.mongodb.MongoClient", "com.mongodb.client.MongoClient" })
public MongoClient mongo(MongoProperties properties, ObjectProvider<MongoClientOptions> options,
Environment environment) {
return new MongoClientFactory(properties, environment).createMongoClient(options.getIfAvailable());
}
Make sure prometheus and micrometer is correctlly setup. You should see mongo metrics in prometheus endpoint like below:
# HELP process_cpu_usage The "recent cpu usage" for the Java Virtual Machine process
# TYPE process_cpu_usage gauge
process_cpu_usage{application="",} 0.004362672325272289
# HELP mongodb_driver_commands_seconds_max Timer of mongodb commands
# TYPE mongodb_driver_commands_seconds_max gauge
mongodb_driver_commands_seconds_max{application="",cluster_id="60b0d12d73b6df671cb4d882",command="find",server_address="",status="SUCCESS",} 34.684200332
mongodb_driver_commands_seconds_max{application="",cluster_id="60b0d12d73b6df671cb4d882",command="buildInfo",server_address="",status="SUCCESS",} 0.263514375
# HELP mongodb_driver_commands_seconds Timer of mongodb commands
# TYPE mongodb_driver_commands_seconds summary
mongodb_driver_commands_seconds_count{application="",cluster_id="60b0d12d73b6df671cb4d882",command="find",server_address="",status="SUCCESS",} 1.0
mongodb_driver_commands_seconds_sum{application="",cluster_id="60b0d12d73b6df671cb4d882",command="find",server_address="",status="SUCCESS",} 34.684200332
mongodb_driver_commands_seconds_count{application="",cluster_id="60b0d12d73b6df671cb4d882",command="buildInfo",server_address="",status="SUCCESS",} 1.0
mongodb_driver_commands_seconds_sum{application="",cluster_id="60b0d12d73b6df671cb4d882",command="buildInfo",server_address="",status="SUCCESS",} 0.263514375

Discovery / Registration only works when K8s pod already registered.

Thanks for Spring Boot Admin!
I am using it with Spring Cloud Kubernetes, our k8s pods only get discovered when we start the Spring Boot Admin, after the service pods have been started.
It seems looking at InstanceDiscoveryListener, the discovery of clients,
will happen based on events. Like ApplicationReadyEvent (When starting) and i.e. InstanceRegisteredEvent.
Is it correct to say, that Spring Boot Admin, will not try to discover periodically? If so how do I make sure an event is fired from the application to Spring boot admin picks it up and registers the instance?
Especially to make sure, that instanced are registered when they are started after spring boot admin, was started. (The order in which k8s pods are started is arbitrary/hard to control, something in general we don't want to do).
Thank You!
Christophew
Version:
springBootAdminVersion = '2.0.1'
springCloudVersion = 'Finchley.RELEASE'
springCloudK8s = '0.3.0.RELEASE'
Not sure if this is the best way to solve it but seems to work:
class TimedInstanceDiscoveryListener extends InstanceDiscoveryListener {
private static final Logger log = LoggerFactory.getLogger(TimedInstanceDiscoveryListener.class);
public TimedInstanceDiscoveryListener(DiscoveryClient discoveryClient, InstanceRegistry registry, InstanceRepository repository) {
super(discoveryClient, registry, repository);
log.info("Starting custom TimedInstanceDiscoveryListener");
}
#Scheduled(fixedRate = 5000)
public void periodicDiscovery() {
log.info("Discovering new pod / services");
super.discover();
}
}
#Bean
#ConfigurationProperties(prefix = "spring.boot.admin.discovery")
public InstanceDiscoveryListener instanceDiscoveryListener(ServiceInstanceConverter serviceInstanceConverter,
DiscoveryClient discoveryClient,
InstanceRegistry registry,
InstanceRepository repository) {
InstanceDiscoveryListener listener = new TimedInstanceDiscoveryListener(discoveryClient, registry, repository);
listener.setConverter(serviceInstanceConverter);
return listener;
}

Unable to download embedded MongoDB, behind proxy, using automatic configuration script

I have a Spring Boot project, built using Maven, where I intend to use embedded mongo db. I am using Eclipse on Windows 7.
I am behind a proxy that uses automatic configuration script, as I have observed in the Connection tab of Internet Options.
I am getting the following exception when I try to run the application.
java.io.IOException: Could not open inputStream for https://downloads.mongodb.org/win32/mongodb-win32-i386-3.2.2.zip
at de.flapdoodle.embed.process.store.Downloader.downloadInputStream(Downloader.java:131) ~[de.flapdoodle.embed.process-2.0.1.jar:na]
at de.flapdoodle.embed.process.store.Downloader.download(Downloader.java:69) ~[de.flapdoodle.embed.process-2.0.1.jar:na]
....
MongoDB gets downloaded just fine, when I hit the following URL in my web browser:
https://downloads.mongodb.org/win32/mongodb-win32-i386-3.2.2.zip
This leads me to believe that probably I'm missing some configuration in my Eclipse or may be the maven project itself.
Please help me to find the right configuration.
What worked for me on a windows machine:
Download the zip file (https://downloads.mongodb.org/win32/mongodb-win32-i386-3.2.2.zip)
manually and put it (not unpack) into this folder:
C:\Users\<Username>\.embedmongo\win32\
Indeed the problem is about your proxy (a corporate one I guess).
If the proxy do not require authentication, you can solve your problem easily just by adding the appropriate -Dhttp.proxyHost=... and -Dhttp.proxyPort=... (or/and the same with "https.[...]") as JVM arguments in your eclipse junit Runner, as suggested here : https://github.com/learning-spring-boot/learning-spring-boot-2nd-edition-code/issues/2
One solution to your problem is to do the following.
Download MongoDB and place it on a ftp server which is inside your corporate network (for which you would not need proxy).
Then write a configuration in your project like this
#Bean
#ConditionalOnProperty("mongo.proxy")
public IRuntimeConfig embeddedMongoRuntimeConfig() {
final Command command = Command.MongoD;
final IRuntimeConfig runtimeConfig = new RuntimeConfigBuilder()
.defaults(command)
.artifactStore(new ExtractedArtifactStoreBuilder()
.defaults(command)
.download(new DownloadConfigBuilder()
.defaultsForCommand(command)
.downloadPath("your-ftp-path")
.build())
.build())
.build();
return runtimeConfig;
}
With the property mongo.proxy you can control whether Spring Boot downloads MongoDB from your ftp server or from outside. If it is set to true then it downloads from the ftp server. If not then it tries to download from the internet.
The easiest way seems to me to customize the default configuration:
#Bean
DownloadConfigBuilderCustomizer mongoProxyCustomizer() {
return configBuilder -> {
configBuilder.proxyFactory(new HttpProxyFactory(host, port));
};
}
Got the same issue (with Spring Boot 2.6.1 the spring.mongodb.embedded.version property is mandatory).
To configure the proxy, I've added the configuration bean by myself:
#Value("${spring.mongodb.embedded.proxy.domain}")
private String proxyDomain;
#Value("${spring.mongodb.embedded.proxy.port}")
private Integer proxyPort;
#Bean
RuntimeConfig embeddedMongoRuntimeConfig(ObjectProvider<DownloadConfigBuilderCustomizer> downloadConfigBuilderCustomizers) {
Logger logger = LoggerFactory.getLogger(this.getClass().getPackage().getName() + ".EmbeddedMongo");
ProcessOutput processOutput = new ProcessOutput(Processors.logTo(logger, Slf4jLevel.INFO), Processors.logTo(logger, Slf4jLevel.ERROR), Processors.named("[console>]", Processors.logTo(logger, Slf4jLevel.DEBUG)));
return Defaults.runtimeConfigFor(Command.MongoD, logger).processOutput(processOutput).artifactStore(this.getArtifactStore(logger, downloadConfigBuilderCustomizers.orderedStream())).isDaemonProcess(false).build();
}
private ExtractedArtifactStore getArtifactStore(Logger logger, Stream<DownloadConfigBuilderCustomizer> downloadConfigBuilderCustomizers) {
de.flapdoodle.embed.process.config.store.ImmutableDownloadConfig.Builder downloadConfigBuilder = Defaults.downloadConfigFor(Command.MongoD);
downloadConfigBuilder.progressListener(new Slf4jProgressListener(logger));
downloadConfigBuilderCustomizers.forEach((customizer) -> {
customizer.customize(downloadConfigBuilder);
});
DownloadConfig downloadConfig = downloadConfigBuilder
.proxyFactory(new HttpProxyFactory(proxyDomain, proxyPort)) // <--- HERE
.build();
return Defaults.extractedArtifactStoreFor(Command.MongoD).withDownloadConfig(downloadConfig);
}
In my case, I had to add the HTTPS corporate proxy to Intellij Run Configuration.
Https because it was trying to download:
https://downloads.mongodb.org/win32/mongodb-win32-x86_64-4.0.2.zip
application.properties:
spring.data.mongodb.database=test
spring.data.mongodb.port=27017
spring.mongodb.embedded.version=4.0.2
Please keep in mind this is a (DEV) setup.

annotation #RibbonClient not work together with RestTemplate

I am trying Ribbon configuration with RestTemplate based on bookmark service example but without luck, here is my code:
#SpringBootApplication
#RestController
#RibbonClient(name = "foo", configuration = SampleRibbonConfiguration.class)
public class BookmarkServiceApplication {
public static void main(String[] args) {
SpringApplication.run(BookmarkServiceApplication.class, args);
}
#Autowired
RestTemplate restTemplate;
#RequestMapping("/hello")
public String hello() {
String greeting = this.restTemplate.getForObject("http://foo/hello", String.class);
return String.format("%s, %s!", greeting);
}
}
with error page as below:
Whitelabel Error Page
This application has no explicit mapping for /error, so you are seeing this as a fallback.
Tue Mar 22 19:59:33 GMT+08:00 2016
There was an unexpected error (type=Internal Server Error, status=500).
No instances available for foo
but if I remove annotation #RibbonClient, everything will be just ok,
#RibbonClient(name = "foo", configuration = SampleRibbonConfiguration.class)
and here is SampleRibbonConfiguration implementation:
public class SampleRibbonConfiguration {
#Autowired
IClientConfig ribbonClientConfig;
#Bean
public IPing ribbonPing(IClientConfig config) {
return new PingUrl();
}
#Bean
public IRule ribbonRule(IClientConfig config) {
return new AvailabilityFilteringRule();
}
}
Is it because RibbonClient can not work with RestTemplate together?
and another question is that does Ribbon configuration like load balancing rule could be configured via application.yml configuration file?
as from Ribbon wiki, seems we can configure Ribbon parameters like NFLoadBalancerClassName, NFLoadBalancerRuleClassName etc in property file, does Spring Cloud also supports this?
I'm going to assume you're using Eureka for Service Discovery.
Your particular error:
No instances available for foo
can happen for a couple of reasons
1.) All services are down
All of the instances of your foo service could legitimately be DOWN.
Solution: Try visiting your Eureka Dashboard and ensure all the services are actually UP.
If you're running locally, the Eureka Dashboard is at http://localhost:8761/
2.) Waiting for heartbeats
When you very first register a service via Eureka, there's a period of time where the service is UP but not available. From the documentation
A service is not available for discovery by clients until the
instance, the server and the client all have the same metadata in
their local cache (so it could take 3 heartbeats)
Solution: Wait a good 30 seconds after starting your foo service before you try calling it via your client.
In your particular case I'm going to guess #2 is likely what's happening to you. You're probably starting the service and trying to call it immediately from the client.
When it doesn't work, you stop the client, make some changes and restart. By that time though, all of the heartbeats have completed and your service is now available.
For your second question. Look at the "Customizing the Ribbon Client using properties" section in the reference documentation. (link)

Has anyone successfully deployed a GWT app on Heroku?

Heroku recently began supporting Java apps. Looking through the docs, it seems to resemble the Java Servlet Standard. Does anyone know of an instance where a GWT app has been successfully deployed on Heroku? If so, are there any limitations?
Yes, I've got a successful deployment using the getting started with Java instructions here:
http://devcenter.heroku.com/articles/java
I use the Maven project with appassembler plugin approach but added gwt-maven-plugin to compile a GWT app during the build.
When you push to heroku you see the GWT compile process running, on one thread only so quite slow but it works fine.
The embedded Jetty instance is configured to serve up static resources at /static from src/main/resources/static and I copy the compiled GWT app to this location during the build and then reference the .nocache.js as normal.
What else do you want to know?
You've got a choice, either build the Javascript representation of your GWT app locally into your Maven project, commit it and the read it from your app, or to generate it inside Heroku via the gwt-maven-plugin as I mentioned.
The code to serve up files from a static location inside your jar via embedded Jetty is something like this inside a Guice ServletModule:
(See my other answer below for a simpler and less Guice-driven way to do this.)
protected void configureServlets() {
bind(DefaultServlet.class).in(Singleton.class);
Map<String, String> initParams = new HashMap<String, String>();
initParams.put("pathInfoOnly", "true");
initParams.put("resourceBase", staticResourceBase());
serve("/static/*").with(DefaultServlet.class, initParams);
}
private String staticResourceBase() {
try {
return WebServletModule.class.getResource("/static").toURI().toString();
}
catch (URISyntaxException e) {
e.printStackTrace();
return "couldn't resolve real path to static/";
}
}
There's a few other tricks to getting embedded Jetty working with guice-servlet, let me know if this isn't enough.
My first answer to this turned out to have problems when GWT tried to read its serialization policy. In the end I went for a simpler approach that was less Guice-based. I had to step through the Jetty code to understand why setBaseResource() was the way to go - it's not immediately obvious from the Javadoc.
Here's my server class - the one with the main() method that you point Heroku at via your app-assembler plugin as per the Heroku docs.
public class MyServer {
public static void main(String[] args) throws Exception {
if (args.length > 0) {
new MyServer().start(Integer.valueOf(args[0]));
}
else {
new MyServer().start(Integer.valueOf(System.getenv("PORT")));
}
}
public void start(int port) throws Exception {
Server server = new Server(port);
ServletContextHandler context = new ServletContextHandler(ServletContextHandler.SESSIONS);
context.setBaseResource(createResourceForStatics());
context.setContextPath("/");
context.addEventListener(new AppConfig());
context.addFilter(GuiceFilter.class, "/*", null);
context.addServlet(DefaultServlet.class, "/");
server.setHandler(context);
server.start();
server.join();
}
private Resource createResourceForStatics() throws MalformedURLException, IOException {
String staticDir = getClass().getClassLoader().getResource("static/").toExternalForm();
Resource staticResource = Resource.newResource(staticDir);
return staticResource;
}
}
AppConfig.java is a GuiceServletContextListener.
You then put your static resources under src/main/resources/static/.
In theory, one should be able to run GWT using the embedded versions of Jetty or Tomcat, and bootstrap the server in main as described in the Heroku Java docs.