[spring-data][r2dbc] How to connect to Oracle DB with r2dbc from Application.yml? - spring-data

[spring-data][r2dbc] How to connect to Oracle DB with r2dbc from Application.yml?
application.yml
spring:
r2dbc:
url: r2dbc:oracle:thin//{host}:{port}/{service-name}
username: {username}
password: {password}
This does not initialise the DB Config.
Overriding also does not work.
Config
#Configuration
#EnableR2dbcRepositories(basePackages = {package})
public class DbConfig extends AbstractR2dbcConfiguration
{
#Override
#Bean
public ConnectionFactory connectionFactory() {
return ConnectionFactories.get(ConnectionFactoryOptions.parse(
{url})
.mutate()
.option(ConnectionFactoryOptions.USER, {username})
.option(ConnectionFactoryOptions.PASSWORD, {password})
.build());
}
}
Added these dependencies in
build.gradle
implementation group: 'com.oracle.database.r2dbc', name: 'oracle-r2dbc', version: '0.4.0'
implementation group: 'org.springframework.data', name: 'spring-data-r2dbc', version: '1.4.2'
implementation group: 'io.r2dbc', name: 'r2dbc-spi', version: '0.9.1.RELEASE'
This too does not create a connection with R2dbc.
r2dbc-spi is added eventhough it is implicitly present in springframework.data which does not take the latest version which caused a property missing error.

I have not tested the following in Oracle DB, but on MySQL. Because both application configurations and related Java metadata are for general purpose, it should work for Oracle DB too.
Just like JDBC, R2DBC needs to have:
a property class
a connection class that loads properties.
#Configuration
#EnableR2dbcRepositories
public class R2dbcConfig extends AbstractR2dbcConfiguration {
private R2dbcProperties r2dbcProperties; // 1. a property class R2dbcProperties
public R2dbcConfig(R2dbcProperties rp) {
this.r2dbcProperties = rp;
}
#Override
#Bean
public ConnectionFactory connectionFactory() {
// a connection class that loads properties.
return ConnectionFactoryBuilder.of(r2dbcProperties, ()->EmbeddedDatabaseConnection.NONE).build();
}
// other configurations...
}
One thing to note, org.springframework.boot.autoconfigure.r2dbc.R2dbcProperties contains basic configurations. There are a lot more OPTIONs that a driver can support, for example, on r2dbc-mysql GH page, it has at least these OPTIONs.
ConnectionFactoryOptions options = ConnectionFactoryOptions.builder()
.option(DRIVER, "mysql")
.option(HOST, "127.0.0.1")
.option(USER, "root")
.option(PORT, 3306) // optional, default 3306
.option(PASSWORD, "database-password-in-here") // optional, default null, null means has no password
.option(DATABASE, "r2dbc") // optional, default null, null means not specifying the database
.option(CONNECT_TIMEOUT, Duration.ofSeconds(3)) // optional, default null, null means no timeout
.option(Option.valueOf("socketTimeout"), Duration.ofSeconds(4)) // optional, default null, null means no timeout
.option(SSL, true) // optional, default sslMode is "preferred", it will be ignore if sslMode is set
.option(Option.valueOf("sslMode"), "verify_identity") // optional, default "preferred"
.option(Option.valueOf("sslCa"), "/path/to/mysql/ca.pem") // required when sslMode is verify_ca or verify_identity, default null, null means has no server CA cert
.option(Option.valueOf("sslCert"), "/path/to/mysql/client-cert.pem") // optional, default null, null means has no client cert
.option(Option.valueOf("sslKey"), "/path/to/mysql/client-key.pem") // optional, default null, null means has no client key
.option(Option.valueOf("sslKeyPassword"), "key-pem-password-in-here") // optional, default null, null means has no password for client key (i.e. "sslKey")
.option(Option.valueOf("tlsVersion"), "TLSv1.3,TLSv1.2,TLSv1.1") // optional, default is auto-selected by the server
.option(Option.valueOf("sslHostnameVerifier"), "com.example.demo.MyVerifier") // optional, default is null, null means use standard verifier
.option(Option.valueOf("sslContextBuilderCustomizer"), "com.example.demo.MyCustomizer") // optional, default is no-op customizer
.option(Option.valueOf("zeroDate"), "use_null") // optional, default "use_null"
.option(Option.valueOf("useServerPrepareStatement"), true) // optional, default false
.option(Option.valueOf("tcpKeepAlive"), true) // optional, default false
.option(Option.valueOf("tcpNoDelay"), true) // optional, default false
.option(Option.valueOf("autodetectExtensions"), false) // optional, default false
.build();

Related

Transient annotation and error executing DDL

I use the annotation # trasient to declare a boolean variable but when I start my spring application, I got the error as below
Error executing DDL "
alter table categories
add column has_children boolean not null" via JDBC Statement
This indicates that the column of has_children should not be null but Im not sure how to set the default of boolean value to false. Any ideas ? Thanks
----Edit-----
Below is my entity code
#Transient
private boolean hasChildren;
public boolean isHasChildren() {
return hasChildren;
}
public void setHasChildren(boolean hasChildren) {
this.hasChildren = hasChildren;
}
You didn't provide the code showing the entity, but just a heads up.
The default value of a boolean (primitive) is false, but the default value of a Boolean (the wrapper class) is null.
So it's worth checking if your variable has_children is created using Boolean instead of boolean

Configuring spring-cloud loadbalancer without autoconfiguration

I've read whole documentation, tutorial [1], and spend several hours in sources, but I still do not understand how to configure loadbalancer, especially if I don't use magic annotations.
I have following configuration:
#Configuration
public class AppConfig {
public static final String SERVICE_ID = "service";
#Primary
#Bean
public ServiceInstanceListSupplier serviceInstanceListSupplier() {
return ServiceInstanceListSuppliers.from(SERVICE_ID,
new DefaultServiceInstance(SERVICE_ID + "1", SERVICE_ID, "localhost", 8886, false),
new DefaultServiceInstance(SERVICE_ID + "2", SERVICE_ID, "localhost", 8887, false));
}
#Bean
public LoadBalancerClientFactory loadBalancerClientFactory() {
return new LoadBalancerClientFactory();
}
#Bean
public ReactorLoadBalancerExchangeFilterFunction loadBalancerExchangeFilterFunction(LoadBalancerProperties properties) {
return new ReactorLoadBalancerExchangeFilterFunction(loadBalancerClientFactory(), properties);
}
}
and use bean loadBalancerExchangeFilterFunction as:
WebClient.builder()
.baseUrl("http://service/test-consumer")
.filter(lbFunction)
.build();
and it works. The problem is, that it works regardless of what hostname I use. So if I replace hostname "service" with whatever work I like, I will be still sending data to localhost:8886 or localhost:8887.
Can someone explain what is the role of serviceId and how this is paired to collection of DefaultServiceInstance?
(I want to understand the internals, what are the key components, their purpose and their interplay. I'm not primarily looking for magic annotation, but that one actually explained would be also great. Debugging it is really hard, I have several A4 with class diagrams and it still makes no sense at all).
Question: Is there a misconfiguration? What is the purpose of serviceId? It seems that none, as webclient using ReactorLoadBalancerExchangeFilterFunction will create roundrobin loadbalancer over configured ServiceInstances regardless of what is actual hostname used in given webclient.
Question2: how could I create 2 loadbalanced services and control to which service (not node) will request go? Do I need 2 separate webclients or some url pattern(like using serviceId in place of hostname) will do? If I need 2 webclients, how is pairing to DefaultServiceInstance done?
[1] https://spring.io/guides/gs/spring-cloud-loadbalancer/
EDIT:
after suggested update, the configuration looks like:
#Configuration
public class AppConfig {
#Bean
public ServiceInstanceListSupplier instanceSupplier(ConfigurableApplicationContext context) {
return ServiceInstanceListSupplier.builder()
.withDiscoveryClient()
.withHealthChecks()
.build(context);
}
#Bean
public LoadBalancerClientFactory loadBalancerClientFactory() {
return new LoadBalancerClientFactory();
}
#Bean
public ReactorLoadBalancerExchangeFilterFunction loadBalancerExchangeFilterFunction(LoadBalancerProperties properties) {
return new ReactorLoadBalancerExchangeFilterFunction(loadBalancerClientFactory(), properties);
}
}
application.properties contains:
spring.cloud.discovery.client.simple.instances.complicated[0].uri=http://localhost:8886
spring.cloud.discovery.client.simple.instances.complicated[1].uri=http://localhost:8887
webclient call to URL: http://localhost:8888/test-consumer (ie. hostname not matching serviceID) produces:
o.s.c.l.core.RoundRobinLoadBalancer : No servers available for service: localhost
eactorLoadBalancerExchangeFilterFunction : LoadBalancer does not contain an instance for the service localhost
webclient call to URL: http://complicated/test-consumer (ie. hostname matching serviceID) produces:
o.s.c.l.core.RoundRobinLoadBalancer : No servers available for service: complicated
eactorLoadBalancerExchangeFilterFunction : LoadBalancer does not contain an instance for the service complicated
The reason for this is that this.serviceId = environment.getProperty(PROPERTY_NAME); in DiscoveryClientServiceInstanceListSupplier(ReactiveDiscoveryClient,Environment) evaluates as null, thus even though I'm looking for some serviceId, delegate.getInstance is called with null, so no ServiceInstances are found. IF I removed #Bean instanceSupplier, and hope for autoconfiguration do it somehow magically, the this.serviceId = environment.getProperty(PROPERTY_NAME); is somehow magically set, serviceId is propagated correctly, and it works. For calls which leads elsewhere than configured serviceId, it fails saying, that this serviceId is not know, instead of making call.
SO it does mean, that if I configure loadbalancer, I cannot call anything else but (auto)configured services???
The LoadBalancer config should not be in a #Configuration-annotated class; instead, it should be a class passed for config via #LoadBalancerClient or #LoadBalancerClients annotation, as described here.
Also, the only bean you need to instantiate is the ServiceInstanceListSupplier (if you add spring-cloud-starter-loadbalancer, LoadBalancerClientFactory, and ReactorLoadBalancerExchangeFilterFunction will be instantiated by the starter).
So your LoadBalancer configuration class will look like so (without #Configuration):
public class AppConfig {
#Bean
public ServiceInstanceListSupplier instanceSupplier(ConfigurableApplicationContext context) {
return ServiceInstanceListSupplier.builder()
.withDiscoveryClient()
.withHealthChecks()
.build(context);
}
}
and your actual #Configuration class (for example, where you configure other webflux-related beans), will have the following annotation: #LoadBalancerClients(defaultConfiguration = AppConfig.class).
Then, if you enable health-checks in the complicated instances, it should work without any problems.
Finally, able to resolve this issue with the below Configuration. not sure why it works only with Non blocking approach and when we pass the new RestTemplate
#Bean
public ServiceInstanceListSupplier instanceSupplier(ConfigurableApplicationContext context) {
return ServiceInstanceListSupplier.builder()
.withDiscoveryClient()
.withBlockingHealthChecks(new RestTemplate())//this change
.build(context);
}

R2DBC and enum (PostgreSQL)

Update 15/08/2020: Looks like Enum support was added on Jun 16. R2DBC commit.
Does H2DBC support PostgreSQL enums? I checked they git page but it doesn't mention anything about it. If it does, how enums could be used (INSERT, SELECT)?
Lets say PostgreSQL enum
CREATE TYPE mood AS ENUM ('UNKNOWN', 'HAPPY', 'SAD', ...);
Java class
#Data
public class Person {
private String name;
private Mood mood;
// ...
enum Mood{ UNKNOWN, HAPPY, SAD, ...}
}
I tried:
// insert
var person = ...;
client.insert()
.table("people")
.using(person)
.then()
.subscribe(System.out::println);
// select
var query = "SELECT * FROM people";
client.execute(query)
.as(Person.class)
.fetch().all()
.subscribe(System.out::println);
But I'm getting error messages:
# on insert
WARN [reactor-tcp-epoll-1] (Loggers.java:294) - Error: SEVERITY_LOCALIZED=ERROR, SEVERITY_NON_LOCALIZED=ERROR, CODE=42804, MESSAGE=column "mood" is of type mood but expression is of type character varying, HINT=You will need to rewrite or cast the expression., POSITION=61, FILE=parse_target.c, LINE=591, ROUTINE=transformAssignedExpr
# on select
ERROR [reactor-tcp-epoll-1] (Loggers.java:319) - [id: 0x8581acdb, L:/127.0.0.1:39726 ! R:127.0.0.1/127.0.0.1:5432] Error was received while reading the incoming data. The connection will be closed.
reactor.core.Exceptions$ErrorCallbackNotImplemented: org.springframework.data.mapping.MappingException: Could not read property private ...
I found similar post but without luck to solve my problem.. maybe I was applying it wrong..
Any help or tips are welcome.
Tested with org.springframework.data:spring-data-r2dbc:1.0.0.RELEASE and io.r2dbc:r2dbc-postgresql:0.8.1.RELEASE.
Kotlin version.
Define a enum class
enum class Mood {
UNKNOWN,
HAPPY,
SAD
}
Create a custom codec
class MoodCodec(private val allocator: ByteBufAllocator) : Codec<Mood> {
override fun canEncodeNull(type: Class<*>): Boolean = false
override fun canEncode(value: Any): Boolean = value is Mood
override fun encode(value: Any): Parameter {
return Parameter(Format.FORMAT_TEXT, oid) {
ByteBufUtils.encode(allocator, (value as Mood).name)
}
}
override fun canDecode(dataType: Int, format: Format, type: Class<*>): Boolean = dataType == oid
override fun decode(buffer: ByteBuf?, dataType: Int, format: Format, type: Class<out Mood>): Mood? {
buffer ?: return null
return Mood.valueOf(ByteBufUtils.decode(buffer))
}
override fun type(): Class<*> = Mood::class.java
override fun encodeNull(): Parameter =
Parameter(Format.FORMAT_TEXT, oid, Parameter.NULL_VALUE)
companion object {
// Get form `select oid from pg_type where typname = 'mood'`
private const val oid = YOUR_ENUM_OID
}
}
Registe the codec
You may need change runtimeOnly("io.r2dbc:r2dbc-postgresql") to implementation("io.r2dbc:r2dbc-postgresql")
#Configuration
#EnableR2dbcRepositories
class AppConfig : AbstractR2dbcConfiguration() {
override fun connectionFactory(): ConnectionFactory = PostgresqlConnectionConfiguration.builder()
.port(5432) // Add your config here.
.codecRegistrar { _, allocator, registry ->
registry.addFirst(MoodCodec(allocator))
Mono.empty()
}.build()
.let { PostgresqlConnectionFactory(it) }
}
I used the below for Spring boot 2.6.4 + r2dbc-postgresql 0.8.11 by adding a customizer rather than creating the connection factory myself.
Thanks #Hantsy for pointing EnumCodec out. I added it to a customizer therefore it can play nicely with existing autoconfigure procedure. Also, the spring-data keeps converting my enum to string until I added the converter.
Hopefully these can provide a little help to others.
Register EnumCodec to builder customizer as extensions
It is possible to register multiple enum, just repeat the withEnum() call.
/**
* Use the customizer to add EnumCodec to R2DBC
*/
#Bean
public ConnectionFactoryOptionsBuilderCustomizer connectionFactoryOptionsBuilderCustomizer() {
return builder -> {
builder.option(Option.valueOf("extensions"),
List.of(EnumCodec.builder()
.withEnum("enum_foo", FooEnum.class)
.withRegistrationPriority(RegistrationPriority.FIRST)
.build()));
logger.info("Adding enum to R2DBC postgresql extensions: {}", builder);
};
}
Implement spring data converter by extending EnumWriteSupport
public class FooWritingConverter extends EnumWriteSupport<Foo> {
}
Register converters so that spring data won't always convert enum to string.
This step is a slightly enhanced version of R2dbcDataAutoConfiguration in spring-boot-autoconfigure project.
/**
* Register converter to make sure Spring data treat enum correctly
*/
#Bean
public R2dbcCustomConversions r2dbcCustomConversions(DatabaseClient databaseClient) {
logger.info("Apply R2DBC custom conversions");
R2dbcDialect dialect = DialectResolver.getDialect(databaseClient.getConnectionFactory());
List<Object> converters = new ArrayList<>(dialect.getConverters());
converters.addAll(R2dbcCustomConversions.STORE_CONVERTERS);
return new R2dbcCustomConversions(
CustomConversions.StoreConversions.of(dialect.getSimpleTypeHolder(), converters),
List.of(
new FooWritingConverter()
));
}
Step 1 and 3 can be added to your application class or any other valid configuration.
Check my article about Postgres specific features supported in R2dbc.
There are two options.
use custom Postgres enum type and Java enum type, and register EnumCodec in the connection factory builder.
use a textual type as data type(such as varchar), and Java Enum type, Spring data r2dbc will convert them directly.

Azure Service Fabric and Azure Key Vault Secret Error

I am getting the following when trying to implement Azure Key Vault Secret from a stateless service fabric works just fine from a console app.
System.TypeLoadException
HResult=0x80131522
Message=Inheritance security rules violated by type: 'System.Net.Http.WebRequestHandler'. Derived types must either match the security accessibility of the base type or be less accessible.
Source=Microsoft.Rest.ClientRuntime
StackTrace:
at Microsoft.Rest.ServiceClient`1.CreateRootHandler
public async Task<string> GetAccessToken(string authority, string resource, string scope)
{
var clientId = MyConfig.Settings.Sections["MyConfigSection"].Parameters["AuthClientId"].Value;
var clientSecret = MyConfig.Settings.Sections["MyConfigSection"].Parameters["AuthClientSecret"].Value;
ClientCredential clientCredential = new ClientCredential(clientId, clientSecret);
var context = new AuthenticationContext(authority, TokenCache.DefaultShared);
var result = await context.AcquireTokenAsync(resource, clientCredential);
return result.AccessToken;
}
public string GetCRMConnectionString()
{
var secretvaultAddress = MyConfig.Settings.Sections["MyConfigSection"].Parameters["SecretVaultUrl"].Value;
var client = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(GetAccessToken));
return client.GetSecretAsync(secretvaultAddress).GetAwaiter().GetResult().Value;
}
WebRequestHandler type(whose instance gets created in your case) is a part of System.Net.Http.WebRequest.dll. If you explore assembly's attributes, you'll find the next one applied to it -
[assembly: AllowPartiallyTrustedCallers]
This attribute makes the assembly being considered as SecurityTransparent. WebRequestHandler is derived from HttpClientHandler defined in another assembly - System.Net.Http.dll. So probably on the environment where you have the code deployed, System.Net.Http.dll is missing AllowPartiallyTrustedCallers which makes it security-critical, meaning that rules get violated - transparent code can not call into security-critical one.
Try to resolve it by either creating a binding rule to a specific System.Net.Http.dll version that has AllowPartiallyTrustedCallers attribute or try to create HttpClient explicitly and pass it then into KeyVaultClient ctr.
Refer to this link for more details and options - Inheritance security rules violated by type: 'System.Net.Http.WebRequestHandler'.

MongoDB issues with Grails Scaffolding (do not occur in MySQL)

I tried using MongoDB 2.0.6 to replace MySQL 5.5.25 for a test Grails 2.1 App and am encountering some strange problems.
Issues when using MongoDB but not MySQL:
When using Scaffolding, I cannot get the fields to order by using static constraints
When I specify inList as a constraint, I get a drop-down when using a MySQL backend, but a field when using a MongoDB backend.
No * (asterisk) on fields where blank=false constraint specified.
Domain Class:
package study
class Student {
String login
String firstName
String lastName
String gender
Boolean active
Date dateCreated
Date lastUpdated
static constraints = {
login()
firstName(blank: false)
lastName(blank: false)
gender(inList: ['M', 'F'])
active()
}
}
Controller
package study
class StudentController {
def scaffold = true
}
DataSource.groovy (MySQL stuff commented out):
grails {
mongo {
host = "dev-linux"
port = 27017
username = "study"
password= "********"
databaseName = "study"
}
}
//dataSource {
// pooled = true
// driverClassName = "com.mysql.jdbc.Driver"
// dialect = "org.hibernate.dialect.MySQL5InnoDBDialect"
// username = "study"
// password = "********"
// dbCreate = "create-drop" // one of 'create', 'create-drop','update'
// url = "jdbc:mysql://dev-linux:3306/study"
//
//}
//hibernate {
// cache.use_second_level_cache = true
// cache.use_query_cache = true
// cache.provider_class = "net.sf.ehcache.hibernate.EhCacheProvider"
//}
BuildConfig.groovy (plugins section shown was all I changed to put MongoDB in place of MySQL, the remainder of this file is the default created by Grails)
plugins {
// build ":hibernate:$grailsVersion"
// compile ":mysql-connectorj:5.1.12"
compile ":mongodb:1.0.0.GA"
build ":tomcat:$grailsVersion"
}
The only changes I made to put in MongoDB and take out MySQL is the changes to the DataSource.groovy and BuildConfig.groovy shown above.
Is there any configuration item that I am missing?
I did see someone mention on this Nabble forum post that the field ordering may be an issue with MongoDB.
However, this post did not have any details.
Also, I did not understand why or how the back end Database engine could impact how the view is rendered when using scaffolding. Specifically, the ordering on a page and drop-down vs textfield.
I would have thought that would come from the Domain Class's field types and constraints.
Has anyone come across this odd behavior when using Grails+Scaffolding with MongoDB before? Does anyone know of a fix or have any insight?
Thank you very much in advance, I appreciate it.
Scaffolding with MongoDB works, the problem is if you just install mongodb plugin, grails will see ambiguous domain mappings and errors like these pop up. You need to either:
Remove hibernate plugin like this:
grails uninstall-plugin hibernate
Also remove these lines from BuildConfig.groovy:
runtime ":database-migration:1.1"
runtime ":hibernate:$grailsVersion"
Explicitly tell a given domain is persisted by Mongo by adding this line to it:
static mapWith="mongo"