Jetty + HikariCP + JVM SSL Connectivity Issues to RDS PostgreSQL with RDS 2019 CA root certificate - postgresql

Question

I am having no luck in trying to connect to RDS PostgreSQL with Jetty (9.4.26) + JVM (java-1.8.0)
I have tried a few things
Importing 2019 RDS CA into Java cacerts
Setting up a separate TrustStore file for Java
Setting TrustStore file for Jetty in start.ini
Turning -Djavax.net.debug=ssl on shows that the 2015 RDS CA was added to trust, but I just cannot put a fingert as to where it is being told to trust. I am suspecting that the webapp has the 2015 RDS CA embedded...
Logs from Jetty
(-Djavax.net.debug=ssl)
~
~
adding as trusted cert:
Subject: CN=Amazon RDS Root CA, OU=Amazon RDS, O="Amazon Web Services, Inc.", L=Seattle, ST=Washington, C=US
Issuer: CN=Amazon RDS Root CA, OU=Amazon RDS, O="Amazon Web Services, Inc.", L=Seattle, ST=Washington, C=US
Algorithm: RSA; Serial number: 0x42
Valid from Thu Feb 05 20:11:31 AEDT 2015 until Thu Mar 05 20:11:31 AEDT 2020
~
~
~
~
~
Jan 23, 2020 9:54:46 AM org.postgresql.Driver connect
SEVERE: Connection error:
org.postgresql.util.PSQLException: SSL error: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:67)
at org.postgresql.core.v3.ConnectionFactoryImpl.enableSSL(ConnectionFactoryImpl.java:391)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:162)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
at org.postgresql.Driver.makeConnection(Driver.java:452)
at org.postgresql.Driver.connect(Driver.java:254)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:314)
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:171)
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:441)
at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:66)
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:576)
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:569)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
at org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:62)
... 18 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
at sun.security.validator.Validator.validate(Validator.java:262)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:237)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621)
... 26 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
... 32 more
~
~
~
Any ideas?

Answer 1

I'm going to take a stab at answering this as I have a working fix
The fix is upgrading the PostgreSQL JDBC Driver. The current version was 42.2.1, at the time of upgrade the version that worked was 42.2.9
Now going through Psql JDBS driver changelog, I found that from version 42.2.5 onwards there was code change that altered sslmode used
Looking at JDBC driver source, specifically in the ssl/MakeSSL.java file in the 42.2.1 version of the driver, this code section is now no longer present
33 SSLSocketFactory factory;
34
35 String sslmode = PGProperty.SSL_MODE.get(info);
36 // Use the default factory if no specific factory is requested
37 // unless sslmode is set
38 String classname = PGProperty.SSL_FACTORY.get(info);
39 if (classname == null) {
40 // If sslmode is set, use the libp compatible factory
41 if (sslmode != null) {
42 factory = new LibPQFactory(info);
43 } else {
44 factory = (SSLSocketFactory) SSLSocketFactory.getDefault();
45 }
46 } else {
47 try {
48 factory = (SSLSocketFactory) instantiate(classname, info, true,
49 PGProperty.SSL_FACTORY_ARG.get(info));
50 } catch (Exception e) {
51 throw new PSQLException(
52 GT.tr("The SSLSocketFactory class provided {0} could not be instantiated.", classname),
53 PSQLState.CONNECTION_FAILURE, e);
54 }
55 }