In Akka Projection’s documentation under offsetting in a relational database with JDBC, there is no information about how and where the configuration for establishing a connection to the relational database used should be included. I mean configs such as the username, password, or the url.
In the documentation under offset in a relational database with Slick, the following configuration is provided for the database connection, which is unclear whether it can be used for JDBC as well:
# add here your Slick db settings
db {
# url = "jdbc:h2:mem:test1"
# driver = org.h2.Driver
# connectionPool = disabled
# keepAliveConnection = true
}
How and where should I specify the JDBC connection parameters?
https://doc.akka.io/docs/akka-projection/current/jdbc.html#defining-a-jdbcsession
The following line in the Scala code snippet is where you can specify connection parameters:
val c = DriverManager.getConnection("jdbc:h2:mem:test;DB_CLOSE_DELAY=-1")
Related
When I try to connect to the database on postgres via jdbc, I get the following error:
org.postgresql.util.PSQLException: ERROR: Unsupported startup parameter: search_path
This is how I create the connection:
val connection = DriverManager.getConnection(profile.connection + Option(profile.catalog).getOrElse("")+ "?currentSchema="+Option(profile.schema).getOrElse(""),
profile.user, profile.password)
I use scala and a custom version of postgres.
pgbauncer
In short, pgbouncer at least my version does not work with the search_path parameter, this discussion led me to this idea. There are two ways to fix this problem:
Change the pgbouncer config file by adding
IGNORE_STARTUP_PARAMETERS: search_path
Make a connection without using the currentSchema parameter in the connection string and create connection like this:
val connection =
DriverManager.getConnection(
profile.connection + Option(profile.catalog).getOrElse(""),
profile.user, profile.password)
Then he will choose the scheme according to the rule set, in search_path, they usually set something like "$user", public, in this case, when connecting, he first tries to choose the same scheme as the user name, and if he does not find such a scheme, he chooses public.
I am trying to connect a database over SSL(namely TLSv1.3).
When tested with psql, it connects over TLSv1.3.
With JDBC driver as below, the connection is not secured by SSL at all for some reason.
(confirmed this by executing select * from pg_stat_ssl;)
val props = new Properties()
props.setProperty("user", "postgres")
props.setProperty("password", "example")
props.setProperty("ssl", "true")
props.setProperty("sslmode", "allow")
props.setProperty("sslfactory", "org.postgresql.ssl.NonValidatingFactory")
DriverManager.getConnection("jdbc:postgresql://localhost/", props)
What would be the cause of this problem?
PostgreSQL JDBC driver versions: 42.2.5/42.2.22
If you set sslmode to allow, the client will prefer non-encrypted connections. Change the setting to require or prefer.
I have a Glue/Connection for an RDS/PostgreSQL DB pre-built via CloudFormation, which works fine in a Glue/Scala/Sparkshell via getJDBCSink API to write down a DataFrame to that DB.
But also I need to write down to the same db, plain sql like create index ... or create table ... etc.
How can I forward that sort of statements in the same Glue/Spark shell?
In python, you can provide pg8000 dependency to the spark glue jobs and then run the sql commands by establishing the connection to the RDS using pg8000.
In scala you can directly establish a JDBC connection without the need of any external library as far as driver is concerned, postgres driver is available in aws glue.
You can create connection as
import java.sql.{Connection, DriverManager, ResultSet}
object pgconn extends App {
println("Postgres connector")
classOf[org.postgresql.Driver]
val con_st = "jdbc:postgresql://localhost:5432/DB_NAME?user=DB_USER"
val conn = DriverManager.getConnection(con_str)
try {
val stm = conn.createStatement(ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY)
val rs = stm.executeQuery("SELECT * from Users")
while(rs.next) {
println(rs.getString("quote"))
}
} finally {
conn.close()
}
}
or follow this blog
I have the following data source configured in application.conf that I use to run Slick statements.
I need to access the same database through an OLAP engine that will use the same user and password.
Since it's already configured, I'd like to get these two from there, is it possible to read application.conf from Scala? I know I can read the physical file, but is there a library to get the parsed data?
## JDBC Datasource
# ~~~~~
#
dbConfig = {
url = "jdbc:mysql://localhost:3306/db"
driver = com.mysql.jdbc.Driver
connectionPool = disabled
keepAliveConnection = true
user=root
password=xxxx
}
Working with play, you simply inject the configuration like this:
import javax.inject.Inject
import play.api.Configuration
class Something #Inject()(configuration: Configuration) {
val url: Option[String] = configuration.getString("dbConfig.url")
val keepAliveConnection: Option[Boolean] = configuration.getBoolean("dbConfig.keepAliveConnection")
...
}
Also see Configuration API on how to get your properties in various types and formats.
I am using spring data for storing and fetching records from the database. Initially the database was MySQL, but now I want to configure the same application for mongodb. Please the application resource properties for MysQL.
# ===============================
# = DATA SOURCE
# ===============================
# Connection url for the database connection
spring.datasource.url = jdbc:mysql://localhost:27017/purchase_books
# Username and password
spring.datasource.username = root
spring.datasource.password = root
# Keep the connection alive if idle for a long time (needed in production)
spring.datasource.testWhileIdle = true
spring.datasource.validationQuery = SELECT 1
# ===============================
# = JPA / HIBERNATE
# ===============================
# Show or not log for each sql query
spring.jpa.show-sql = true
# Hibernate ddl auto (create, create-drop, update): with "update" the database
# schema will be automatically updated accordingly to java entities found in the project
spring.jpa.hibernate.ddl-auto = update
# Naming strategy
spring.jpa.hibernate.naming-strategy = org.hibernate.cfg.ImprovedNamingStrategy
# Allows Hibernate to generate SQL optimized for a particular DBMS
spring.jpa.properties.hibernate.dialect = org.hibernate.dialect.MySQL5Dialect
Can anyone please tell the configuration changes in case of mongodb ?
MySQL is an RDBMS and MongoDB is an Document Database.
Hibernate supports NOSQL Database in the form OGM (Object/Grid Mapper), Documentation on OGM http://docs.jboss.org/hibernate/ogm/4.2/reference/en-US/html/
Refer the below example for Spring, Hibernate and MongoDB
https://pragmaticintegrator.wordpress.com/2011/07/14/use-spring-and-hibernate-with-mongodb/
Moreover if you are using spring and if you want to remove Hibernate you can go for Spring MongoTemplate
Refer the sample in the below url
https://spring.io/guides/gs/accessing-data-mongodb/
spring.data.mongodb.uri=mongodb://localhost:27017/<database name>
spring.datasource.username= <username>
spring.datasource.password= <password>
#OR
spring.data.mongodb.host=localhost
spring.data.mongodb.port=27017
spring.data.mongodb.database=<database name>
spring.data.mongodb.username=<username>
spring.data.mongodb.password=<password>