I have the following data source configured in application.conf that I use to run Slick statements.
I need to access the same database through an OLAP engine that will use the same user and password.
Since it's already configured, I'd like to get these two from there, is it possible to read application.conf from Scala? I know I can read the physical file, but is there a library to get the parsed data?
## JDBC Datasource
# ~~~~~
#
dbConfig = {
url = "jdbc:mysql://localhost:3306/db"
driver = com.mysql.jdbc.Driver
connectionPool = disabled
keepAliveConnection = true
user=root
password=xxxx
}
Working with play, you simply inject the configuration like this:
import javax.inject.Inject
import play.api.Configuration
class Something #Inject()(configuration: Configuration) {
val url: Option[String] = configuration.getString("dbConfig.url")
val keepAliveConnection: Option[Boolean] = configuration.getBoolean("dbConfig.keepAliveConnection")
...
}
Also see Configuration API on how to get your properties in various types and formats.
Related
In Akka Projection’s documentation under offsetting in a relational database with JDBC, there is no information about how and where the configuration for establishing a connection to the relational database used should be included. I mean configs such as the username, password, or the url.
In the documentation under offset in a relational database with Slick, the following configuration is provided for the database connection, which is unclear whether it can be used for JDBC as well:
# add here your Slick db settings
db {
# url = "jdbc:h2:mem:test1"
# driver = org.h2.Driver
# connectionPool = disabled
# keepAliveConnection = true
}
How and where should I specify the JDBC connection parameters?
https://doc.akka.io/docs/akka-projection/current/jdbc.html#defining-a-jdbcsession
The following line in the Scala code snippet is where you can specify connection parameters:
val c = DriverManager.getConnection("jdbc:h2:mem:test;DB_CLOSE_DELAY=-1")
anyone tried to use this:
X-Ray
with Slick ?
import slick.dbio.DBIO
import slick.jdbc.JdbcBackend.Database
import slick.jdbc.PostgresProfile.api._
private[database] class PostgresConnector extends DatabaseConnector {
protected final val configurationPath = "mycompany.backend.database.postgres"
protected lazy val database = Database.forConfig(configurationPath)
Probably no way because its based on tomcat:
These interceptors are in the aws-xray-recorder-sql-postgres and
aws-xray-recorder-sql-mysql submodules, respectively. They implement
org.apache.tomcat.jdbc.pool.JdbcInterceptor and are compatible with
Tomcat connection pools.
If you are trying to instrument a SQL connection using a provider other than MySQL or Postgres, you can try to use the generic JDBC-based SQL Library, documented here: https://github.com/aws/aws-xray-sdk-java#intercept-jdbc-based-sql-queries
Alternatively, you can use the X-Ray auto-instrumentation agent for Java, which automatically captures all JDBC-based SQL queries.
I have a Glue/Connection for an RDS/PostgreSQL DB pre-built via CloudFormation, which works fine in a Glue/Scala/Sparkshell via getJDBCSink API to write down a DataFrame to that DB.
But also I need to write down to the same db, plain sql like create index ... or create table ... etc.
How can I forward that sort of statements in the same Glue/Spark shell?
In python, you can provide pg8000 dependency to the spark glue jobs and then run the sql commands by establishing the connection to the RDS using pg8000.
In scala you can directly establish a JDBC connection without the need of any external library as far as driver is concerned, postgres driver is available in aws glue.
You can create connection as
import java.sql.{Connection, DriverManager, ResultSet}
object pgconn extends App {
println("Postgres connector")
classOf[org.postgresql.Driver]
val con_st = "jdbc:postgresql://localhost:5432/DB_NAME?user=DB_USER"
val conn = DriverManager.getConnection(con_str)
try {
val stm = conn.createStatement(ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY)
val rs = stm.executeQuery("SELECT * from Users")
while(rs.next) {
println(rs.getString("quote"))
}
} finally {
conn.close()
}
}
or follow this blog
(I have multiple related questions, so I highlight them as bold)
I have a play app.
play: 2.6.19
scala: 2.12.6
h2: 1.4.197
postgresql: 42.2.5
play-slick/play-slick-evolutions: 3.0.1
slick-pg: 0.16.3
I am adding a test for DAO, and I believe it should run on an h2 in-memory database that is created when tests start, cleared when tests end.
However, my test always runs on PostgreSQL database I configure and use.
# application.conf
slick.dbs.default.profile="slick.jdbc.PostgresProfile$"
slick.dbs.default.db.driver="org.postgresql.Driver"
slick.dbs.default.db.url="jdbc:postgresql://localhost:5432/postgres"
Here is my test test/dao/TodoDAOImplSpec.scala.
package dao
import play.api.inject.guice.GuiceApplicationBuilder
import play.api.test.{Injecting, PlaySpecification, WithApplication}
class TodoDAOImplSpec extends PlaySpecification {
val conf = Map(
"slick.dbs.test.profile" -> "slick.jdbc.H2Profile$",
"slick.dbs.test.db.driver" -> "org.h2.Driver",
"slick.dbs.test.db.url" -> "jdbc:h2:mem:test;MODE=PostgreSQL;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=FALSE"
)
val fakeApp = new GuiceApplicationBuilder().configure(conf).build()
//val fakeApp = new GuiceApplicationBuilder().configure(inMemoryDatabase()).build()
//val fakeApp = new GuiceApplicationBuilder().configure(inMemoryDatabase("test")).build()
"TodoDAO" should {
"returns current state in local pgsql table" in new WithApplication(fakeApp) with Injecting {
val todoDao = inject[TodoDAOImpl]
val result = await(todoDao.index())
result.size should_== 0
}
}
}
For fakeApp, I try all three, but none of them work as expected - my test still runs on my local PostgreSQL table (in which there are 3 todo items), so the test fails.
What I have tried/found:
First, inMemoryDatabase() simply returns a Map("db.<name>.driver"->"org.h2.Driver", "db.<name>.url"->""jdbc:h2:mem:play-test-xxx"), which looks very similar to my own conf map. However, there are 2 main differeneces:
inMemoryDatabase uses db.<name>.xxx while my conf map uses slick.dbs.<name>.db.xxx. Which one should be correct?
Second, rename conf map's keys to "slick.dbs.default.profile", "slick.dbs.default.db.driver" and "slick.dbs.default.db.url" will throw error.
[error] p.a.d.e.DefaultEvolutionsApi - Unknown data type: "status_enum"; SQL statement:
ALTER TABLE todo ADD COLUMN status status_enum NOT NULL [50004-197] [ERROR:50004, SQLSTATE:HY004]
cannot create an instance for class dao.TodoDAOImplSpec
caused by #79bg46315: Database 'default' is in an inconsistent state!
The finding is interesting - is it related to my use of PostgreSQL ENUM type and slick-pg? (See slick-pg issue with h2). Does it mean this is the right configuration for running h2 in-memory tests? If so, the question becomes How to fake PostgreSQL ENUM in h2.
Third, I follow this thread, run sbt '; set javaOptions += "-Dconfig.file=conf/application-test.conf"; test' with a test configuration file conf/application-test.conf:
include "application.conf"
slick.dbs.default.profile="slick.jdbc.H2Profile$"
slick.dbs.default.db.driver="org.h2.Driver"
slick.dbs.default.db.url="jdbc:h2:mem:test;MODE=PostgreSQL;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=FALSE"
Not surprisingly, I get the same error as the 2nd trial.
It seems to me that the 2nd and 3rd trials point to the right direction (Will work on this). But why must we set name to default? Any other better approach?
In play the default database is default. You could however change that to any other database name to want, but then you need to add the database name as well. For example, I want to have a comment database that has the user table:
CREATE TABLE comment.User(
id int(250) NOT NULL AUTO_INCREMENT,
username varchar(255),
comment varchar(255),
PRIMARY KEY (id));
Then I need to have the configuration of it to connect to it (add it to the application.conf file):
db.comment.url="jdbc:mysql://localhost/comment"
db.comment.username=admin-username
db.comment.password="admin-password"
You could have the test database for your testing as mentioned above and use it within your test.
Database Tests Locally: Why not have the database, in local, as you have in production? The data is not there and running the test on local does not touch the production database; why you need an extra database?
Inconsistent State: This is when the MYSQL you wrote, changes the state of the current database within the database, that could be based on creation of a new table or when you want to delete it.
Also status_enum is not recognizable as a MySQL command obviously. Try the commands you want to use in MySQL console if you are not sure about it.
I am going to use soapUI to test the REST API framework.
Is there a way through which i can insert/update records inside the MongoDB with data in a file type(csv, txt etc) using soapUI tool?
What i am trying to do is validate the API calls and update the database from a data file.
If you are willing to use Groovy script, then you can do this pretty easily.
Put your jdbc driver in SoapUI's bin\ext directory.
https://github.com/mongodb/mongo-java-driver/downloads
(probably where you can it for mongodb)
Then you need roughly these things in your script:
import groovy.sql.Sql
def groovyUtils = new com.eviware.soapui.support.GroovyUtils(context)
groovyUtils.registerJdbcDriver("org.postgresql.Driver") // NOT SURE WHAT STRING FOR MONGODB
def connectString = "....."
sql = Sql.newInstance(connectString) // TEST YOUR CONNECT STRING IN A SQL BROWSER
def misc = sql.firstRow("SELECT * from table")
groovy.sql.Sql is very nice!
http://groovy.codehaus.org/api/groovy/sql/Sql.html
You can easily use below groovy code in "Groovy test step" of your test case and connect to mongodb. Before that please ensure that mongodb java client jar file and gmongo are in {Installation Directory}\bin\ext folder of your soapUI installation
Gmongo: http://mvnrepository.com/artifact/com.gmongo/gmongo/1.5
Mongodb Java Client : http://mvnrepository.com/artifact/org.mongodb/mongo-java-driver/3.2.2
import com.gmongo.GMongoClient
import com.gmongo.GMongo
import com.mongodb.MongoCredential
import com.mongodb.ServerAddress
//def credentials = MongoCredential.createMongoCRCredential('admin', 'students', 'admin' as char[])
//def client = new GMongoClient(new ServerAddress("127.0.0.1:27017"))
context.gmongo=new GMongo()
def db=context.gmongo.getDB("test")
log.info db.fruit.find().count()
db.fruit.find().each{
doc->log.info doc
}