I am trying to leverage TypeSafe's slick library to interface with a MySQL server. All the gettingstarted/tutorial examples use withSession{} where the framework will automatically create a session, execute the queries within the {}'s, then terminate the session at the end of the block.
My program is rather chatty, and I would like to maintain a persistent connection throughout the execution of the script. So far I have pieced together this code to explicitly create and close sessions.
val db = Database.forURL("jdbc:mysql://localhost/sandbox", user = "root", password="***", driver = "com.mysql.jdbc.Driver")
val s = db.createSession()
...
s.close()
Where I can execute queries in between. However, when I try to execute a command, such as
(Q.u + "insert into TEST (name) values('"+name+"')").execute
It crashes because it cannot find the implicit session. I don't completely understand the syntax of the execute definition in the documentation, but it seems like there might be an optional parameter to pass an explicit session. I've tried using .execute(s), but that spits out a warning that (s) doesn't do anything in a pure expession.
How do I explicitly specify a pre-existing session to run a query on?
Appended: Trial code for JAB's solution
class ActorMinion(name: String) extends Actor
{
Database.forURL("jdbc:mysql://localhost/sandbox", user = "root", password="****", driver = "com.mysql.jdbc.Driver") withSession
{
def receive =
{
case Execute =>
{
(Q.u + "insert into TEST (name) values('"+name+"')").execute
sender ! DoneExecuting(name,output,err.toString)
}
}
}
}
Which returns compile error
[error] /home/ubuntu/helloworld/src/main/scala/hw.scala:41: missing parameter type for expanded function
[error] The argument types of an anonymous function must be fully known. (SLS 8.5)
[error] Expected type was: ?
[error] {
[error] ^
[error] one error found
I was able derive what I needed from this answer
//imports at top of file
//import Database.threadLocalSession <--this should be commented/removed
import scala.slick.session.Session // <-- this should be added
......
//These two lines in actor constructor
val db = Database.forURL("jdbc:mysql://localhost/sandbox", user = "root", password="****", driver = "com.mysql.jdbc.Driver")
implicit var session: Session = db.createSession()
......
session.close() //This line in actor destructor
Just enclose the relevant part of your script in withSession{}. Note that if you are keeping the session open for a while/are performing lots of database manipulation queries, you should also look into taking advantage of transactions.
And you should really be using prepared statements for inserts if the data has a potentially external source.
Related
Is there any benefit to using partially applied functions vs injecting dependencies into a class? Both approaches as I understand them shown here:
class DB(conn: String) {
def get(sql: String): List[Any] = _
}
object DB {
def get(conn: String) (sql: String): List[Any] = _
}
object MyApp {
val conn = "jdbc:..."
val sql = "select * from employees"
val db1 = new DB(conn)
db1.get(sql)
val db2 = DB.get(conn) _
db2(sql)
}
Using partially-applied functions is somewhat simpler, but the conn is passed to the function each time, and could have a different conn each time it is called. The advantage of using a class is that it can perform one-off operations when it is created, such as validation or caching, and retain the results in the class for re-use.
For example the conn in this code is a String but this is presumably used to connect to a database of some sort. With the partially-applied function it must make this connection each time. With a class the connection can be made when the class is created and just re-used for each query. The class version can also prevent the class being created unless the conn is valid.
The class is usually used when the dependency is longer-lived or used by multiple functions. Partial application is more common when the dependency is shorter-lived, like during a single loop or callback. For example:
list.map(f(context))
def f(context: Context)(element: Int): Result = ???
It wouldn't really make sense to create a class just to hold f. On the other hand, if you have 5 functions that all take context, you should probably just put those into a class. In your example, get is unlikely to be the only thing that requires the conn, so a class makes more sense.
I'm trying to get Slick set up to access a MySQL database, and having a bit of trouble. I've put together the following code:
case class FieldValue(fieldName: String, fieldValue: String)
class FieldValues(tag: Tag) extends Table[FieldValue](tag, "FieldValues") {
def fieldName = column[String]("fieldName")
def fieldValue = column[String]("fieldValueID", O.PrimaryKey)
def * = (fieldName, fieldValue) <> (FieldValue.tupled, FieldValue.unapply)
}
object SlickTest extends App {
val db = Database.forConfig("validation_db")
val simplesql = sql"select fieldValueID from FieldValues".as[(String)]
val simplequeryf = db.run(simplesql)
val simplequeryout = Await.result(simplequeryf, 1 second)
simplequeryout.foreach(println)
lazy val slickquery = TableQuery[FieldValues]
val slickqueryf = db.run(slickquery.result)
val slickqueryout = Await.result(slickqueryf, 1 second)
slickqueryout.foreach(println)
}
As you'll see, it runs two queries - the first uses simple SQL (and works fine), whereas the second uses the Slick method. The second, unfortunately, throws up the following error:
java.sql.SQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that c
orresponds to your MySQL server version for the right syntax to use near '"FieldValues"' at line 1
The fact that it quotes the table name in double quotes within single quotes makes me wonder if it's related to MySql's ANSI_QUOTES option, but I'm unsure. For reference, here's my application.conf:
validation_db = {
driver = "com.mysql.cj.jdbc.Driver",
url = "jdbc:mysql://127.0.0.1:3306/validation?serverTimezone=UTC",
user = "root",
password = "password",
connectionPool = disabled,
useSSL=false
}
Any help greatly appreciated!
The way Slick composes SQL, and how fields are quoted or not, is controlled by importing a "profile". There's a different profile for each relational database.
One possibility is that you may see this error if you have imported the incorrect Slick profile.
For MySQL, you should have an import in your code equivalent to:
import slick.jdbc.MySQLProfile.api._
In my application, I have to interact (read-only) with multiple MySQL DBs one-by-one. For each DB, I need a certain no of connections. Interactions with a DB do not occur in a single stretch: I query the DB, take some time processing the results, again query the DB, again process the result and so on.
Each one of these interactions require multiple connections [I fire multiple queries concurrently], hence I need a ConnectionPool that spawns when I start interacting with the DB and lives until I'm done with all queries to that DB (including the interim time intervals when I'm not querying, only processing the results).
I'm able to successfully create a ConnectionPool with desired no of connections and obtain the implicit session as shown below
def createConnectionPool(poolSize: Int): DBSession = {
implicit val session: AutoSession.type = AutoSession
ConnectionPool.singleton(
url = "myUrl",
user = "myUser",
password = "***",
settings = ConnectionPoolSettings(initialSize = poolSize)
)
session
}
I then pass this implicit session throughout the methods where I need to interact with DB. That ways, I'm able to fire poolSize no of queries concurrently using this session. Fair enough.
def methodThatCallsAnotherMethod(implicit session: DBSession): Unit = {
...
methodThatInteractsWithDb
...
}
def methodThatInteractsWithDb(implicit session: DBSession): Unit = {
...
getResultsParallely(poolSize = 32, fetchSize = 2000000)
...
}
def getResultsParallely(poolSize: Int, fetchSize: Int)(implicit session: DBSession): Seq[ResultClass] = {
import java.util.concurrent.Executors
import scala.concurrent.ExecutionContext
import scala.concurrent.duration._
implicit val ec: ExecutionContext = ExecutionContext.fromExecutorService(Executors.newFixedThreadPool(poolSize))
val resultsSequenceFuture: Seq[Future[ResultClass]] = {
(0 until poolSize).map { i =>
val limit: Long = fetchSize
val offset: Long = i * fetchSize
Future(methodThatMakesSingleQuery(limit, offset))
}
}
val resultsFutureSequence: Future[Seq[ResultClass]] = Future.sequence(resultsSequenceFuture)
Await.result(resultsFuture, 2.minutes)
}
There are 2 problems with this technique:
My application is quite big and has many nested method calls, so passing implicit session through all methods like this (see below) isn't feasible.
In addition to the said interactions with different DBs one-by-one, I also need a single connection to another (fixed) DB throughout the lifetime of my entire application. This connection would be used to make a small write operation (logging the progress of my interactions with other DBs) after every few minutes. Therefore, I need multiple ConnectionPools, one for each DB
From what I could make out of ScalikeJdbc's docs, I came up with following way of doing it that doesn't require me to pass the implicit session everywhere.
def createConnectionPool(poolName: String, poolSize: Int): Unit = {
ConnectionPool.add(
name = poolName,
url = "myUrl",
user = "myUser",
password = "***",
settings = ConnectionPoolSettings(initialSize = poolSize)
)
}
def methodThatInteractsWithDb(poolName: String): Unit = {
...
(DB(ConnectionPool.get(poolName).borrow())).readOnly { implicit session: DBSession =>
// interact with DB
...
}
...
}
Although this works, but I'm no longer able to parallelize the db-interaction. This behaviour is obvious since I'm using the borrow() method, that gets a single connection from the pool. This, in turn, makes me wonder why that AutoSession thing worked earlier: why was I able to fire multiple queries simultaneously using a single implicit session? And if that thing worked, then why doesn't this work? But I find no examples of how to obtain a DBSession from a ConnectionPool that supports multiple connections.
To sum up, I have 2 problems and 2 solutions: one for each problem. But I need a single (commmon) solution that solves both the problems.
ScalikeJdbc's limited docs aren't offering a lot of help and blogs / articles on ScalikeJdbc are practically non-existent.
Please suggest the correct way / some work-around.
Framework versions
Scala 2.11.11
"org.scalikejdbc" %% "scalikejdbc" % "3.2.0"
Thanks to #Dennis Hunziker, I was able to figure out the correct way to release connections borrowed from ScalikeJdbc's ConnectionPool. It can be done as follows:
import scalikejdbc.{ConnectionPool, using}
import java.sql.Connection
using(ConnectionPool.get("poolName").borrow()) { (connection: Connection) =>
// use connection (only once) here
}
// connection automatically returned to pool
With this, now I'm able to parallelize interaction with the pool.
To solve my problem of managing several ConnectionPools and using connections across several classes, I ended up writing a ConnectionPoolManager, complete code for which can be found here. By offloading the tasks of
creating pools
borrowing connections from pools
removing pools
to a singleton object that I could use anywhere across my project, I was able to clear a lot of clutter and eliminated the need pass implicit session across chain of methods.
EDIT-1
While I've already linked the complete code for ConnectionPoolManager, here's a quick hint of how you can go about it
Following method of ConnectionPoolManager lets you borrow connections from ConnectionPools
def getDB(dbName: String, poolNameOpt: Option[String] = None): DB = {
// create a pool for db (only) if it doesn't exist
addPool(dbName, poolNameOpt)
val poolName: String = poolNameOpt.getOrElse(dbName)
DB(ConnectionPool.get(poolName).borrow())
}
Thereafter, throughout your code, you can use the above method to borrow connections from pools and make your queries
def makeQuery(dbName: String, poolNameOpt: Option[String]) = {
ConnectionPoolManager.getDB(dbName, poolNameOpt).localTx { implicit session: DBSession =>
// perform ScalikeJdbc SQL query here
}
}
I'm using slick, and have a question about slick session. I'll give an example first,
An Order class contains line items, Order can either fetch line items or remove one of the line item, and Order also can price it self. Below is the pseudocode:
class Order{
def getLineItems= database withSesison{
//get Line Items from db repository
}
def removeLineItem(itemId: String) = database withTransaction{
implicit ss: Session =>
//Remove item from db
//Price the order
}
def priceOrder() = database withTransaction{
implicit ss: Session =>
//getLineItems
//recalculate order price by each line item
}
}
So when I try to remove a line item, it will create a new Session and transaction, and then it will invoke priceOrder, which will also create a new Session and transaction, priceOrder will invoke getLineItems, which create another new session.
From slick document, I know each session is opening a jdbc connection, so in one method invocation it will create 3 database connection, it's a waste of connection resource. Is there a way to use only one connection to finish this operation?
I know slick has a threadLocalSession which bound the session to thread local, but from https://groups.google.com/forum/#!topic/scalaquery/Sg42HDEK34Q I see we should avoid to use threadLocalSession.
Please help, thanks.
Instead of creating a new session/transaction for each method, you can use currying to pass an implicit session.
def create(user: User)(implicit session: Session) = {
val id = Users.returning(Users.map(_.id)).insert(user)
user.copy(id = Some(id))
}
Then, in a controller or some place, when you want to call the methods, you setup a session/transaction and it will be used for all database work within that block.
// Create the user.
DB.withTransaction { implicit session: Session =>
Users.create(user)
}
Implicit sessions are how some of the Slick examples are setup. https://github.com/slick/slick-examples/blob/master/src/main/scala/com/typesafe/slick/examples/lifted/MultiDBCakeExample.scala
I'm trying to implement a controller in Play2 which exposes a simple REST-style api for my db-tables. I'm using squeryl for database access and spray-json for converting objects to/from json
My idea is to have a single generic controller to do all the work, so I've set up the following routes in conf/routes:
GET /:tableName controllers.Crud.getAll(tableName)
GET /:tableName/:primaryKey controllers.Crud.getSingle(tableName, primaryKey)
.. and the following controller:
object Crud extends Controller {
def getAll(tableName: String) = Action {..}
def getSingle(tableName: String, primaryKey: Long) = Action {..}
}
(Yes, missing create/update/delete, but let's get read to work first)
I've mapped tables to case classes by extended squeryl's Schema:
object MyDB extends Schema {
val accountsTable = table[Account]("accounts")
val customersTable = table[Customer]("customers")
}
And I've told spray-json about my case classes so it knows how to convert them.
object MyJsonProtocol extends DefaultJsonProtocol {
implicit val accountFormat = jsonFormat8(Account)
implicit val customerFormat = jsonFormat4(Customer)
}
So far so good, it actually works pretty well as long as I'm using the table-instances directly. The problem surfaces when I'm trying to generify the code so that I end up with excatly one controller for accessing all tables: I'm stuck with some piece of code that doesn't compile and I am not sure what's the next step.
It seems to be a type issue with spray-json which occurs when I'm trying to convert the list of objects to json in my getAll function.
Here is my generic attempt:
def getAll(tableName: String) = Action {
val json = inTransaction {
// lookup table based on url
val table = MyDB.tables.find( t => t.name == tableName).get
// execute select all and convert to json
from(table)(t =>
select(t)
).toList.toJson // causes compile error
}
// convert json to string and set correct content type
Ok(json.compactPrint).as(JSON)
}
Compile error:
[error] /Users/code/api/app/controllers/Crud.scala:29:
Cannot find JsonWriter or JsonFormat type class for List[_$2]
[error] ).toList.toJson
[error] ^
[error] one error found
I'm guessing the problem could be that the json-library needs to know at compile-time which model type I'm throwing at it, but I'm not sure (notice the List[_$2] in that compile error). I have tried the following changes to the code which compile and return results:
Remove the generic table-lookup (MyDB.tables.find(.....).get) and instead use the specific table instance e.g. MyDB.accountsTable. Proves that JSON serialization for work . However this is not generic, will require a unique controller and route config per table in db.
Convert the list of objects from db query to a string before calling toJson. I.e: toList.toJson --> toList.toString.toJson. Proves that generic lookup of tables work But not a proper json response since it is a string-serialized list of objects..
Thoughts anyone?
Your guess is correct. MyDb.tables is a Seq[Table[_]], in other words it could hold any type of table. There is no way for the compiler to figure out the type of the table you locate using the find method, and it seems like the type is needed for the JSON conversion. There are ways to get around that, but you'd need to some type of access to the model class.