Slick does not commit transaction on AWS postgres DB - postgresql

We have an issue with slick 3.0 and a postgres database (9.5) on AWS, where slick opens a transaction but does not seem to commit it, leaving an open connection "idle in transaction" and the futures never complete.
We are just calling db.run(saveRow(row).transactionally.asTry), where
private def saveRow(row: Row): DBIO[Int] = {
val getExistingRow: DBIO[Option[Row]] = table.filter(_.id === row.id).result.headOption
getExistingRow.flatMap((existingRow: Option[Row]) =>
existingRow match {
case None => table += row
case Some(row) =>
table.filter(_.id === row.id).map(_.propety).update(row.property)
}
)
}
Now the first select statement created from getExistingRow already does not complete. It works locally, but when running it in production on AWS, all prepared statements are never commited. Logs from slick.backend just show
#1: Start transaction
#2: StreamingInvokerAction$HeadOptionAction [select ...]
We would expect to get the following further logs from slick.backend (we see them locally), but we don't see them.
#3: SingleInsertAction [insert into ...]
#4: Commit
Is there some configuration setting I need to provide for this to work on the side of Slick, HikariCP or the postgres database that could fix this? Any other ideas on how to fix this issue?

It was actually caused by using the play execution context. When switching to the scala default execution context it worked fine.

Related

Cannot run tests on h2 in-memory database, rather it runs on PostgreSQL

(I have multiple related questions, so I highlight them as bold)
I have a play app.
play: 2.6.19
scala: 2.12.6
h2: 1.4.197
postgresql: 42.2.5
play-slick/play-slick-evolutions: 3.0.1
slick-pg: 0.16.3
I am adding a test for DAO, and I believe it should run on an h2 in-memory database that is created when tests start, cleared when tests end.
However, my test always runs on PostgreSQL database I configure and use.
# application.conf
slick.dbs.default.profile="slick.jdbc.PostgresProfile$"
slick.dbs.default.db.driver="org.postgresql.Driver"
slick.dbs.default.db.url="jdbc:postgresql://localhost:5432/postgres"
Here is my test test/dao/TodoDAOImplSpec.scala.
package dao
import play.api.inject.guice.GuiceApplicationBuilder
import play.api.test.{Injecting, PlaySpecification, WithApplication}
class TodoDAOImplSpec extends PlaySpecification {
val conf = Map(
"slick.dbs.test.profile" -> "slick.jdbc.H2Profile$",
"slick.dbs.test.db.driver" -> "org.h2.Driver",
"slick.dbs.test.db.url" -> "jdbc:h2:mem:test;MODE=PostgreSQL;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=FALSE"
)
val fakeApp = new GuiceApplicationBuilder().configure(conf).build()
//val fakeApp = new GuiceApplicationBuilder().configure(inMemoryDatabase()).build()
//val fakeApp = new GuiceApplicationBuilder().configure(inMemoryDatabase("test")).build()
"TodoDAO" should {
"returns current state in local pgsql table" in new WithApplication(fakeApp) with Injecting {
val todoDao = inject[TodoDAOImpl]
val result = await(todoDao.index())
result.size should_== 0
}
}
}
For fakeApp, I try all three, but none of them work as expected - my test still runs on my local PostgreSQL table (in which there are 3 todo items), so the test fails.
What I have tried/found:
First, inMemoryDatabase() simply returns a Map("db.<name>.driver"->"org.h2.Driver", "db.<name>.url"->""jdbc:h2:mem:play-test-xxx"), which looks very similar to my own conf map. However, there are 2 main differeneces:
inMemoryDatabase uses db.<name>.xxx while my conf map uses slick.dbs.<name>.db.xxx. Which one should be correct?
Second, rename conf map's keys to "slick.dbs.default.profile", "slick.dbs.default.db.driver" and "slick.dbs.default.db.url" will throw error.
[error] p.a.d.e.DefaultEvolutionsApi - Unknown data type: "status_enum"; SQL statement:
ALTER TABLE todo ADD COLUMN status status_enum NOT NULL [50004-197] [ERROR:50004, SQLSTATE:HY004]
cannot create an instance for class dao.TodoDAOImplSpec
caused by #79bg46315: Database 'default' is in an inconsistent state!
The finding is interesting - is it related to my use of PostgreSQL ENUM type and slick-pg? (See slick-pg issue with h2). Does it mean this is the right configuration for running h2 in-memory tests? If so, the question becomes How to fake PostgreSQL ENUM in h2.
Third, I follow this thread, run sbt '; set javaOptions += "-Dconfig.file=conf/application-test.conf"; test' with a test configuration file conf/application-test.conf:
include "application.conf"
slick.dbs.default.profile="slick.jdbc.H2Profile$"
slick.dbs.default.db.driver="org.h2.Driver"
slick.dbs.default.db.url="jdbc:h2:mem:test;MODE=PostgreSQL;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=FALSE"
Not surprisingly, I get the same error as the 2nd trial.
It seems to me that the 2nd and 3rd trials point to the right direction (Will work on this). But why must we set name to default? Any other better approach?
In play the default database is default. You could however change that to any other database name to want, but then you need to add the database name as well. For example, I want to have a comment database that has the user table:
CREATE TABLE comment.User(
id int(250) NOT NULL AUTO_INCREMENT,
username varchar(255),
comment varchar(255),
PRIMARY KEY (id));
Then I need to have the configuration of it to connect to it (add it to the application.conf file):
db.comment.url="jdbc:mysql://localhost/comment"
db.comment.username=admin-username
db.comment.password="admin-password"
You could have the test database for your testing as mentioned above and use it within your test.
Database Tests Locally: Why not have the database, in local, as you have in production? The data is not there and running the test on local does not touch the production database; why you need an extra database?
Inconsistent State: This is when the MYSQL you wrote, changes the state of the current database within the database, that could be based on creation of a new table or when you want to delete it.
Also status_enum is not recognizable as a MySQL command obviously. Try the commands you want to use in MySQL console if you are not sure about it.

JanusGraph DynamoDB backend exceptions when committing data to the database

Hello: I am using JanusGraph with DynamoDB example from https://github.com/awslabs/dynamodb-janusgraph-storage-backend
Also, I am connecting to JanusGraph using Spark - Scala - Gremlin Scala framework. Everything thing works when I used Cassandra as the backend, but when I switch to using DynamoDB, I start getting backend exception errors.
My conf looks like this
val conf = new BaseConfiguration
conf.setProperty("gremlin.graph","org.janusgraph.core.JanusGraphFactory")
conf.setProperty("storage.write-time","1 ms")
conf.setProperty("storage.read-time","1 ms")
conf.setProperty("storage.backend","com.amazon.janusgraph.diskstorage.dynamodb.DynamoDBStoreManager")
conf.setProperty("storage.dynamodb.client.signing-region","us-east-1")
conf.setProperty("storage.dynamodb.client.endpoint","http://127.0.0.1:8000")
val graph = JanusGraphFactory.open(conf)
I can connect DynamoDB fine, but when I start to insert data, I run into backend exceptions.
Below is part of the error log
ERROR org.janusgraph.graphdb.database.StandardJanusGraph - Could not commit transaction [1] due to storage exception in system-commit
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:95)
at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:143)
at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:200)
at org.janusgraph.diskstorage.BackendTransaction.commit(BackendTransaction.java:150)
at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:703)
at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1363)
at org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph$GraphTransaction.doCommit(JanusGraphBlueprintsGraph.java:272)
at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:105)
at $line81.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1$$anonfun$apply$1.apply(:84)
at $line81.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1$$anonfun$apply$1.apply(:80)
Any idea what is going on here. I am pretty new to DynamoDB. This was working fine in Cassandra
Why do you know you are connected? I think you must provide credentials into your config. For example:
conf.setProperty("storage.dynamodb.client.credentials.class-name", "com.amazonaws.auth.BasicAWSCredentials")
conf.setProperty("storage.dynamodb.client.credentials.constructor-args", "ACCESS_KEY,SECRET_KEY")

travis-ci postgres `SELECT EXISTS` query return different result

I am working on some tests with pg-promise involve dropping a table and recreate a table.
All tests passes on my local machine. But ontravis-ci, it seems to skipp all the DROP TABLE ... SQL, resulting tests fail.
Anyone has any idea why? Is it a permission issue?
Is there a way for me to further debug this, like connect to travis-ci postgres sever?
Update: I didnt put any code cuz all tests pass on my local env, so I thought it is just a travis-ci issue. The below are the bit that I think traivs-ci is skipping.
afterEach('cleanup tables', (done) => {
db.none('DROP TABLE $1~', 'syncTest')
.then(done)
.catch(() => done());
});
beforeEach('cleanup tables', (done) => {
db.none('DROP TABLE $1~', 'syncTest')
.then(done)
.catch(() => done());
});
Update2: After some further tests, it turns out that the test fail was because that
db.one('SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name=$1)', [tableName])
was not returning expected value. The query is returning { '?column?': false } on travis, but returning { exists: false } on my local env.
Is this a travis-ci issue? or postgres version issue?
Most likely it is because your test sequence is wrong, which is subject to a race condition, which you only see on Travis Ci because it is much busier than your local machine when running the tests.
To start with, try replacing your DROP TABLE name with DROP TABLE IF EXISTS name.
And then you may try using CREATE TABLE IF NOT EXISTS name...

SQL Timeout when running an Entity Framework Migration

I am currently experiencing a SQL Timeout when running a SQL() command inside of an EF Migration.
Situation:
I am replacing a bunch (>50) tables with one table, and need to convert the data from those tables I'm about to drop into the new table. I've organized the Migration the following way:
1.Create the new table.
In the same Migration, use the SQL() function to run a sql script that migrates the data.
3.Drop all the old tables.
Currently, the migration gives the following error:
System.Data.SqlClient.SqlException (0x80131904): Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
The statement has been terminated. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out
The error is happening in an environment where I give them an installer, and they run it without me involved, so I can't manually run individual migrations, and pause in the middle to run the SQL script.
Is there any way to change the timeout for a connection, or get around this issue?
Environment:
EF 6.0 Code First
SQL Server 2012
See this answer.
Use Configuration.cs file to set custom time out:
internal sealed class Configuration :
DbMigrationsConfiguration<ApplicationDbContext>
{
public Configuration()
{
AutomaticMigrationsEnabled = false;
ContextKey = "YourDbContext";
// New timeout in seconds
this.CommandTimeout = 60 * 5;
}
}
With this method, you can change the timeout only for migration and not everyone using your default connection string.
You should be able to set the connection timeout in the connection string, something like:
Connection Timeout=180;

Why do I get 'Database is already closed' when invoking StaticQuery updateNA "shutdown;"

import scala.slick.driver.H2Driver
import scala.slick.jdbc.StaticQuery
object Main extends App {
val db = H2Driver.simple.Database forURL (url = s"jdbc:h2:mem:test", user = "sa", driver = "org.h2.Driver")
StaticQuery updateNA "shutdown;" execute db.createSession()
}
Executing this with scala 2.11.5, h2 1.4.186 and slick 2.1.0 yields a "org.h2.jdbc.JdbcSQLException: Database is already closed". What is happening here?
After executing the "shutdown" prepared statement, the slick StatementInvoker asks the database for the updateCount of the statement.
The H2 database doesn't like being asked this because it's already shut down.
I don't know which of the two is not behaving correctly. However, if you happen to have the same problem, to close the database just use
db.createSession().createStatement() execute "shutdown;"