I'm currently encountring a performance issue with play slick hitting a Postgresql database on AWS RDS. The app was running quite well 2 days ago but now some queries are freezing the app.
A join query with a result of only 8 rows would take up to 2300ms and one resulting with 30 row is taking more than 4000ms.
Here is the query performed as an example:
protected def jobOfferQuery = for {
(((mission, client), process), clientCompany) <- jobOffers
.joinLeft(users).on(_.clientId === _.userId)
.joinLeft(processes).on(_._1.offerId === _.jobOfferId)
.joinLeft(companies)
} yield (mission, client, process, clientCompany)
Here is my slick configuration for my play app:
database {
dataSourceClass = org.postgresql.ds.PGSimpleDataSource
properties = {
serverName = ${app.postgres.host}
portNumber = ${app.postgres.port}
databaseName = ${app.postgres.db}
user = ${app.postgres.user}
password = ${app.postgres.password}
}
poolName = ${app.postgres.poolName}
numThreads=5
driver="slick.driver.PostgresDriver$"
}
How can it be so slow ? Should I add more hikarycp configuration ?
Thank you so much for helping me out, much appreciated ..
Related
For performance optimisation we are trying to read data from Mongo secondary server for selected scenarios. I am using the inline query using "withReadPreference(ReadPreference.secondaryPreferred())" to read the data, PFB the code snippet.
What I want to confirm the data we are getting is coming from secondary server after executing the inline query highlighted, is there any method available to check the same from Java or Springboot
public User read(final String userId) {
final ObjectId objectId = new ObjectId(userId);
final User user = collection.withReadPreference(ReadPreference.secondaryPreferred()).findOne(objectId).as(User.class);
return user;
}
Pretty much the same way in Java. Note we use secondary() not secondaryPrefered(); this guarantees reads from secondary ONLY:
import com.mongodb.ReadPreference;
{
// This is your "regular" primaryPrefered collection:
MongoCollection<BsonDocument> tcoll = db.getCollection("myCollection", BsonDocument.class);
// ... various operations on tcoll, then create a new
// handle that FORCES reads from secondary and will timeout and
// fail if no secondary can be found:
MongoCollection<BsonDocument> xcoll = tcoll.withReadPreference(ReadPreference.secondary());
BsonDocument f7 = xcoll.find(queryExpr).first();
}
Problem:
So I am trying to connect to a PostgreSql DB with scala slick v3.3.3 and it is failing to find the relation (table) users in schema 'one' within the 'onetest' Database.
I have the following Table setup:
CREATE SCHEMA one;
CREATE TABLE one.users (
...
);
and the table definition:
class UsersTable(tag: Tag) extends Table[UserRequest](tag, Some("one"), "users") {
...
}
with database configuration:
onedbtest = {
profile = "slick.jdbc.PostgresProfile$"
db = {
dataSourceClass = "org.postgresql.ds.PGSimpleDataSource" //Simple datasource with no connection pooling. The connection pool has already been specified with HikariCP.
driver = "slick.driver.PostgresDriver$"
serverName = "localhost"
portNumber = "5432"
databaseName = "onetest"
user = onetestuser
password = "password"
connectionPool = disabled
}
}
and when running (with necessary imports):
dbConfig.db.run((usersTable += createUserRequest).asTry)
Why can it not find relations (tables) in db?
Note: Error does not appear (with tests passing) when:
keepAliveConnection = true is added to config for DB initialisation however, it writes to another db called "one" (dev environment) doesn't work when connectionPool = disabled is added. It should work with the connectionPool attribute added but it doesn't. Strange it is referencing another DB when the db isn't defined anywhere within the code. I am using sbt.version = 1.3.13 and scalaVersion := "2.12.6". sbt clean compile and rebuilding does not solve caching issues. I have also killed all processes to stop any open connections and used db.close where necessary.
I swapped dataSourceClass = "org.postgresql.ds.PGSimpleDataSource"
with dataSourceClass = "slick.jdbc.DatabaseUrlDataSource".
This enables the application to actually close collections and not just return connection to connection pool. This in turn allows the application to initialise a new connection to the relevant db and pick up any modifications to the db.
I am using node-pg-pool to query my Postgres db (host in AWS, db.t2.micro) in my REST API.
I have a concern how to specify a optimal number for max size of Pool.
I got some useful formulas to get pool size from https://github.com/brettwooldridge/HikariCP/wiki/About-Pool-Sizing
I need to know how many threads, cores and hard-drives using in an AWS instance (example: db.t2.micro). I searched AWS documents, and still cannot find all those infos
I'm successfully setting pool size when I connect to my postgres db on elephantsql using the following code, aws should be similar. BTW how are you connecting to AWS RDS? as I am having loads of trouble
var poolConfig = {
max: 5, //connections (was 20 in brianc's repo)
min: 2, //connections (was 4 in brianc's repo)
idleTimeoutMillis: 1000 //close idle clients after 1 second
}
poolConfig.user = 'aaaa';
poolConfig.password = 'bbbb';
poolConfig.database = 'ccccc';
poolConfig.host = 'ddddd';
poolConfig.port = 5432;
poolConfig.ssl = true;
var pool = new Pool(poolConfig);
pool.connect().then(client => {
client.query("SELECT table_name FROM information_schema.tables WHERE table_schema = 'public';")
.then(data=>{});
I'm trying to write some unit test utilities for an orientDB client in scala.
The following is intended to take a function to operate on a DB, and it should wrap the function with code to create and destroy the DB for a single unit test.
However, there doesn't see to be much good documentation on how to clean up a memory DB (and looking at many open source projects, people seem to simply just leak databases and create new ones on a new port).
Simply calling db.close leaves the DB listening to a port and subsequent tests fail. Calling db.drop seems to work, but only if the func succeeded in adding data to the DB.
So, what cleanup is required in the finally clause?
#Test
def fTest2(): Unit = {
def withJSONDBLoan(func: ODatabaseDocumentTx => Unit) : Unit = {
val db: ODatabaseDocumentTx = new ODatabaseDocumentTx("memory:jsondb")
db.create()
try {
func(db)
} finally {
if (!db.isClosed){
db.close // Nope. DB is leaked.
}
// db.drop seems to close the DB but can't
// see when to safely call this.
}
}
val query1 = "insert into ouser set name='test',password='test', status='ACTIVE'"
withJSONDBLoan { db =>
db.command(new OCommandSQL(query1)).execute[ODocument]()
}
// Fails at create because DB already exists.
val query2 = "insert into ouser set name='test2',password='test2', status='ACTIVE'"
withJSONDBLoan { db =>
db.command(new OCommandSQL(query2)).execute[ODocument]()
}
}
I tried your code and it worked for me.
Hope it helps.
We have two different query strategies that we'd ideally like to operate in conjunction on our site without opening redundant connections. One strategy uses the enterprise library to pull Database objects and Execute_____(DbCommand)s on the Database, without directly selecting any sort of connection. Effectively like this:
Database db = DatabaseFactory.CreateDatabase();
DbCommand q = db.GetStoredProcCommand("SomeProc");
using (IDataReader r = db.ExecuteReader(q))
{
List<RecordType> rv = new List<RecordType>();
while (r.Read())
{
rv.Add(RecordType.CreateFromReader(r));
}
return rv;
}
The other, newer strategy, uses a library that asks for an IDbConnection, which it Close()es immediately after execution. So, we do something like this:
DbConnection c = DatabaseFactory.CreateDatabase().CreateConnection();
using (QueryBuilder qb = new QueryBuilder(c))
{
return qb.Find<RecordType>(ConditionCollection);
}
But, the connection returned by CreateConnection() isn't the same one used by the Database.ExecuteReader(), which is apparently left open between queries. So, when we call a data access method using the new strategy after one using the old strategy inside a TransactionScope, it causes unnecessary promotion -- promotion that I'm not sure we have the ability to configure for (we don't have administrative access to the SQL Server).
Before we go down the path of modifying the query-builder-library to work with the Enterprise Library's Database objects ... Is there a way to retrieve, if existent, the open connection last used by one of the Database.Execute_______() methods?
Yes, you can get the connection associated with a transaction. Enterprise Library internally manages a collection of transactions and the associated database connections so if you are in a transaction you can retrieve the connection associated with a database using the static TransactionScopeConnections.GetConnection method:
using (var scope = new TransactionScope())
{
IEnumerable<RecordType> records = GetRecordTypes();
Database db = DatabaseFactory.CreateDatabase();
DbConnection connection = TransactionScopeConnections.GetConnection(db).Connection;
}
public static IEnumerable<RecordType> GetRecordTypes()
{
Database db = DatabaseFactory.CreateDatabase();
DbCommand q = db.GetStoredProcCommand("GetLogEntries");
using (IDataReader r = db.ExecuteReader(q))
{
List<RecordType> rv = new List<RecordType>();
while (r.Read())
{
rv.Add(RecordType.CreateFromReader(r));
}
return rv;
}
}