travis-ci postgres `SELECT EXISTS` query return different result - postgresql

I am working on some tests with pg-promise involve dropping a table and recreate a table.
All tests passes on my local machine. But ontravis-ci, it seems to skipp all the DROP TABLE ... SQL, resulting tests fail.
Anyone has any idea why? Is it a permission issue?
Is there a way for me to further debug this, like connect to travis-ci postgres sever?
Update: I didnt put any code cuz all tests pass on my local env, so I thought it is just a travis-ci issue. The below are the bit that I think traivs-ci is skipping.
afterEach('cleanup tables', (done) => {
db.none('DROP TABLE $1~', 'syncTest')
.then(done)
.catch(() => done());
});
beforeEach('cleanup tables', (done) => {
db.none('DROP TABLE $1~', 'syncTest')
.then(done)
.catch(() => done());
});
Update2: After some further tests, it turns out that the test fail was because that
db.one('SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name=$1)', [tableName])
was not returning expected value. The query is returning { '?column?': false } on travis, but returning { exists: false } on my local env.
Is this a travis-ci issue? or postgres version issue?

Most likely it is because your test sequence is wrong, which is subject to a race condition, which you only see on Travis Ci because it is much busier than your local machine when running the tests.
To start with, try replacing your DROP TABLE name with DROP TABLE IF EXISTS name.
And then you may try using CREATE TABLE IF NOT EXISTS name...

Related

Strange error when initializing Postgres database in FeathersJS

I am converting a FeathersJS system from MySQL to Postgres and facing some problems.
After changing the connection string
"mysql": "mysql://user:password#server:port/database"
to
"postgres": "postgres://user:password#server:port/database"
at config/default.json, I changed the dialect from mysql to pg at feathers/sequelize.js.
But when I started with npm run dev I got the following JSON error message, never seen before.
{
"_bitField":18087936,
"_fulfillmentHandler0":{
"name":"SequelizeConnectionError",
"parent":{
"name":"error",
"length":93,
"severity":"FATAL",
"code":"3D000",
"file":"postinit.c",
"line":"855",
"routine":"InitPostgres"
},
"original":{
"name":"error",
"length":93,
"severity":"FATAL",
"code":"3D000",
"file":"postinit.c",
"line":"855",
"routine":"InitPostgres"
}
},
"name":"SequelizeConnectionError",
"parent":{
"name":"error",
"length":93,
"severity":"FATAL",
"code":"3D000",
"file":"postinit.c",
"line":"855",
"routine":"InitPostgres"
},
"original":{
"name":"error",
"length":93,
"severity":"FATAL",
"code":"3D000",
"file":"postinit.c",
"line":"855",
"routine":"InitPostgres"
}
}
After researching a bit with no success, here I am to ask if someone has ever seen this message.
I already tried the dialect as pg and postgres at feathers/sequelize.js, just to check if it would make any difference, but it didn't.
A true beginner error! My fault!
When I created the Postgres (Docker) container, I forgot to create the database itself!
I created the role and gave it the needed permissions, but forgot to create the database corresponding to my connection string.
The Postgres error code mentioned in my question (3D000) corresponds to invalid_catalog_name, i.e., no database corresponding to the connection string.
One may easily check this at Postgres documentation, but I was mistakenly looking for this error code at FeathersJS documentation.

Cannot run tests on h2 in-memory database, rather it runs on PostgreSQL

(I have multiple related questions, so I highlight them as bold)
I have a play app.
play: 2.6.19
scala: 2.12.6
h2: 1.4.197
postgresql: 42.2.5
play-slick/play-slick-evolutions: 3.0.1
slick-pg: 0.16.3
I am adding a test for DAO, and I believe it should run on an h2 in-memory database that is created when tests start, cleared when tests end.
However, my test always runs on PostgreSQL database I configure and use.
# application.conf
slick.dbs.default.profile="slick.jdbc.PostgresProfile$"
slick.dbs.default.db.driver="org.postgresql.Driver"
slick.dbs.default.db.url="jdbc:postgresql://localhost:5432/postgres"
Here is my test test/dao/TodoDAOImplSpec.scala.
package dao
import play.api.inject.guice.GuiceApplicationBuilder
import play.api.test.{Injecting, PlaySpecification, WithApplication}
class TodoDAOImplSpec extends PlaySpecification {
val conf = Map(
"slick.dbs.test.profile" -> "slick.jdbc.H2Profile$",
"slick.dbs.test.db.driver" -> "org.h2.Driver",
"slick.dbs.test.db.url" -> "jdbc:h2:mem:test;MODE=PostgreSQL;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=FALSE"
)
val fakeApp = new GuiceApplicationBuilder().configure(conf).build()
//val fakeApp = new GuiceApplicationBuilder().configure(inMemoryDatabase()).build()
//val fakeApp = new GuiceApplicationBuilder().configure(inMemoryDatabase("test")).build()
"TodoDAO" should {
"returns current state in local pgsql table" in new WithApplication(fakeApp) with Injecting {
val todoDao = inject[TodoDAOImpl]
val result = await(todoDao.index())
result.size should_== 0
}
}
}
For fakeApp, I try all three, but none of them work as expected - my test still runs on my local PostgreSQL table (in which there are 3 todo items), so the test fails.
What I have tried/found:
First, inMemoryDatabase() simply returns a Map("db.<name>.driver"->"org.h2.Driver", "db.<name>.url"->""jdbc:h2:mem:play-test-xxx"), which looks very similar to my own conf map. However, there are 2 main differeneces:
inMemoryDatabase uses db.<name>.xxx while my conf map uses slick.dbs.<name>.db.xxx. Which one should be correct?
Second, rename conf map's keys to "slick.dbs.default.profile", "slick.dbs.default.db.driver" and "slick.dbs.default.db.url" will throw error.
[error] p.a.d.e.DefaultEvolutionsApi - Unknown data type: "status_enum"; SQL statement:
ALTER TABLE todo ADD COLUMN status status_enum NOT NULL [50004-197] [ERROR:50004, SQLSTATE:HY004]
cannot create an instance for class dao.TodoDAOImplSpec
caused by #79bg46315: Database 'default' is in an inconsistent state!
The finding is interesting - is it related to my use of PostgreSQL ENUM type and slick-pg? (See slick-pg issue with h2). Does it mean this is the right configuration for running h2 in-memory tests? If so, the question becomes How to fake PostgreSQL ENUM in h2.
Third, I follow this thread, run sbt '; set javaOptions += "-Dconfig.file=conf/application-test.conf"; test' with a test configuration file conf/application-test.conf:
include "application.conf"
slick.dbs.default.profile="slick.jdbc.H2Profile$"
slick.dbs.default.db.driver="org.h2.Driver"
slick.dbs.default.db.url="jdbc:h2:mem:test;MODE=PostgreSQL;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=FALSE"
Not surprisingly, I get the same error as the 2nd trial.
It seems to me that the 2nd and 3rd trials point to the right direction (Will work on this). But why must we set name to default? Any other better approach?
In play the default database is default. You could however change that to any other database name to want, but then you need to add the database name as well. For example, I want to have a comment database that has the user table:
CREATE TABLE comment.User(
id int(250) NOT NULL AUTO_INCREMENT,
username varchar(255),
comment varchar(255),
PRIMARY KEY (id));
Then I need to have the configuration of it to connect to it (add it to the application.conf file):
db.comment.url="jdbc:mysql://localhost/comment"
db.comment.username=admin-username
db.comment.password="admin-password"
You could have the test database for your testing as mentioned above and use it within your test.
Database Tests Locally: Why not have the database, in local, as you have in production? The data is not there and running the test on local does not touch the production database; why you need an extra database?
Inconsistent State: This is when the MYSQL you wrote, changes the state of the current database within the database, that could be based on creation of a new table or when you want to delete it.
Also status_enum is not recognizable as a MySQL command obviously. Try the commands you want to use in MySQL console if you are not sure about it.

Flyway - Flyway Schema migration failed

I have successfully configured spring boot with a new project to work
with flyway
Migrated with the Postgres database from the version 0001.0 to 0008.0
I have made manually alter the script in local but
flyway migration getting failed.
Sample Error message:
org.springframework.beans.factory.BeanCreationException: Error
creating bean with name 'flywayInitializer' defined in class path
resource
[org/springframework/boot/autoconfigure/flyway/FlywayAutoConfiguration$FlywayConfiguration.class]:
Invocation of init method failed; nested exception is
org.flywaydb.core.api.FlywayException: Validate failed: Migration
checksum mismatch for migration version 0006.0
How to alter the database tables without affecting flyway script from the flyway_schema_history?
For example, I need to change the table name using alter command but executing the flyway migration script without failed.
Any suggestions, Kindly appreciated.
Note:- I don't want to remove the script entries from the table flyway_schema_history.
There are a few ways to do this:-
1) Create a new script file with incremented version. Put your DDL commands to alter the table in this file. Then run migration.
2) If you don't want to delete the entry from the schema_version table, you can change the checksum value in that table. To calculate checksum, use the following method copied from org.flywaydb.core.internal.resolver.sql.SqlMigrationResolver. You can pass null for resource parameter:-
/**
* Calculates the checksum of this string.
*
* #param str The string to calculate the checksum for.
* #return The crc-32 checksum of the bytes.
*/
/* private -> for testing */
static int calculateChecksum(Resource resource, String str) {
final CRC32 crc32 = new CRC32();
BufferedReader bufferedReader = new BufferedReader(new StringReader(str));
try {
String line;
while ((line = bufferedReader.readLine()) != null) {
crc32.update(line.getBytes("UTF-8"));
}
} catch (IOException e) {
String message = "Unable to calculate checksum";
if (resource != null) {
message += " for " + resource.getLocation() + " (" + resource.getLocationOnDisk() + ")";
}
throw new FlywayException(message, e);
}
return (int) crc32.getValue();
}
3) If you are using Flyway Pro version 5+, you can rollback the migration https://flywaydb.org/getstarted/undo.
The answers here are outdated but can still help you.
It sounds like you might be in one of two situations:
You want to re-run a versioned migration. This isn't really how flyway works, as Kartik has suggested, create a new versioned migration to alter the table.
A migration file has been modified and you want to leave it that way and run new ones (eg 0009.0). In this situation you can try:
Run repair. Which will recalculate the checksums (among other things).
Turn off the validateOnMigrate option which will not fail a migration if there are modified migration files.
To solve this error locally without dropping your whole db:
Fix the migration error which caused the root problem
Disconnect your db server
Open the table "flyway_schema_history" which would be created automatically
Delete the rows with the versions that are causing the mismatch problem
Open the tables that have columns depending on the conflict migrations and drop those columns (if needed)
Run again your db server with the new migrations

Slick does not commit transaction on AWS postgres DB

We have an issue with slick 3.0 and a postgres database (9.5) on AWS, where slick opens a transaction but does not seem to commit it, leaving an open connection "idle in transaction" and the futures never complete.
We are just calling db.run(saveRow(row).transactionally.asTry), where
private def saveRow(row: Row): DBIO[Int] = {
val getExistingRow: DBIO[Option[Row]] = table.filter(_.id === row.id).result.headOption
getExistingRow.flatMap((existingRow: Option[Row]) =>
existingRow match {
case None => table += row
case Some(row) =>
table.filter(_.id === row.id).map(_.propety).update(row.property)
}
)
}
Now the first select statement created from getExistingRow already does not complete. It works locally, but when running it in production on AWS, all prepared statements are never commited. Logs from slick.backend just show
#1: Start transaction
#2: StreamingInvokerAction$HeadOptionAction [select ...]
We would expect to get the following further logs from slick.backend (we see them locally), but we don't see them.
#3: SingleInsertAction [insert into ...]
#4: Commit
Is there some configuration setting I need to provide for this to work on the side of Slick, HikariCP or the postgres database that could fix this? Any other ideas on how to fix this issue?
It was actually caused by using the play execution context. When switching to the scala default execution context it worked fine.

how to order query execution with opscode chef and database cookbook

I'm looking for some help on defining a priority in my chef recipe.
The recipe is supposed to import a sql dump in a database then execute 2 mysql queries against the database.
I'm using Chef Solo.
The first mysql_database action :query importing the dump works well
But the 2 other queries seem to do nothing as it seems that the dump is still importing datas in the DB and the datas are not there when runs. Random execution ?
My recipe:
mysql_database node['wp-dev']['database'] do
connection ({:host => "localhost", :username => 'root',
:password => node['mysql']['server_root_password']})
#NOW THE SQL SCRIPT importing the dump - WORKING
sql { ::File.open("/tmp/database.sql").read }
#NOW THE 2 QUERIES - Not working at first Run
sql "UPDATE WORKING QUERY WHEN COPIED IN MYSQL 1"
sql "UPDATE WORKING QUERY WHEN COPIED IN MYSQL 2"
action :query
end
I can not figure how to fix this or how I can work with only_if to check that the import is finished before running the queries.
A solution is to use different mysql_datase blocks with one notifying another when completed.
mysql_database 'insertTestData' do
connection mysql_connection_info
database_name node['some']['database']
sql { ::File.open(/somePath/testData.sql).read }
action :nothing
end
mysql_database 'createTables' do
connection mysql_connection_info
database_name node['some']['database']
sql { ::File.open(/somePath/tableCreate.sql).read }
action :query
notifies :query,"mysql_database[insertTestData]"
end
You need to create a separate mysql_database block for each sql statement you're executing. You can't have multiple sql queries in the same one.