In my persistence code all through out the tables, etc. I have the following import
import scala.slick.driver.PostgresDriver.simple._
This is nice because it works, but this is a problem because all my code is bound to Postgres exclusively. If I want my production to do Postgres and my test to be HSQLDB, for example, I can't. I'd like to declare which DataSource/Driver when I'm running my persistence manager (which will do the create) instead of at the table declaration. What am I missing?
This is certainly possible using the cake pattern. My team uses H2 in development and MySQL in production.
See MultiDBExample and MultiDBCakeExample in https://github.com/slick/slick-examples
As far as I was able to find, I think this is a definite restriction in Slick. So much so that I dropped my test environments database and switched it to the same type as my production one. In retrospect, this is what I should have been doing in the first place, but I do understand this is sometimes easier said than done.
Related
So I am using Postgres for my production database. I use Alembic for migrations and SQLAlchemy to define models and query and so on. This setup works very well for the most part, but I have run into a problem recently.
One of my tables requires a config column, which will be a json blob of varying contents. As such, I have opted for the Postgres JSONB type.
The issue is that my local tests run with an sqlite in memory database. Sqlite apparently has cannot interpret this column type, raising the following exception up running the tests (and building the database):
sqlalchemy.exc.CompileError: (in table 'applications', column 'config'): Compiler <sqlalchemy.dialects.sqlite.base.SQLiteTypeCompiler object at 0x109e52ef0> can't render element of type JSONB
Has anyone got a suggestion about how to overcome this issue? I was thought about maybe trying to alter the migrations based on whether the program is running tests or in production, but I really don't know where to start in that regard.
Has anyone got any advice?
Thanks in advance,
Eric
For anyone who comes across a similar issue: as mentioned in the comments on the question, Sqlite presents many differences from Postgres. As such, my testing environment does not mirror my production environment well and differences like these force me to consider hacks, which is bad news.
The solution I have chosen is to containerize a Postgres db to be spun up when tests are run as described here. It's a slight hit in terms of speed, but it does allow me to reliably test db behaviour locally and in CI pipelines, so I think it is worth it.
Thanks to all the people who took the time to comment.
We have a series of modifications to a Postgres database, which can generally be written all in SQL. So it seems Flyway would be a great fit to automate these.
However, they also include imports from files to tables, such as
COPY mytable FROM '${PWD}/mydata.sql';
And secondarily, we'd like not to rely on Postgres' use of file paths like this, which apparently must reside on the server. It should be possible to run any migration from a remote client -- as in Amazon's RDS documentation (last section).
Are there good approaches to handling this kind of scenario already in Flyway? Or alternate approaches to avoid this issue altogether?
Currently, it looks like it'd work to implement the whole migration in Java and use the Postgres driver's CopyManager to import the data. However, that means most of our migrations have to be done in Java, which seems much clumsier. (As far as I can tell, hybrid Java+SQL migrations are not expected?)
Am new to looking at Flyway so thought I'd ask what other alternatives might exist with Flyway, since I'd expect it's pretty common to import a table during a migration.
Starting with Flyway 3.1, you can use COPY FROM STDIN statements within your migration files to accomplish this. The SQL execution engine will automatically use PostgreSQL's CopyManager to transfer the data.
I'm using PostgreSQL as the DB backend for a new Symfony2 + Doctrine2 project. I'd like to take advantage of some of the features of PostGIS, and as such have created a Postgres template for PostGIS.
Right now, I can't see how to tell Doctrine2 to use the PostGIS template instead of the default, looking through the Doctrine2 code hasn't lead me anywhere (to be honest, I'm struggling to figure out how it even works).
Is this even possible? If so, how?
One of your very basic problems is likely to be the fact that when you abstract, you lose low level control. To my knowledge template databases are not likely to be common concepts elsewhere. So you really have to drop to a lower level, connect to the postgres db manually and issue your CREATE DATABASE foo WITH TEMPLATE bar statement yourself.
The comment here says I can use LightSpeed, which I've downloaded and installed. Then I get to this screen:
And I'm not sure what to do. I'm not too familiar with connection strings, first of all. I found this, but... I think "Model assembly" and "Unit of work class" are required. I don't have any DLL's and I'm not working with LightSpeed models, so I'm not sure this is even useful.
The IQ addon Driver works great with my MySQL database. Really easy to set up and use. Are there any drivers that work with PostgreSQL yet? Looks like you can add them by choosing an "*.lpx" file.
I've developed a dynamic LINQPad driver for PostgreSQL databases. Using it you can query the database without the necessity to define your model first (similar to the IQ or the built in LINQ to SQL driver).
You can find it on GitHub:
https://github.com/fknx/linqpad-postgresql-driver
The LightSpeed driver is a static driver, meaning that you must first create a model (and hence a typed DataContext) using the LightSpeed tools.
If you just want to query PostgreSQL in LINQ, you need a dynamic LINQPad driver which no-one has written has yet, AFAIK.
I am using Squeryl as an ORM with MySQL. This is a new project working with existing schemas that contain several hundred tables.
As far as I can tell, neither Squeryl nor any of the available Scala ORMs can generate the O (Scala classes) from the R (mysql tables). I suppose it wouldn't be too hard to roll my own by crawling the information schema, but I'd rather not duplicate that effort if someone else has already done so.
I'm also curious if anyone can tell me why the R->O direction is so often neglected. In my experience, O->R is the exception and not the rule.
I'll probably start down the path of rolling my own solution. If that's anywhere near complete before I hear of another option, I'll post a link to that code.
Thanks.
QueryDSL provides you with a utility, that can generate code from existing tables. You would however need to accept, that it's primarily a Java lib, and Scala is treated only as an extension there.
I guess the support for R-O is just a matter of time and users' feedback.
There is Squealer which does query database tables and generate scala code. It uses Squeryl and other libraries.
I managed to use it with minimal tweaking.
Its gitub is here
I'm curious what type of projects you are working on where you've found R->O to be the rule. My experience, and I'm including not just my own projects but those that other Squeryl users have mentioned on the mailing list, is that most Squeryl projects are predominantly new applications where an SQL database is being used to persist an application specific model rather than a model being created to match an existing schema. Like most OS projects the developers tend to focus first on features that they themselves need and second on features that are most requested by the community so I would encourage you to take this up at the Squeryl Google Group as well.