Strange behaviour of Grails' application connected to PostgreSQL - postgresql

I'm trying to switch my grails application from h2 to PostgreSQL.
Steps I've done to reach my goal:
Download JDBC from http://jdbc.postgresql.org/download.html (JDBC4 Postgresql Driver, Version 9.3-1100)
Attach JDBC to /lib folder
Change DataSource. Now it looks like:
dataSource {
pooled = true
driverClassName = "org.postgresql.Driver"
dialect="org.hibernate.dialect.PostgreSQLDialect"
username = "postgres"
password = "admin"
}
hibernate {
cache.use_second_level_cache = true
cache.use_query_cache = false
cache.region.factory_class = 'net.sf.ehcache.hibernate.EhCacheRegionFactory'
}
// environment specific settings
environments {
development {
dataSource {
dbCreate = "update" // one of 'create', 'create-drop', 'update', 'validate', ''
//url = "jdbc:h2:mem:devDb;MVCC=TRUE;LOCK_TIMEOUT=10000"
url = "jdbc:postgresql://localhost:5432/admin_panel"
}
}
test {
dataSource {
dbCreate = "update"
url = "jdbc:postgresql://localhost:5432/admin_panel"
}
}
production {
dataSource {
dbCreate = "update"
url = "jdbc:postgresql://localhost:5432/admin_panel"
pooled = true
properties {
maxActive = -1
minEvictableIdleTimeMillis=1800000
timeBetweenEvictionRunsMillis=1800000
numTestsPerEvictionRun=3
testOnBorrow=true
testWhileIdle=true
testOnReturn=true
validationQuery="SELECT 1"
}
}
}
}
And now the game starts. I type 'run-app' in GGTS and I get an error. Objects I'm trying to create using BootStrap cannot be initialized because of Validation: Error initializing the application: Validation Error(s) occurred during save() .
It is really strange because the message says that reference to previously created object is null: Field error in object 'adminpanel.component.Text' on field 'subpage': rejected value [null];.
There should be no possibility that "subpage" is null in this line, so I go to the pgAdmin III to check if this record is created and there I notice that no table is created at all.
Everetyhing works if application is connected to H2, but starts to freak out when I switch it to postgres. Additionally, when I remove everything from BootStrap, application starts and I can create objects normally, but I still cannot see them into pgAdmin. Do you have any advice what else can I check or why GORM does not create tables in my app when I use PostgreSQL ?
Thanks in advance.
EDIT:
I found the source of the problem after few tests more...
PostgreSQL gives a strange value for 'id' column in every table. When I was using H2, I had values from 1..x in every table, in PostgreSQL I have something like this:
table1
id:
1
2
3
-
7
8
9
table2
id:
4
5
6
-
10
11
As you probably noticed, values are given interchangeably for all rows in different tables, so I cannot have e.g. object table1 with id 1 and object table2 with id 1. Do you have idea why?

Grails/Hibernate uses Sequence for object ID for databases like Postgres (or Oracle, etc). By default, Grails uses a shared sequence (hibernate_sequence). So all object will have uniq id, but unique per whole database, not per table.
You can configure domain to use a different Sequence for a domain, like:
static mapping = {
id generator: 'sequence', params: [sequence: 'my_own_sequence']
}
See also
http://www.postgresql.org/docs/9.1/static/sql-createsequence.html
http://grails.org/doc/2.3.4/ref/Database%20Mapping/id.html

Related

Why does Jooq code-generation break with PostGIS?

Context - I am trying out Postgres' Geographic Information System extension PostGis that enables stories latitude and longitudes as Point and operations on it.
If I understand correctly then I need to add a custom converter that can convert the point between JOOQ and PostGis and add it to the gradle file.
Problem - When I generate the jooq-code, few files are generated incorrectly and have the fields defined twice which fail compilation. These are:
<configured-generation-dir>/tables/StValuecount.java
<configured-generation-dir>/tables/records/StValuecountRecord.java
<configured-generation-dir>/tables/records/StValuepercentRecord.java
<configured-generation-dir>/tables/_StValuecount.java
<configured-generation-dir>/tables/records/_StValuecountRecord.java
<configured-generation-dir>/tables/_StHistogram.java
<configured-generation-dir>/tables/records/_StHistogramRecord.java
<configured-generation-dir>/tables/_StQuantile.java
Gradle config =>
jooq{
myAwesomeApp(sourceSets.main){
logging = 'WARN'
jdbc {
driver = 'org.postgresql.Driver'
url = db_url
user = db_user
password = db_password
}
generator {
name = 'org.jooq.codegen.DefaultGenerator'
strategy {
name = 'org.jooq.codegen.DefaultGeneratorStrategy'
}
database {
name = 'org.jooq.meta.postgres.PostgresDatabase'
inputSchema = 'public'
forcedTypes {
forcedType {
userType = 'org.postgis.Point'
converter = 'com.example.JooqBreaksWithPostGis.jooq.converters.PostgresPointJooqConverter'
expression = '.*\\.point'
types = '.*'
}
}
}
generate {
routines = false
relations = true
deprecated = false
records = true
immutablePojos = false
fluentSetters = true
}
target {
packageName = 'jooq.fancy.app'
directory = 'src/main/java/generated'
}
}
}
}
What am I doing wrong?
I have also created a minimal project where I have reproduced the problem in case someone wants to quickly try it.
Steps to reproduce
Checkout project
git clone git#github.com:raj-saxena/JooqBreaksWithPostGis.git
Go to the project directory and start postgis docker container with
docker-compose up
Similarly, to remove postgis docker container run
docker-compose down
Run migrations that add a simple City table containing Point type with
./gradlew flywayMigrate
I have added few rows in a second migration to verify if the DB structure was working. Details to connect to Postgres instance in the build.gradle file.
Generate jooq files with
./gradlew generateMyAwesomeAppJooqSchemaSource
Verify that the files are generated in the configured src/main/java/generated directory.
Verify that the files mentioned above fail to compile.
Taking Lukas' advice, I added the exclude configuration to the jooq config as below:
database {
name = 'org.jooq.meta.postgres.PostgresDatabase'
...
excludes = '.*ST_ValueCount' +
'|.*St_Valuepercent' +
'|.*St_Histogram' +
'|.*St_Quantile' +
'|.*St_Approxhistogram' +
'|.*St_PixelOfValue' +
'|.*St_Approxquantile' +
'|.*ST_Tile'
}
This allowed the code to compile.
This sounds a lot like https://github.com/jOOQ/jOOQ/issues/4055. jOOQ 3.11 currently cannot handle overloaded table valued functions in any RDBMS that supports table valued functions. Your best option here is to exclude all the affected functions from the code generation, using <excludes>:
https://www.jooq.org/doc/latest/manual/code-generation/codegen-advanced/codegen-config-database/codegen-database-includes-excludes/

EF6 code-first: access to database before update database

I'm trying to create a project with EF6.1 with code-first. All works fine I have migration is enabled, create and update DB works too. Now my Problem:
I have create a table "VersionHistory" and a CompanyInfo table.
I'm writing an "Upgrade Wizzard" for update the database. Is it possible to get data from this tables to Display Information (e.g. YourCompanyName and Update from Program Version 1.x to 1.y) before I start the database update?
Should I use classic SQLConnection for this?
Many thanks
You can use a SQL Connection (it can be the same of EF) or you can disable EF database structure checking.
System.Data.Entity.Database.SetInitializer<MyModel>(null);
EDIT
If you access to an entity that is not updated on the database, you can receive Ado exceptions from EF (i.e. missing columns, missing tables and so on).
In compliance with "bubi" I will use a SQLConnection based on my Context e.g.
using (var ctx = new AppContext()) {
ctx.Database.Connection.Open();
var cmd = ctx.Database.Connection.CreateCommand();
cmd.CommandText = "Select * From CompanyInfo";
var rdr = cmd.ExecuteReader();
var infos = (from row in rdr.Cast<System.Data.Common.DbDataRecord>()
let entityId = (int)row["EntityId"] //internal key
let entityKey = (string)row["EntityKey"] //visible key
let displayname = (string)row["DisplayName"] //company name
// some more stuff (version, etc.)
select new NOCompanyInfo {
EntityId = entityId,
EntityKey = entityKey,
DisplayName = displayname,
DBName = dbName,
...
}).ToList();
return new ObservableCollection<NOCompanyInfo>(infos);
}
Thanks

MongoDB & FosUserBundle - lack of columns

When I load data using data-fixtures to my mongoDB
private function loadAdminUser()
{
$userManager = $this->container->get('fos_user.user_manager');
// Create our user and set details
$user = $userManager->createUser();
$user->setUsername('username');
$user->setEmail('email#domain.com');
$user->setPlainPassword('password');
//$user->setPassword('3NCRYPT3D-V3R51ON');
$user->setEnabled(true);
$user->setRoles(array('ROLE_ADMIN'));
}
or when I use console/terminal for it
php app/console fos:user:create adminuser --super-admin
Everyhing is ok (no errors while updating).
But when I try to get that data from mongo it only returns me 4 columns:
while in FOSUserBudnle there is about 15(?) default columns... What's wrong?
I've done it. The problem was because Sonata was extending from use Sonata\UserBundle\Document\BaseUser as BaseUser not from use FOS\UserBundle\Document\User as BaseUser. I have changed it in Application\Sonata\UserBundle\Document and now it is ok

Querying average with a function of column in Rails

I'm using Rails 4 in a web app, Postgresql database and squeel gem for queries.
I have this function in my model statistic.rb
def properties_mean_ppm(mode, rooms, type, output_currency_id)
sql_result = properties(mode, rooms, type).select{
avg(price_dolar / property_area).as(prom)
}
avg = sql_result[0].prom
final_avg = change_currency(avg, DOLAR_ID, output_currency_id)
return final_avg.to_f
end
price_dolar and property_area are columns in the properties table.
It works fine in Rails console and displays the result, but when I use it on the controller it gives an error:
ActiveModel::MissingAttributeError (missing attribute: id)
And indicates the line
avg = sql_result.to_a[0].prom
I also tried using sql_result[0].prom or sql_result.take or sql_result.first, they all have the same error.
The sql_result is this:
#<ActiveRecord::Relation [#<Property >]>
This is the action called in the controller
def properties_mean_ppm
#statistic = Statistic.find(params[:id])
mode = params[:mode] ? params[:mode] : ANY_MODE
type = params[:type] ? params[:type] : ANY_TYPE
one_room = #statistic.properties_mean_ppm(mode, 1, type, UF)
end
I know how to get the result using only SQL without activerecord but that would be very inefficient for me because I have lots of filters called before in the properties() function
Seems like calling that from the properties object made the controller to expect a Property with an id as a result. So I made it work without squeel.
sql_result = properties(mode, rooms, type).select(
"avg(precio_dolar / dimension_propiedad) as prom, 1 as id"
)
And giving a fixed id to the result

Emoticons from iPhone to Python/Django

I'm trying to save comments from an iPhone app that may and nowadays most likely will include emoticons. No matter what I do, I can't save the emoticons to the MySQL database ... Constant Unicode errors.
Python 2.6.5
Django 1.2.1
MySQL database (set to utf8 character set for tables and rows)
Saving the data to a VARCHAR(255) field
The error I keep receiving is:
Incorrect string value: '\xF0\x9F\x97\xBC \xF0...' for column 'body' at row 1
The string I'm passing into the database is:
test_txt = u"Emoji - \U0001f5fc \U0001f60c \U0001f47b ...".encode('utf-8')
Update: Here's the model I'm using:
class ItemComment(db.Model):
item = db.ForeignKey(Item)
user = db.ForeignKey(Profile)
body = db.CharField(max_length=255, blank=True, null=True)
active = db.BooleanField(default=True)
date_added = db.DateTimeField(auto_now_add=True)
def __unicode__(self):
return "%s" % (self.item)
The odd thing is, if I try and pass this to a field that I've created in MySQL and not Django models.py it works fine. But as soon as I register the field in Django models it dies. Is there another way to store these perhaps?
Any ideas would be amazing.
I could not be more stuck on this ...
Update 2: Tracking it in Terminal using the following UPDATE statement (notice the U0001f5fc)
UPDATE 'table' SET 'body' = '🗼', WHERE 'table'.'id' = 1 ; args=(u'\U0001f5fc')
Using as hardcore as I can get to pass the value:
force_unicode(smart_str(value), encoding='utf-8', strings_only=False, errors='ignore')
But the error still throws:
_mysql_exceptions.Warning: Incorrect string value: '\xF0\x9F\x97\xBC' for column 'body' at row 1
Totally lost!!!
Cheers,
Change charset utf8mb4 for MySQL server (version 5.5.3 later)
In my.ini (my.cnf)
[mysqld]
character_set_server = utf8mb4
collation-server = utf8mb4_unicode_ci
or SQL query
SET NAMES 'utf8mb4';
see also http://dev.mysql.com/doc/refman/5.5/en/charset-connection.html
or deletes the character to do it.
python
import re
# emoji_text is unicode
no_emoji_text = re.sub('[\xF0-\xF7][\x80-\xBF][\x80-\xBF][\x80-\xBF]', '', str(emoji_text))
Thank you.
See also
MySQL throws Incorrect string value error
I use Django 1.11 and following setting.py and the create sql can store emoji well,
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'db_name',
'USER': 'db_user',
'PASSWORD': 'your_password',
'OPTIONS': {'charset': 'utf8mb4'}, # note here!!!
}
}
The sql come from this answer,
CREATE DATABASE db_name CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;