I'm trying to save comments from an iPhone app that may and nowadays most likely will include emoticons. No matter what I do, I can't save the emoticons to the MySQL database ... Constant Unicode errors.
Python 2.6.5
Django 1.2.1
MySQL database (set to utf8 character set for tables and rows)
Saving the data to a VARCHAR(255) field
The error I keep receiving is:
Incorrect string value: '\xF0\x9F\x97\xBC \xF0...' for column 'body' at row 1
The string I'm passing into the database is:
test_txt = u"Emoji - \U0001f5fc \U0001f60c \U0001f47b ...".encode('utf-8')
Update: Here's the model I'm using:
class ItemComment(db.Model):
item = db.ForeignKey(Item)
user = db.ForeignKey(Profile)
body = db.CharField(max_length=255, blank=True, null=True)
active = db.BooleanField(default=True)
date_added = db.DateTimeField(auto_now_add=True)
def __unicode__(self):
return "%s" % (self.item)
The odd thing is, if I try and pass this to a field that I've created in MySQL and not Django models.py it works fine. But as soon as I register the field in Django models it dies. Is there another way to store these perhaps?
Any ideas would be amazing.
I could not be more stuck on this ...
Update 2: Tracking it in Terminal using the following UPDATE statement (notice the U0001f5fc)
UPDATE 'table' SET 'body' = '🗼', WHERE 'table'.'id' = 1 ; args=(u'\U0001f5fc')
Using as hardcore as I can get to pass the value:
force_unicode(smart_str(value), encoding='utf-8', strings_only=False, errors='ignore')
But the error still throws:
_mysql_exceptions.Warning: Incorrect string value: '\xF0\x9F\x97\xBC' for column 'body' at row 1
Totally lost!!!
Cheers,
Change charset utf8mb4 for MySQL server (version 5.5.3 later)
In my.ini (my.cnf)
[mysqld]
character_set_server = utf8mb4
collation-server = utf8mb4_unicode_ci
or SQL query
SET NAMES 'utf8mb4';
see also http://dev.mysql.com/doc/refman/5.5/en/charset-connection.html
or deletes the character to do it.
python
import re
# emoji_text is unicode
no_emoji_text = re.sub('[\xF0-\xF7][\x80-\xBF][\x80-\xBF][\x80-\xBF]', '', str(emoji_text))
Thank you.
See also
MySQL throws Incorrect string value error
I use Django 1.11 and following setting.py and the create sql can store emoji well,
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'db_name',
'USER': 'db_user',
'PASSWORD': 'your_password',
'OPTIONS': {'charset': 'utf8mb4'}, # note here!!!
}
}
The sql come from this answer,
CREATE DATABASE db_name CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
Related
I don't understand why linq2db is not generating the correct SQL string for this query against Oracle, and it is working perfectly against SQLServer.
string idUser="U1";
await context.Users.ToLinqToDbTable().Where(u=>u.IdUser == idUser).FirstOrDefaultAsyncLinqToDB();
Always returns null. However, it works if el value used directly:
await context.Users.ToLinqToDbTable().Where(u=>u.IdUser == "U1").FirstOrDefaultAsyncLinqToDB();
Looking at the logs, el SQL statement generated is like this:
-- Oracle Oracle11
DECLARE #idUser Varchar2(8) -- String
SET #idUser = 'U1'
SELECT
u.IdUser,
etc..
FROM
USERS u
WHERE
u.IDUSER = :idUser
Why is it generating T-SQL code and not Oracle code (the context is initialized with .UseOracle())?
Thanks
I have been able to link PostgreSQL to java. I have been able to display all the records in the table, however I unable to perform delete operation.
Here is my code:
con = DriverManager.getConnection(url, user, password);
String stm = "DELETE FROM hostdetails WHERE MAC = 'kzhdf'";
pst = con.prepareStatement(stm);
pst.executeUpdate();
Please note that MAC is a string field and is written in capital letters. This field does exist in the table.
The error that I am getting:
SEVERE: ERROR: column "mac" does not exist
When it comes to Postgresql and entity names (Tables, Columns, etc.) with UPPER CASE letters, you need to "escape" the word by placing it in "". Please refer to the documentation on this particular subject. So, your example would be written like this:
String stm = "DELETE FROM hostdetails WHERE \"MAC\" = 'kzhdf'";
On a side note, considering you are using prepared statements, you should not be setting the value directly in your SQL statement.
con = DriverManager.getConnection(url, user, password);
String stm = "DELETE FROM hostdetails WHERE \"MAC\" = ?";
pst = con.prepareStatement(stm);
pst.setString(1, "kzhdf");
pst.executeUpdate();
I am developing an application using Dashdb on Bluemix and nodered, my PHP application uses the call to webservice to invoke the node-red, whenever my function on PHP invokes the node to insert on table and the field GEO_ID is null, the application fails, I understand the issue, it seems the third parameter was not informed, I have just tried to check the param before and passing something like NULL but it continues not working.
See the code:
msg.account_id = msg.req.query.account_id;
msg.user_id = msg.req.query.user_id;
msg.geo_id=msg.req.query.geo_id;
msg.payload = "INSERT INTO ACCOUNT_USER (ACCOUNT_ID, USER_ID, GEO_ID) VALUES (?,?,?) ";
return msg;
And on Dashdb component I have set the parameter as below:
msg.account_id,msg.user_id,msg.geo_id
The third geo_id is the issue, I have tried something like the code below:
if(msg.req.query.geo_id===null){msg.geo_id=null}
or
if(msg.req.query.geo_id===null){msg.geo_id="null"}
The error I got is the one below:
dashDB query node: Error: [IBM][CLI Driver][DB2/LINUXX8664] SQL0420N Invalid character found in a character string argument of the function "DECIMAL". SQLSTATE=22018
I really appreciate if someone could help me on it .
Thanks,
Eduardo Diogo Garcia
Is it possible that msg.req.query.geo_id is set to an empty string?
In that case neither if statement above would get executed, and you would be trying to insert an empty string into a DECIMAL column. Maybe try something like this:
if (! msg.req.query.geo_id || msg.req.query.geo_id == '') {
msg.geo_id = null;
}
I'm trying to switch my grails application from h2 to PostgreSQL.
Steps I've done to reach my goal:
Download JDBC from http://jdbc.postgresql.org/download.html (JDBC4 Postgresql Driver, Version 9.3-1100)
Attach JDBC to /lib folder
Change DataSource. Now it looks like:
dataSource {
pooled = true
driverClassName = "org.postgresql.Driver"
dialect="org.hibernate.dialect.PostgreSQLDialect"
username = "postgres"
password = "admin"
}
hibernate {
cache.use_second_level_cache = true
cache.use_query_cache = false
cache.region.factory_class = 'net.sf.ehcache.hibernate.EhCacheRegionFactory'
}
// environment specific settings
environments {
development {
dataSource {
dbCreate = "update" // one of 'create', 'create-drop', 'update', 'validate', ''
//url = "jdbc:h2:mem:devDb;MVCC=TRUE;LOCK_TIMEOUT=10000"
url = "jdbc:postgresql://localhost:5432/admin_panel"
}
}
test {
dataSource {
dbCreate = "update"
url = "jdbc:postgresql://localhost:5432/admin_panel"
}
}
production {
dataSource {
dbCreate = "update"
url = "jdbc:postgresql://localhost:5432/admin_panel"
pooled = true
properties {
maxActive = -1
minEvictableIdleTimeMillis=1800000
timeBetweenEvictionRunsMillis=1800000
numTestsPerEvictionRun=3
testOnBorrow=true
testWhileIdle=true
testOnReturn=true
validationQuery="SELECT 1"
}
}
}
}
And now the game starts. I type 'run-app' in GGTS and I get an error. Objects I'm trying to create using BootStrap cannot be initialized because of Validation: Error initializing the application: Validation Error(s) occurred during save() .
It is really strange because the message says that reference to previously created object is null: Field error in object 'adminpanel.component.Text' on field 'subpage': rejected value [null];.
There should be no possibility that "subpage" is null in this line, so I go to the pgAdmin III to check if this record is created and there I notice that no table is created at all.
Everetyhing works if application is connected to H2, but starts to freak out when I switch it to postgres. Additionally, when I remove everything from BootStrap, application starts and I can create objects normally, but I still cannot see them into pgAdmin. Do you have any advice what else can I check or why GORM does not create tables in my app when I use PostgreSQL ?
Thanks in advance.
EDIT:
I found the source of the problem after few tests more...
PostgreSQL gives a strange value for 'id' column in every table. When I was using H2, I had values from 1..x in every table, in PostgreSQL I have something like this:
table1
id:
1
2
3
-
7
8
9
table2
id:
4
5
6
-
10
11
As you probably noticed, values are given interchangeably for all rows in different tables, so I cannot have e.g. object table1 with id 1 and object table2 with id 1. Do you have idea why?
Grails/Hibernate uses Sequence for object ID for databases like Postgres (or Oracle, etc). By default, Grails uses a shared sequence (hibernate_sequence). So all object will have uniq id, but unique per whole database, not per table.
You can configure domain to use a different Sequence for a domain, like:
static mapping = {
id generator: 'sequence', params: [sequence: 'my_own_sequence']
}
See also
http://www.postgresql.org/docs/9.1/static/sql-createsequence.html
http://grails.org/doc/2.3.4/ref/Database%20Mapping/id.html
I'm trying to use Flask-MongoKit as follows (with both attempts to find_one failing):
app = Flask('app-name')
db = MongoKit(app)
db.register([database.Users])
with app.app_context():
print db['users'].find_one()
print db.Users.find_one()
When I used plain MongoKit (non-Flask version), and this worked (as follows)
db = Connection()
db.register([database.Users])
print db.Users.find_one()
Thanks!
EDIT:
The database and collection are defined as follows.
class Users(Document):
__collection__ = 'users'
__database__ = 'database'
Flask-MongoKit doesn't use MongoKit's __database__ value. Instead, it uses an application config setting named MONGODB_DATABASE. If that isn't set, it defaults to a database named flask. If you change your code to
app = Flask('app-name')
app.config['MONGODB_DATABASE'] = 'database'
db = MongoKit(app)
your calls to find_one() should work.
The relative bits can be found here and here.