I have a problem when I convert mysql database to postgres.
mysql tablenames and columns are all uppercase but kettle creates postgres tables all lowercase when i run this job. The components of tableoutput prints log just like this
INSERT INTO USER_MENU ("FLOW_ID", "USER_ID" .... ,
I try mysql all lowercase, run successfully. I know postgres is case-sensitive but how to solve this case when mysql all uppercase or how to make the tableoutput to output lowercase sql.
Using kettle 6.1.0.1-R
Quick answer: The create table statement is an editable text. In particular you can copy/paste it into a notepad (or any editor), change all to lowercase and copy/paste it back before to push the create button. (Useful also for non-standard SQL dialect, like Date/Time/Boolean types).
Neat answer: Edit the connection. On the right panel you have a General/Advanced/Options/Pooling/Cluster menu. Go to the Advanced panel. There you can tell weather your database is using uppercase or lowercase.
Stupid but quick use of the clever answer: Use the Advanced menu to force quote around identifiers.
Really smart answer: Edit the MySQL connection and select the Options menu, and refer to that page. Postgres conforms strictly to standand SQL, so be reluctant to change the Options default for Postgres connections.
Related
I am using firebird database version 2.0. When I try to update a row, I get an error message: Attempted update of read-only database.
http://www.firebirdfaq.org/faq359/ suggests I may query a blob field that uses a different character set than the connection character set.
I do query a blob field and when the blob field has a value then the updating causes the error. If there is no value in the blob then the updating is just fine.
I use IBConsole to open the firebird database and check the database metadata, I find the metadata says "Default character set NONE".
To fix the problem I firstly need to know what are the character sets used in my database.
So my questions are:
what is the character set being used for my database (Connection character set)
the data type of the blob field is MEMOBLOB, and MEMOBLOB is created as Create domain MEMOBLOB as blob sub_type TEXT segment size 80; So what is the character set use for the MEMOBLOB?
No, that is not about queries or BLOBs.
Firebird databases have a number of modes, one of them is "read-only". In this mode no any change to database is allowed.
You can use gfix utility to change this database mode. You can also use corresponding menu in IBExpert and other development tools that use Firebird Services API
The very link you posted - http://www.firebirdfaq.org/faq359/ - says that:
It does not mean that the database file is read-only, but it (database) contains a read-only mark
gfix -mode read_only /path/to/database.fdb
gfix -mode read_write /path/to/database.fdb
See also https://www.firebirdsql.org/manual/gfix-dbmode.html
See also https://www.ibexpert.net/ibe/pmwiki.php?n=Doc.DatabaseProperties
Of tangential questions:
character set being used for my database
According to your text, it is NONE.
To be exact, database does not use some charset. It is every textual column (char/varchar/blob sub_type text) that do. But usually developer does not bother with specifying individual per-column charsets, so they inherit that default one.
Read also: https://www.firebirdsql.org/file/documentation/reference_manuals/fblangref25-en/html/fblangref25-ddl-tbl.html#fblangref25-ddl-tbl-character
metadata says "Default character set NONE"
This is as close to "database charset" as can be.
Granted, that is only default one and you could override it when creating your columns, but i do not think you did. So probably all your textual columns have charset "NONE".
That is rather dangerous setting, meaning all texts in such columns are stored as raw bytes dump, and hoping application would correctly guess how to convert bytes to letters and letters to bytes.
Read more: https://www.firebirdsql.org/file/documentation/reference_manuals/fblangref25-en/html/fblangref25-datatypes-chartypes.html
Flame Robin seems not to show charset by default, but maybe in DDL section it would.
http://www.flamerobin.org/images/screenshots/0.6.0/gtk2/property_page.png
IBExpert does: https://www.ibexpert.net/ibe/uploads/Doc/dmiles783.gif
(Connection character set)
....is not "character set used by database", it is character set used by, well, connection your application (such as IBConsole or FlameRobin or IBExpert) makes to the database.
You have to set it in every application's connection properties. Easiest option would be UTF-8, but when you have NONE-charset columns it might fail...
For example in FR: http://www.flamerobin.org/images/screenshots/0.7.1/winxp/databaseinfo.png
You can use monitoring tables to query for charset id of your CURRENT_CONNECTION, see more at https://www.firebirdsql.org/file/documentation/reference_manuals/fblangref25-en/html/fblangref-appx05-monattach.html
After I add transaction parameters for the relevant tables. It solves the problem.
The parameter value I added are isc_tpb_lock_write and isc_tpb_shared.
Thank you Arioch 'The.
I use Firebird 2.5, and I want to retrieve the following values
Username:
I used SELECT rdb$get_context('SYSTEM', 'CURRENT_USER') FROM ...
Database name:
I used SELECT rdb$get_context('SYSTEM', 'DB_NAME') FROM ...
But for server name, I did not find any client API, do you know how can I retrieve the server name with a SELECT statement.
There is nothing built-in in Firebird to obtain the server host name (or host names, as a server can have multiple host names) through SQL.
The closest you can get is by requesting the isc_info_firebird_version information item through the isc_database_info API function. This returns version information that - if connecting through TCP/IP - includes a host name of the server.
But as your primary reason for this is to discern between environments in SQL, it might be better to look for a different solution. Some ideas:
Use an external table
You can create an external table to contain the environment information you need
In this example I just put in a short, fixed width name for the environment types, but you could include other information, just be aware the external table format is a binary format. When using CHAR it will look like a fixed width format, where values shorter than declared need to be padded with spaces.
You can follow these steps:
Configure ExternalFileAccess in firebird.conf (for this example, you'd need to set ExternalFileAccess = Restrict D:\data\DB\exttables).
You can then create a table as
create table environment
external file 'D:\data\DB\exttables\environment.dat' (
environment_type CHAR(3) CHARACTER SET ASCII NOT NULL
)
Next, create the file D:\data\DB\exttables\environment.dat and populate it with exactly three characters (eg TST for test, PRO for production, etc). You can also insert the value instead, the external table file will be created automatically. Inserting might be simpler if you want more columns, or data with varying length, etc. Just keep in mind it is binary, but using CHAR for all columns will make it look like text.
Do this for each environment, and make sure the file is read-only to avoid accidental inserts.
After this is done, you can obtain the environment information using
select environment_type
from environment
You can share the same file for multiple databases on the same host, and external files are - by default - not included in a gbak backup (they are only included if you apply the -convert backup option), so this would allow moving database between environments without dragging this information along.
Use an UDF or UDR
You can write an UDF (User-Defined Function) or UDR (User Defined Routine) in a suitable programming language to return the information you want and define this function in your database. Firebird can then call this function from SQL.
UDFs are considered deprecated, and you should use UDRs - introduced in Firebird 3 - instead if possible.
I have never written an UDF or UDR myself, so I can't describe it in detail.
I am performing an ETL job via Pentaho 7.1.
The job is to populate a table 'PRO_T_TICKETS' in PostgreSQL 9.2 via the Pentaho Jobs and transformations?
I have mapped the table fields with respect to the stream fields
Mapped Fields
My Table PRO_T_TICKETS contains the Schema (Column Names) in UPPERCASE.
Is this the reason I can't populate the table PRO_T_TICKETS with my ETL Job?
I duplicated the step TABLE_OUTPUT to PRO_T_TICKETS and changed the Target table field to 'PRO_T_TICKETS2'. Pentaho created a new table with lowercase schema and populated the data in it.
But I want this data to be uploaded in the table PRO_T_TICKETS only and with the UPPERCASE schema if possible.
I am attaching the whole job here and the error thrown by Pentaho. Pentaho Error I have also tried my query by adding double quotes to the column names as you can see in the error. But it didn't help.
What do you think I should do?
When you create (or modify) the connection, select Advanced on the left panel and click on the Force to upper case or Force to lower case or, even better, Preserve case of reserved words.
To know which option to choose, copy the 4th line of your error log, the line starting with INSERT INTO "public"."PRO_T_TICKETS("OID"... in your SQL-developer tool and change the connection advanced parameters until it works.
Also, at debug time, don't use batch updates, don't use lazy conversion on previous steps, and try with one (1) field rather than all (25).
Just as a complement: it worked for me following the tips from AlainD and using specific configurations that I'd like to share with you. I have a transformation streaming data from MySQL to PostgreSQL using a Table Input and Output. In both of DBs I have uppercase objects.
I did the following steps to work in the right way:
In the table input (MySQL) the objects are uppercase too, but I typed in lowercase and it worked and I didn't set any special option in the DB Connection.
In the table output (PostgreSQL) I typed everything in uppercase (schema, table name and columns) and I also set "specify the database fields" (clicking on "Get fields").
In the target DB Connection (PostgreSQL) I put the options (in "Advanced" section): "Quote all in database" and "Preserve case of reserved words".
PS: Ah, the last option is because I've found out that there was one more problem with my fields: there was a column called "Admin" (yes guys, they created a camelcase column using a reserved word!) and for that reason I must to put "Preserve case of reserved words" and type it as "Admin" (without quotes and in camelcase) in the Table Output.
I am looking to customize SQL Workbench/J a bit more for Amazon Redshift.
Specifically, I am using the FLOAT8 data type and the "Format SQL" command doesn't recognize that as a type so it doesn't capitalize it or align it with the other types in my CREATE TABLE statements
I've looked here: http://www.sql-workbench.net/manual/settings.html
but i'm not seeing a setting for adding additional keywords.
You can try doing this in workbench.settings file. I have tried adding date/ time literals there, not so sure about FLOAT8.
Reference: http://www.sql-workbench.net/manual/settings.html
In my iPhone app I am using an Sqlite database. I have a requirement to store the text in database. The text contains apostrophes.
For example:
Insert into tbl_insert values ('It is Steve's Shirt');
How to store this kind of statements in Sqlite database?
This is something that I go through in SQL Server and MySQL as well. You should definitely use parameterised SQL queries
See this page for examples in many languages.
I strongly discourage the use of literal strings in the update statement. Use parameterized queries. There's no reason to compromise security
You can write a function which replaces each instance of character ' with ''
http://www.kamath.com/codelibrary/cl003_apostrophe.asp
Simply replace ' characters to ` :)
text = text.replace("'", "`");
With python and sqlite3 i found that the following line worked perfectly (replacing ' with '')
myString = myString.replace('\'', '\'\'')
the string can then be concatenated in an UPDATE command
The line is stored and displayed correctly. It also works great with Grafana.
I'm not yet sure if this is specific to the sqlite3 python module or if it can be generalized