I am trying to read hstore data from source and insert into target hstore column. But for some weird reason the data has some single quotes in it and I cannot delete or remove them. Source hstore data looks something like
Value 1: "Target_Payment_Type"=>"Auto_Renew", "Target_Membership_term"=>"1 Year"
Value 2: "Target_Payment_Type"=>"'Auto_Renew'", "Target_Membership_term"=>"'1 Year'"
The transformation works fine with the 1st value but fails when at Value2. Can could anyone suggest me a way I can escape the single quotes which may appear in data using pentaho or postgresql (source & target database). Thanks in advance.
At least, you can use postgres replace function in Table Input step:
SELECT
,all_your_non_string_columns
,replace(string_column,'''', '') //note that '''' represents '
FROM
your_table
Real solution you could find in up-to-date driver perhaps.
Related
I have the following string:
SELECT hello as "helloHello", '' as "empty" FROM tbl_test
I want to do something like this:
INSERT INTO tbl_x (a, b, c) VALUES (x, y, string_from_above)
The strings are going to be dynamic (basically they're strings of sql statements), so I do not want to escape all the ' and ". If I open the postgres database and double click on the varchar column, I can copy and paste the string and it goes in exactly as it looks. I want to be able to programmatically do so. Is there a way using an insert command? I tried using pgFormat, but that didn't work. See attached pic with what it looks like in the table.
All PostgreSQL APIs worth their money have a way to escape strings for inclusion in a dynamic SQL statement.
For example, if you write in C, you can use libpq's PQescapeLiteral().
There are also SQL functions for this: quote_literal() and format(). They can be used in PL/pgSQL code.
Don't try to write your own code for this. Use prior art and avoid the risk of getting it wrong.
We are seeing issue in table value which are populated from DB2 (source) to Postgres (Target).
I have including here all the job details for each component.
Based on the above approach and once the data has been populated, when we run the below query in the Postgres DB.
SELECT * FROM VMRCTTA1.VMRRCUST_SUMM where cust_gssn_cd='XY03666699' ;
SELECT * FROM VMRCTTA1.VMRRCUST_SUMM where cust_cntry_cd='847' ;
There will be no records were returned however, when we run the same query with Trim as below it works.
SELECT * FROM VMRCTTA1.VMRRCUST_SUMM where trim(cust_gssn_cd)='XY03666699' ;
SELECT * FROM VMRCTTA1.VMRRCUST_SUMM where trim(cust_cntry_cd)='847' ;
Below are the ways we have tried to overcome this but no luck.
Used tmap between source and target component.
Used trim in source component under Advanced setting.
Change the datatype in Postgres DB of cust_cntry_cd from char(5) to Character varying, this will allow value without any length restriction.
Please suggest what is missing as we have this issue in almost all the table where we have character/varchar columns.
We are using TOS.
The data type is probably character(5) in DB2.
That means that the trailing spaces are part of the column and will be migrated. You have to compare with
cust_cntry_cd = '847 '
or cast the right argument to character(5):
cust_cntry_cd = CAST ('847' AS character(5))
Maybe you could delete all spaces in the advanced settings of the tDB2Input component.
Like the screen :
I am trying to remove the double quotes for the numeric columns using export command by using replace function but it wont worked out, below is the query I used in Linux environment,
EXPORT TO '/Staging/ebi/src/CLP/legal_bill_charge_adjustment11.csv' OF DEL
MESSAGES '/Staging/ebi/src/CLP/legal_bill_charge_adjustment11.log'
select
CLIENT_ID,
CLIENT_DIVISION_ID,
CLIENT_OFFICE_ID,
MATTER_ID,
LEGAL_BILL_CHARGE_ADJ_ID,
LEGAL_BILL_CHARGE_ID,
ADJUSTMENT_DT,
replace ( ORIGINAL_ADJUSTMENT_AMT,""),
replace (CURRENT_ADJUSTMENT_AMT,""),
replace (SYSTEM_ADJUSTMENT_AMT,""),
replace (CLIENT_ADJUSTMENT_AMT,""),
replace (DELETED_ADJUSTMENT,""),
FLAGGED_AMOUNT,
ADJUSTMENT_USER,
STATUS_DESC,
ADJUSTMENT_COMMENT,
WF_TASK_NAME,
WF_TASK_DESC from CLP.legal_bill_charge_adjustment1;
If anyone suggest me the exact db2 query it would be helpful.
Thanks in advance.
Export will not have quotes around numeric data types. You have not provided any data type information so I suppose your numeric content may be stored in a CHAR/VARCHAR column.
Try casting the columns to numeric data types in the export SQL statement.
i.e.
SELECT cast(Textcol as integer) as colname
..
I am attempting to use COPY FROM STDIN to import data into my table. One of the columns in my table is of type geometry. My command looks something like this...
COPY "WeatherStations" ("Station_ID", "Station_Code", "Station_Name", "Station_Location") FROM stdin;
1 KAVP WILKES-BARRE ST_GeomFromText('POINT(41.338055 -75.724166)')
2 KOKV WINCHESTER ST_GeomFromText('POINT(39.143333 -78.144444)')
3 KSHD SHENANDOAH ST_GeomFromText('POINT(38.263611 -78.896388)')
...
However, I think it is attempting to insert the text "ST_GeomFromText('POINT..." and failing instead of evaluating the expression and inserting the result of the expression. Does anyone know what might be going on here and how I can get the actual geoms inserted?
I had a bad time figuring out how to bulk copy/load geometry data into PostGIS using the COPY FROM STDIN command, I couldn't find official documentation on this topic.
Altering the column during the bulk load (the ALTER TABLE / SET DATA TYPE / USING) was not an option to me because it is only supported in PostGIS 2.0+ for the Geometry type, nor was acceptable the use of a temporary table.
There is indeed a direct way to do it (at least in PostGIS 1.5.2+).
You can simply rewrite the data for your copy statement this way, using a simple WKT (Well-known text) representation for your Geometry data:
1 KAVP WILKES-BARRE POINT(41.338055 -75.724166)
2 KOKV WINCHESTER POINT(39.143333 -78.144444)
3 KSHD SHENANDOAH POINT(38.263611 -78.896388)
If you have enforced a SRID constraint on the geometry column you'll have to use the following syntax (in this example the SRID is 4326) known as EWKT (Extended Well-Known Text, which is a PostGIS specific format):
1 KAVP WILKES-BARRE SRID=4326;POINT(41.338055 -75.724166)
2 KOKV WINCHESTER SRID=4326;POINT(39.143333 -78.144444)
3 KSHD SHENANDOAH SRID=4326;POINT(38.263611 -78.896388)
Closing note: there must be no space between "POINT" and the opening parenthesis "(", or the COPY will still return error saying your geometry data has an invalid format.
You could omit the function wrapping the text, import into a temporary table with text column, and then run INSERT/SELECT into the permanent table with the function doing the conversion in that step.
INSERT INTO "WeatherStations"
("Station_ID", "Station_Code", "Station_Name", "Station_Location")
SELECT "Station_ID", "Station_Code", "Station_Name",
ST_GeomFromText("Station_Location")
FROM "TempWeatherStations";
You will keep all the values in .csv file and try like this:
CAT /path/file/demo.csv | psql -u <username> -h <localhost> -d<database>
-c "COPY "WeatherStations" ("Station_ID", "Station_Code", "Station_Name",
"Station_Location") FROM stdin;"
This will work.
Point's value looks something like this: 0101000020E6100000DA722EC555552B40CDCCCCCCCC0C4840.
I typically keep latitude and longitude columns in my tables and build spatial data with triggers.
I don't know how to copy POINTs from stdin otherwise.
Wondering if there is a way to query a SQL Server database and somehow format columns to omit commas in the data if there is any.
Reason for asking is I have 10000+ records and through out the data the varchar have data like 3,25% and other 1%.
I'd prefer not to alter the data in the original table thus asking if a select with other functions would do the trick.
I have thought about selecting all the data into a temp table and stripping the commas but that is a lot of work for every time I do the query.
Any info or if its is possible please reply.
Take a look at the REPLACE function:
SELECT REPLACE(YourColumn, ',', '')
FROM YourTable
Use SQL REPLACE :
REPLACE(YourField,',','')