I am trying to do simultaneous inserts from external tables (files in csv format) in DB2 table which is truncated before starting inserts.
insert into dbo.TEST_DATA select * from EXTERNAL '/u03/app/TEST_DATA_1.csv' USING (DELIMITER ',' REMOTESOURCE 'JDBC')
insert into dbo.TEST_DATA select * from EXTERNAL '/u03/app/TEST_DATA_2.csv' USING (DELIMITER ',' REMOTESOURCE 'JDBC')
I see multiple entries as below in SYSIBMADM.LOCKS_HELD table and both queries are stuck.
179 db2jcc_application DB2INST1 DBO TEST_DATA ROW_LOCK X GRNT
Data in both external tables are completely different with no overlapping. No other process is reading data from this table, TEST_DATA
Wondering what's causing the lock as I am only doing inserts and how can I avoid this issue?
Any insight or solution is much appreciated.
Thanks
Related
We daily receive 7 millions of records , we are going to append to the existing target table.The target table is partitioned based on date
We are using DB2 Load command to load data from one DB2 table (stage) to another DB2 table target
call SYSPROC.ADMIN_CMD('LOAD FROM (SELECT * FROM stage_table )
OF CURSOR INSERT INTO target_table NONRECOVERABLE INDEXING MODE INCREMENTAL ALLOW READ ACCESS')
As per IBM documentation , ALLOW READ ACCESS is going to be deprecated suggested to use INGEST method instead of that
https://www.ibm.com/docs/en/db2/10.1.0?topic=functionality-fp1-allow-read-access-parameter-load-command
Question:
How to use INGEST method to load data from DB2 to DB2 tables ?
what would be other alternatives to load millions of records with improved performance.
I have create a redshift cluster and created a db inside.
My schema is new_schema
I have created 2 tables inside two tables inside table1, table2
My Question.
I want to list the datatypes of table1
I need to change the datatype of description which is inside the table1 which is of VARCHAR to TEXT
I have tried to list the datatypes of table1 with below query but nothing listing
SELECT * FROM PG_TABLE_DEF WHERE schemaname = 'new_schema';
A few possibilities as to why you are not seeing the expected results. Most likely is that new_schema isn't in your search_path. Pg_table_info only return info for tables in your search_path - see: https://docs.aws.amazon.com/redshift/latest/dg/r_PG_TABLE_DEF.html
Another possibility is that the tables have no data rows (no blocks assigned) and this can lead to incomplete info from some system tables.
Another possibility is that the tables were not committed by the creating session and being checked by a different session. Since you say that you are creating a new db this comes to mind.
Are the tables visible in svv_table_info?
Also the premise of changing varchar to text is a bit off. From https://docs.aws.amazon.com/redshift/latest/dg/r_Character_types.html#r_Character_types-text-and-bpchar-types
You can create an Amazon Redshift table with a TEXT column, but it is
converted to a VARCHAR(256) column that accepts variable-length values
with a maximum of 256 characters.
So it seems like the objective you are trying to achieve is a bit off.
I have many tables in different databases and want to bring them to a database.
It seems like I have to create foreign table in the database (where I want to merge them all) with schemas of all the tables.
I am sure, there is a way to automate this (by the way, I am going to use psql command) but I do not know where to start.
what I have found so far is I can use
select * from information_schema.columns
where table_schema = 'public' and table_name = 'mytable'
I added more detail explanation.
I wanted to copy tables from another database
the tables have same column names and data type
using postgres_fdw, I needed to set up a field name and data type for each tables (the table names are also same)
then, I want to union the tables have same name all to have one single table.
for that, I am going to add prefix on table
for instance, mytable in db1, mytable in db2, mytable in db3 as in
db1_mytable, db2_mytable, db3_mytable in my local database.
Thanks to Albe's comment, I managed it and now I need to figure out doing 4th step using psql command.
I don't want to use pg_dump to export data into sql script, since feeding it to the greenplum cluster is too slow when I have a large amount of data to import. So it seems using greenplum's gpfdist is prefered. Is there any way I can do this?
Or as an alternative, can I export a particular Postgres table's data into a CSV format file containing the large orbjects of that table?
pg_dump will create a file that will use "COPY" to load the data back into a database. When loading into Greenplum, it will load through the Master server and for very large loads, it will become a bottleneck. Yes, the preferred method is to use gpfdist but you can most certainly use COPY to load data into Greenplum. It won't load in the 10+ TB per hour rate that gpfdist can achieve but it still can achieve 1 to 2 TB per hour.
Another alternative is to use gpfdist to execute a program to get data. It would execute the SELECT statement against PostgreSQL to make that available to an External Table in Greenplum. I created a wrapper for this process called, "gplink". You can check it out here: http://www.pivotalguru.com/?page_id=982
Accoridng to greenplum reference:
The simplest data loading method is the SQL INSERT statement...
You can use the COPY command to load the data into a table when the data
is in external text files...
You can use a pair of Greenplum utilities, gpfdist and gpload, to load external data into tables...
Nevertheless if you want to use csv to import data, you can generate csv with large object "filename" joining you table against pg_largeobject. Eg:
b=# create table lo (n text,p oid);
CREATE TABLE
b=# insert into lo values('wheel',lo_import ('/tmp/wheel.PNG'));
INSERT 0 1
b=# copy (select lo.*, pg_largeobject.pageno, pg_largeobject.data from lo join pg_largeobject on lo.p = loid) to '/tmp/lo.csv' WITH (format csv, header);
COPY 20
Generated /tmp/lo.csv will have name, oid and data bytea in csv format.
I've a lot of records that are originally from MySQL. I massaged the data so it will be successfully inserted into PostgreSQL using ActiveRecord. This I can easily do with insertions on row basis i.e one row at a time. This is very slow I want to do bulk insert but this fails if any of the rows contains invalid data. Is there anyway I can achieve bulk insert and only the invalid rows failing instead of the whole bulk?
COPY
When using SQL COPY for bulk insert (or its equivalent \copy in the psql client), failure is not an option. COPY cannot skip illegal lines. You have to match your input format to the table you import to.
If data itself (not decorators) is violating your table definition, there are ways to make this a lot more tolerant though. For instance: create a temporary staging table with all columns of type text. COPY to it, then fix offending rows with SQL commands before converting to the actual data type and inserting into the actual target table.
Consider this related answer:
How to bulk insert only new rows in PostreSQL
Or this more advanced case:
"ERROR: extra data after last expected column" when using PostgreSQL COPY
If NULL values are offending, remove the NOT NULL constraint from your target table temporarily. Fix the rows after COPY, then reinstate the constraint. Or take the route with the staging table, if you cannot afford to soften your rules temporarily.
Sample code:
ALTER TABLE tbl ALTER COLUMN col DROP NOT NULL;
COPY ...
-- repair, like ..
-- UPDATE tbl SET col = 0 WHERE col IS NULL;
ALTER TABLE tbl ALTER COLUMN col SET NOT NULL;
Or you just fix the source table. COPY tells you the number of the offending line. Use an editor of your preference and fix it, then retry. I like to use vim for that.
INSERT
For an INSERT (like commented) the check for NULL values is trivial:
To skip a row with a NULL value:
INSERT INTO (col1, ...
SELECT col1, ...
WHERE col1 IS NOT NULL
To insert sth. else instead of a NULL value (empty string in my example):
INSERT INTO (col1, ...
SELECT COALESCE(col1, ''), ...
A common work-around for this is to import the data into a TEMPORARY or UNLOGGED table with no constraints and, where data in the input is sufficiently bogus, text typed columns.
You can then do INSERT INTO ... SELECT queries against the data to populate the real table with a big query that cleans up the data during import. You can use a lot of CASE statements for this. The idea is to transform the data in one pass.
You might be able to do many of the fixes in Ruby as you read the data in, then push the data to PostgreSQL using COPY ... FROM STDIN. This is possible with Ruby's Pg gem, see eg https://bitbucket.org/ged/ruby-pg/src/tip/sample/copyfrom.rb .
For more complicated cases, look at Pentaho Kettle or Talend Studio ETL tools.