I'm trying to copy data from postgres to db2. Just a full table.
I found these instructions and am following them, but they're failing for me on the db2 side.
currently I'm doing:
engine_pstgrs._metadata = MetaData(bind=engine_pstgrs)
engine_pstgrs._metadata.reflect(engine_pstgrs)
table_one = Table('table_one', engine_pstgrs._metadata, schema='prod_schema')
print("Got prod_schema.table_one")
#Create new table
engine_db2._metadata = MetaData(bind=engine_db2)
new_pages = Table('new_pages', engine_db2._metadata, schema='mru')
print("Created new table")
for column in table_one.columns:
print("Column appended to new table")
new_pages.append_column(column.copy())
new_pages.create()
And I'm getting the error:
(ibm_db_dbi.ProgrammingError) ibm_db_dbi::ProgrammingError: Statement Execute Failed: [IBM][CLI Driver][DB2/LINUXX8664] SQL0104N An unexpected token ")" was found following "TABLE MRU.new_pages (". Expected tokens may include: "<table_element_list>". SQLSTATE=42601 SQLCODE=-104 [SQL: '\nCREATE TABLE mru.new_pages (\n)\n\n']
I'm open to trying another way (i.e. Pandas) but I'm not seeing how to do so in a sustainable amount of time (the table is very very large)
Related
I am trying to run a create Shema/table query in the ADF lookup activity, with a dummy select in the end.
CREATE SCHEMA [schemax] AUTHORIZATION [auth1];
SELECT 0 AS dummyValue
but I got the below error
A database operation failed with the following error: 'Parse error at line: 2, column: 1: Incorrect syntax near 'SELECT'.',Source=,''Type=System.Data.SqlClient.SqlException,Message=Parse error at line: 2, column: 1: Incorrect syntax near 'SELECT'.,Source=.Net SqlClient Data Provider,SqlErrorNumber=103010,Class=16,ErrorCode=-2146232060,State=1
Data factory pipeline
I was able to run a similar query without SELECT query in the end but got another error mandating lookup must return a value.
You can only write select statements in lookup activity query settings.
To create schema or table, use copy data activity pre-copy script in sink settings. You can select a dummy table for the source and sink dataset and write your create script in pre-copy activity as below.
Source settings: (using dummy table which pulls 0 records)
Sink settings:
Pre-copy script: CREATE SCHEMA test1 AUTHORIZATION [user]
I am trying to query from a table within a copy command however, I have continually gotten errors. Here is the example SQL statement.
copy schema.table
from 's3://bucket/folder`
iam_role (select value from roles.iam where key = 'IAMRole');
The inner select statement on its own returns a value however, when I run the above, I get the following error:
SQL Error [500310] [42601]: [Amazon](500310) Invalid operation: syntax error at or near "("
The COPY command, as you must suspect, does not support embedded SQL.
If you want to do something like this, you can, but you'll need a procedure.
When I run the command below in PostgreSQL
CREATE TABLESPACE 'forex_eurusd_2020_07'
OWNER 'forex'
LOCATION 'f:\data\forex\eurusd\2020_07';
I get the following error.
ERROR: syntax error at or near "'forex_eurusd_2020_07'"
LINE 3: CREATE TABLESPACE 'forex_eurusd_2020_07'
^
SQL state: 42601
Character: 21
What I am trying to do here is to create table-space and later partition by passing the values in my function as string .
Though this is my test statement
Here is something else I tried below
do $$
declare
tbl_spc varchar(50) := 'forex_eurusd_2020_07';
forex_owner varchar(10) :='forex';
begin
CREATE TABLESPACE tbl_spc
OWNER forex_owner
LOCATION 'f:\data\forex\eurusd\2020_07';
end $$;
And got the following result
ERROR: CREATE TABLESPACE cannot be executed from a function
CONTEXT: SQL statement "CREATE TABLESPACE tbl_spc
OWNER forex_owner
LOCATION 'f:\data\forex\eurusd\2020_07'"
PL/pgSQL function inline_code_block line 6 at SQL statement
SQL state: 25001
In the end I am open to suggestions on how to go about passing values to creating table-space using my custom values?
'forex_eurusd_2020_07' is an identifier so it needs to be double quoted(same with OWNER name). quote_ident(string text)(Docs) is your friend. So:
quote_ident('forex_eurusd_2020_07')
The function issue is explained by the error:
ERROR: CREATE TABLESPACE cannot be executed from a function
That is because Tablespace:
"CREATE TABLESPACE cannot be executed inside a transaction block."
And a function runs inside a transaction block. So you can't use a function for this.
I found the solution by using python. As I was using python to get data which I was trying to post in a partitioned table with its own table-space. I was facing issues dynamically creating the table-space. Which I have achieved by writing the following in my code. I used the code given to create table, instead of table, I modified to create the table-space.
connection = psycopg2.connect(user="forex",
password="MXXXXXX#XXXX",
host="127.0.0.1",
port="5432",
database="MyDB")
connection.autocommit = True
cursor = connection.cursor()
tbl_spc = 'forex_eurusd_2020_07'
forex_owner = 'forex'
location_tbl_spc = "'f:\\data\\forex\\eurusd\\2020_07'"
print(location_tbl_spc)
create_tbl_spc_query = ''' CREATE TABLESPACE '''+tbl_spc+'''
OWNER '''+forex_owner+'''
LOCATION '''+location_tbl_spc+''';'''
print(create_tbl_spc_query)
cursor.execute(create_tbl_spc_query)
connection.commit()
print("Table-space created successfully in PostgreSQL ")
except (Exception, psycopg2.Error) as error:
print("Error while creating PostgreSQL table-space", error)
finally:
closing database connection.
if (connection):
cursor.close()
connection.close()
print("PostgreSQL connection is closed")
I have some issue regarding update column of a table in PostgreSQL
My PostgreSQL version is 9.3
Eg:
In below example jifileresource is the table and column name is DATA
public is the schema
bytea_import is the function to retrieve the content of a file
UPDATE jifileresource
SET DATA=(SELECT public.bytea_import('D:/BOB_XML/BOB.xml')
WHERE ID=2235;
i am getting error like this:
ERROR: syntax error at end of input
LINE 4: WHERE ID=2235
^
********** Error **********
ERROR: syntax error at end of input
SQL state: 42601
You are opening two parentheses and closing only one.
UPDATE jifileresource
SET DATA = (SELECT public.bytea_import('D:/BOB_XML/BOB.xml')) <-- here a ")" was missing
WHERE ID=2235;
I have installed DB2 on my local machine and created a table Users with a column Name.
From my portlets I am executing a simple select statement:
Class.forName("com.ibm.db2.jcc.DB2Driver");
String dburl = "jdbc:db2://localhost:50001/Portlets";
conn = DriverManager.getConnection(dburl,"db2admin","password");
String selectTableSQL = "SELECT Name from Users";
statement = conn.createStatement();
ResultSet rs = statement.executeQuery(selectTableSQL);
when i execute this i get the following exception:
com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-204, SQLSTATE=42704, SQLERRMC=DB2ADMIN.USERS, DRIVER=3.62.56.
I dont understand why I am getting this exception. There is nothing much in the stack trace as well.
The SQLCODE in your message is what tells you the actual error message. -204 is the error you are looking for. If you cross-reference the Information Center article about -204, you will see the following message:
name is an undefined name.
And if you look back at your exception, you see name = DB2ADMIN.USERS (the SQLERRMC field).
My guess is that you aren't finding anything because you forgot to append a Schema to your table (the DB2ADMIN part was assumed in the error message because that's your login name).