I got the requirement like this,
Create a single Talend job with reads multiple tables and writes to multiple files dynamically(when we give a tablename via context variable the job should take that table as select * from tablename and writes to file tablename.txt)
My oracle query given in toracle input stage-
"SELECT * FROM '"+context.Table_Name+"'"
In Context Variable part given as
Table_Name- String- checked Prompt for value for dynamic table name
In the metadata definition for Oracle table I gave as
Type="dynamic" db type="varchar2"
Issues Facing:
The context variable is not been identified by the job
ORA-00903: invalid table name
Exception in component tOracleInput_1
java.sql.SQLSyntaxErrorException: ORA-00903: invalid table name
at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:91)
at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:112)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:173)
But when I hard code the tablename, job is running fine
The target file path I gave as
"C:/Talend/OutputFIles/context.Table_Name.txt"
Instead of printing value of context variable, I am getting as context.Table_Name.txt as filename!!!
Please help me on this
Did you try to remove ' around the table name?Try this:
"SELECT * FROM " + context.Table_Name
Same for filename construction, you should write:
"C:/Talend/OutputFIles/" + context.Table_Name + ".txt"
Better, you should have pathname defined by a context variable, giving:context.OutputPath + context.Table_Name + ".txt"TRF
Related
I'm trying to run the following PHP script to do a simple database query:
$db_host = "localhost";
$db_name = "showfinder";
$username = "user";
$password = "password";
$dbconn = pg_connect("host=$db_host dbname=$db_name user=$username password=$password")
or die('Could not connect: ' . pg_last_error());
$query = 'SELECT * FROM sf_bands LIMIT 10';
$result = pg_query($query) or die('Query failed: ' . pg_last_error());
This produces the following error:
Query failed: ERROR: relation "sf_bands" does not exist
In all the examples I can find where someone gets an error stating the relation does not exist, it's because they use uppercase letters in their table name. My table name does not have uppercase letters. Is there a way to query my table without including the database name, i.e. showfinder.sf_bands?
From what I've read, this error means that you're not referencing the table name correctly. One common reason is that the table is defined with a mixed-case spelling, and you're trying to query it with all lower-case.
In other words, the following fails:
CREATE TABLE "SF_Bands" ( ... );
SELECT * FROM sf_bands; -- ERROR!
Use double-quotes to delimit identifiers so you can use the specific mixed-case spelling as the table is defined.
SELECT * FROM "SF_Bands";
Re your comment, you can add a schema to the "search_path" so that when you reference a table name without qualifying its schema, the query will match that table name by checked each schema in order. Just like PATH in the shell or include_path in PHP, etc. You can check your current schema search path:
SHOW search_path
"$user",public
You can change your schema search path:
SET search_path TO showfinder,public;
See also http://www.postgresql.org/docs/8.3/static/ddl-schemas.html
I had problems with this and this is the story (sad but true) :
If your table name is all lower case like : accounts
you can use: select * from AcCounTs and it will work fine
If your table name is all lower case like : accounts
The following will fail:
select * from "AcCounTs"
If your table name is mixed case like : Accounts
The following will fail:
select * from accounts
If your table name is mixed case like : Accounts
The following will work OK:
select * from "Accounts"
I dont like remembering useless stuff like this but you have to ;)
Postgres process query different from other RDMS. Put schema name in double quote before your table name like this, "SCHEMA_NAME"."SF_Bands"
Put the dbname parameter in your connection string. It works for me while everything else failed.
Also when doing the select, specify the your_schema.your_table like this:
select * from my_schema.your_table
If a table name contains underscores or upper case, you need to surround it in double-quotes.
SELECT * from "Table_Name";
I had a similar problem on OSX but tried to play around with double and single quotes. For your case, you could try something like this
$query = 'SELECT * FROM "sf_bands"'; // NOTE: double quotes on "sf_Bands"
This is realy helpfull
SET search_path TO schema,public;
I digged this issues more, and found out about how to set this "search_path" by defoult for a new user in current database.
Open DataBase Properties then open Sheet "Variables"
and simply add this variable for your user with actual value.
So now your user will get this schema_name by defoult and you could use tableName without schemaName.
You must write schema name and table name in qutotation mark. As below:
select * from "schemaName"."tableName";
I had the same issue as above and I am using PostgreSQL 10.5.
I tried everything as above but nothing seems to be working.
Then I closed the pgadmin and opened a session for the PSQL terminal.
Logged into the PSQL and connected to the database and schema respectively :
\c <DATABASE_NAME>;
set search_path to <SCHEMA_NAME>;
Then, restarted the pgadmin console and then I was able to work without issue in the query-tool of the pagadmin.
For me the problem was, that I had used a query to that particular table while Django was initialized. Of course it will then throw an error, because those tables did not exist. In my case, it was a get_or_create method within a admin.py file, that was executed whenever the software ran any kind of operation (in this case the migration). Hope that helps someone.
In addition to Bill Karwin's answer =>
Yes, you should surround the table name with double quotes. However, be aware that most probably php will not allow you to just write simply:
$query = "SELECT * FROM "SF_Bands"";
Instead, you should use single quotes while surrounding the query as sav said.
$query = 'SELECT * FROM "SF_Bands"';
You have to add the schema first e.g.
SELECT * FROM place.user_place;
If you don't want to add that in all queries then try this:
SET search_path TO place;
Now it will works:
SELECT * FROM user_place;
Easiest workaround is Just change the table name and all column names to lowercase and your issue will be resolved.
For example:
Change Table_Name to table_name and
Change ColumnName to columnname
It might be silly for a few, but in my case - once I created the table I could able to query the table on the same session, but if I relogin with new session table does not exits.
Then I used commit just after creating the table and now I could able to find and query the table in the new session as well. Like this:
select * from my_schema.my_tbl;
Hope this would help a few.
Make sure that Table name doesn't contain any trailing whitespaces
Try this: SCHEMA_NAME.TABLE_NAME
I'd suggest checking if you run the migrations or if the table exists in the database.
I tried every good answer ( upvote > 10) but not works.
I met this problem in pgAdmin4.
so my solution is quite simple:
find the target table / scheme.
mouse right click, and click: query-tool
in this new query tool window, you can run your SQL without specifying set search_path to <SCHEMA_NAME>;
you can see the result:
I want to write a CSV file to a table in Postgres via Airflow.
I came across this Airflow documentation denoting that the hook already has a builtin function for CSV export.
And used this thread on how to use it.
I have a python operator whose python_callable is as follows:
def copy_expert_csv():
hook = PostgresHook(postgres_conn_id='warehouse',host='data-warehouse',
database='datalake',
user='root',
password='root',
port=9999)
with hook.get_conn() as connection:
hook.copy_expert("""COPY datalake.public.wcc_users FROM stdin WITH CSV HEADER
DELIMITER as ',' """,
'includes/cleaned_data/wwc/' + str(date.today()) + '_wwc_cleaned ')
connection.commit()
The task finishes successfully as shown in the image.
:
And there is no error logs on my database either:
materials-data-warehouse-1 | 2022-04-29 17:43:01.942 UTC [198] STATEMENT: COPY datalake.public.wcc_users FROM STDIN WITH (FORMAT CSV) HEADER
My file has around 1000 rows. However, when I select from my table, there are 0 rows inserted.
The column naming in table are different from the file and also 2 columns have date and timestamp datatypes rather than text. Can it be the cause? Then why no errors are thrown?
Seems that table definition was incorrect. This throws no errors but inserts nothing either.
We are seeing issue in table value which are populated from DB2 (source) to Postgres (Target).
I have including here all the job details for each component.
Based on the above approach and once the data has been populated, when we run the below query in the Postgres DB.
SELECT * FROM VMRCTTA1.VMRRCUST_SUMM where cust_gssn_cd='XY03666699' ;
SELECT * FROM VMRCTTA1.VMRRCUST_SUMM where cust_cntry_cd='847' ;
There will be no records were returned however, when we run the same query with Trim as below it works.
SELECT * FROM VMRCTTA1.VMRRCUST_SUMM where trim(cust_gssn_cd)='XY03666699' ;
SELECT * FROM VMRCTTA1.VMRRCUST_SUMM where trim(cust_cntry_cd)='847' ;
Below are the ways we have tried to overcome this but no luck.
Used tmap between source and target component.
Used trim in source component under Advanced setting.
Change the datatype in Postgres DB of cust_cntry_cd from char(5) to Character varying, this will allow value without any length restriction.
Please suggest what is missing as we have this issue in almost all the table where we have character/varchar columns.
We are using TOS.
The data type is probably character(5) in DB2.
That means that the trailing spaces are part of the column and will be migrated. You have to compare with
cust_cntry_cd = '847 '
or cast the right argument to character(5):
cust_cntry_cd = CAST ('847' AS character(5))
Maybe you could delete all spaces in the advanced settings of the tDB2Input component.
Like the screen :
I'm trying to run the following PHP script to do a simple database query:
$db_host = "localhost";
$db_name = "showfinder";
$username = "user";
$password = "password";
$dbconn = pg_connect("host=$db_host dbname=$db_name user=$username password=$password")
or die('Could not connect: ' . pg_last_error());
$query = 'SELECT * FROM sf_bands LIMIT 10';
$result = pg_query($query) or die('Query failed: ' . pg_last_error());
This produces the following error:
Query failed: ERROR: relation "sf_bands" does not exist
In all the examples I can find where someone gets an error stating the relation does not exist, it's because they use uppercase letters in their table name. My table name does not have uppercase letters. Is there a way to query my table without including the database name, i.e. showfinder.sf_bands?
From what I've read, this error means that you're not referencing the table name correctly. One common reason is that the table is defined with a mixed-case spelling, and you're trying to query it with all lower-case.
In other words, the following fails:
CREATE TABLE "SF_Bands" ( ... );
SELECT * FROM sf_bands; -- ERROR!
Use double-quotes to delimit identifiers so you can use the specific mixed-case spelling as the table is defined.
SELECT * FROM "SF_Bands";
Re your comment, you can add a schema to the "search_path" so that when you reference a table name without qualifying its schema, the query will match that table name by checked each schema in order. Just like PATH in the shell or include_path in PHP, etc. You can check your current schema search path:
SHOW search_path
"$user",public
You can change your schema search path:
SET search_path TO showfinder,public;
See also http://www.postgresql.org/docs/8.3/static/ddl-schemas.html
I had problems with this and this is the story (sad but true) :
If your table name is all lower case like : accounts
you can use: select * from AcCounTs and it will work fine
If your table name is all lower case like : accounts
The following will fail:
select * from "AcCounTs"
If your table name is mixed case like : Accounts
The following will fail:
select * from accounts
If your table name is mixed case like : Accounts
The following will work OK:
select * from "Accounts"
I dont like remembering useless stuff like this but you have to ;)
Postgres process query different from other RDMS. Put schema name in double quote before your table name like this, "SCHEMA_NAME"."SF_Bands"
Put the dbname parameter in your connection string. It works for me while everything else failed.
Also when doing the select, specify the your_schema.your_table like this:
select * from my_schema.your_table
If a table name contains underscores or upper case, you need to surround it in double-quotes.
SELECT * from "Table_Name";
I had a similar problem on OSX but tried to play around with double and single quotes. For your case, you could try something like this
$query = 'SELECT * FROM "sf_bands"'; // NOTE: double quotes on "sf_Bands"
This is realy helpfull
SET search_path TO schema,public;
I digged this issues more, and found out about how to set this "search_path" by defoult for a new user in current database.
Open DataBase Properties then open Sheet "Variables"
and simply add this variable for your user with actual value.
So now your user will get this schema_name by defoult and you could use tableName without schemaName.
You must write schema name and table name in qutotation mark. As below:
select * from "schemaName"."tableName";
I had the same issue as above and I am using PostgreSQL 10.5.
I tried everything as above but nothing seems to be working.
Then I closed the pgadmin and opened a session for the PSQL terminal.
Logged into the PSQL and connected to the database and schema respectively :
\c <DATABASE_NAME>;
set search_path to <SCHEMA_NAME>;
Then, restarted the pgadmin console and then I was able to work without issue in the query-tool of the pagadmin.
For me the problem was, that I had used a query to that particular table while Django was initialized. Of course it will then throw an error, because those tables did not exist. In my case, it was a get_or_create method within a admin.py file, that was executed whenever the software ran any kind of operation (in this case the migration). Hope that helps someone.
In addition to Bill Karwin's answer =>
Yes, you should surround the table name with double quotes. However, be aware that most probably php will not allow you to just write simply:
$query = "SELECT * FROM "SF_Bands"";
Instead, you should use single quotes while surrounding the query as sav said.
$query = 'SELECT * FROM "SF_Bands"';
You have to add the schema first e.g.
SELECT * FROM place.user_place;
If you don't want to add that in all queries then try this:
SET search_path TO place;
Now it will works:
SELECT * FROM user_place;
Easiest workaround is Just change the table name and all column names to lowercase and your issue will be resolved.
For example:
Change Table_Name to table_name and
Change ColumnName to columnname
It might be silly for a few, but in my case - once I created the table I could able to query the table on the same session, but if I relogin with new session table does not exits.
Then I used commit just after creating the table and now I could able to find and query the table in the new session as well. Like this:
select * from my_schema.my_tbl;
Hope this would help a few.
Make sure that Table name doesn't contain any trailing whitespaces
Try this: SCHEMA_NAME.TABLE_NAME
I'd suggest checking if you run the migrations or if the table exists in the database.
I tried every good answer ( upvote > 10) but not works.
I met this problem in pgAdmin4.
so my solution is quite simple:
find the target table / scheme.
mouse right click, and click: query-tool
in this new query tool window, you can run your SQL without specifying set search_path to <SCHEMA_NAME>;
you can see the result:
We've got a system (MS SQL 2008 R2-based) that has a number of "input" database and a one "output" database. I'd like to write a query that will read from the output DB, and JOIN it to data in one of the source DB. However, the source table may be one or more individual tables :( The name of the source DB is included in the output DB; ideally, I'd like to do something like the following (pseudo-SQL ahoy)
select o.[UID]
,o.[description]
,i.[data]
from [output].dbo.[description] as o
left join (select [UID]
,[data]
from
[output.sourcedb].dbo.datatable
) as i
on i.[UID] = o.[UID];
Is there any way to do something like the above - "dynamically" specify the database and table to be joined on for each row in the query?
Try using the exec function, then specify the select as a string, adding variables for database names and tables where appropriate. Simple example:
DECLARE #dbName VARCHAR(255), #tableName VARCHAR(255), #colName VARCHAR(255)
...
EXEC('SELECT * FROM ' + #dbName + '.dbo.' + #tableName + ' WHERE ' + #colName + ' = 1')
No, the table must be known at the time you prepare the query. Otherwise how would the query optimizer know what indexes it might be able to use? Or if the table you reference even has an UID column?
You'll have to do this in stages:
Fetch the sourcedb value from your output database in one query.
Build an SQL query string, interpolating the value you fetched in the first query into the FROM clause of the second query.
Be careful to check that this value contains a legitimate database name. For instance, filter out non-alpha characters or apply a regular expression or look it up in a whitelist. Otherwise you're exposing yourself to a SQL Injection risk.
Execute the new SQL string you built with exec() as #user353852 suggests.