I am using the below flow.
ODATA -> Blob storage (JSON)
JSON -> Snowflake table
Copy Data -> Copy Data - Lookup
Both copy data is working fine.
In the lookup (query), i have given. (Need to add 1 value in table, its a variant column)
Update T1 set source_json = object_insert(source_json,device_type,web_browser,TRUE);)
When i use the above query in snowflake database it works fine, the table has 25K rows.
When run from pipeline, it gives the below error.
Multiple SQL statements in a single API call are not supported; use one API call per statement instead.
Any suggestions please.
Some of the workarounds are provided below.
Execute multiple SQL files using SnowSql (command line utility) as described below:
snowsql -c cc -f file1.sql -f file2.sql -f file3.sql
Once we have downloaded and installed the snowsql tool, we can wrap up all our SQL queries in a single .sql file and call that file using bash.
For example, suppose that we have written all the queries which we would like to run around in a file named abc.sql stored in /tmp.
We can then run the following command:
snowsql -a enter_accountname -u enter_your_username -f /tmp/abc.sql
For reference:
Workaround for multiple sql statement in a single api call are not supported
Multiple single api call are not supported use one api call per statement instead
Thanks for the reply. Requirement got changed.
Our flow
Lookup1 -> Copy data -> Copy data > Lookup2
We passed the values from the lookup1 and ran the stored procedure.
Related
I'm familiarizing myself with the standalone version of Datagrip and having a bit of trouble understanding the different approaches to composing SQL via console, external files, scratch files, etc.
I'm managing, referencing the documentation, and am happy to figure things out as such.
However, I'm trying to ingest CSV data into tables via batch files using the Postgres \copy command. Datagrip will execute this command without error but no data is being populated.
This is my syntax, composed and ran in the console view:
\copy tablename from 'C:\Users\username\data_file.txt' WITH DELIMITER E'\t' csv;
Note that the data is tab-separated and stored in a .txt file.
I'm able to use the import functions of Datagrip (via context menu) just fine but I'd like to understand how to issue commands to do similarly.
\copy is a command of the command-line PostgreSQL client psql.
I doubt that Datagrip invokes psql, so it won't be able to use \copy or any other “backslash command”.
You probably have to use Datagrip's import facilities. Or you start using psql.
Ok, but what about the SQL COPY command https://www.postgresql.org/docs/12/sql-copy.html ?
How can I run something like that with datagrip ?
BEGIN;
CREATE TEMPORARY TABLE temp_json(values text) ON COMMIT DROP;
COPY temp_json FROM 'MY_FILE.JSON';
SELECT values->>'aJsonField' as f
FROM (select values::json AS values FROM temp_json) AS a;
COMMIT;
I try to replace 'MY_FILE.JSON' with full path, parameter (?), I put it in sql directory etc.
The data grip answer is :
[2021-05-05 10:30:45] [58P01] ERROR: could not open file '...' for reading : No such file or directory
EDIT :
I know why. RTFM! -_-
COPY with a file name instructs the PostgreSQL server to directly read from or write to a file. The file must be accessible by the PostgreSQL user (the user ID the server runs as) and the name must be specified from the viewpoint of the server.
Sorry.....
I have a table on pgadmin4 which consist of 100.000 lines and 23 columns.I need to benchmark postgresql on this specific table using pgbench,but i cant understand what parameters should i use.The database name is desdb and table called test.
PgAdmin4 is not a database server, it is a client. You don't have tables "on" pgadmin4, pgadmin4 is just one way of accessing tables which are on an actual server.
You don't benchmark tables, you benchmark queries. Knowing nothing about the table other than its name, all I could propose for a query is something like:
select * from test
Or
select count(*) from test
You could put that in a file test.sql, then run:
pgbench -n -f test.sql -T60 -P5 desdb
If you are like me and don't like littering your filesystem with bunches of tiny files with contents of no particular interest and you if use the bash shell, you could not create a test.sql file and instead make it dynamic:
pgbench -n -f <(echo 'select * from test') -T60 -P5 desdb
Whether that is a meaningful query to be benchmarking, I don't know. Do you care about how fast you can read (and then throw away) all columns for all rows in the table?
you can refer details regarding pgbench from : https://www.cloudbees.com/blog/tuning-postgresql-with-pgbench.
I have multiple tables that are created in the same way (same columns, indexes, etc.)
I would like to have one sql file for creating them all without duplicating the create statements.
Is there a way to use some kind of placeholder in sql file which would be substituted when executing the sql file with a parameter?
For example I would like to have below sql statement:
drop table if exists schema.%PLACEHOLDER%;
create table schema.%PLACEHOLDER%(id text, data text);
And execute such script with:
psql -f mysqlfile.sql -magic_parameter my_desired_table_name
Is this possible when executing PostgreSQL sql files, or maybe other way to achieve the same (except using sed)?
Sincr you are using psql, you can use variables as follows:
drop table if exists schema.:placeholder;
The invocation is:
psql -f mysqlfile.sql -v placeholder=table_name
I created a mapping that pulls data from a flat file that shows me usage data for specific SSRS reports. The file is overwritten each day with the previous days usage data. My issue is, sometimes the report doesn't have any usage for that day and my ETL sends me a "Failed" email because there wasn't any data in the Source. The job from running if there is no data in the source or to prevent it from failing.
--Thanks
A simple way to solve this is to create a "Passthrough" mapping that only contains a flat file source, source qualifier, and a flat file target.
You would create a session that runs this mapping at the beginning of your workflow and have it read your flat file source. The target can just be a dummy flat file that you keep overwriting. Then you would have this condition in the link to your next session that would actually process the file:
$s_Passthrough.SrcSuccessRows > 0
Yes, there are several ways, you can do this.
You can provide an empty file to ETL job when there is no source data. To do this, use a pre-session command like touch <filename> in the Informatica workflow. This will create an empty file with the <filename> if it is not present. The workflow will run successfully with 0 rows.
If you have a script that triggers the Informatica job, then you can put a check there as well like this:
if [ -e <filename> ]
then
pmcmd ...
fi
This will skip the job from executing.
Have another session before the actual dataload. Read the file, use a FALSE filter and some dummy target. Link this one to the session you already have and set the following link condition:
$yourDummySessionName.SrcSuccessRows > 0
I am trying to export a row to a DEL file using;
EXPORT TO ExportedFile.DEL OF DEL SELECT * From MyTable
However, when I execute the query I get this;
Error: SQL0104N An unexpected token "ExportedFile" was found following "EXPORT TO ". Expected tokens may include: "JOIN". SQLSTATE=42601
I understand that Export is not a supported command in WinSQL but is there a supported command?
EXPORT is a utility – not an SQL statement – and is handled by the DB2 command line processor.
IBM have made it possible to execute the EXPORT utility in an SQL statement by using the ADMIN_CMD stored procedure:
call sysproc.admin_cmd('export to /tmp/file.del of del select * from table')
Note that, unlike using the EXPORT utility directly on your machine with the DB2 client, this will write the file to the specified path on the database server. You will have to retrieve the file from the server manually.
WinSQL only passes its information to the back end databases via ODBC. Therefore, I'd like to add to #Ian Bjorhovde's answer and recommend that you enter that call into the query window for DB2 in WinSQL, namely:
call sysproc.admin_cmd('export to /tmp/file.del of del select * from table')