I am attempting to automate the copy of files from S3 to Redshift. Standing in my way is my ignorance of how to make the date (in bold below) always be today's date.
COPY cid_localytics_export
FROM 's3://exports.localytics.t-z/<redacted>/**2014/11/19**/eb0ecf515724b3ef5a6d8cb-a8a0fa68-3551-11e4-4b45-00a426b17dd8/00.log.gz'
CREDENTIALS 'aws_access_key_id=<redacted>;aws_secret_access_key=<redacted>'
JSON 's3://exports.localytics.t-z/<redacted>/redshift_example.json'
GZIP
TRUNCATECOLUMNS;
When I attempt the following I get a syntax error. Any ideas or suggestions would be greatly appreciated.
COPY cid_localytics_export_test
FROM (select ''''||'s3://exports.localytics.t-z/<redacted>/'||(select ltrim(rtrim(replace(replace(cast(date_trunc('day', now()) - interval '1 day' as varchar),'-','/'),' 00:00:00+00',''))))||'/eb0ecf515724b3ef5a6d8cb-a8a0fa68-3551-11e4-4b45-00a426b17dd8/00.log.gz'||'''')
CREDENTIALS 'aws_access_key_id=<redacted>;aws_secret_access_key=<redacted>'
JSON 's3://exports.localytics.t-z/whitepages/redshift_example.json'
GZIP
TRUNCATECOLUMNS;
Redshift copy command expects exact s3 path for folder or file (s3://abc/def or s3://abc/def/ijk.csv )
You need to give correct path for the file.
You can write simple java /php or shell script to ( postgres drivers) create s3 path using date dynamically build query and then fire. So that you replace it with actual date value.
So that there will be on syntax errors.
Related
I am using the below flow.
ODATA -> Blob storage (JSON)
JSON -> Snowflake table
Copy Data -> Copy Data - Lookup
Both copy data is working fine.
In the lookup (query), i have given. (Need to add 1 value in table, its a variant column)
Update T1 set source_json = object_insert(source_json,device_type,web_browser,TRUE);)
When i use the above query in snowflake database it works fine, the table has 25K rows.
When run from pipeline, it gives the below error.
Multiple SQL statements in a single API call are not supported; use one API call per statement instead.
Any suggestions please.
Some of the workarounds are provided below.
Execute multiple SQL files using SnowSql (command line utility) as described below:
snowsql -c cc -f file1.sql -f file2.sql -f file3.sql
Once we have downloaded and installed the snowsql tool, we can wrap up all our SQL queries in a single .sql file and call that file using bash.
For example, suppose that we have written all the queries which we would like to run around in a file named abc.sql stored in /tmp.
We can then run the following command:
snowsql -a enter_accountname -u enter_your_username -f /tmp/abc.sql
For reference:
Workaround for multiple sql statement in a single api call are not supported
Multiple single api call are not supported use one api call per statement instead
Thanks for the reply. Requirement got changed.
Our flow
Lookup1 -> Copy data -> Copy data > Lookup2
We passed the values from the lookup1 and ran the stored procedure.
I have a Crate.io database, running CrateDB version 3.2.7, under Windows Server 2012. (I know it's not the best, it's only for testing purposes right now and will not be the final setup.)
I have create the table dbo.snapshots
I exported the data from SQL Server to a CSV file via the BCP command.
bcp DatabaseName.dbo.Snapshots out F:\Path\dbo_OldSnapshots.dat -S ServerName -U UserName -P Password -a65535 -c -C 65001 -t ,
Then I tried to import the data into CrateDb with the "COPY FROM" command.
COPY dbo.snapshots FROM 'file:///F:/Path/dbo_OldSnapshots.dat';
The file is about 11go big. I know it found the file, as I could see the I/O on the drive in Task Manager.
It ran for about 13 minutes, and then said "0 rows affected". I have no idea why it didn't work, I didn't get any errors.
Any idea what I can do to make it work?
********************************* EDITED ADDED ADDITIONAL INFO ****************************
Ok, so I've found out that you can specify the "RETURN SUMMARY" clause at the end of the COPY command. I tested it with a smaller file.
With that, I got an error that says that the primary key cannot be NULL. I know that it's not NULL in the data that I extract, so I'm gonna have to find out why it says that my primary key is NULL.
So I changed the BCP delimiter to be a comma, since the CSV file for CrateDB must be comma separated, and I manually edited the file to add the column headers since CrateDB asks for a header.I also edited the file in Notepadd++ to save it in UTF-8 encoding, to make sure it was the right encoding.But even with all that, I still get the error saying that the primary key value must not be NULL.
Ok so I managed to make it work. Here is what you need to check if you try to Export data from SQL or another dbms to CrateDB :
- File encoding in UTF-8
- Comma separated file
- The first line needs to be a header with all the columns be carefull, the names are case sensitive so if the column is "MyColumn" in SQL Server, but "mycolumn" in CrateDB, it must be in lower case in the header or else CrateDb won't be able to find it correctly
- If you have DateTime types, it must be between "" in the file (ex: 1,0,"2019-05-10T16:40:00",0,0)
- If you have DateTime types, please note that you need to have T between the date and time part. So "2019-05-10T16:40:00" instead of "2019-05-10 16:40:00"
With all of that in check, I was able to import a sample data in my CrateDB database.
I am using pgAdmin version 4.3 and i want to export one table data to CSV file. I used this query
COPY (select * from product_template) TO 'D:\Product_template_Output.csv' DELIMITER ',' CSV HEADER;
but it shows error
a relative path is not allowed to use COPY to a file
How can I resolve this problem any help please ?
From the query editor, once you have executed your query, you just have to click on the "Download as CSV (F8)" button or use F8 key.
Source pgAdmin 4 Query Toolbar
Use absolute paths or cd to a known location so that you can ignore the path.
For example cd into documents directory then run the commands there.
If you are able to cd into your documents directory, then the command would be like this:
Assuming you are want to use PSQL from the command line.
cd ~/Documents && psql -h host -d dbname -U user
\COPY (select * from product_template) TO 'Product_template_Output.csv' DELIMITER ',' CSV HEADER;
The result would be Product_template_Output.csv in your current working directory(Documents folder).
Again using psql.
You have to remove the double quotes:
COPY (select * from product_template) TO 'D:\Product_template_Output.csv'
DELIMITER ',' CSV HEADER;
If your PgAdmin instance resides in a remote server, the aforementioned solutions might not be handy for you if you do not have remote access to the server. In this case, simply select all the query data and copy it. Open an excel file and you could paste it. Simple !! Tweaked.
You might have tough time if your query result is too much though.
Try this command:
COPY (select * from product_template) TO 'D:\Product_template_Output.csv' WITH CSV;
In PgAdmin export option is available in file menu.Execute the query, and then we can view the data in the Output pane. Click on the menu FILE -> EXPORT from query window.
PSQL to export data
COPY noviceusers(code, name) FROM 'C:\noviceusers.csv' DELIMITER ',' CSV HEADER;
https://www.novicetechie.com/2019/12/export-postgresql-data-in-to-excel-file.html for reference.
Write your query to select data on the query tool and execute
Click on the download button on the pgAdmin top bar (selected in red)
Rename the file to your liking
Select which folder to save the file
Congrats!!!
We're copying S3 uploaded files to Redshift by Workbench/J in our batch as follows;
java -jar %WBJ_DIR%\sqlworkbench.jar -profile='%WBJ_PROF%' -script='%ScriptFile%' -displayResult=true
Script File is copy command, created precedingly, according to target tables and conditions(used as file name, 'yyyy-mm-dd_TABLE_NAME.gz').
So far, uploading task is going successfully, but we'd like to add any error captures to confirm upload result on Redshift.
Just wondering if there is any option like 'SET NoCount ON' in SQL Server.
If only get record count, with no header and no message, I think we can get the upload error count from STL_LOAD_ERRORS on our batch like this;
--Check SQL
set COMMAND=select count(*) from STL_LOAD_ERRORS where filename='%COPY_FILE%' and starttime > '%bat_start_time%';
-- Get Error Count
for /F "delims=" %%A in ('%COMMAND%') do (
set REC_CNT=%%A
)
echo Upload Error Count : %REC_CNT%
Of cource, any other good way to confirm the uploaded result on Redshift would be appreciated.
Thank you so much for your useful advice in advance.
I need to export the resulting data from a query in PostgreSQL to Excel/CSV.
I use PostgreSQL 8.2.11.
SQL error:
ERROR: relative path not allowed for COPY to file
In statement:
COPY (select distinct(m_price) from m_product)TO '"c:\auto_new.txt"';
Example with Unix-style file name:
COPY (SELECT * FROM tbl) TO '/var/lib/postgres/myfile1.csv' format csv;
Read the manual about COPY (link to version 8.2).
You have to use an absolute path for the target file. Be sure to double quote file names with spaces. Example for MS Windows:
COPY (SELECT * FROM tbl)
TO E'"C:\\Documents and Settings\\Tech\Desktop\\myfile1.csv"' format csv;
In PostgreSQL 8.2, with standard_conforming_strings = off per default, you need to double backslashes, because \ is a special character and interpreted by PostgreSQL. Works in any version. It's all in the fine manual:
filename
 The absolute path name of the input or output file. Windows users might need to use an E'' string and double backslashes used as path separators.
Or the modern syntax with standard_conforming_strings = on (default since Postgres 9.1):
COPY tbl -- short for (SELECT * FROM tbl)
TO '"C:\Documents and Settings\Tech\Desktop\myfile1.csv"' (format csv);
Or you can also use forward slashes for filenames under Windows.
An alternative is to use the meta-command \copy of the default terminal client psql.
You can also use a GUI like pgadmin and copy / paste from the result grid to Excel for small queries.
Closely related answer:
Copy results from a PostgreSQL view in one DB to a table in another
Similar solution for MySQL:
Exporting MYSQL data into Excel/CSV via php
In PostgreSQL 9.4 to create to file CSV with the header in Ubuntu:
COPY (SELECT * FROM tbl) TO '/home/user/Desktop/result_sql.csv' WITH CSV HEADER;
Note: The folder must be writable.
This worked for me:
COPY (SELECT * FROM table)
TO E'C:\\Program Files (x86)\\PostgreSQL\\8.4\\data\\try.csv';
In my case the problem was with the writing permission to a special folder (though I work as administrator), after changing the path to the original data folder under PostgreSQL I had success.
Several GUI tools like Squirrel, SQL Workbench/J, AnySQL, ExecuteQuery can export to Excel files.
Most of those tools are listed in the PostgreSQL wiki:
http://wiki.postgresql.org/wiki/Community_Guide_to_PostgreSQL_GUI_Tools
If you have error like "ERROR: could not open server file "/file": Permission denied" you can fix it that:
Ran through the same problem, and this is the solution I found:
Create a new folder (for instance, tmp) under /home
$ cd /home
make postgres the owner of that folder
$ chown -R postgres:postgres tmp
copy in tmp the files you want to write into the database, and make sure they also are owned by postgres.
That's it. You should be in business after that.
The correct script for postgres (Ubuntu) is:
COPY (SELECT * FROM tbl) TO '/var/lib/postgres/myfile1.csv';