I am using the following script:
UNLOAD ('SELECT * FROM table WHERE
UPPER(description) LIKE \'%something%\' AND
transaction_date=\'2014-01-01\'')
TO 's3://yourdirectory/jan14.txt' CREDENTIALS
'aws_access_key_id=xxxx;
aws_secret_access_key=xxx'
PARALLEL OFF
But it returns this error:
An error occurred when executing the SQL command:
UNLOAD ('SELECT * FROM yi_base_views.card_panel WHERE
UPPER(description) LIKE \'%LULULEMON%\' AND
transaction_date=\'2014-01-01\'')
TO 's3://yourdirec...
ERROR: Invalid credentials. Must be of the format: credentials 'aws_access_key_id=;aws_secret_access_key=[;token=]'
Detail:
error: Invalid credentials. Must be of the format: credentials 'aws_access_key_id=;aws_secret_access_key=[;token=]'
code: 8001
context:
query: 0
location: aws_credentials_parser.cpp:59
process: padbmaster [pid=43295]
Remove the line break between the access key and secret key and it should work.
Related
I'm running pg_dump to create a script to automate the creation of a system like this:
pg_dump --dbname=postgresql://postgres:ohdsi#127.0.0.1:5432/OHDSI -t webapi.* > webapi.sql
This creates a sql script, but it is not really a sql script as it has code in it like what is shown below.
When this script is run as a sql script, it fails giving the error shown below.
Is there a way to get pg_dump to create a sql script that is standard sql and can be executed as a sql script?
Code sample from sql generated by pg_dump:
COPY webapi.cohort_version (asset_id, comment, description, version, asset_json, archived, created_by_id, created_date) FROM stdin;
\.
--
-- Data for Name: concept_of_interest; Type: TABLE DATA; Schema: webapi; Owner: ohdsi_admin_user
--
COPY webapi.concept_of_interest (id, concept_id, concept_of_interest_id) FROM stdin;
1 4329847 4185932
2 4329847 77670
3 192671 4247120
4 192671 201340
Error seen when running the script generated by pg_dump:
--
-- Name: penelope_laertes_uni_pivot id; Type: DEFAULT; Schema: webapi; Owner: ohdsi_admin_user
--
ALTER TABLE ONLY webapi.penelope_laertes_uni_pivot ALTER COLUMN id SET DEFAULT nextval('webapi.penelope_laertes_uni_pivot_id_seq'::regclass)
--
-- Data for Name: achilles_cache; Type: TABLE DATA; Schema: webapi; Owner: ohdsi_admin_user
--
COPY webapi.achilles_cache (id, source_id, cache_name, cache) FROM stdin
Error executing: COPY webapi.achilles_cache (id, source_id, cache_name, cache) FROM stdin
. Cause: org.postgresql.util.PSQLException: ERROR: COPY from stdin failed: COPY commands are only supported using the CopyManager API.
Where: COPY achilles_cache, line 1
Exception in thread "main" java.lang.RuntimeException: org.apache.ibatis.jdbc.RuntimeSqlException: Error executing: COPY webapi.achilles_cache (id, source_id, cache_name, cache) FROM stdin
. Cause: org.postgresql.util.PSQLException: ERROR: COPY from stdin failed: COPY commands are only supported using the CopyManager API.
Where: COPY achilles_cache, line 1
at org.yaorma.database.Database.executeSqlScript(Database.java:344)
at org.yaorma.database.Database.executeSqlScript(Database.java:332)
at org.nachc.tools.fhirtoomop.tools.build.postgres.build.A04_CreateAtlasWebApiTables.exec(A04_CreateAtlasWebApiTables.java:29)
at org.nachc.tools.fhirtoomop.tools.build.postgres.build.A04_CreateAtlasWebApiTables.main(A04_CreateAtlasWebApiTables.java:19)
Caused by: org.apache.ibatis.jdbc.RuntimeSqlException: Error executing: COPY webapi.achilles_cache (id, source_id, cache_name, cache) FROM stdin
. Cause: org.postgresql.util.PSQLException: ERROR: COPY from stdin failed: COPY commands are only supported using the CopyManager API.
Where: COPY achilles_cache, line 1
at org.apache.ibatis.jdbc.ScriptRunner.executeLineByLine(ScriptRunner.java:109)
at org.apache.ibatis.jdbc.ScriptRunner.runScript(ScriptRunner.java:71)
at org.yaorma.database.Database.executeSqlScript(Database.java:342)
... 3 more
Caused by: org.postgresql.util.PSQLException: ERROR: COPY from stdin failed: COPY commands are only supported using the CopyManager API.
Where: COPY achilles_cache, line 1
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2675)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2365)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:355)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:490)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:408)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:329)
at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:315)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:291)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:286)
at org.apache.ibatis.jdbc.ScriptRunner.executeStatement(ScriptRunner.java:190)
at org.apache.ibatis.jdbc.ScriptRunner.handleLine(ScriptRunner.java:165)
at org.apache.ibatis.jdbc.ScriptRunner.executeLineByLine(ScriptRunner.java:102)
... 5 more
--- EDIT ------------------------------------
The --inserts method in the accepted answer gave me exactly what I needed.
I ended up doing this:
pg_dump --inserts --dbname=postgresql://postgres:ohdsi#127.0.0.1:5432/OHDSI -t webapi.* > webapi.sql
The client tool you are using to restore the dump cannot deal with the data from the (nonstandard) COPY command being mixed into the script. You need psql to restore such a dump.
You can use the --inserts option of pg_dump to create a dump that contains INSERT statements rather than COPY. That will be slower to restore, but will work with more client tools.
However, your wish to get a standard SQL script is hopeless. PostgreSQL extends the standard in many ways, so a database cannot be dumped with a standard SQL script. Note, for example, that indexes are not defined by the SQL standard. If you are looking to transfer a PostgreSQL dump to a different RDBMS, you will be disappointed. That is more difficult.
Hi I was using this code to extract data from a column it was working, but I am now getting an error message. Any input is appreciated.
SQL Error [XX000]: ERROR: JSON parsing error
Detail:
error: JSON parsing error
code: 8001
context: invalid json object false
query: 6047879
location: funcs_json.hpp:127
process: query1_117_6047879 [pid=6614]
select json_extract_path_text (info,'device_category') as device_category
from admin.project
I have a database dump, and I need to import it into a new empty database.
COPY public.accounts_account (id, username, password, first_name, last_name, street_address, city, state, zip, daytime_phone, evening_phone, email, membership, total_purchase_amount, current_discount, registered_at, membership_approved) FROM stdin;
53 user53 password53 Name53 Last53 afd462740737a3801e90c6d050e81b88 Wilmette IL 60091 123.456.786 user53#obfuscated.com 7590 102.00 0 2011-03-24 03:52:23+00 t
I get this error:
ERROR: syntax error at or near "53"
LINE 4566: 53 user53 password53 Name53 Last53 afd462740737a3801e90c6d05...
^
********** Error **********
ERROR: syntax error at or near "53"
SQL state: 42601
Character: 132900
Your COPY command failed for some reason (table doesn't exist, column doesn't exist, you don't have permissions to insert into it, etc.). Since PostgreSQL did not go into COPY mode, it tried to interpret the next line as another command, rather than as data. Look earlier in your log file to see what the initial error is.
This database dump looks like a query you can run from PG Admin's Query Tool but it is not and that is why your getting this error.
Copy paste everything in that database dump into a PSQL session and it should run correctly.
You can usually launch a PSQL session from PG Admin from the Tool drop down.
PostgreSQL syntax problem.
Query below will run but result is:
ERROR: out of memory for query result
SELECT
AD.ADDRESS_DETAIL_PID as ADDRESS_DETAIL_PID,
AD.STREET_LOCALITY_PID as STREET_LOCALITY_PID,
AD.LOCALITY_PID as LOCALITY_PID,
AD.BUILDING_NAME as BUILDING_NAME
When the query is amended to:
set FETCH_COUNT=1000
SELECT
AD.ADDRESS_DETAIL_PID as ADDRESS_DETAIL_PID,
AD.STREET_LOCALITY_PID as STREET_LOCALITY_PID,
AD.LOCALITY_PID as LOCALITY_PID,
AD.BUILDING_NAME as BUILDING_NAME
the result is:
ERROR: syntax error at or near "SELECT" LINE 3: SELECT
^ SQL state: 42601 Character: 23
except that when a colon ; is included at the end the rssult is:
ERROR: unrecognized configuration parameter "fetch_count" SQL state:
42704
I am seeking help with the syntax.
FETCH_COUNT is a psql interval variable and is not executed on the server.
SET is a SQL command that's executed on the server and cannot be used for setting psql internal variables. use \set instead.
\set FETCH_COUNT 1000
I am attempting to insert data from an sql file which is about 1.6GB in size. I used the command:
\i filename.sql
however it had been running for well over ten minutes. Initially when I ran it I got output such as below:
psql:xxx.sql:1102: ERROR: role "xxx" does not exist
psql:xxx.sql:1124: ERROR: relation "xxx" already exists
psql:xxx.sql:1127: ERROR: role "xxx" does not exist
psql:xxx.sql:1138: ERROR: relation "xxx" already exists
psql:xxx.sql:1141: ERROR: role "xxx" does not exist
psql:xxx.sql:1177: ERROR: relation "xxx" already exists
psql:xxx.sql:1180: ERROR: role "xxx" does not exist
psql:xxx.sql:1191: ERROR: relation "xxx" already exists
psql:xxx.sql:1194: ERROR: role "xxx" does not exist
psql:xxx.sql:1223: ERROR: relation "xxx" already exists
psql:xxx.sql:1226: ERROR: role "xxx" does not exist
psql:xxx.sql:1237: ERROR: relation "xxx" already exists
psql:xxx.sql:1240: ERROR: role "xxx" does not exist
This continues until the final output was:
psql:xxx.sql:1183266: ERROR: duplicate key value violates unique constraint "xxx"
DETAIL: Key (id)=(2) already exists.
CONTEXT: COPY mo_logs, line 1: "2 +6421558671 +15759151518 LolWithPlus unable to deliver, dropping : undefined method `bytesize' for..."
Then the console shows nothing new since then. On my first attempt I thought it was hanging so exited but got the following output:
Cancel request sent
This leads me to believe the script is still running but I have no way of telling. Can anyone advise how I can either diagnose what is going on or give me an indication of whether the insert is failing as shown by it hanging.
It seems like the SQL script contains several COPY statements.
One of them (COPY TO mo_logs) failed, and another of them (COPY TO midn_protections) was running when you interrupted processing.
Normally psql continues working after an error and will just execute the next statement. To avoid that, start it with the option --set ON_ERROR_STOP=on.