I'm trying to upload a lot of data from a .sql file using the COPY command for Postgresql.
I have those data in a file.sql in the following format :
COPY my_table(id, name, status) FROM stdin;
1 peter active
1 steve active
1 maria active
\.
And my changeset like this:
<changeSet id="sqlFile-example" author="me" >
<sqlFile encoding="UTF-8"
path="file.sql"
relativeToChangelogFile="true"
endDelimiter=";"
splitStatements="false"
/>
</changeSet>
And get this error:
[ERROR] Failed to execute goal
org.liquibase:liquibase-maven-plugin:3.6.3:update (default-cli) on
project lincoln-soft: Error setting up or running Liquibase: Migration
failed for change set
src/main/resources/db/liquibase/db-changelog.xml::sqlFile-example::me
[ERROR] Reason: liquibase.exception.DatabaseException: ERROR: unexpected message type 0x50 during COPY from stdin
[ERROR] Where:
COPY my_table, line 1 [Failed SQL: COPY my_table(id, name, status)
FROM stdin;
[ERROR] 1 peter active
[ERROR] 1 steve active
[ERROR] 1 maria active
[ERROR] \.]
I there a way to upload those data by liquibase?
Finally got a solution, as #a_horse_with_no_name and #Laurenz Albe
mentioned, can't use COPY FROM STDIN directly in JDBC, so I used pg_dump to generate insert statements like this:
pg_dump --table=public.my_table --data-only --column-inserts my_databse > /tmp/my_table_data.sql
It gives me a file my_table_data.sql with the inserts statements like this:
INSERT INTO public.my_table (id, name, status) VALUES (1,peter,active);
INSERT INTO public.my_table (id, name, status) VALUES (1,peter,active);
INSERT INTO public.my_table (id, name, status) VALUES (1,peter,active);
And then I use this liquibase Chageset to upload the sql file:
<changeSet id="sqlFile-example" author="me" >
<sqlFile encoding="UTF-8"
path="my_table_data.sql"
relativeToChangelogFile="true"
splitStatements="true"
stripComments="true"
/>
</changeSet>
It works for me
As Laurenz already mentioned: you can't use COPY FROM STDIN directly in JDBC (you can use the CopyManager API to implement that manually, but Liquibase doesn't support that and I also don't know of any plugin that would do that)
I would suggest you use Liquibase's built-in ability to load CSV (text) files. Put your input data in CSV file, e.g. my_table_data.txt with a header line for the columns:
id,name,status
1,peter,active
1,steve,active
1,maria,active
Then use <loadData> instead of running a SQL script:
<changeSet id="sqlFile-example" author="me" >
<loadData tableName="my_table"
file="my_table_data.txt"
separator=","
encoding="UTF-8">
</changeSet>
Mixing the COPY statement and the data in the same file only works in psql scripts.
Moreover, COPY FROM STDIN is not supported by the JDBC driver at all.
You should use INSERT statements in your script.
Related
I'm running pg_dump to create a script to automate the creation of a system like this:
pg_dump --dbname=postgresql://postgres:ohdsi#127.0.0.1:5432/OHDSI -t webapi.* > webapi.sql
This creates a sql script, but it is not really a sql script as it has code in it like what is shown below.
When this script is run as a sql script, it fails giving the error shown below.
Is there a way to get pg_dump to create a sql script that is standard sql and can be executed as a sql script?
Code sample from sql generated by pg_dump:
COPY webapi.cohort_version (asset_id, comment, description, version, asset_json, archived, created_by_id, created_date) FROM stdin;
\.
--
-- Data for Name: concept_of_interest; Type: TABLE DATA; Schema: webapi; Owner: ohdsi_admin_user
--
COPY webapi.concept_of_interest (id, concept_id, concept_of_interest_id) FROM stdin;
1 4329847 4185932
2 4329847 77670
3 192671 4247120
4 192671 201340
Error seen when running the script generated by pg_dump:
--
-- Name: penelope_laertes_uni_pivot id; Type: DEFAULT; Schema: webapi; Owner: ohdsi_admin_user
--
ALTER TABLE ONLY webapi.penelope_laertes_uni_pivot ALTER COLUMN id SET DEFAULT nextval('webapi.penelope_laertes_uni_pivot_id_seq'::regclass)
--
-- Data for Name: achilles_cache; Type: TABLE DATA; Schema: webapi; Owner: ohdsi_admin_user
--
COPY webapi.achilles_cache (id, source_id, cache_name, cache) FROM stdin
Error executing: COPY webapi.achilles_cache (id, source_id, cache_name, cache) FROM stdin
. Cause: org.postgresql.util.PSQLException: ERROR: COPY from stdin failed: COPY commands are only supported using the CopyManager API.
Where: COPY achilles_cache, line 1
Exception in thread "main" java.lang.RuntimeException: org.apache.ibatis.jdbc.RuntimeSqlException: Error executing: COPY webapi.achilles_cache (id, source_id, cache_name, cache) FROM stdin
. Cause: org.postgresql.util.PSQLException: ERROR: COPY from stdin failed: COPY commands are only supported using the CopyManager API.
Where: COPY achilles_cache, line 1
at org.yaorma.database.Database.executeSqlScript(Database.java:344)
at org.yaorma.database.Database.executeSqlScript(Database.java:332)
at org.nachc.tools.fhirtoomop.tools.build.postgres.build.A04_CreateAtlasWebApiTables.exec(A04_CreateAtlasWebApiTables.java:29)
at org.nachc.tools.fhirtoomop.tools.build.postgres.build.A04_CreateAtlasWebApiTables.main(A04_CreateAtlasWebApiTables.java:19)
Caused by: org.apache.ibatis.jdbc.RuntimeSqlException: Error executing: COPY webapi.achilles_cache (id, source_id, cache_name, cache) FROM stdin
. Cause: org.postgresql.util.PSQLException: ERROR: COPY from stdin failed: COPY commands are only supported using the CopyManager API.
Where: COPY achilles_cache, line 1
at org.apache.ibatis.jdbc.ScriptRunner.executeLineByLine(ScriptRunner.java:109)
at org.apache.ibatis.jdbc.ScriptRunner.runScript(ScriptRunner.java:71)
at org.yaorma.database.Database.executeSqlScript(Database.java:342)
... 3 more
Caused by: org.postgresql.util.PSQLException: ERROR: COPY from stdin failed: COPY commands are only supported using the CopyManager API.
Where: COPY achilles_cache, line 1
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2675)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2365)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:355)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:490)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:408)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:329)
at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:315)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:291)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:286)
at org.apache.ibatis.jdbc.ScriptRunner.executeStatement(ScriptRunner.java:190)
at org.apache.ibatis.jdbc.ScriptRunner.handleLine(ScriptRunner.java:165)
at org.apache.ibatis.jdbc.ScriptRunner.executeLineByLine(ScriptRunner.java:102)
... 5 more
--- EDIT ------------------------------------
The --inserts method in the accepted answer gave me exactly what I needed.
I ended up doing this:
pg_dump --inserts --dbname=postgresql://postgres:ohdsi#127.0.0.1:5432/OHDSI -t webapi.* > webapi.sql
The client tool you are using to restore the dump cannot deal with the data from the (nonstandard) COPY command being mixed into the script. You need psql to restore such a dump.
You can use the --inserts option of pg_dump to create a dump that contains INSERT statements rather than COPY. That will be slower to restore, but will work with more client tools.
However, your wish to get a standard SQL script is hopeless. PostgreSQL extends the standard in many ways, so a database cannot be dumped with a standard SQL script. Note, for example, that indexes are not defined by the SQL standard. If you are looking to transfer a PostgreSQL dump to a different RDBMS, you will be disappointed. That is more difficult.
I've used pg_dump --no-privileges --format custom --compress=0 some_database > my-dump.pgdump to dump a database, but I'm running into issues when I try to restore it.
Specifically, it appears to be loading function definitions before table definitions:
$ pg_restore ./my-dump.pgdump
…
create function my_function() returns …
language sql $$
select …
from some_table
where …
$$;
… later in the dump …
create table some_table ( … );
…
Which causes an error when I try to restore the dump:
pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 4863; 0 16735 TABLE DATA some_table some_database
pg_restore: [archiver (db)] COPY failed for table "some_table": ERROR: relation "some_table" does not exist
LINE 3: from some_table
^
QUERY:
select …
from some_table
where …
CONTEXT: SQL function "my_function" during inlining
What's going on here? How can I trick pg_dump / pg_restore into doing things in the correct order?
Check your dump file for commands which mess with search_path, for example:
SELECT pg_catalog.set_config('search_path', '', false);
I encountered the same kind error as you (relation xxx does not exist ... during inlining) in a legacy project I inherited, even though it's running PostgreSQL 9.4.x.
I traced it to the above command.
The solution for me was to remove this command from the dump file.
After I did this I was able to restore the database without errors.
Note: the OP is using the custom format. There is no editing the binary file that is emitted.
In my experience, pg_dump using the custom format (-Fc) doesn't set check_function_bodies = false. But since it adds random functions at the top of the dump file (instead of putting all routines at the end), this causes pg_restore to barf.
I was able to workaround this issue by setting PGOPTIONS:
export PGOPTIONS="-c check_function_bodies=false"
pg_restore ...
That is strange. Ever since commit ef88199f611e625b496ff92aa17a447d254b9796 in 2003, pg_dump and pg_restore have emitted
SET check_function_bodies = false;
This setting makes sure that an error like you describe won't happen, because PostgreSQL won't check the validity of the function bodies.
Are you using an ancient PostgreSQL version or are you doing anything else that could mess with that?
If you run pg_restore on your dump (without specifying a destination database), does it emit the line?
Below is my shell script, I am trying to insert columns in posrgresql using shell script.
But getting below error.
Script:
i='dcm_account494401_click_2017050511_20170505_093843_556195422.csv.gz'
load_date='2017-05-12'
load_status='Fail'
message="INFO: Load into table 'stg_ft_raw_activity' completed, 362554 record(s) loaded successfully.
INFO: Load into table 'stg_ft_raw_activity' completed, 1 record(s) were loaded with replacements made for ACCEPTINVCHARS. Check 'stl_replacements' system table for details.
"
psql "host=$HOST port=$DBPORT dbname=$DBNAME user=$DBUSER password=$DBPASS" -F --no-align <<EOF
truncate table stg.notification_table;
\set fname $i
\set load_date $load_date
\set load_status $load_status
\set message $message
insert into stg.notification_table values (:'fname', :'load_date', :'load_status',:"message");
EOF
error:
Expanded display is used automatically.
TRUNCATE TABLE and COMMIT TRANSACTION
ERROR: syntax error at or near "INFO"
LINE 1: INFO: Load into table 'stg_ft_raw_activity' completed, 1 re...
^
message col is string value and also contains spl chars. is that a reason?
Please help to resolve.
Thansk,
There is no need to use these variables with \set.
Here is an example:
message="INFO: Load into table 'stg_ft_raw_activity' completed, 362554 record(s) loaded successfully.
INFO: Load into table 'stg_ft_raw_activity' completed, 1 record(s) were loaded with replacements made for ACCEPTINVCHARS. Check 'stl_replacements' system table for details.
"
psql <<EOF
INSERT INTO message_table VALUES (\$\$$message\$\$);
EOF
This makes use of “dollar quoting” for string literals, with the $ signs that are no shell variable reference escaped with backslashes.
In usual migration I am using SQL usual COPY CSV command, but now all the SpringBoot project must be into a git and generic path for clone (and do it in a protected server):
where the best place or "standard Spring folder" to CSV files used by main/resources/db/migration?
how to use the COPY (SQL) with a relative path?
Relative path by shell is:
psql -h localhost -U postgres gcp -c "\
CREATE TABLE question_import (question text, weight integer); \
COPY question_import FROM STDIN WITH CSV HEADER delimiter as ',' \
" < _docs/data/csc-questoes.csv
but PostgreSQL not supports internal relative path.
Once you're using SpringBoot, why not incorporating the benefits of Liquibase (section 75.5.2)?
Apart from the ability to define/version your database schema, there are also methods for loading data, e.g. CSV in your case:
<changeSet author="liquibase-docs" id="loadUpdateData-example">
<loadUpdateData catalogName="cat"
encoding="UTF-8"
file="com/example/users.csv"
primaryKey="pk_id"
quotchar="A String"
schemaName="public"
separator="A String"
tableName="person">
<column name="address" type="varchar(255)"/>
</loadUpdateData>
The Spring-Boot-Liquibase Sample project should give you a quickstart.
I'm trying to print executed query and also result of the query on PostgreSQL. I have simple ant sql task, using standard postgresql JDBC driver.
<?xml version="1.0"?>
<project name="Ant SQL task" default="sql">
<target name="sql">
<echo>Ant SQL task</echo>
<sql
driver="org.postgresql.Driver"
url="jdbc:postgresql://hostname:port/database"
userid="user"
password="password"
src="script.sql"
print="yes"
>
</sql>
</target>
</project>
The script.sql containts just
select 1 as test;
I'm getting this output:
info:
[sql] Executing resource: script.sql
[sql] test
[sql] 1
[sql]
[sql] 0 rows affected
[sql] 1 of 1 SQL statements executed successfully
I would like to print the select query in the output like this:
info:
[sql] Executing resource: script.sql
[sql] select 1 as test;
[sql] test
[sql] 1
[sql]
[sql] 0 rows affected
[sql] 1 of 1 SQL statements executed successfully
Is there a way to include the select query in the output?
1.) Set loglevel to DEBUG (2) for the jdbc. Except now you don't have too little info in your log file, but too much.
2.) Append your queries a little:
SELECT 1 as foo, (SELECT query FROM pg_stat_activity);
foo | query
-----+--------------------------------------------------------
1 | SELECT 1 as foo, (SELECT query FROM pg_stat_activity);
(1 row)
Note that this will give you all running queries, not just the current one. You can narrow it down by one of the many columns in pg_stat_activity, but that's probably unnecessary unless this is a busy system. Set ApplicationName in your connection parameters if you decide to do that.
3.) Add lines to script.sql that manually repeat the next query:
SELECT 'SELECT 1;';
SELECT 1;
SELECT 'SELECT 2;';
SELECT 2;
4.) Ditch the ant sql module and use exec to run psql -e -f script.sql instead:
SELECT 1;
?column?
----------
1
(1 row)
You can omit the row count with -P footer=OFF.