I have about 6000 IPv6 addresses I need loaded into a postgres table. without even getting to the point of finding my 'if not exist error' it borks on the unescaped ':'
I don't know postgres that well, is there a LOAD DATA INFILE function that will read in the lines and ignore the ':' as well as look for existing records?
INSERT INTO ip_list (ip_addr)
SELECT 'ip_addr',
2600:3c01:e000:44:0:0:0:1,
2600:3c01:e000:44:0:0:0:2,
2600:3c01:e000:44:0:0:0:3,
2600:3c01:e000:44:0:0:0:4,
2600:3c01:e000:44:0:0:0:5,
2600:3c01:e000:44:0:0:0:7,
....
WHERE NOT EXIST(
SELECT 1 FROM ip_list WHERE name = 'ip_addr')
RETURNING id
);
ERROR: syntax error at or near ":"
LINE 3: 2600:3c01:e000:44:0:0:0:1,
Update:
This method never uploads any records:
You are now connected to database "postfix" as user "postgres".
postfix=# create temporary table t(ip_addr inet);
CREATE TABLE
postfix=# \copy t from '/var/www/localhost/htdocs/ipListScript'
postfix=# INSERT INTO ip_list(ip_addr)
select ip_addr from ip_list where
not exists (select 1 from ip_list where ip_list.ip_addr=ip_list.ip_addr);
Looks like you just need to put quotes around your IP addresses. Check out this pages for more info:
http://www.postgresql.org/docs/9.1/static/datatype-net-types.html
6000 addresses as constants in an SQL statement is unreasonable, besides your proposed syntax being incorrect anyway (a VALUES clause would be needed).
The reasonable way is to COPY into a temporary table from the file, and then insert from the temporary table to your final table, eliminating duplicates at the same time.
In psql:
create temporary table t(ip_addr inet);
\copy t from '/path/to/file.txt'
INSERT INTO ip_list(ip_addr)
select ip_addr from t where
not exists (select 1 from ip_list where ip_list.ip_addr=t.ip_addr);
The \copy is a command of the psql program that takes care of loading the file client-side and feeding it to the server.
If the context does not allow using the psql command, you may use the SQL statement COPY tablename FROM '/path/to/file.txt' but then you must be connected as superuser, and the file should be accessible on the server by the postgres user.
If these restrictions are not acceptable you want to use COPY tablename FROM STDIN and feed the data by your own means.
Related
I'm trying to run the following PHP script to do a simple database query:
$db_host = "localhost";
$db_name = "showfinder";
$username = "user";
$password = "password";
$dbconn = pg_connect("host=$db_host dbname=$db_name user=$username password=$password")
or die('Could not connect: ' . pg_last_error());
$query = 'SELECT * FROM sf_bands LIMIT 10';
$result = pg_query($query) or die('Query failed: ' . pg_last_error());
This produces the following error:
Query failed: ERROR: relation "sf_bands" does not exist
In all the examples I can find where someone gets an error stating the relation does not exist, it's because they use uppercase letters in their table name. My table name does not have uppercase letters. Is there a way to query my table without including the database name, i.e. showfinder.sf_bands?
From what I've read, this error means that you're not referencing the table name correctly. One common reason is that the table is defined with a mixed-case spelling, and you're trying to query it with all lower-case.
In other words, the following fails:
CREATE TABLE "SF_Bands" ( ... );
SELECT * FROM sf_bands; -- ERROR!
Use double-quotes to delimit identifiers so you can use the specific mixed-case spelling as the table is defined.
SELECT * FROM "SF_Bands";
Re your comment, you can add a schema to the "search_path" so that when you reference a table name without qualifying its schema, the query will match that table name by checked each schema in order. Just like PATH in the shell or include_path in PHP, etc. You can check your current schema search path:
SHOW search_path
"$user",public
You can change your schema search path:
SET search_path TO showfinder,public;
See also http://www.postgresql.org/docs/8.3/static/ddl-schemas.html
I had problems with this and this is the story (sad but true) :
If your table name is all lower case like : accounts
you can use: select * from AcCounTs and it will work fine
If your table name is all lower case like : accounts
The following will fail:
select * from "AcCounTs"
If your table name is mixed case like : Accounts
The following will fail:
select * from accounts
If your table name is mixed case like : Accounts
The following will work OK:
select * from "Accounts"
I dont like remembering useless stuff like this but you have to ;)
Postgres process query different from other RDMS. Put schema name in double quote before your table name like this, "SCHEMA_NAME"."SF_Bands"
Put the dbname parameter in your connection string. It works for me while everything else failed.
Also when doing the select, specify the your_schema.your_table like this:
select * from my_schema.your_table
If a table name contains underscores or upper case, you need to surround it in double-quotes.
SELECT * from "Table_Name";
I had a similar problem on OSX but tried to play around with double and single quotes. For your case, you could try something like this
$query = 'SELECT * FROM "sf_bands"'; // NOTE: double quotes on "sf_Bands"
This is realy helpfull
SET search_path TO schema,public;
I digged this issues more, and found out about how to set this "search_path" by defoult for a new user in current database.
Open DataBase Properties then open Sheet "Variables"
and simply add this variable for your user with actual value.
So now your user will get this schema_name by defoult and you could use tableName without schemaName.
You must write schema name and table name in qutotation mark. As below:
select * from "schemaName"."tableName";
I had the same issue as above and I am using PostgreSQL 10.5.
I tried everything as above but nothing seems to be working.
Then I closed the pgadmin and opened a session for the PSQL terminal.
Logged into the PSQL and connected to the database and schema respectively :
\c <DATABASE_NAME>;
set search_path to <SCHEMA_NAME>;
Then, restarted the pgadmin console and then I was able to work without issue in the query-tool of the pagadmin.
For me the problem was, that I had used a query to that particular table while Django was initialized. Of course it will then throw an error, because those tables did not exist. In my case, it was a get_or_create method within a admin.py file, that was executed whenever the software ran any kind of operation (in this case the migration). Hope that helps someone.
In addition to Bill Karwin's answer =>
Yes, you should surround the table name with double quotes. However, be aware that most probably php will not allow you to just write simply:
$query = "SELECT * FROM "SF_Bands"";
Instead, you should use single quotes while surrounding the query as sav said.
$query = 'SELECT * FROM "SF_Bands"';
You have to add the schema first e.g.
SELECT * FROM place.user_place;
If you don't want to add that in all queries then try this:
SET search_path TO place;
Now it will works:
SELECT * FROM user_place;
Easiest workaround is Just change the table name and all column names to lowercase and your issue will be resolved.
For example:
Change Table_Name to table_name and
Change ColumnName to columnname
It might be silly for a few, but in my case - once I created the table I could able to query the table on the same session, but if I relogin with new session table does not exits.
Then I used commit just after creating the table and now I could able to find and query the table in the new session as well. Like this:
select * from my_schema.my_tbl;
Hope this would help a few.
Make sure that Table name doesn't contain any trailing whitespaces
Try this: SCHEMA_NAME.TABLE_NAME
I'd suggest checking if you run the migrations or if the table exists in the database.
I tried every good answer ( upvote > 10) but not works.
I met this problem in pgAdmin4.
so my solution is quite simple:
find the target table / scheme.
mouse right click, and click: query-tool
in this new query tool window, you can run your SQL without specifying set search_path to <SCHEMA_NAME>;
you can see the result:
I am using psql with a PostgreSQL database and the following copy command:
\COPY isa (np1, np2, sentence) FROM 'c:\Downloads\isa.txt' WITH DELIMITER '|'
I get:
ERROR: extra data after last expected column
How can I skip the lines with errors?
You cannot skip the errors without skipping the whole command up to and including Postgres 14. There is currently no more sophisticated error handling.
\copy is just a wrapper around SQL COPY that channels results through psql. The manual for COPY:
COPY stops operation at the first error. This should not lead to problems in the event of a COPY TO, but the target table will
already have received earlier rows in a COPY FROM. These rows will
not be visible or accessible, but they still occupy disk space. This
might amount to a considerable amount of wasted disk space if the
failure happened well into a large copy operation. You might wish to
invoke VACUUM to recover the wasted space.
Bold emphasis mine. And:
COPY FROM will raise an error if any line of the input file contains
more or fewer columns than are expected.
COPY is an extremely fast way to import / export data. Sophisticated checks and error handling would slow it down.
There was an attempt to add error logging to COPY in Postgres 9.0 but it was never committed.
Solution
Fix your input file instead.
If you have one or more additional columns in your input file and the file is otherwise consistent, you might add dummy columns to your table isa and drop those afterwards. Or (cleaner with production tables) import to a temporary staging table and INSERT selected columns (or expressions) to your target table isa from there.
Related answers with detailed instructions:
How to update selected rows with values from a CSV file in Postgres?
COPY command: copy only specific columns from csv
It is too bad that in 25 years Postgres doesn't have -ignore-errors flag or option for COPY command. In this era of BigData you get a lot of dirty records and it can be very costly for the project to fix every outlier.
I had to make a work-around this way:
Copy the original table and call it dummy_original_table
in the original table, create a trigger like this:
CREATE OR REPLACE FUNCTION on_insert_in_original_table() RETURNS trigger AS $$
DECLARE
v_rec RECORD;
BEGIN
-- we use the trigger to prevent 'duplicate index' error by returning NULL on duplicates
SELECT * FROM original_table WHERE primary_key=NEW.primary_key INTO v_rec;
IF v_rec IS NOT NULL THEN
RETURN NULL;
END IF;
BEGIN
INSERT INTO original_table(datum,primary_key) VALUES(NEW.datum,NEW.primary_key)
ON CONFLICT DO NOTHING;
EXCEPTION
WHEN OTHERS THEN
NULL;
END;
RETURN NULL;
END;
Run a copy into the dummy table. No record will be inserted there, but all of them will be inserted in the original_table
psql dbname -c \copy dummy_original_table(datum,primary_key) FROM '/home/user/data.csv' delimiter E'\t'
Workaround: remove the reported errant line using sed and run \copy again
Later versions of Postgres (including Postgres 13), will report the line number of the error. You can then remove that line with sed and run \copy again, e.g.,
#!/bin/bash
bad_line_number=5 # assuming line 5 is the bad line
sed ${bad_line_number}d < input.csv > filtered.csv
[per the comment from #Botond_Balázs ]
Here's one solution -- import the batch file one line at a time. The performance can be much slower, but it may be sufficient for your scenario:
#!/bin/bash
input_file=./my_input.csv
tmp_file=/tmp/one-line.csv
cat $input_file | while read input_line; do
echo "$input_line" > $tmp_file
psql my_database \
-c "\
COPY my_table \
FROM `$tmp_file` \
DELIMITER '|'\
CSV;\
"
done
Additionally, you could modify the script to capture the psql stdout/stderr and exit
status, and if the exit status is non-zero, echo $input_line and the captured stdout/stderr to stdin and/or append it to a file.
I'm trying to run the following PHP script to do a simple database query:
$db_host = "localhost";
$db_name = "showfinder";
$username = "user";
$password = "password";
$dbconn = pg_connect("host=$db_host dbname=$db_name user=$username password=$password")
or die('Could not connect: ' . pg_last_error());
$query = 'SELECT * FROM sf_bands LIMIT 10';
$result = pg_query($query) or die('Query failed: ' . pg_last_error());
This produces the following error:
Query failed: ERROR: relation "sf_bands" does not exist
In all the examples I can find where someone gets an error stating the relation does not exist, it's because they use uppercase letters in their table name. My table name does not have uppercase letters. Is there a way to query my table without including the database name, i.e. showfinder.sf_bands?
From what I've read, this error means that you're not referencing the table name correctly. One common reason is that the table is defined with a mixed-case spelling, and you're trying to query it with all lower-case.
In other words, the following fails:
CREATE TABLE "SF_Bands" ( ... );
SELECT * FROM sf_bands; -- ERROR!
Use double-quotes to delimit identifiers so you can use the specific mixed-case spelling as the table is defined.
SELECT * FROM "SF_Bands";
Re your comment, you can add a schema to the "search_path" so that when you reference a table name without qualifying its schema, the query will match that table name by checked each schema in order. Just like PATH in the shell or include_path in PHP, etc. You can check your current schema search path:
SHOW search_path
"$user",public
You can change your schema search path:
SET search_path TO showfinder,public;
See also http://www.postgresql.org/docs/8.3/static/ddl-schemas.html
I had problems with this and this is the story (sad but true) :
If your table name is all lower case like : accounts
you can use: select * from AcCounTs and it will work fine
If your table name is all lower case like : accounts
The following will fail:
select * from "AcCounTs"
If your table name is mixed case like : Accounts
The following will fail:
select * from accounts
If your table name is mixed case like : Accounts
The following will work OK:
select * from "Accounts"
I dont like remembering useless stuff like this but you have to ;)
Postgres process query different from other RDMS. Put schema name in double quote before your table name like this, "SCHEMA_NAME"."SF_Bands"
Put the dbname parameter in your connection string. It works for me while everything else failed.
Also when doing the select, specify the your_schema.your_table like this:
select * from my_schema.your_table
If a table name contains underscores or upper case, you need to surround it in double-quotes.
SELECT * from "Table_Name";
I had a similar problem on OSX but tried to play around with double and single quotes. For your case, you could try something like this
$query = 'SELECT * FROM "sf_bands"'; // NOTE: double quotes on "sf_Bands"
This is realy helpfull
SET search_path TO schema,public;
I digged this issues more, and found out about how to set this "search_path" by defoult for a new user in current database.
Open DataBase Properties then open Sheet "Variables"
and simply add this variable for your user with actual value.
So now your user will get this schema_name by defoult and you could use tableName without schemaName.
You must write schema name and table name in qutotation mark. As below:
select * from "schemaName"."tableName";
I had the same issue as above and I am using PostgreSQL 10.5.
I tried everything as above but nothing seems to be working.
Then I closed the pgadmin and opened a session for the PSQL terminal.
Logged into the PSQL and connected to the database and schema respectively :
\c <DATABASE_NAME>;
set search_path to <SCHEMA_NAME>;
Then, restarted the pgadmin console and then I was able to work without issue in the query-tool of the pagadmin.
For me the problem was, that I had used a query to that particular table while Django was initialized. Of course it will then throw an error, because those tables did not exist. In my case, it was a get_or_create method within a admin.py file, that was executed whenever the software ran any kind of operation (in this case the migration). Hope that helps someone.
In addition to Bill Karwin's answer =>
Yes, you should surround the table name with double quotes. However, be aware that most probably php will not allow you to just write simply:
$query = "SELECT * FROM "SF_Bands"";
Instead, you should use single quotes while surrounding the query as sav said.
$query = 'SELECT * FROM "SF_Bands"';
You have to add the schema first e.g.
SELECT * FROM place.user_place;
If you don't want to add that in all queries then try this:
SET search_path TO place;
Now it will works:
SELECT * FROM user_place;
Easiest workaround is Just change the table name and all column names to lowercase and your issue will be resolved.
For example:
Change Table_Name to table_name and
Change ColumnName to columnname
It might be silly for a few, but in my case - once I created the table I could able to query the table on the same session, but if I relogin with new session table does not exits.
Then I used commit just after creating the table and now I could able to find and query the table in the new session as well. Like this:
select * from my_schema.my_tbl;
Hope this would help a few.
Make sure that Table name doesn't contain any trailing whitespaces
Try this: SCHEMA_NAME.TABLE_NAME
I'd suggest checking if you run the migrations or if the table exists in the database.
I tried every good answer ( upvote > 10) but not works.
I met this problem in pgAdmin4.
so my solution is quite simple:
find the target table / scheme.
mouse right click, and click: query-tool
in this new query tool window, you can run your SQL without specifying set search_path to <SCHEMA_NAME>;
you can see the result:
I am using PostgreSQL 9.0.3. I have an Excel spreadsheet with lots of data to load into couple of tables in Windows OS.
I have written the script to get the data from input file and Insert into some 15 tables. This can't be done with COPY or Import. I named the input file as DATALD.
I find out the psql command -d to point the db and -f for the script sql. But I need to know the commands how to feed the input file along with the script so that the data gets inserted into the tables..
For example this is what I have done:
begin
for emp in (select distinct w_name from DATALD where w_name <> 'w_name')
--insert in a loop
INSERT INTO tblemployer( id_employer, employer_name,date_created, created_by)
VALUES (employer_id,emp.w_name,now(),'SYSTEM1');
Can someone please help?
For an SQL script you must ..
either have the data inlined in your script (in the same file).
or you need to utilize COPY to import the data into Postgres.
I suppose you use a temporary staging table, since the format doesn't seem to fit the target tables. Code example:
How to bulk insert only new rows in PostreSQL
There are other options like pg_read_file(). But:
Use of these functions is restricted to superusers.
Intended for special purposes.
Is there any way, how to check, whether a variable has already been set in my environment?
Example:
\set table_name countries
\i queries.sql
queries.sql:
SELECT * FROM :table_name;
I want to make queries.sql to be called independently and use some default table name I would specify.
Is this possible or do I really need to create another SQL file through which I will call the queries (\i)?
My use case is usage of my SQL queries both in pgTAP unit tests (with some sample table names) and independently.
You could check the current value with:
SELECT :'table_name';
You can set it on the call to psql with something like --set='table_name' on the psql command line.