Trying to import a .csv into my postgres table using the following approach:
System: WSL2 - UBUNTU 20.04
psql -d db_name --user=username -c "\copy test_list FROM 'testmngrs.csv' delimiter '|' csv;"
The content format of my .csv:
1,Name,name#store_id.com,1234567891,City Name
The error I'm receiving:
ERROR: invalid input syntax for type bigint:
CONTEXT: COPY test_list, line 1, column id:
The table:
SELECT * FROM test_list;
id | store_id | name | email | phone | city
The additional id at the head of the table above was not something created during my initial set up of the table.
My ecto migration file is as follows:
I'm not sure what's causing the BigInt error, nor how to avoid it as I copy over the data. I'm also a bit confused as to why there's an additional id column in my table given that it was never defined in my migration
I'm pretty new to postgresql and elixir / ecto so any assistance is greatly/guidance/context is greatly appreciated!
From the docs:
By default, the table will also include an :id primary key field that has a type of :bigserial.
Ecto assumes you want it to generate the id field by default. It's better to just go with it. But you can configure it somewhat counter-intuitively by setting primary_key: false on the table, and primary_key: true on the column:
create table(:managers, primary_key: false) do
add :store_id, :integer, null: false, primary_key: true
...
Related
I want to import the csv file into database table .but it was not working..
I run the bash shell in the linux env .
CREATE TABLE test.targetDB (
no int4 NOT NULL GENERATED ALWAYS AS IDENTITY,
year varchar(15) NOT NULL,
name bpchar(12) NOT NULL,
city varchar(15) NOT NULL,
ts_load timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (seq_no)
)
test.csv file
"2019","112 ","1123",2019-07-26-05.33.43.000000
Linux Cmd
psql -d $database -c " COPY test.targetDB from 'test.csv' delimiter ',' csv "
Error
ERROR: invalid input syntax for integer: "2019"
CONTEXT: COPY targetDB, line 1, column no: "2019"
How can I resolve this issue
You need to tell copy, that the no column is not part of the CSV file by specifying the columns that should be populated:
COPY test.targetDB(year, name, city, ts_load) from 'test.csv' delimiter ',' csv
I would recommend, using datagrip - a postgresql client tool. You can use a evaluation version if you don't wish to purchase. It's pretty simple from the UI to import a file rather using a command line.
Good day collegues.
I have very strange problem. I use docker-compose and part of my file looks like:
version: "2.1"
services:
db:
image: dr.mediascope.net/mediascope/wf/db:latest
container_name: workflow_db_test
ports:
- "15432:5432"
environment:
- POSTGRES_USER=dev
- POSTGRES_PASSWORD=dev
- POSTGRES_DB=wf-db
networks:
wf_newfusion:
aliases:
- wf-db
volumes:
- /root/opt/databus_newfusion/workflow_newfusion/create-schema-docker.sql:/docker-entrypoint-initdb.d/4-schema.sql
After starting docker i catch next err:
ERROR: type "json_param" does not exist at character 252
Part of my SQL, it works fine from Idea:
-- separated schema which is dedicated for API
DROP SCHEMA IF EXISTS api CASCADE;
CREATE SCHEMA api;
ALTER SCHEMA api
OWNER TO wfweb_docker;
DROP TYPE IF EXISTS json_param;
CREATE TYPE json_param AS (id TEXT, label TEXT);
ALTER TYPE json_param
OWNER TO wfweb_docker;
.......
SET search_path = core, pg_catalog;
CREATE OR REPLACE VIEW api.v_scheme_param_set AS
WITH cte AS (
SELECT
param_set_detail.param_set_id,
json_agg(row_to_json(
ROW (param_set_detail.param_name :: TEXT, param_set_detail.param_val :: TEXT) :: json_param)) AS set_values
FROM core.param_set_detail
GROUP BY param_set_detail.param_set_id
)
SELECT
s.scheme_id,
cte.param_set_id,
cte.set_values
FROM core.param_set s
JOIN cte ON s.param_set_id = cte.param_set_id
ORDER BY cte.param_set_id DESC;
If i use Idea Ultimate and execute it, all work fine! No err message and other! If i use it inside "psql" or when docker starts it falls with err. Before this script three other was run, but all ok.
Presumably you created the json_param type in the api schema. In that case you need to specify the schema name in your query when using that type, since you explicitly did not include that schema in your search path.
Also you don't need to include pg_catalog in your search path, as it's always searched.
I'm trying to build a table with csvsql.
When I use command:
csvsql --db mysql://user:password#localhost:3306/database_name --table table_name file.csv
I get the error:
(in table 'blabla', column 'xyz'): VARCHAR requires a length on dialect mysql
I've then tried to build a database schema and force it with --db-schema flag,
The db-schema format is:
CREATE TABLE table_name (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`x` varchar(29) DEFAULT NULL,
`y` int(10) NOT NULL DEFAULT '0',
`z` BOOL NOT NULL,
PRIMARY KEY (`id`),
KEY `indexed` (`indexed`)
);
but I still get the same error.
The complete command with db-schema is:
csvsql --db mysql://user:password#localhost:3306/database_name --table table_name --db-schema db_schema_filename csvfile.csv
I've read the manual for csvkit, but I don't get what I'm doing wrong.
This command should print the conversion result right?
Can someone please help?
Thank you.
Well, found the solution in the github.
https://github.com/wireservice/csvkit/issues/758#issue-201924611
After update from github, no more errors and tables are created normaly.
At amazon ec2 RDS Postgresql:
=> SHOW rds.extensions;
rds.extensions
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
btree_gin,btree_gist,chkpass,citext,cube,dblink,dict_int,dict_xsyn,earthdistance,fuzzystrmatch,hstore,intagg,intarray,isn,ltree,pgcrypto,pgrowlocks,pg_trgm,plperl,plpgsql,pltcl,postgis,postgis_tiger_geocoder,postgis_topology,sslinfo,tablefunc,tsearch2,unaccent,uuid-ossp
(1 row)
As you can see, uuid-ossp extension does exist. However, when I'm calling the function for generation uuid_v4, it fails:
CREATE TABLE my_table (
id uuid DEFAULT uuid_generate_v4() NOT NULL,
name character varying(32) NOT NULL,
);
What's wrong with this?
The extension is available but not installed in this database.
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
If the extension is already there but you don't see the uuid_generate_v4() function when you do a describe functions \df command then all you need to do is drop the extension and re-add it so that the functions are also added. Here is the issue replication:
db=# \df
List of functions
Schema | Name | Result data type | Argument data types | Type
--------+------+------------------+---------------------+------
(0 rows)
CREATE EXTENSION "uuid-ossp";
ERROR: extension "uuid-ossp" already exists
DROP EXTENSION "uuid-ossp";
CREATE EXTENSION "uuid-ossp";
db=# \df
List of functions
Schema | Name | Result data type | Argument data types | Type
--------+--------------------+------------------+---------------------------+--------
public | uuid_generate_v1 | uuid | | normal
public | uuid_generate_v1mc | uuid | | normal
public | uuid_generate_v3 | uuid | namespace uuid, name text | normal
public | uuid_generate_v4 | uuid | | normal
db=# select uuid_generate_v4();
uuid_generate_v4
--------------------------------------
b19d597c-8f54-41ba-ba73-02299c1adf92
(1 row)
What probably happened is that the extension was originally added to the cluster at some point in the past and then you probably created a new database within that cluster afterward. If that was the case then the new database will only be "aware" of the extension but it will not have the uuid functions added which happens when you add the extension. Therefore you must re-add it.
Looks like the extension is not installed in the particular database you require it.
You should connect to this particular database with
\CONNECT my_database
Then install the extension in this database
CREATE EXTENSION "uuid-ossp";
Step #1: re-install uuid-ossp extention into the exact schema:
If this is a fresh installation you can skip SET and DROP. Credits to #atomCode (details)
SET search_path TO public;
DROP EXTENSION IF EXISTS "uuid-ossp";
CREATE EXTENSION "uuid-ossp" SCHEMA public;
After this, you should see uuid_generate_v4() function IN THE RIGHT SCHEMA (when execute \df query in psql command-line prompt).
Step #2: use fully-qualified names (with schemaname. qualifier):
For example:
CREATE TABLE public.my_table (
id uuid DEFAULT public.uuid_generate_v4() NOT NULL,
If you've changed the search_path, specify the public schema in the function call:
public.uuid_generate_v4()
This worked for me.
create extension IF NOT EXISTS "uuid-ossp" schema pg_catalog version "1.1";
make sure the extension should by on pg_catalog and not in your schema...
Just add this code to the Beginning of your script
DROP EXTENSION IF EXISTS "uuid-ossp";
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
Maybe It was the same I was facing. The uuid_generate_v4 was from the public schema and I was trying to run it in a specific schema, so to fix it I did:
SET search_path TO specific_schema;
INSERTO INTO my_table VALUES public.uuid_generate_v4();
You can check the schema where your function is running:
\df uuid_generate_v4
Or
SELECT n.nspname, p.probin, p.proname
FROM
pg_proc p
LEFT JOIN pg_namespace n ON p.pronamespace = n.oid
WHERE p.proname like 'uuid_generate_v4';
You can check info related to the extension of the uuid-ossp like this:
SELECT * FROM pg_extension WHERE extname LIKE 'uuid-ossp';
You can add this extension case you don't have it already:
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
if you do it from unix command (apart from PGAdmin) dont forget to pass the DB as a parameter. otherwise this extension will not be enabled when executing requests on this DB
psql -d -c "create EXTENSION pgcrypto;"
in my case were 3 steps. Create the database, connect to the database and create the extension. The important step is the second one, "connect to the database", and you can notice the line without ";" cause is a command and not a SQL sentence.
CREATE DATABASE database_name_here;
\connect database_name_here
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
I'm working with the PADUS OBI shape file, not that that's probably important.
I'm running the shape file through shp2pgsql using the default options, as in:
shp2pgsql PADUS_1_1_CBI_Edition.shp > PADUS.sql
Then I'm trying to import the SQL into Postgres by doing:
psql -d padusdb -f PADUS.sql
And getting the following error:
psql:PADUS.sql:36: ERROR: function addgeometrycolumn(unknown, unknown, unknown, unknown, unknown, integer) does not exist
LINE 1: SELECT AddGeometryColumn('','padus_1_1_cbi_edition','the_geo...
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
I have PostGIS installed.
The SQL commands leading to the error (being put into an otherwise empty database) are:
SET CLIENT_ENCODING TO UTF8;
SET STANDARD_CONFORMING_STRINGS TO ON;
BEGIN;
CREATE TABLE "padus_1_1_cbi_edition" (gid serial PRIMARY KEY,
"us_id" int4,
"category" varchar(10),
"gis_acres" numeric,
...
BUNCH OF COLUMNS, none of which is called "the_geom"
...
"comments" varchar(200),
"shape_leng" numeric,
"shape_area" numeric);
SELECT AddGeometryColumn('','padus_1_1_cbi_edition','the_geom','-1','MULTIPOLYGON',2);
COMMIT;
Any thoughts on what this might mean and how to resolve the problem?
So, as it turns out, it is not enough to simply have installed PostGIS on one's machine.
Originally, I'd chosen sudo apt-get install postgresql postgis on Ubuntu 10.10. This left me with a working version of PostGRE 8.4, but no sign of PostGIS.
Therefore, I tried sudo apt-get install postgresql-8.4-postgis.
But one's work doesn't end there! You need to set up the PostGIS database.
This website provides instructions on doing this and using the database afterwards.
It also sounds like the database needs to be spatially enabled. The reason it's throwing that errors is because the function is missing. This resource has a quick and easy answer and solution.
this error indicates that the function cannot be recognized (either function name or parameters types are incorrect)
this is the definitions for AddGeometryColumn in v7.2
text AddGeometryColumn(varchar table_name, varchar column_name, integer srid, varchar type, integer dimension);
text AddGeometryColumn(varchar schema_name, varchar table_name, varchar column_name, integer srid, varchar type, integer dimension);
text AddGeometryColumn(varchar catalog_name, varchar schema_name, varchar table_name, varchar column_name, integer srid, varchar type, integer dimension);
it looks to me like you're trying to use the 2nd definition, try changing it to use the first definition (no schema) and try unquote the srid (-1) since it should be passed as an integer.
You may need to cast everything...
Thanks atorres757! Your answer solved my problem in minutes. I deleted my database and created a new database and choose the template_postgis as my template. All shapefiles are importing fine with my python script like this:
for lyr in iList:
os.system("shp2pgsql -c -s 4326 -k -I -W UTF-8 "+lyr[:-4]+" "+lyr[:-4]+" | psql -d AWM -p 5432 -U postgres")