I'm trying to build a table with csvsql.
When I use command:
csvsql --db mysql://user:password#localhost:3306/database_name --table table_name file.csv
I get the error:
(in table 'blabla', column 'xyz'): VARCHAR requires a length on dialect mysql
I've then tried to build a database schema and force it with --db-schema flag,
The db-schema format is:
CREATE TABLE table_name (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`x` varchar(29) DEFAULT NULL,
`y` int(10) NOT NULL DEFAULT '0',
`z` BOOL NOT NULL,
PRIMARY KEY (`id`),
KEY `indexed` (`indexed`)
);
but I still get the same error.
The complete command with db-schema is:
csvsql --db mysql://user:password#localhost:3306/database_name --table table_name --db-schema db_schema_filename csvfile.csv
I've read the manual for csvkit, but I don't get what I'm doing wrong.
This command should print the conversion result right?
Can someone please help?
Thank you.
Well, found the solution in the github.
https://github.com/wireservice/csvkit/issues/758#issue-201924611
After update from github, no more errors and tables are created normaly.
Related
I am running a script that creates a database, some tables with foreign keys and insert some data, but somehow creating the tables is not working, although it doesn't throw any error: I go to pgAdmin, look for the tables created and there's no one...
When I copy the text of my script and execute it into the Query Tool, it works fine and creates the tables.
Can you please explain me what I am doing wrong?
Script:
DROP DATABASE IF EXISTS test01 WITH (FORCE); --drops even if in use
CREATE DATABASE test01
WITH
OWNER = postgres
ENCODING = 'UTF8'
LC_COLLATE = 'German_Germany.1252'
LC_CTYPE = 'German_Germany.1252'
TABLESPACE = pg_default
CONNECTION LIMIT = -1
IS_TEMPLATE = False
;
CREATE TABLE customers
(
customer_id INT GENERATED ALWAYS AS IDENTITY,
customer_name VARCHAR(255) NOT NULL,
PRIMARY KEY(customer_id)
);
CREATE TABLE contacts
(
contact_id INT GENERATED ALWAYS AS IDENTITY,
customer_id INT,
contact_name VARCHAR(255) NOT NULL,
phone VARCHAR(15),
email VARCHAR(100),
PRIMARY KEY(contact_id),
CONSTRAINT fk_customer
FOREIGN KEY(customer_id)
REFERENCES customers(customer_id)
ON DELETE CASCADE
);
INSERT INTO customers(customer_name)
VALUES('BlueBird Inc'),
('Dolphin LLC');
INSERT INTO contacts(customer_id, contact_name, phone, email)
VALUES(1,'John Doe','(408)-111-1234','john.doe#bluebird.dev'),
(1,'Jane Doe','(408)-111-1235','jane.doe#bluebird.dev'),
(2,'David Wright','(408)-222-1234','david.wright#dolphin.dev');
I am calling the script from a Windows console like this:
"C:\Program Files\PostgreSQL\15\bin\psql.exe" -U postgres -f "C:\Users\my user name\Desktop\db_create.sql" postgres
My script is edited in Notepad++ and saved with Encoding set to UTF-8 without BOM, as per a suggestion found here
I see you are using -U postgres command line parameter, and also using database name as last parameter (postgres).
So all your SQL commands was executed while you are connected to postgres database. Of course, CREATE DATABASE command did creation of test01 database, but CREATE TABLE and INSERT INTO did executed not for test01 database, but for postgres database, and all your tables are in postgres database, but not in test01.
You need to split your SQL script into 2 scripts (files): first for 'CREATE DATABASE', second for the rest of.
You need to execute first script as before, like
psql.exe -U postgres -f "db_create_1.sql" postgres
And for second one need to choose the database which was created at 1st step, like
psql.exe -U postgres -f "db_create_2.sql" test01
Trying to import a .csv into my postgres table using the following approach:
System: WSL2 - UBUNTU 20.04
psql -d db_name --user=username -c "\copy test_list FROM 'testmngrs.csv' delimiter '|' csv;"
The content format of my .csv:
1,Name,name#store_id.com,1234567891,City Name
The error I'm receiving:
ERROR: invalid input syntax for type bigint:
CONTEXT: COPY test_list, line 1, column id:
The table:
SELECT * FROM test_list;
id | store_id | name | email | phone | city
The additional id at the head of the table above was not something created during my initial set up of the table.
My ecto migration file is as follows:
I'm not sure what's causing the BigInt error, nor how to avoid it as I copy over the data. I'm also a bit confused as to why there's an additional id column in my table given that it was never defined in my migration
I'm pretty new to postgresql and elixir / ecto so any assistance is greatly/guidance/context is greatly appreciated!
From the docs:
By default, the table will also include an :id primary key field that has a type of :bigserial.
Ecto assumes you want it to generate the id field by default. It's better to just go with it. But you can configure it somewhat counter-intuitively by setting primary_key: false on the table, and primary_key: true on the column:
create table(:managers, primary_key: false) do
add :store_id, :integer, null: false, primary_key: true
...
I just set up a new foreign table and it works as intended if I just select the "ID" (integer) field.
When I add the "Description"(text) field and try to select the table, it fails with this error message:
utf-8 'Codec cannot decode byte 0xfc in position 10: invalid start byte
After checking the remote table, I found that "Description" contains special characters like: "ö, ü, ä"
What can i do to fix this?
Table definitions (Only first 2 rows)
Remote table:
CREATE TABLE test (
[Id] [char](8) NOT NULL,
[Description] [nvarchar](50) NOT NULL
)
Foreign table:
Create Foreign Table "Test" (
"Id" Char(8),
"Description" VarChar(50)
) Server "Remote" Options (
schema_name 'dbo', table_name 'test'
);
Additional information:
Foreign data wrapper: tds_fdw
Local server: Postgres 12, encoding: UTF8
Remote server: Sql Server, encoding: Latin1_General_CI_AS
As Laurenz Albe suggested in the comments, I created a freetds.conf in my PostgreSQL folder with the following content:
[global]
tds version = auto
client charset = UTF-8
Don't forget to set the path to the configuration file in the environment variable FREETDS.
Powershell:
[System.Environment]::SetEnvironmentVariable('FREETDS','C:\Program Files\PostgreSQL\12',[System.EnvironmentVariableTarget]::Machine)
I am following the examples in CREATE TABLE:
CREATE TABLE distributors (
did integer PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY,
name varchar(40) NOT NULL CHECK (name <> '')
);
However, it gives me ERROR: syntax error at or near "GENERATED". Why is that and how should I fix it?
\! psql -V returns psql (PostgreSQL) 10.5 (Ubuntu 10.5-1.pgdg14.04+1)
SELECT version(); returns PostgreSQL 9.4.19 on x86_64-pc-linux-gnu (Ubuntu 9.4.19-1.pgdg14.04+1), compiled by gcc (Ubuntu 4.8.4-2ubuntu1~14.04.4) 4.8.4, 64-bit
Edits:
Thanks to #muistooshort, I checked the 9.4 docs. So I execute:
CREATE TABLE distributors (
did integer PRIMARY KEY DEFAULT nextval('serial'),
name varchar(40) NOT NULL CHECK (name <> '')
);
Nevertheless, it now gives me ERROR: relation "serial" does not exist...
The SQL standard IDENTITY was added in PostgreSQL 10 but your server (which does all the real work) is 9.4. Before 10 you have to use serial or bigserial types:
CREATE TABLE distributors (
did serial not null primary key,
name varchar(40) NOT NULL CHECK (name <> '')
);
The serial type will create a sequence to supply values, attach the sequence to the table, and hook up a default value for did to get values from the sequence.
I'm working with the PADUS OBI shape file, not that that's probably important.
I'm running the shape file through shp2pgsql using the default options, as in:
shp2pgsql PADUS_1_1_CBI_Edition.shp > PADUS.sql
Then I'm trying to import the SQL into Postgres by doing:
psql -d padusdb -f PADUS.sql
And getting the following error:
psql:PADUS.sql:36: ERROR: function addgeometrycolumn(unknown, unknown, unknown, unknown, unknown, integer) does not exist
LINE 1: SELECT AddGeometryColumn('','padus_1_1_cbi_edition','the_geo...
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
I have PostGIS installed.
The SQL commands leading to the error (being put into an otherwise empty database) are:
SET CLIENT_ENCODING TO UTF8;
SET STANDARD_CONFORMING_STRINGS TO ON;
BEGIN;
CREATE TABLE "padus_1_1_cbi_edition" (gid serial PRIMARY KEY,
"us_id" int4,
"category" varchar(10),
"gis_acres" numeric,
...
BUNCH OF COLUMNS, none of which is called "the_geom"
...
"comments" varchar(200),
"shape_leng" numeric,
"shape_area" numeric);
SELECT AddGeometryColumn('','padus_1_1_cbi_edition','the_geom','-1','MULTIPOLYGON',2);
COMMIT;
Any thoughts on what this might mean and how to resolve the problem?
So, as it turns out, it is not enough to simply have installed PostGIS on one's machine.
Originally, I'd chosen sudo apt-get install postgresql postgis on Ubuntu 10.10. This left me with a working version of PostGRE 8.4, but no sign of PostGIS.
Therefore, I tried sudo apt-get install postgresql-8.4-postgis.
But one's work doesn't end there! You need to set up the PostGIS database.
This website provides instructions on doing this and using the database afterwards.
It also sounds like the database needs to be spatially enabled. The reason it's throwing that errors is because the function is missing. This resource has a quick and easy answer and solution.
this error indicates that the function cannot be recognized (either function name or parameters types are incorrect)
this is the definitions for AddGeometryColumn in v7.2
text AddGeometryColumn(varchar table_name, varchar column_name, integer srid, varchar type, integer dimension);
text AddGeometryColumn(varchar schema_name, varchar table_name, varchar column_name, integer srid, varchar type, integer dimension);
text AddGeometryColumn(varchar catalog_name, varchar schema_name, varchar table_name, varchar column_name, integer srid, varchar type, integer dimension);
it looks to me like you're trying to use the 2nd definition, try changing it to use the first definition (no schema) and try unquote the srid (-1) since it should be passed as an integer.
You may need to cast everything...
Thanks atorres757! Your answer solved my problem in minutes. I deleted my database and created a new database and choose the template_postgis as my template. All shapefiles are importing fine with my python script like this:
for lyr in iList:
os.system("shp2pgsql -c -s 4326 -k -I -W UTF-8 "+lyr[:-4]+" "+lyr[:-4]+" | psql -d AWM -p 5432 -U postgres")