I want to import the csv file into database table .but it was not working..
I run the bash shell in the linux env .
CREATE TABLE test.targetDB (
no int4 NOT NULL GENERATED ALWAYS AS IDENTITY,
year varchar(15) NOT NULL,
name bpchar(12) NOT NULL,
city varchar(15) NOT NULL,
ts_load timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (seq_no)
)
test.csv file
"2019","112 ","1123",2019-07-26-05.33.43.000000
Linux Cmd
psql -d $database -c " COPY test.targetDB from 'test.csv' delimiter ',' csv "
Error
ERROR: invalid input syntax for integer: "2019"
CONTEXT: COPY targetDB, line 1, column no: "2019"
How can I resolve this issue
You need to tell copy, that the no column is not part of the CSV file by specifying the columns that should be populated:
COPY test.targetDB(year, name, city, ts_load) from 'test.csv' delimiter ',' csv
I would recommend, using datagrip - a postgresql client tool. You can use a evaluation version if you don't wish to purchase. It's pretty simple from the UI to import a file rather using a command line.
Related
Trying to import a .csv into my postgres table using the following approach:
System: WSL2 - UBUNTU 20.04
psql -d db_name --user=username -c "\copy test_list FROM 'testmngrs.csv' delimiter '|' csv;"
The content format of my .csv:
1,Name,name#store_id.com,1234567891,City Name
The error I'm receiving:
ERROR: invalid input syntax for type bigint:
CONTEXT: COPY test_list, line 1, column id:
The table:
SELECT * FROM test_list;
id | store_id | name | email | phone | city
The additional id at the head of the table above was not something created during my initial set up of the table.
My ecto migration file is as follows:
I'm not sure what's causing the BigInt error, nor how to avoid it as I copy over the data. I'm also a bit confused as to why there's an additional id column in my table given that it was never defined in my migration
I'm pretty new to postgresql and elixir / ecto so any assistance is greatly/guidance/context is greatly appreciated!
From the docs:
By default, the table will also include an :id primary key field that has a type of :bigserial.
Ecto assumes you want it to generate the id field by default. It's better to just go with it. But you can configure it somewhat counter-intuitively by setting primary_key: false on the table, and primary_key: true on the column:
create table(:managers, primary_key: false) do
add :store_id, :integer, null: false, primary_key: true
...
I have a table xml_table_date. Below is the structure and sample data in the table.
But I want to insert multiple xml files (here it is 9) into the table in one go. These files resides in a DB directory
My code to insert xml file into the table:
CREATE TABLE xml_table_data (
File_name varchar2(100),
Insert_date timestamp
xml_data XMLTYPE
);
INSERT INTO xml_tab VALUES ( 'DataTransfer_HH_TWWholesale_001_004_12142020113003.xml',
XMLTYPE (BFILENAME ('TESTING', 'DataTransfer_HH_TWWholesale_001_004_12142020113003.xml'),NLS_CHARSET_ID ('AL32UTF8')));
Please help me on this.. Thanks for reading my query.
You can use an external table with preproxessing to read the filenames from the directory.
ALTER SESSION SET CONTAINER=pdb1;
CREATE DIRECTORY data_dir AS '/u02/data';
CREATE DIRECTORY script_dir AS '/u02/scripts';
CREATE DIRECTORY log_dir AS '/u02/logs';
GRANT READ, WRITE ON DIRECTORY data_dir TO demo1;
GRANT READ, EXECUTE ON DIRECTORY script_dir TO demo1;
GRANT READ, WRITE ON DIRECTORY log_dir TO demo1;
Create a list_files.sh file in the scripts-directory. Make sure oracle is the owner and the privileges are 755 on the file.
The preprocessing script file don't inherit the $PATH enviroment variable. So you have to prepend /usr/bin before all commands.
/usr/bin/ls -1 /u02/data/test*.xml | /usr/bin/xargs -n1 /usr/bin/basename
You also need a source file for the external table, but this can be an empty dummy file.
CREATE TABLE data_files
( file_name VARCHAR2(255))
ORGANIZATION EXTERNAL
(
TYPE ORACLE_LOADER
DEFAULT DIRECTORY data_dir
ACCESS PARAMETERS
( RECORDS DELIMITED BY NEWLINE CHARACTERSET AL32UTF8
PREPROCESSOR script_dir: 'list_files.sh'
BADFILE log_dir:'list_files_%a_%p.bad'
LOGFILE log_dir:'list_files_%a_%p.log'
FIELDS TERMINATED BY WHITESPACE
)
LOCATION ('dummy.txt')
)
REJECT LIMIT UNLIMITED;
Now you can insert the xml data into your table.
INSERT INTO xml_table_data
( file_name,
insert_date,
xml_data
)
SELECT file_name,
SYSTIMESTAMP,
XMLTYPE (BFILENAME ('DATA_DIR', file_name), NLS_CHARSET_ID ('AL32UTF8'))
FROM data_files;
You still need to adapt the example to your environment, please.
I have exported the oracle database table data in to csv file and importing the same data into postgres database using '\copy' command via command prompt. While importing I'm getting below error because of the timestamp issue
psql command:
\copy "CSV_IMPORT"."DUMMY_TABLE" FROM 'D:\Database_Auto\DUMMY_TABLE_DATA.csv' DELIMITER ',' CSV HEADER;
CSV_IMPORT is the schema name
DUMMY_TABLE is the table name
Error:
ERROR: invalid input syntax for type timestamp with time zone: "21-JUN-07 06.42.43.950926000 PM"
CONTEXT: COPY DUMMY_TABLE, line 2, column updated_date: "21-JUN-07 06.42.43.950926000 PM"
If I modify the timestamp data with : instead of . as 21-JUN-07 06:42:43.950926000 PM it is importing the record without any error. I can't do it manually for millions of records in csv file. Any solution via psql command.
Table Create Script:
CREATE TABLE "CSV_IMPORT"."DUMMY_TABLE"
(
ID VARCHAR(100) NOT NULL
, DOCK_TYPE VARCHAR(1) NOT NULL
, START_DATE TIMESTAMP(6) WITH TIME ZONE NOT NULL
, UPDATE_SEQ_NBR DOUBLE PRECISION NOT NULL
, END_DATE TIMESTAMP(6) WITH TIME ZONE
, CONSTRAINT PK_DUMMY_TABLE PRIMARY KEY
(
ID
, DOCK_TYPE
, START_DATE
, UPDATE_SEQ_NBR
)
);
Table Data in CSV file:
"ID","DOCK_TYPE","START_DATE","UPDATE_SEQ_NBR","END_DATE"
"756748","L",21-JUN-07 06.42.43.950926000 PM,1,21-JUN-07 06.42.43.950926000 PM
"658399","T",15-NOV-03 02.59.54.000000000 AM,2,15-NOV-03 02.59.54.000000000 AM
"647388","F",19-NOV-04 11.09.05.000000000 PM,3,19-NOV-04 11.09.05.000000000 PM
Your best option is to re-do the export from Oracle and use to_string() to format the timestamp correctly.
If that is not feasible, then change your DUMMY_TABLE column to text instead of timestamptz and use to_timestamp(<tstz_column>, 'DD-MON-YY HH.MI.SS.US000 PM') to parse it inside of PostgreSQL.
If you were not stuck on Windows, you could use \copy ... from program and use sed to clean up your export on the fly.
I have an empty table in postgreSQL :
CREATE TABLE public.tbltesting
(
"ID" integer,
"testValue" numeric
)
WITH (
OIDS = FALSE
)
TABLESPACE pg_default;
I have a CSV file with the following data :
ID,testValue
1,2.0
2,3.33
3,4
The file is huge and requires a bulk copy and so I am trying to run the following command from PSQL :
\copy tblfoodnutrients FROM 'C:\temp\tbltestingbulk.csv' with CSV HEADER
ERROR: relation "tblfoodnutrients" does not exist
I have also tried the following :
\copy public.tblfoodnutrients FROM 'C:\temp\tbltestingbulk.csv' with CSV HEADER
ERROR: relation "public.tblfoodnutrients" does not exist
DUH! My connection was missing the database name to begin with.
I'm trying to build a table with csvsql.
When I use command:
csvsql --db mysql://user:password#localhost:3306/database_name --table table_name file.csv
I get the error:
(in table 'blabla', column 'xyz'): VARCHAR requires a length on dialect mysql
I've then tried to build a database schema and force it with --db-schema flag,
The db-schema format is:
CREATE TABLE table_name (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`x` varchar(29) DEFAULT NULL,
`y` int(10) NOT NULL DEFAULT '0',
`z` BOOL NOT NULL,
PRIMARY KEY (`id`),
KEY `indexed` (`indexed`)
);
but I still get the same error.
The complete command with db-schema is:
csvsql --db mysql://user:password#localhost:3306/database_name --table table_name --db-schema db_schema_filename csvfile.csv
I've read the manual for csvkit, but I don't get what I'm doing wrong.
This command should print the conversion result right?
Can someone please help?
Thank you.
Well, found the solution in the github.
https://github.com/wireservice/csvkit/issues/758#issue-201924611
After update from github, no more errors and tables are created normaly.