I am dumping postgresql .csv file into a table but getting below error
COPY postgres_log FROM 'C:\Program Files\PostgreSQL\13\data\log\postgresql-2021-06-15_191640.csv' WITH csv;
ERROR: extra data after last expected column
CONTEXT: COPY postgres_log, line 1: "2021-06-15 19:16:40.261 IST,,,5532,,60c8af3f.159c,1,,2021-06-15 19:16:39 IST,,0,LOG,00000,"ending lo..."
SQL state: 22P04
I have followed below postgresql document
https://www.postgresql.org/docs/10/runtime-config-logging.html
Please suggest, how to dump in table
You're following instructions from Postgres 10, in which the CSV log format has one fewer column than Postgres 13, which is the version you're actually running. That's why you get that error message. To fix the problem, update your postgres_log table definition as per the PG 13 docs:
https://www.postgresql.org/docs/13/runtime-config-logging.html#RUNTIME-CONFIG-LOGGING-CSVLOG
CREATE TABLE postgres_log
(
log_time timestamp(3) with time zone,
user_name text,
database_name text,
process_id integer,
connection_from text,
session_id text,
session_line_num bigint,
command_tag text,
session_start_time timestamp with time zone,
virtual_transaction_id text,
transaction_id bigint,
error_severity text,
sql_state_code text,
message text,
detail text,
hint text,
internal_query text,
internal_query_pos integer,
context text,
query text,
query_pos integer,
location text,
application_name text,
backend_type text,
PRIMARY KEY (session_id, session_line_num)
);
Related
I'm trying to add CSV data to my Postgres table:
CREATE TABLE IF NOT EXISTS Review (
review_id BIGSERIAL NOT NULL,
product_id_Metadata INTEGER,
date DATE,
summary VARCHAR(255),
body VARCHAR(500),
rating INTEGER,
recommend BOOLEAN,
reported BOOLEAN
reviewer_name VARCHAR(50),
reviewer_email VARCHAR(50),
response VARCHAR(255),
helpfulness INTEGER,
FOREIGN KEY product_id(product_id) REFERENCES Metadata(product_id)
);
Running the CMD:
COPY review (product_id_Metadata, summary, body, rating, recommend, reported, reviewer_name, reviewer_email, response, helpfulness) FROM '/Users/luke82601/Desktop/csv/reviews.csv' DELIMITER ',' CSV HEADER;
The CSV file has the same number of columns as my Postgres table, so I'm not sure why this error occurs
I have logs from various linux servers being fed by rsyslog to a PostgreSQL database. The incoming timestamp is an rsyslog'd RFC3339 formatted time like so: 2020-10-12T12:01:18.162329+02:00.
In the original test setup of the database logging table, I created that timestamp field as 'text'. Most things I need parsed are working right, so I was hoping to convert that timestamp table column from text to a timestamp datatype (and retain the subseconds and timezone if possible).
The end result should be a timestamp datatype so that I can do date-range queries using PostgreSQL data functions.
Is this doable in PostgreSQL 11? Or is it just better to re-create the table with the correct timestamp column datatype to begin with?
Thanks in advance for any pointers, advice, places to look, or snippets of code.
Relevant rsyslog config:
$template CustomFormat,"%timegenerated:::date-rfc3339% %syslogseverity-text:::uppercase% %hostname% %syslogtag% %msg%\n"
$ActionFileDefaultTemplate CustomFormat
...
template(name="rsyslog" type="list" option.sql="on") {
constant(value="INSERT INTO log (timestamp, severity, hostname, syslogtag, message)
values ('")
property(name="timegenerated" dateFormat="rfc3339") constant(value="','")
property(name="syslogseverity-text" caseConversion="upper") constant(value="','")
property(name="hostname") constant(value="','")
property(name="syslogtag") constant(value="','")
property(name="msg") constant(value="')")
}
and the log table structure:
CREATE TABLE public.log
(
id integer NOT NULL DEFAULT nextval('log_id_seq'::regclass),
"timestamp" text COLLATE pg_catalog."default" DEFAULT timezone('UTC'::text, CURRENT_TIMESTAMP),
severity character varying(10) COLLATE pg_catalog."default",
hostname character varying(20) COLLATE pg_catalog."default",
syslogtag character varying(24) COLLATE pg_catalog."default",
program character varying(24) COLLATE pg_catalog."default",
process text COLLATE pg_catalog."default",
message text COLLATE pg_catalog."default",
CONSTRAINT log_pkey PRIMARY KEY (id)
)
some sample data already fed into the table (ignore the timestamps in the messsage, they are done with an independent handmade logging system by my predecessor):
You can in theory convert the TEXT column to TIMESTAMP WITH TIME ZONE with ALTER TABLE .. ALTER COLUMN ... SET DATA TYPE ... USING, e.g.:
postgres=# CREATE TABLE tstest (tsval TEXT NOT NULL);
CREATE TABLE
postgres=# INSERT INTO tstest values('2020-10-12T12:01:18.162329+02:00');
INSERT 0 1
postgres=# ALTER TABLE tstest
ALTER COLUMN tsval SET DATA TYPE TIMESTAMP WITH TIME ZONE
USING tsval::TIMESTAMPTZ;
ALTER TABLE
postgres=# \d tstest
Table "public.tstest"
Column | Type | Collation | Nullable | Default
--------+--------------------------+-----------+----------+---------
tsval | timestamp with time zone | | not null |
postgres=# SELECT * FROM tstest ;
tsval
-------------------------------
2020-10-12 12:01:18.162329+02
(1 row)
PostgreSQL can parse the RFC3339 format, so subsequent inserts should just work:
postgres=# INSERT INTO tstest values('2020-10-12T12:01:18.162329+02:00');
INSERT 0 1
postgres=# SELECT * FROM tstest ;
tsval
-------------------------------
2020-10-12 12:01:18.162329+02
2020-10-12 12:01:18.162329+02
(2 rows)
But note that any bad data in the table (i.e. values which cannot be parsed as timestamps) will cause the ALTER TABLE operation to fail, so you should consider verifying the values before converting the data. Something like SELECT "timestamp"::TIMESTAMPTZ FROM public.log would fail with an error like invalid input syntax for type timestamp with time zone: "somebadvalue".
Also bear in mind this kind of ALTER TABLE requires a table rewrite which may take some time to complete (depending on how large the table is), and which requires a ACCESS EXCLUSIVE lock, rendering the table inaccessible for the duration of the operation.
If you want to avoid a long-running ACCESS EXCLUSIVE lock, you could probably do something like this (not tested):
add a new TIMESTAMPTZ column (adding a column doesn't rewrite the table and is fairly cheap provided you don't use a volatile default value)
creating a trigger to copy any values inserted into the original column
copy the existing values (using a bunch of batched updateds like UPDATE public.foo SET newlog = log::TIMESTAMPTZ
(in a single transaction) drop the trigger and the existing column, and rename the new column to the old one
SQL Server guy, new to PostgreSQL. Created the below table, performed the below insert, then ran a SELECT to see the data, yet the row shows the integer formatted with columns to break up the integer. Is this just a formatting style in the HeidiSQL utility I'm using, or is the data actually being stored as x,xxx,xxx,xxx rather than xxxxxxxxxx.
Table:
CREATE TABLE customer (
business_partner_id INTEGER PRIMARY KEY,
first_name VARCHAR(100),
last_name VARCHAR(100),
organisation_name VARCHAR(200),
date_of_bith DATE,
gender VARCHAR(50),
customer_type VARCHAR(50),
store_joined VARCHAR(10),
store_joined_date DATE,
created_date_time TIMESTAMP,
updated_date_time TIMESTAMP);
Insert:
-- Insert a customer
INSERT INTO customer VALUES
(1029884766,'James','Matson','Unknown','1980-02-17','M','Standard','303',CURRENT_DATE,CURRENT_TIMESTAMP,CURRENT_TIMESTAMP)
Query results:
I imported a csv file to an sql table, but this csv file will change on a regular basis. Is there a way to refresh the table based on the changes in the csv file without removing the table, creating it again, and using the 'import' function in pgadmin?
If possible, would such a solution also exist for the entire schema, consisting of tables based on imported csv files?
Thank you in advance!
Edit To Add: This assumes you have decent access to the postgres server so not just a pure PGADMIN solution.
You can do this file an FDW (File Data Wrapper).
https://www.postgresql.org/docs/9.5/file-fdw.html or for your correct version.
For example I have a FDW setup to look at the Postgres logfile from within SQL rather than having to open an ssh session to the server.
The file exists as a table in the schema when you access it refreshes the data from the file.
The code I used for the file is as follows, obviously the file needs to be on the db server local system.
create foreign table pglog
(
log_time timestamp(3) with time zone,
user_name text,
database_name text,
process_id integer,
connection_from text,
session_id text,
session_line_num bigint,
command_tag text,
session_start_time timestamp with time zone,
virtual_transaction_id text,
transaction_id bigint,
error_severity text,
sql_state_code text,
message text,
detail text,
hint text,
internal_query text,
internal_query_pos integer,
context text,
query text,
query_pos integer,
location text,
application_name text
)
server pglog
options (filename '/var/db/postgres/data11/log/postgresql.csv', format 'csv');
I have placed a file in my docker-entrypoint-initdb.d/ directory. Here is what is in the file:
CREATE TABLE user_test (
user_id INTEGER,
name VARCHAR(100),
email VARCHAR(128),
active_flg BOOLEAN,
type VARCHAR(20),
CONSTRAINT pk_user PRIMARY KEY (user_id)
);
The error I am getting is psql:/docker-entrypoint-initdb.d/0001-initial-database-design.sql:8: ERROR: syntax error at or near "CREATE".
What am I missing in being able to run a file? How do I change this file to work?
USER is a reserved keyword in Postgres, see the documentation. In general, you should avoid naming your tables and columns using reserved SQL keywords. If you really wanted to proceed as is, then place user into double quotes:
CREATE TABLE "user" (
user_id INTEGER,
name VARCHAR(100),
email VARCHAR(128),
active_flg BOOLEAN,
type VARCHAR(20),
CONSTRAINT pk_user PRIMARY KEY (user_id)
);
But, keep in mind that if you choose to name your table user, then you will forever have to escape it with double quotes.