I have created a stream and i am writing KSQL on that stream .
But when i run this query and data arrive i can see records but when i data is not arriving and i run that query i do not see any older records .
So this is my KSQL
LOG_DIR=./ksql_logs /usr/local/confluent/bin/ksql http://localhost:8088
CREATE STREAM AUDIT_EVENT ( ID VARCHAR , VERSION VARCHAR , ACTION_TYPE VARCHAR , EVENT_TYPE VARCHAR , CLIENT_ID VARCHAR , DETAILS VARCHAR , OBJECT_TYPE VARCHAR , UTC_DATE_TIME VARCHAR , POINT_IN_TIME_PRECISION VARCHAR , TIME_ZONE VARCHAR , TIMELINE_PRECISION VARCHAR , GROUP_ID VARCHAR , OBJECT_DISPLAY_NAME VARCHAR , OBJECT_ID VARCHAR , USER_DISPLAY_NAME VARCHAR , USER_ID VARCHAR , PARENT_EVENT_ID VARCHAR , NOTES VARCHAR , SUMMARY VARCHAR , AUDIT_EVENT_TO_UTC_DT VARCHAR , AUDIT_EVENT_TO_DATE_PITP VARCHAR , AUDIT_EVENT_TO_DATE_TZ VARCHAR , AUDIT_EVENT_TO_DATE_TP VARCHAR ) WITH (KAFKA_TOPIC='AVRO-AUDIT_EVENT', VALUE_FORMAT='AVRO');
SELECT * FROM "AUDIT_EVENT" WHERE CLIENT_ID='fgh-5d1e-17a2-9749-0e4d00';
I have created table and tried but i table also i can not see my older records .
Is there any way i can records when ever i run this query ?
SET 'auto.offset.reset' = 'earliest' before your SELECT query statement.
Related
i will import data csv to postgresql via pgAdmin 4. But, there are problem
ERROR: invalid input syntax for type integer: ""
CONTEXT: COPY films, line 1, column gross: ""
i understand about the error that is line 1 column gross there is null value and in some other columns there are also null values. My questions, how to import file csv but in the that file there is null value. I've been search in google but not found similar my case.
CREATE TABLE public.films
(
id int,
title varchar,
release_year float4,
country varchar,
duration float4,
language varchar,
certification varchar,
gross int,
budget int
);
And i try in this code below, but failed
CREATE TABLE public.films
(
id int,
title varchar,
release_year float4 null,
country varchar null,
duration float4 null,
language varchar null,
certification varchar null,
gross float4 null,
budget float4 null
);
error message in image
I've searched on google and on the stackoverflow forums. I hope that someone will help solve my problem
There is no difference between the two table definitions. A column accepts NULL by default.
The issue is not a NULL value but an empty string:
select ''::integer;
ERROR: invalid input syntax for type integer: ""
LINE 1: select ''::integer;
select null::integer;
int4
------
NULL
Create a staging table that has data type of varchar for the fields that are now integer. Load the data into that table. Then modify the empty string data that will be integer using something like:
update table set gross = nullif(trim(gross), '');
Then move the data to the production table.
This is not a pgAdmin4 issue it is a data issue. Working in psql because it is easier to follow:
CREATE TABLE public.films_text
(
id varchar,
title varchar,
release_year varchar,
country varchar,
duration varchar,
language varchar,
certification varchar,
gross varchar,
budget varchar
);
\copy films_text from '~/Downloads/films.csv' with csv
COPY 4968
CREATE TABLE public.films
(
id int,
title varchar,
release_year float4,
country varchar,
duration float4,
language varchar,
certification varchar,
gross int,
budget int
);
-- Below done because of this value 12215500000 in budget column
alter table films alter COLUMN budget type int8;
INSERT INTO films
SELECT
id::int,
title,
nullif (trim(release_year), '')::real, country, nullif(trim(duration), '')::real,
LANGUAGE,
certification,
nullif (trim(gross), '')::float, nullif(trim(budget), '')::float
FROM
films_text;
INSERT 0 4968
It worked for me:
https://learnsql.com/blog/how-to-import-csv-to-postgresql/
a small workaround but it works
I created a table
I added headers to csv file
Right click on the newly created table-> Import/export data, select csv file to upload, go to tab2 - select Header and it should work
I have a problem. I use script which creates +/- 200 tables. On Firebird 3.0, time evaluation is 2 minutes. On Firebird 2.5, 6 seconds. I don't know why. My first 3 tables
CREATE TABLE defdok_analityka
(id_analityka INTEGER NOT NULL
, id_nagl INTEGER NOT NULL
, nazwa VARCHAR(60)
);
CREATE TABLE d_czas
(dc_id INTEGER NOT NULL
, dc_rok_mc VARCHAR(7)
, dc_rok_kwartal VARCHAR(7)
, dc_rok_tydzien_r VARCHAR(7)
, dc_data DATE
, dc_rok INTEGER
, dc_dzien_r INTEGER
, dc_kwartal INTEGER
, dc_miesiac INTEGER
, dc_dzien_mc INTEGER
, dc_dzien_tg INTEGER
, dc_tydzien_r INTEGER
);
CREATE TABLE d_kontrahent
(dk_id INTEGER NOT NULL
, dk_poczta_kod VARCHAR(20)
, dk_key_konsolid VARCHAR(30)
, dk_kraj_nazwa VARCHAR(35)
, dk_oper_prow VARCHAR(40)
, dk_przed_handl VARCHAR(40)
, dk_woj VARCHAR(50)
, dk_poczta VARCHAR(50)
, dk_firma_ident VARCHAR(50)
, dk_obszar_logist VARCHAR(60)
, dk_gmina VARCHAR(64)
, dk_powiat VARCHAR(64)
, dk_miejscowosc VARCHAR(64)
, dk_adres VARCHAR(90)
, dk_nazwa_skr VARCHAR(100)
, dk_obszar_handl VARCHAR(100)
, dk_nazwa VARCHAR(150)
, dk_src VARCHAR(20) NOT NULL
, dk_nip VARCHAR(35) NOT NULL
, dk_nip_wew VARCHAR(35) NOT NULL
, dk_data_utworzenia DATE
, dk_jdt0 TIMESTAMP
, dk_jdt1 TIMESTAMP
, dk_nr INTEGER
, dk_ver INTEGER
, dk_id_kon INTEGER
, dk_id_kontrah INTEGER
, dk_id_attribute INTEGER
, dk_id_firmy INTEGER NOT NULL
);
I tried using a lot of different server configuration and client connection. Even when I run isql directly FB3 still slower.
So I found a solution to my problem. My administrator installed FB2.5 on NVME disk and FB3.0 on HDD. This is a misunderstanding.
I am trying to create a PGSQL function that uses HTTP to get some json, format it, and insert it into a table using the donation_id key as a row constraint to prevent duplicates.
I have tried this function:
BEGIN
INSERT INTO donations(
donation_id, amount, avatar_image_url, created_date_utc, display_name, donor_id, event_id, incentive_id, message, participant_id, team_id)
ON CONFLICT (donation_id) DO NOTHING
SELECT elms::jsonb->>'donationID' AS donation_id ,
(elms::jsonb->>'amount')::float8 AS amount ,
elms::jsonb->>'avatarImageURL' AS avatar_image_url ,
(elms::jsonb->>'createdDateUTC')::timestamptz AS created_date_utc ,
elms::jsonb->>'displayName' AS display_name ,
elms::jsonb->>'donorID' AS donor_id ,
(elms::jsonb->>'eventID')::int AS event_id ,
elms::jsonb->>'incentiveID' AS incentive_id ,
elms::jsonb->>'message' AS message ,
(elms::jsonb->>'participantID')::int AS participant_id ,
(elms::jsonb->>'teamID')::int AS team_id
FROM (
select jsonb_array_elements(content::jsonb) AS elms
from http_get('https://extralife.donordrive.com/api/teams/59881/donations/')) as alias;
END;
I'm not quite understanding what I am doing wrong with the ON CONFLICT part of the query, just that it is apparently not valid syntax. I appreciate the insight as I'm not quite grasping the explainer written in docs.
Assuming a test table:
drop table if exists donations;
create table donations (
donation_id text primary key,
amount float8,
avatar_image_url text,
created_date_utc timestamptz,
display_name text,
donor_id text,
event_id int,
incentive_id text,
message text,
participant_id int,
team_id int);
It will work once you move the ON CONFLICT (donation_id) DO NOTHING to the end of the query:
INSERT INTO donations(donation_id, amount, avatar_image_url, created_date_utc,
display_name, donor_id, event_id, incentive_id, message, participant_id,
team_id)
SELECT elms::jsonb->>'donationID' AS donation_id ,
(elms::jsonb->>'amount')::float8 AS amount ,
elms::jsonb->>'avatarImageURL' AS avatar_image_url ,
(elms::jsonb->>'createdDateUTC')::timestamptz AS created_date_utc ,
elms::jsonb->>'displayName' AS display_name ,
elms::jsonb->>'donorID' AS donor_id ,
(elms::jsonb->>'eventID')::int AS event_id ,
elms::jsonb->>'incentiveID' AS incentive_id ,
elms::jsonb->>'message' AS message ,
(elms::jsonb->>'participantID')::int AS participant_id ,
(elms::jsonb->>'teamID')::int AS team_id
FROM ( select jsonb_array_elements('[
{
"displayName": "Christine",
"donorID": "A05C2C1E5DE15CDC",
"links": {
"recipient": "https://assets.yourdomain.com/somelink"
},
"eventID": 552,
"createdDateUTC": "2022-09-18T14:08:35.227+0000",
"recipientName": "Have A Drink Show",
"participantID": 494574,
"amount": 50,
"avatarImageURL": "https://assets.yourdomain.com/asset.gif",
"teamID": 59881,
"donationID": "FDBB61C5C8FFB3AE"
}
]'::jsonb) AS elms) as alias
ON CONFLICT (donation_id) DO NOTHING;
Demo.
I am currently trying to convert a postgres query to be compatible with Snowflake and work the same way.
Postgres
CREATE SEQUENCE IF NOT EXISTS public.etl_jobs_delta_loading_id_seq
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;
CREATE TABLE IF NOT EXISTS public.etl_jobs_delta_loading
(
id INTEGER DEFAULT nextval('public.etl_jobs_delta_loading_id_seq'::regclass) NOT NULL,
job_name VARCHAR(500) NOT NULL,
loaded_until TIMESTAMP,
etl_execution_time TIMESTAMP,
execution_status VARCHAR(30)
);
I translated the sequence to Snowflake, but keep getting errors while trying to get the nextVal in snowflake.
Snowflake
CREATE OR REPLACE SEQUENCE etl_jobs_delta_loading_id_seq
START = 1
INCREMENT = 1
CREATE TABLE IF NOT EXISTS public.etl_jobs_delta_loading
(
id INTEGER DEFAULT nextval('public.etl_jobs_delta_loading_id_seq'::regclass) NOT NULL, -- statement that needs to be converted
job_name VARCHAR(500) NOT NULL,
loaded_until TIMESTAMP,
etl_execution_time TIMESTAMP,
execution_status VARCHAR(30)
);
I have tried various approaches on creating the etl_jobs_delta_loading table but no luck till now. Any ideas on how to implement this in snowflake?
The correct syntax to get value from sequence is <seq_name>.NEXTVAL:
CREATE OR REPLACE SEQUENCE PUBLIC.etl_jobs_delta_loading_id_seq
START = 1
INCREMENT = 1;
CREATE TABLE IF NOT EXISTS public.etl_jobs_delta_loading
(
id INTEGER DEFAULT public.etl_jobs_delta_loading_id_seq.NEXTVAL NOT NULL,
job_name VARCHAR(500) NOT NULL,
loaded_until TIMESTAMP,
etl_execution_time TIMESTAMP,
execution_status VARCHAR(30)
);
Related: Sequences as Expressions
Alternatively using IDENTITY/AUTOINCREMENT property:
CREATE OR REPLACE TABLE public.etl_jobs_delta_loading
(
id INTEGER NOT NULL IDENTITY(1,1),
job_name VARCHAR(500) NOT NULL,
loaded_until TIMESTAMP,
etl_execution_time TIMESTAMP,
execution_status VARCHAR(30)
);
I am using the COPY FROM command to load data from a file.
The table is defined with identity column, which is not part of the file.
CREATE TABLE APP2DBMAP (
FIELD_ID integer NOT NULL GENERATED BY DEFAULT AS IDENTITY,
FIELD_NAME varchar(128) ,
TABLE_NAME varchar(128) ,
COLUMN_NAME varchar(128) ,
CONSTRAINT PK_APP2DBMAP PRIMARY KEY ( FIELD_ID )
);
I executed the following COPY FROM command, the file contains 3 values in 1 row.
copy app2dbmap (field_name, table_name, column_name) from '/opt/NetMgr/data/templ_db.txt' DELIMITER ',' ;
And I got the following error:
ERROR: null value in column "field_id" violates not-null constraint
DETAIL: Failing row contains (null, 'aaa', 'bbb', 'ccc').
CONTEXT: COPY app2dbmap, line 1: "'aaa','bbb','ccc'"
I tried to change the column description of field_id to serial, and it did work fine.
I don't understand why it doesn't work with the original table definition.
The problem is you have specified the field_id to be a not null value and hence when the file is passing null as a value, your error is there.
If you want an auto increment id, Use,
CREATE TABLE APP2DBMAP (
FIELD_ID smallserial NOT NULL,
FIELD_NAME varchar(128) ,
TABLE_NAME varchar(128) ,
COLUMN_NAME varchar(128) ,
CONSTRAINT PK_APP2DBMAP PRIMARY KEY ( FIELD_ID )
);
You can also use bigserial(int4) instead of smallint(int8)
or you can give a default value,
CREATE TABLE APP2DBMAP (
FIELD_ID integer NOT NULL default 0,
FIELD_NAME varchar(128) ,
TABLE_NAME varchar(128) ,
COLUMN_NAME varchar(128) ,
CONSTRAINT PK_APP2DBMAP PRIMARY KEY ( FIELD_ID )
);
You will have to pass something, you cannot pass null in a not null column