Error when querying PostgreSQL using range operators - postgresql

I am trying to query a postgresql (v 9.3.6) table with a tstzrange to determine if a given timestamp exists within the table defined as
CREATE TABLE sensor(
id serial,
hostname varchar(64) NOT NULL,
ip varchar(15) NOT NULL,
period tstzrange NOT NULL,
PRIMARY KEY(id),
EXCLUDE USING gist (hostname WITH =, period with &&)
);
I am using psycopg2 and when I try the query:
sql = "SELECT id FROM sensor WHERE %s <# period;"
cursor.execute(sql,(isotimestamp,))
I get the error
psycopg2.DataError: malformed range literal:
...
DETAIL: Missing left parenthesis or bracket.
I've tried various type castings to no avail.
I've managed a workaround using the following query:
sql = "SELECT * FROM sensor WHERE %s BETWEEN lower(period) AND upper(period);"
but would like to know why I am having problem with the range operators. Is it my code or psycopg2 or what?
Any help is appreciated.
EDIT 1:
In response to the comments, I have attempted the same query on a simple 1-row table in postgresql like below
=> select * from sensor;
session_id | hostname | ip | period
------------+----------+-----------+-------------------------------------------------------------------
1 | bob | 127.0.0.1 | ["2015-02-08 19:26:42.032637+00","2015-02-08 19:27:28.562341+00")
(1 row)
Now by using the "#>" operator I get the following error:
=> select * from sensor where period #> '2015-02-08 19:26:43.04+00';
ERROR: malformed range literal: "2015-02-08 19:26:43.04+00"
LINE 1: select * from sensor where period #> '2015-02-08 19:26:42.03...
Which appears to be the same as the psycopg2 error, a malformed range literal, so I thought I would try typecasting to timestamp as below
=> select * from sensor where sensor.period #> '2015-02-08 19:26:42.032637+00'::timestamptz;
session_id | hostname | ip | period
------------+----------+-----------+-------------------------------------------------------------------
1 | feral | 127.0.0.1 | ["2015-02-08 19:26:42.032637+00","2015-02-08 19:27:28.562341+00")
So it appears that it is my mistake, the literal has to be typecast or it is assumed to be a range. Using psycopg2, the query can be executed with:
sql="select * from sensor where period #> %s::timestamptz"

Related

Querying jsonb field with #> through Postgrex adapter

I'm trying to query jsonb field via Postgrex adapter, however I receive errors I cannot understand.
Notification schema
def all_for(user_id, external_id) do
from(n in __MODULE__,
where: n.to == ^user_id and fragment("? #> '{\"external_id\": ?}'", n.data, ^external_id)
)
|> order_by(desc: :id)
end
it generates the following sql
SELECT n0."id", n0."data", n0."to", n0."inserted_at", n0."updated_at" FROM "notifications"
AS n0 WHERE ((n0."to" = $1) AND n0."data" #> '{"external_id": $2}') ORDER BY n0."id" DESC
and then I receive the following error
↳ :erl_eval.do_apply/6, at: erl_eval.erl:680
** (Postgrex.Error) ERROR 22P02 (invalid_text_representation) invalid input syntax for type json. If you are trying to query a JSON field, the parameter may need to be interpolated. Instead of
p.json["field"] != "value"
do
p.json["field"] != ^"value"
query: SELECT n0."id", n0."data", n0."to", n0."inserted_at", n0."updated_at" FROM "notifications" AS n0 WHERE ((n0."to" = $1) AND n0."data" #> '{"external_id": $2}') ORDER BY n0."id" DESC
Token "$" is invalid.
(ecto_sql 3.9.1) lib/ecto/adapters/sql.ex:913: Ecto.Adapters.SQL.raise_sql_call_error/1
(ecto_sql 3.9.1) lib/ecto/adapters/sql.ex:828: Ecto.Adapters.SQL.execute/6
(ecto 3.9.2) lib/ecto/repo/queryable.ex:229: Ecto.Repo.Queryable.execute/4
(ecto 3.9.2) lib/ecto/repo/queryable.ex:19: Ecto.Repo.Queryable.all/3
however if I just copypaste generated sql to psql console and run it, it will succeed.
SELECT n0."id", n0."data", n0."to", n0."inserted_at", n0."updated_at" FROM "notifications" AS n0 WHERE ((n0."to" = 233) AND n0."data" #> '{"external_id": 11}') ORDER BY n0."id" DESC
notifications-# ;
id | data | to | inserted_at | updated_at
----+---------------------+-----+---------------------+---------------------
90 | {"external_id": 11} | 233 | 2022-12-15 14:07:44 | 2022-12-15 14:07:44
(1 row)
data is jsonb column
Column | Type | Collation | Nullable | Default
-------------+--------------------------------+-----------+----------+-------------------------------------------
data | jsonb | | | '{}'::jsonb
What am I missing in my elixir notification query code?
Searching for solution I came across only using raw sql statement, as I couldn't figure out what's wrong with my query when it gets passed through Postgrex
so as a solution I found the following:
def all_for(user_id, external_ids) do
{:ok, result} =
Ecto.Adapters.SQL.query(
Notifications.Repo,
search_by_external_id_query(user_id, external_ids)
)
Enum.map(result.rows, &Map.new(Enum.zip(result.columns, &1)))
end
defp search_by_external_id_query(user_id, external_id) do
"""
SELECT * FROM "notifications" AS n0 WHERE ((n0."to" = #{user_id})
AND n0.data #> '{\"external_id\": #{external_id}}')
ORDER BY n0."id" DESC
"""
end
But as a result I'm receiving Array with Maps inside not with Ecto.Schema as if I've been using Ecto.Query through Postgrex, so be aware.

cannot recognize epoch time when trying to copy from kafka to vertica

I am trying to copy a JSON data from Kafka to vertica. I am using the following query
COPY public.from_kafka
SOURCE KafkaSource(stream='example_data|0|-2, example_data|1|-2',
brokers='kafka01.example.com:9092',
duration=interval '10000 milliseconds') PARSER KafkaJSONParser()
REJECTED DATA AS TABLE public.rejections;
each message in the topic looks like that:
{"location_id":30277, "start_date":1667911800000}
when I am running the query, no new rows are created. when I am checking the rejections table I see the following rejected_reason:
Missing or null value for column with NOT NULL constraint [start_date]
however the rejected_data is {"location_id":30277, "start_date":1667911800000}
why does Vertica not recognize the start_date field and how can I solve it?
vertica table:
CREATE TABLE public.from_kafka
(
location_id int NOT NULL,
start_date timestamp NOT NULL
)
CREATE PROJECTION public.from_kafka /*+createtype(L)*/
(
location_id ENCODING RLE,
start_date ENCODING GCDDELTA
)
AS
SELECT from_kafka.location_id,
from_kafka.start_date,
FROM public.from_kafka
ORDER BY from_kafka.start_date,
from_kafka.location_id
SEGMENTED BY hash(from_kafka.location_id, from_kafka.start_date) ALL NODES KSAFE 1;
EDIT - SOLUTION
PARSER KafkaJSONParser() does not know how to convert long into timestamp, due to that I had to convert the JSON message with java, insert the updated JSON to a new topic and then use KafkaJSONParser() function
A timestamp, in any SQL database, is a timestamp, not an integer.
To load your JSON format and have a timestamp, redefine your table to receive an integer and convert it to a timestamp on the fly.
I do it from file, here, but it will work with a Kafka stream, too.
-- create your table like so:
DROP TABLE IF EXISTS public.from_kafka;
CREATE TABLE public.from_kafka
(
location_id int NOT NULL,
start_date int NOT NULL,
start_date_ts timestamp DEFAULT TO_TIMESTAMP(start_date//1000)
);
This is the JSON file I use:
$ cat kafka.json
{"location_id":30277, "start_date":1667911800000},
{"location_id":30278, "start_date":1667911900000},
{"location_id":30279, "start_date":1667912000000},
{"location_id":30280, "start_date":1667912100000},
{"location_id":30281, "start_date":1667912200000},
{"location_id":30282, "start_date":1667912300000}
And this is the copy command I use:
COPY public.from_kafka (
location_id
, start_date
)
FROM LOCAL 'kafka.json' PARSER FJsonParser(record_terminator=E'\n')
EXCEPTIONS 'kafka.log';
And this, finally, is what from_kafka will contain:
SELECT * FROM public.from_kafka;
-- out location_id | start_date | start_date_ts
-- out -------------+------------+---------------------
-- out 30277 | 1667911800 | 2022-11-08 13:50:00
-- out 30278 | 1667911900 | 2022-11-08 13:51:40
-- out 30279 | 1667912000 | 2022-11-08 13:53:20
-- out 30280 | 1667912100 | 2022-11-08 13:55:00
-- out 30281 | 1667912200 | 2022-11-08 13:56:40
-- out 30282 | 1667912300 | 2022-11-08 13:58:20

How do I use the Class::DBI->sequence() method to fill 'id' field automatically in perl?

I'm following the example Class::DBI.
I create the cd table like that in my MariaDB database:
CREATE TABLE cd (
cdid INTEGER PRIMARY KEY,
artist INTEGER, # references 'artist'
title VARCHAR(255),
year CHAR(4)
);
The primary key cdid is not set to auto-incremental. I want to use a sequence in MariaDB. So, I configured the sequence:
mysql> CREATE SEQUENCE cd_seq START WITH 100 INCREMENT BY 10;
Query OK, 0 rows affected (0.01 sec)
mysql> SELECT NEXTVAL(cd_seq);
+-----------------+
| NEXTVAL(cd_seq) |
+-----------------+
| 100 |
+-----------------+
1 row in set (0.00 sec)
And set-up the Music::CD class to use it:
Music::CD->columns(Primary => qw/cdid/);
Music::CD->sequence('cd_seq');
Music::CD->columns(Others => qw/artist title year/);
After that, I try this inserts:
# NORMAL INSERT
my $cd = Music::CD->insert({
cdid => 4,
artist => 2,
title => 'October',
year => 1980,
});
# SEQUENCE INSERT
my $cd = Music::CD->insert({
artist => 2,
title => 'October',
year => 1980,
});
The "normal insert" succeed, but the "sequence insert" give me this error:
DBD::mysql::st execute failed: You have an error in your SQL syntax; check the manual that
corresponds to your MariaDB server version for the right syntax to use near ''cd_seq')' at line
1 [for Statement "SELECT NEXTVAL ('cd_seq')
"] at /usr/local/share/perl5/site_perl/DBIx/ContextualFetch.pm line 52.
I think the quotation marks ('') are provoking the error, because when I put the command "SELECT NEXTVAL (cd_seq)" (without quotations) in mysql client it works (see above). I proved all combinations (', ", `, no quotation), but still...
Any idea?
My versions: perl 5.30.3, 10.5.4-MariaDB
The documentation for sequence() says this:
If you are using a database with AUTO_INCREMENT (e.g. MySQL) then you do not need this, and any call to insert() without a primary key specified will fill this in automagically.
MariaDB is based on MySQL. Therefore you do not need the call to sequence(). Use the AUTO_INCREMENT keyword in your table definition instead.

PostgreSQL complains about inexistent comparison function for element in primary key

I have a table in a PostgreSQL database in which I want to store the following columns:
STATION LOCATION SERVICE NORTH EAST
text point text real real
Each tuple(STATION, LOCATION, SERVICE) is unique, so I decided to make it a composite type and make it the primary key.
However, when I try to insert a new entry in the database I get the following error:
psycopg2.ProgrammingError: could not identify a comparison function for type point
I guess it is complaining that you cannot order two points in a 2D plane, but I cannot see how that is relevant. I have managed to use composite types that made use of points as primary keys in a test example, so I cannot see how this is different.
I want to know:
Why this is happening.
How it can be fixed, preferrably without changing the table schema.
Debugging information:
testdb=> \d ERROR_KEY
Composite type "public.error_key"
Column | Type | Modifiers
----------+-------+-----------
station | text |
location | point |
service | text |
testdb=> \d testtable
Table "public.testtable"
Column | Type | Modifiers
--------+-----------+-----------
key | error_key | not null
north | real |
east | real |
Indexes:
"testtable_pkey" PRIMARY KEY, btree (key)
For reference, this is the code I am using for the insertion:
from collections import namedtuple
import psycopg2
DB_NAME = 'testdb'
DB_USER = 'testuser'
DB_HOST = 'localhost'
DB_PASSWORD = '123456'
PVT_TABLE_NAME = 'testtable'
Coordinate = namedtuple('Coordinate', ['lat', 'lon'])
PVT_Error_Key = namedtuple('PVT_Error_Key',
['station', 'location', 'service'])
PVT_Error_Entry = namedtuple(
'PVT_Error_Entry', ['key', 'north', 'east'])
def _adapt_coordinate(coord):
"""
Adapter from Python class to Postgre geometric point
"""
lat = psycopg2.extensions.adapt(coord.lat)
lon = psycopg2.extensions.adapt(coord.lon)
return psycopg2.extensions.AsIs("'(%s, %s)'" % (lat, lon))
def _connect_to_db(db_name, db_user, db_host, db_password):
"""
Connects to a database and returns a cursor object to handle the connection
"""
connection_str = ('dbname=\'%s\' user=\'%s\' host=\'%s\' password=\'%s\''
% (db_name, db_user, db_host, db_password))
return psycopg2.connect(connection_str).cursor()
def main():
# Register the adapter for the location
psycopg2.extensions.register_adapter(Coordinate, _adapt_coordinate)
cursor = _connect_to_db(DB_NAME, DB_USER, DB_HOST, DB_PASSWORD)
# Create a dummy entry
entry = PVT_Error_Entry(
key=PVT_Error_Key(station='GKIR',
location=Coordinate(lat=12, lon=10),
service='E1'),
north=1, east=2)
# Insert the dummy entry in the database
cursor.execute(
'INSERT INTO %s '
'(KEY, NORTH, EAST) '
'VALUES((%%s, %%s, %%s), %%s, %%s)'
% PVT_TABLE_NAME,
(entry.key.station, entry.key.location, entry.key.service,
entry.north, entry.east))
# Retrieve and print all entries of the database
cursor.execute('SELECT * FROM %s', (PVT_TABLE_NAME))
rows = cursor.fetchall()
print(rows)
if __name__ == '__main__':
main()
You cannot use a column of type point in a primary key, e.g.:
create table my_table(location point primary key);
ERROR: data type point has no default operator class for access method "btree"
HINT: You must specify an operator class for the index or define a default operator class for the data type.
The error message is clear enough, you need to create a complete btree operator class for the type.
The full procedure is described in this answer: Creating custom “equality operator” for PostgreSQL type (point) for DISTINCT calls.
Update. With the workaround you mentioned in your comment
create table my_table(
x numeric,
y numeric,
primary key (x, y));
insert into my_table values
(1.1, 1.2);
you can always create a view, which can be queried just like a table:
create view my_view as
select point(x, y) as location
from my_table;
select *
from my_view;
location
-----------
(1.1,1.2)
(1 row)

Postgres json select not ignoring quotes

I have the following table and setup
create table test (
id serial primary key,
name text not null,
meta json
);
insert into test (name, meta) values ('demo1', '{"name" : "Hello"}')
However, when I run this query, this is the result
select * from test;
id | name | meta
----+-------+--------------------
1 | demo1 | {"name" : "Hello"}
(1 row)
but
select * from test where meta->'name' = 'Hello';
ERROR: operator does not exist: json = unknown
LINE 1: select * from test where meta->'name' = 'Hello';
^
HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.
-
select * from test where cast(meta->'name' as text) = 'Hello';
id | name | meta
----+------+------
(0 rows)
and this works
select * from test where cast(meta->'name' as text) = '"Hello"';
id | name | meta
----+-------+--------------------
1 | demo1 | {"name" : "Hello"}
(1 row)
Can anyone tell me what the relevance of this quote is and why it's not doing a simple string search/comparison? Alternatively, does this have something to do with the casting?
That's because the -> gets a field not a value, so you need to add the cast to say to postgresql which data type you are after.
So to run your query like you want you need to use the ->> which gets the json element as text see it here on the docs JSON Functions and Operators
So your query should looks like:
select *
from test
where meta->>'name' = 'Hello';
See it working here: http://sqlfiddle.com/#!15/bf866/8