I'm very new at postgres and I've been googling this for about 1h before posting here.
Hopefully, you can help me with this probably trivial issue.
I have a database LTC that was created with this schema:
CREATE TABLE LTC (
id SERIAL PRIMARY KEY,
time timestamptz,
side CHAR(4),
price REAL,
v REAL,
n INT
);
I want to query to get all the data for time='2017-08-12T03:58:26.563Z' (ISO string from javascript). I know there are over a hundred lines of data in the database for that time, i see it in my postgres client.
Here is the query I'm doing:
select * from LTC where time = '2017-08-12T03:58:26.563Z'::timestamptz
Why am I getting no results?
Edit:
Still unsure why it wasn't working, but I wrote a work-around that does:
In JavaScript:
var date = new Date('2017-08-12T03:58:26.563Z').toISOString(); // actual time passed as parameter in my function, hard-coded for the example
var reg = new RegExp("([0-9]{4})-([0-9]{2})-([0-9]{2})T([0-9]{2}):([0-9]{2}):([0-9]{2})\.[0-9]*Z","gmi");
var start = date.replace(reg, function(match,year,month,day,hour,min,sec,ms) {
return year+'-'+month+'-'+day+'T'+hour+':'+min+':00.000Z';
});
var end = date.replace(reg, function(match,year,month,day,hour,min,sec,ms) {
var min = parseInt(min)+1;
if (min<=9) {
min = '0'+min;
}
return year+'-'+month+'-'+day+'T'+hour+':'+min+':00.000Z';
});
var query = "select * from LTC where time >= '"+start+"'::timestamptz and time < '"+end+"'::timestamptz"; // This works
Try using date_trunc('second',timestamp) on both to ensure that numerical precision is not what is throwing you off. There may be additional decimal places that are not being shown by your client and throwing off the equality.
The other possible solution is giving a range (between x and y) to avoid the numerical equality issue.
It would be easier for us to help if you can get an exact copy of the data being represented (using pg_dump perhaps, if you are familiar) so that we can test with the data that you are using.
The final thing you may want to check is explicitly stating the time zones that you are referencing. I generally use timestamp without time zone to avoid this issue, but auto-setting time zones to different values may be throwing you off as well. A good way to test is by selecting the two values, something like
select *, l.time = p.ts as test
from LTC l, (select '2017-08-12T03:58:26.563Z'::timestamptz as ts) p
;
EDIT:
I have built a test to try to reproduce your behavior:
CREATE TABLE LTC (
id serial
, time timestamptz
);
INSERT INTO LTC (time)
values ('2017-08-12T03:58:26.56312345'::timestamptz)
returning *;
select *
from LTC
where time = '2017-08-12T03:58:26.563Z'::timestamptz
;
select *, l.time = p.ts as test
from LTC l, (select '2017-08-12T03:58:26.563Z'::timestamptz as ts) p
;
What I get here is actually:
1;"2017-08-12 03:58:26.563123-04";"2017-08-11 23:58:26.563-04";f
Hopefully you can see what is happening - the '2017-08-12T03:58:26.563Z'::timestamptz is being interpreted as a UTC time and then converted to my time zone (UTC-04), so what is being compared is actually a different date altogether! In the future, showing this type of equality side-by-side is a great way to test that you are executing what you think you are (especially with dates / times where auto-conversion happens often).
Related
I have a big table in a postgres db with location of units. Now I need to retrieve a location for every 60 seconds.
In Mysql, this is a piece of cake: select * from location_table where unit_id = '123' GROUP BY round(timestamp / 60)
But in postgres this seems to be a very hard problem. I also have the timestamps in dateformat rather than in epoch format.
Here is an example of how the table looks
CREATE TABLE location_table (
unit_id int,
"timestamp" timestamp(3) without time zone NOT NULL,
lat double precision,
lng double precision
);
Use date_trunc() to make sets per minute:
SELECT * -- most likely not what you want
FROM location_table
WHERE unit_id = 123 -- numbers don't need quotes '
GROUP BY date_trunc('minute', 'timestamp');
The * is of course wrong, but I don't know what you want to know about the GROUP so I can't come up with something better.
Edit:
When you need a random result from your table, DISTINCT ON () could do the job:
SELECT DISTINCT ON(date_trunc('minute', timestamp))
* -- your columns
FROM location_table;
There are other (standard SQL) solutions as well, like using row_number().
I have a table with large amount of records:
date instrument price
2019.03.07 X 1.1
2019.03.07 X 1.0
2019.03.07 X 1.2
...
When I query for the day opening price, I use:
1 sublist select from prices where date = 2019.03.07, instrument = `X
It takes a long time to execute because it selects all the prices on that day and get the first one.
I also tried:
select from prices where date = 2019.03.07, instrument = `X, i = 0 //It does not return any record (why?)
select from prices where date = 2019.03.07, instrument = `X, i = first i //Seem to work. Does it?
In Oracle an equivalent will be:
select * from prices where date = to_date(...) and instrument = "X" and rownum = 1
and Oracle will stop immediately when it finds the first record.
How to do this in KDB (e.g. stop immediately after it finds the first record)?
In kdb, where subclauses in select statements are executed sequentially. i.e. only those records which pass the first "test" get passed to the second test. With that in mind, looking at your two attempts:
select from prices where date = 2019.03.07, instrument = `X, i = 0 //It does not return any record (why?)
This doesn't (necessarily) return anything, because by the time it gets to the i=0 check, you've already filtered out some records (possibly including the first record in the original table, which would have i=0)
select from prices where date = 2019.03.07, instrument = `X, i = first i //Seem to work. Does it?
This one should work. First you filter by date. Then within the records for that date, you select the records for instrument `X. Then within those records, you take the record where i is the first i (where i has already been filtered down, so first i is simply the index of the first record [still the index from the original table, not the filtered down version])
Q-SQL equivalent for that is select[n] which also performs better than other approaches in most of the cases. Positive 'n' will give first n records and negative will give last n records.
q) select[1] from prices where date = 2019.03.07, instrument = `X
There is no inbuilt functionality to stop after first match. You can write custom function for that but that would probably execute slower than above supported version.
I want to fetch data from 3 tables in a single database at once. I used 3 conn.cursor() to it.. Are there any sophisticated ways to do it?
conn = psycopg2.connect(database="plottest", user="postgres")
self.statusbar.showMessage("Database opened Sucessfully", 1000)
cur = conn.cursor()
cur1 = conn.cursor()
cur2 = conn.cursor()
cur.execute("SELECT id ,actual from \"%s\" " % date)
rows = cur.fetchall()
cur1.execute("SELECT qty from DAILY where date = \'%s\'" % date)
dailyqty = cur1.fetchone()
cur2.execute("SELECT qty from MONTHLY where month = \'%s\'" % month)
monthqty = cur2.fetchone()
Awoogah awoogah, SQL injection warning! Don't write code using string interpolation. What happens if someone calls your code with the "date" ');-- DROP TABLE DAILY;-- ?
Use bind parameters. Always.
The only exception is for dynamic identifiers, like in the case above where you seem to use a table named after the current date. In that case you must "double quote" them and double any contained double-quotes. In your case that means that date should be date.replace('"', '""') where you substitute it into the SQL.
Now, back to our regular programming.
Since you fetchall from each cursor you can just re-use it. You don't need new cursors each time.
You can also combine the daily and monthly stats if you want, with a UNION ALL. I fixed your capitalisation and parameter binding in the process:
cur.execute("""SELECT 1, qty FROM daily WHERE date = %s
UNION ALL
SELECT 2, qty FROM monthly WHERE month = %s
ORDER BY 1""",
(date, month))
Note that string interpolation isn't used, instead a 2-tuple of parameters is passed to psycopg2 to bind directly. There's no need for quotes around the parameters, psycopg2 adds them if needed.
This avoids a client-server round trip by bundling the two queries. The extra column andORDER BY is technically needed so you can safely assume the first row is the daily results and second is the monthly. In practice PostgreSQL won't re-order them with UNION ALL though.
You can combine
SELECT a1 FROM t1 WHERE b1 = 'v1';
and
SELECT a2 FROM t2 WHERE b2 = 'v2';
to a single statement like this:
SELECT t1.a1, t2.a2 FROM t1, t2
WHERE t1.b1 = 'v1' AND t2.b2 = 'v2';
provided that both queries return exactly one row.
When running the following the query.select * from surgicals where to_char(dt_surgery ,'DD-MM-YYYY' ) = to_char('12-02-2012','DD-MM-YYYY');
the error coming as 'SQL state 42725: ERROR: function to_char(unknown, unknown) is not unique'
How to run above select query?
You probably mean to_char('12-02-2012'::date, 'DD-MM-YYYY'). to_char cannot convert a plain string to string. Still, it does not seem to make sense, you need one of these two, depending on the format of your date constant (which cannot be determined from the actual example date you provided):
select * from surgicals where to_char(dt_surgery ,'DD-MM-YYYY' ) = '12-02-2012';
select * from surgicals where to_char(dt_surgery ,'MM-DD-YYYY' ) = '12-02-2012';
The wrongness here is that you're doing string comparison of dates. Use date/time math, which can take into account fun things like time zones etc. and still get it right.
Maybe this is what you need:
SELECT *
FROM surgicals
WHERE date_trunc('day', dt_surgery) = '2012-02-12'
;
I have a problem with the query below in postgres
SELECT u.username,l.description,l.ip,SUBSTRING(l.createdate,0,11) as createdate,l.action
FROM n_logs AS l LEFT JOIN n_users AS u ON u.id = l.userid
WHERE SUBSTRING(l.createdate,0,11) >= '2009-06-07'
AND SUBSTRING(l.createdate,0,11) <= '2009-07-07';
I always used the above query in an older version of postgres and it worked 100%. Now with the new version of posgres it gives me errors like below
**ERROR: function pg_catalog.substring(timestamp without time zone, integer, integer) does not exist
LINE 1: SELECT u.username,l.description,l.ip,SUBSTRING(l.createdate,...
^
HINT: No function matches the given name and argument types. You might need to add explicit type casts.**
I assume it has something to do with datatypes, that the data is a time zone and that substring only support string datatypes, now my question is what can I do about my query so that my results would come up?
The explicit solution to your problem is to cast the datetime to string.
...,SUBSTRING(l.createdate::varchar,...
Now, this isn't at all a good practice to use the result to compare dates.
So, the good solution to your need is to change your query using the explicit datetime manipulation, comparison and formatting functions, like extract() and to_char()
You'd have to change your query to have a clause like
l.createdate::DATE >= '2009-06-07'::DATE
AND l.createdate::DATE < '2009-07-08'::DATE;
or one of the alternatives below (which you should really accept instead of this.)
SELECT u.username, l.description, l.ip,
CAST(l.createdate AS DATE) as createdate,
l.action
FROM n_logs AS l
LEFT JOIN
n_users AS u
ON u.id = l.userid
WHERE l.createdate >= '2009-06-07'::TIMESTAMP
AND l.createdate < '2009-07-07'::TIMESTAMP + '1 DAY'::INTERVAL
I'm not sure what you want to achieve, but basically "substring" on date datatypes is not really well defined, as it depends on external format of said data.
In most of the cases you should use extract() or to_char() functions.
Generally - for returning data you want to_char(), and for operations on it (including comparison) - extract(). There are some cases where this general rule does not apply, but these are usually signs of not really well thought data-structure.
Example:
# select to_char( now(), 'YYYY-MM-DD');
to_char
------------
2009-07-07
(1 row)
For extract let's write a simple query that will list all objects created after 8pm:
select * from objects where extract(hour from created) >= 20;
A variation on the Quassnoi's answer:
SELECT
u.username,
l.description,
l.ip,
CAST(l.createdate AS DATE) as createdate,
l.action
FROM
n_logs AS l
LEFT JOIN
n_users AS u
ON
(u.id = l.userid)
WHERE
l.createdate::DATE BETWEEN '2009-06-07'::DATE AND '2009-07-07'::DATE
If you use Postgresql, you will receive:
select('SUBSTRING(offer.date_closed, 0, 11)')
function substr(timestamp without time zone integer integer) does not
exist
Use:
select('SUBSTRING(CONCAT(offer.date_closed, \'\'), 0, 11)')