Split postgres records into groups based on time fields - postgresql

I have a table with records that look like this:
| id | coord-x | coord-y | time |
---------------------------------
| 1 | 0 | 0 | 123 |
| 1 | 0 | 1 | 124 |
| 1 | 0 | 3 | 125 |
The time column represents a time in milliseconds. What I want to do is find all coord-x, coord-y as a set of points for a given timeframe for a given id. For any given id there is a unique coord-x, coord-y, and time.
What I need to do however is group these points as long as they're n milliseconds apart. So if I have this:
| id | coord-x | coord-y | time |
---------------------------------
| 1 | 0 | 0 | 123 |
| 1 | 0 | 1 | 124 |
| 1 | 0 | 3 | 125 |
| 1 | 0 | 6 | 140 |
| 1 | 0 | 7 | 141 |
I would want a result similar to this:
| id | points | start-time | end-time |
| 1 | (0,0), (0,1), (0,3) | 123 | 125 |
| 1 | (0,140), (0,141) | 140 | 141 |
I do have PostGIS installed on my database, the times I posted above are not representative but I kept them small just as a sample, the time is just a millisecond timestamp.

The tricky part is picking the expression inside your GROUP BY. If n = 5, you can do something like time / 5. To match the example exactly, the query below uses (time - 3) / 5. Once you group it, you can aggregate them into an array with array_agg.
SELECT
array_agg(("coord-x", "coord-y")) as points,
min(time) AS time_start,
max(time) AS time_end
FROM "<your_table>"
WHERE id = 1
GROUP BY (time - 3) / 5
Here is the output
+---------------------------+--------------+------------+
| points | time_start | time_end |
|---------------------------+--------------+------------|
| {"(0,0)","(0,1)","(0,3)"} | 123 | 125 |
| {"(0,6)","(0,7)"} | 140 | 141 |
+---------------------------+--------------+------------+

Related

postgreSQL question: get data by last date of each record and subtract from last date number of days

Please help me make a request. i'm at a dead end.
There are 2 tables:
“Trains”:
+----+---------+
| id | numbers |
+----+---------+
| 1 | 101 |
| 2 | 102 |
| 3 | 103 |
| 4 | 104 |
| 5 | 105 |
+----+---------+
“Passages”:
+----+--------------+-------+---------------------+
| id | train_number | speed | date_time |
+----+--------------+-------+---------------------+
| 1 | 101 | 26 | 2021-11-10 16:26:30 |
| 2 | 101 | 28 | 2021-11-12 16:26:30 |
| 3 | 102 | 24 | 2021-11-14 16:26:30 |
| 4 | 103 | 27 | 2021-11-15 16:26:30 |
| 5 | 101 | 29 | 2021-11-16 16:26:30 |
+----+--------------+-------+---------------------+
The goal is to go through the train numbers from the Trains table, take from the existing ones from the Passages table by the latest date (date_time) and the number of passages for “the last date for each train” - N days. as I understand date_time - interval "N days". should get something like:
+----+--------+---------------------+----------------+
| id | train | last_passage | count_passages |
+----+--------+---------------------+----------------+
| 1 | 101 | 2021-11-10 16:26:30 | 2 |
| 2 | 102 | 2021-11-14 16:26:30 | 1 |
| 3 | 103 | 2021-11-15 16:26:30 | 1 |
| 4 | 104 | null | 0 |
| 5 | 105 | null | 0 |
+----+--------+---------------------+----------------+
ps: "count_passages" - for example, last passage date minus 4 days
I tried through "where in" but I can’t create the necessary and correct request

Multiple Joins with Filters in DAX

I'm trying to recreate simple SQL query in DAX. The output Query needs to work in Power BI Report Builder and I have been trying all day reading all sorts of PowerBI / DAX online resources to rewrite this.
A little bit about the data:
The data is structured in three tables, CustomCar, Engine and Chassis.
Basically "CarId" is the key that connects all three tables.
Let's assume all tables have more than 20 columns. so only a few of the columns are needed in the final output.
All three tables (CustomCar, Chassis and Engine) have an IsActive property (the relationship between Engine/Chassis to CustomCar is MANY-TO-ONE. Because an engine might blow up and they change it therefore somehow we want to track which Engine is on the car today and what engine was on it last year, however, at any time, there is only one active engine for each car.. The same goes for Chassis)
Both Engine and Chassis have 'Manufacturer' and 'Model' columns so in the output query they need to be identified from each other.
I am not trying to sum any sort of sales number, just a list of cars with their current configuration.
Any help is appreciated.
Select
CC.Name, CC.Model as 'CustomCarModel', CC.MaxSpeed,
Ch.Manufacturer as 'ChassisManufacturer', Ch.Model as 'ChassisModel', Ch.ManufacturedDate as 'ChassisManfDate',
E.Manufactuer as 'EngineManufacturer', E.Model as 'EngineModel', E.Power, E.CylCount, E.ManufacturedDate
From CustomCars CC
Join Chassis Ch on Ch.CarID = CC.CarId
Join Engine E on E.CarID = CC.CarID
where
CC.IsActive = 1 and CC.FirstTestDriveYear < 1980 and
Ch.IsActive = 1 and
E.IsActive = 1
More info, here are my tables.
Classic Car:
CarId (Primary Key) | Model | MaxSpeed | NumOfPax | TankCapacity | IsActive | FirstTestDriveYear |....
1 | SuperChev | 220 | 2 | 60 | 1 | 1985 |
2 | CustomBranco | 185 | 2 | 90 | 1 | 1979 |
3 | RebuiltToyo | 251 | 4 | 20 | 0 | 1990 |
Chassis:
ChassisId (Primary Key) | CarId (Foreign Key)| IsActive | Manufacturer | Model | ManufacturedDate | ...
1 | 1 | 0 | ACME Chassis | M1 | '04-Jan-1985' | ...
2 | 1 | 1 | SuperChassis | T5 | '03-Feb-1987' | ...
3 | 2 | 0 | Ford | S2 | '25-Mar-1965' | ...
4 | 2 | 0 | Ford | S2 | '25-Mar-1968' | ...
5 | 3 | 0 | JapanChass | X123 | '25-Feb-1988' | ...
6 | 2 | 1 | Ford | S8 | '08-Jul-1978' | ...
7 | 2 | 0 | Ford | S2 | '25-Mar-1968' | ...
8 | 3 | 1 | JapanChass | Y765 | '25-Feb-1992' | ...
Engine:
EngineId (Primary Key) | CarId (Foreign Key)| IsActive | Manufacturer | Model | ManufacturedDate | Power | CylCount | ...
1 | 1 | 0 | GM | AB1 | '04-Jan-1985' | 320 | 8 | ...
2 | 1 | 1 | Bently | ZY2 | '03-Feb-1987' | 285 | 8 | ...
3 | 2 | 0 | Ford | S2 | '25-Mar-1965' | 290 | 6 | ...
4 | 2 | 0 | Ford | S2 | '25-Mar-1968' | 292 | 6 | ...
5 | 3 | 0 | Toyota | X123 | '25-Feb-1988' | 180 | 4 | ...
6 | 2 | 1 | Ford | S8 | '08-Jul-1978' | 222 | 8 | ...
7 | 2 | 0 | Ford | S2 | '25-Mar-1968' | 320 | 8 | ...
8 | 3 | 1 | Toyota | Y765 | '25-Feb-1992' | 211 | 6 | ...
I have found a work around for this. I added the query when adding the data pipeline in Power BI dashboard and will use the values from the query as is.

calculate aggregation and percentage simultaneous after groupBy in scala/Spark Dataset/Dataframe

I am learning to work with Scala and spark. It's my first incidents using them. I have some structured Scala DataSet(org.apache.spark.sql.Dataset) like following format.
Region | Id | RecId | Widget | Views | Clicks | CTR
1 | 1 | 101 | A | 5 | 1 | 0.2
1 | 1 | 101 | B | 10 | 4 | 0.4
1 | 1 | 101 | C | 5 | 1 | 0.2
1 | 2 | 401 | A | 5 | 1 | 0.2
1 | 2 | 401 | D | 10 | 2 | 0.1
NOTE: CTR = Clicks/Views
I want to merge the mapping regardless of Widget (i.e using Region, Id, RecID).
The Expected Output I want is like following:
Region | Id | RecId | Views | Clicks | CTR
1 | 1 | 101 | 20 | 6 | 0.3
1 | 1 | 101 | 15 | 3 | 0.2
What I am getting is like below:
>>> ds.groupBy("Region","Id","RecId").sum().show()
Region | Id | RecId | sum(Views) | sum(Clicks) | sum(CTR)
1 | 1 | 101 | 20 | 6 | 0.8
1 | 1 | 101 | 15 | 3 | 0.3
I understand that it is summing up all the CTR from original but I want to groupBy as explained but still want to get the expected CTR value. I also don't want to change column names as it is changing in my approach.
Is there any possible way of calculating in such manner. I also have #Purchases and CoversionRate (#Purchases/Views) and I want to do the same thing with that field also. Any leads will be appreciated.
You can calculate the ctr after the aggregation. Try the below code.
ds.groupBy("Region","Id","RecId")
.agg(sum(col("Views")).as("Views"), sum(col("Clicks")).as("Clicks"))
.withColumn("CTR" , col("Views") / col("Clicks"))
.show()

Retrieve additional columns on aggregation and date operator

I have the following PostgreSQL table structure, which gathers temperature records for every second:
+----+--------+-------------------------------+---------+
| id | value | date | station |
+----+--------+-------------------------------+---------+
| 1 | 0 | 2017-08-22 14:01:09.314625+02 | 1 |
| 2 | 0 | 2017-08-22 14:01:09.347758+02 | 1 |
| 3 | 25.187 | 2017-08-22 14:01:10.315413+02 | 1 |
| 4 | 24.937 | 2017-08-22 14:01:10.322528+02 | 1 |
| 5 | 25.187 | 2017-08-22 14:01:11.347271+02 | 1 |
| 6 | 24.937 | 2017-08-22 14:01:11.355005+02 | 1 |
| 18 | 24.875 | 2017-08-22 14:01:17.35265+02 | 1 |
| 19 | 25.187 | 2017-08-22 14:01:18.34673+02 | 1 |
| 20 | 24.875 | 2017-08-22 14:01:18.355082+02 | 1 |
| 21 | 25.187 | 2017-08-22 14:01:19.361491+02 | 1 |
| 22 | 24.875 | 2017-08-22 14:01:19.371154+02 | 1 |
| 23 | 25.187 | 2017-08-22 14:01:20.354576+02 | 1 |
| 30 | 24.937 | 2017-08-22 14:01:23.372612+02 | 1 |
| 31 | 0 | 2017-08-22 15:58:53.576238+02 | 1 |
| 32 | 0 | 2017-08-22 15:58:53.590872+02 | 1 |
| 33 | 26.625 | 2017-08-22 15:58:54.59986+02 | 1 |
| 38 | 26.375 | 2017-08-22 15:58:56.593205+02 | 1 |
| 39 | 0 | 2017-08-21 15:59:40.181317+02 | 1 |
| 40 | 0 | 2017-08-21 15:59:40.190221+02 | 1 |
| 41 | 26.562 | 2017-08-21 15:59:41.182622+02 | 1 |
| 42 | 26.375 | 2017-08-21 15:59:41.18905+02 | 1 |
+----+--------+-------------------------------+---------+
I want now to retrieve the maximum value for every hour, along with the data associated to that entry (id, date). As such, I tried the following:
select max(value) as m, (date_trunc('hour', date)) as d
from temperature
where station='1'
group by (date_trunc('hour', date));
Which works fine (fiddle), but I only get the columns m and d as a result. If I now try to add the date or id columns to the SELECT statement, I get the usual column "temperature.id" must appear in the GROUP BY clause or be used in an aggregate function error.
I have already tried approaches such as the ones described here, unfortunately to no avail, as for instance I seem to be unable to perform a join on the date_trunc-generated columns.
The result I am aiming for is this:
+----+--------+-------------------------------+---------+
| id | value | date | station |
+----+--------+-------------------------------+---------+
| 3 | 25.187 | 2017-08-22 14:01:10.315413+02 | 1 |
| 33 | 26.625 | 2017-08-22 15:58:54.59986+02 | 1 |
| 41 | 26.562 | 2017-08-21 15:59:41.182622+02 | 1 |
+----+--------+-------------------------------+---------+
It does not matter which record was retrieved in case two or more entries have the same value.
distinct on:
select distinct on (date_trunc('hour', date)) *
from temperature
where station = '1'
order by date_trunc('hour', date), value desc
Fiddle

In postgresql, how do you find aggregate base on time range

For example, if I have a database table of transactions done over the counter. And I would like to search whether there was any time that was defined as extremely busy (Processed more than 10 transaction in the span of 10 minutes). How would I go about querying it? Could I aggregate based on time range and count the amount of transaction id within those ranges?
Adding example to clarify my input and desired output:
+----+--------------------+
| Id | register_timestamp |
+----+--------------------+
| 25 | 08:10:50 |
| 26 | 09:07:36 |
| 27 | 09:08:06 |
| 28 | 09:08:35 |
| 29 | 09:12:08 |
| 30 | 09:12:18 |
| 31 | 09:12:44 |
| 32 | 09:15:29 |
| 33 | 09:15:47 |
| 34 | 09:18:13 |
| 35 | 09:18:42 |
| 36 | 09:20:33 |
| 37 | 09:20:36 |
| 38 | 09:21:04 |
| 39 | 09:21:53 |
| 40 | 09:22:23 |
| 41 | 09:22:42 |
| 42 | 09:22:51 |
| 43 | 09:28:14 |
+----+--------------------+
Desired output would be something like:
+-------+----------+
| Count | Min |
+-------+----------+
| 1 | 08:10:50 |
| 3 | 09:07:36 |
| 7 | 09:12:08 |
| 8 | 09:20:33 |
+-------+----------+
How about this:
SELECT time,
FROM (
SELECT count(*) AS c, min(time) AS time
FROM transactions
GROUP BY floor(extract(epoch from time)/600);
)
WHERE c > 10;
This will find all ten minute intervals for which more than ten transactions occurred within that interval. It assumes that the table is called transactions and that it has a column called time where the timestamp is stored.
Thanks to redneb, I ended up with the following query:
SELECT count(*) AS c, min(register_timestamp) AS register_timestamp
FROM trak_participants_data
GROUP BY floor(extract(epoch from register_timestamp)/600)
order by register_timestamp
It works close enough for me to be able tell which time chunks are the most busiest for the counter.