How to transpose specific columns in PySpark - pyspark

I have a following PySpark dataframe:
year week date time value
2020 1 20201203 2:00 - 2:15 23.9
2020 1 20201203 2:15 - 2:30 45.87
2020 1 20201203 2:30 - 2:45 87.76
2020 1 20201203 2:45 - 3:00 12.87
I want to transpose the time and value column. The desired output should be:
year week date 2:00 - 2:15 2:15 - 2:30 2:30 - 2:45 2:45 - 3:00
2020 1 20201203 23.9 45.87 87.76 12.87


You can use groupby and pivot.
df = df.groupby('year', 'week', 'date').pivot('time').max('value')

Related

How to INSERT repeated values like (a,b,c,d,a,b,c,d....) in DB table?

I try to make work schedule table.
I have a table like:
shift_starts_dt
shift_type
2022-01-01 08:00:00
Day
2022-01-01 20:00:00
Night
2022-01-02 08:00:00
Day
2022-01-02 20:00:00
Night
2022-01-03 08:00:00
Day
2022-01-03 20:00:00
Night
2022-01-04 08:00:00
Day
2022-01-04 20:00:00
Night
etc.. until the end of the year
I can't figure out how to add repeated values to table.
I want to add the 'shift_name' column that contains 'A','B','C','D' (It's like name for team)
What query should I use to achieve the next result:
shift_starts_dt
shift_type
shift_name
2022-01-01 08:00:00
Day
'A'
2022-01-01 20:00:00
Night
'B'
2022-01-02 08:00:00
Day
'C'
2022-01-02 20:00:00
Night
'D'
2022-01-03 08:00:00
Day
'A'
2022-01-03 20:00:00
Night
'B'
2022-01-04 08:00:00
Day
'C'
2022-01-04 20:00:00
Night
'D'
. . . . . .

Use number of half days since Jan 1 modulus 4 to index an array:
select
shift_starts_dt,
shift_type,
(array['A','B','C','D'])[(extract(epoch from shift_starts_dt - '2022-01-01')::int / 43200) % 4 + 1]
from work_schedule
See live demo.
You could replace '2022-01-01' with (select min(shift_starts_dt) from work_schedule) for a more general solution.

How to convert List as String from API to List of String

I'm getting the following String from an API:
"[4:00 PM - 5:00 PM, 5:00 PM - 6:00 PM, 6:00 PM - 7:00 PM, 7:00 PM - 8:00 PM, 10:00 AM - 11:00 AM, 11:00 AM - 12:00 PM, 12:00 PM - 1:00 PM, 1:00 PM - 2:00 PM, 2:00 PM - 3:00 PM]"
I want to convert this String to a List.
How to convert it?

Code :
void main() {
String mylist = "[4:00 PM - 5:00 PM, 5:00 PM - 6:00 PM, 6:00 PM - 7:00 PM, 7:00 PM - 8:00 PM, 10:00 AM - 11:00 AM, 11:00 AM - 12:00 PM, 12:00 PM - 1:00 PM, 1:00 PM - 2:00 PM, 2:00 PM - 3:00 PM]";
mylist = mylist.replaceAll('[', '');
mylist = mylist.replaceAll(']', '');
List<String> newList = mylist.split(',');
print(newList[0]);
}
Output :
4:00 PM - 5:00 PM

void main() {
String list = "[4:00 PM - 5:00 PM, 5:00 PM - 6:00 PM,"
" 6:00 PM - 7:00 PM, 7:00 PM - 8:00 PM,"
" 10:00 AM - 11:00 AM, 11:00 AM - 12:00 PM,"
" 12:00 PM - 1:00 PM, 1:00 PM - 2:00 PM, 2:00 PM - 3:00 PM]";
List<String> out = [];
String batch = "";
for (String s in list.split("")) {
if (s == "[" || s == "]") continue;
if (s == ",") {
out.add(batch);
batch = "";
} else
batch += s;
}
print(out);
}
Output -> [4:00 PM - 5:00 PM, 5:00 PM - 6:00 PM, 6:00 PM - 7:00 PM, 7:00 PM - 8:00 PM, 10:00 AM - 11:00 AM, 11:00 AM - 12:00 PM, 12:00 PM - 1:00 PM, 1:00 PM - 2:00 PM]
Hope this helps! 😋

this works
String data =
"[4:00 PM - 5:00 PM, 5:00 PM - 6:00 PM, 6:00 PM - 7:00 PM, 7:00 PM - 8:00 PM, 10:00 AM - 11:00 AM, 11:00 AM - 12:00 PM, 12:00 PM - 1:00 PM, 1:00 PM - 2:00 PM, 2:00 PM - 3:00 PM]";
List<String> dataList = data.replaceAll('[', '').replaceAll(']', '').split(',');

You can use split() method in Dart. It will look like this:
void main() {
String times = "4:00 PM - 5:00 PM, 5:00 PM - 6:00 PM, 6:00 PM - 7:00 PM, 7:00 PM - 8:00 PM, 10:00 AM - 11:00 AM, 11:00 AM - 12:00 PM, 12:00 PM - 1:00 PM, 1:00 PM - 2:00 PM, 2:00 PM - 3:00 PM";
// Splitting the string
// across comma
print(times.split(","));
List<String> timesList = times.split(",");
}
// Output:
//[4:00 PM - 5:00 PM, 5:00 PM - 6:00 PM, 6:00 PM - 7:00 PM, 7:00 PM - 8:00 PM, 10:00 AM - 11:00 AM, 11:00 AM - 12:00 PM, 12:00 PM - 1:00 PM, 1:00 PM - 2:00 PM, 2:00 PM - 3:00 PM]

add 1 day to date when using CASE WHEN REDSHIFT

I am trying to use a CASE WHEN statement like below to add 1 day to a timestamp based on the time part of the timestamp:
CASE WHEN to_char(pickup_date, 'HH24:MI') between 0 and 7 then y.pickup_date else dateadd(day,1,y.pickup_date) end as ead_target
pickup_Date is a timestamp with default format YYYY-MM-DD HH:MM:SS
My output
pickup_Date ead_target
2020-07-01 10:00:00 2020-07-01 10:00:00
2020-07-02 3:00:00 2020-07-02 3:00:00
When the hour of the day is between 0 and 7 then ead_target = pickup_Date ELSE add 1 day
Expected output
pickup_Date ead_target
2020-07-01 10:00:00 2020-07-02 10:00:00
2020-07-02 3:00:00 2020-07-02 3:00:00

You will want to use the date_part() function to extract the hour of the day - https://docs.aws.amazon.com/redshift/latest/dg/r_DATE_PART_function.html
Your case statement should work if you extract 'hour' from the timestamp and compare it to the range 0 - 7.

How to add a separator line after specific item in NSTableView (Cocoa/OS X)

I'm building a program that gets weather forecasts from the Met Office using their DataPoint API. This gives me forecast data for five days into the future, with eight forecast points (every three hours) for every day. It displays them in an NSTableView, which currently looks like this:
27/3/2016 at 0:00 6°C ...
27/3/2016 at 3:00 4°C ...
27/3/2016 at 6:00 4°C ...
27/3/2016 at 9:00 7°C ...
27/3/2016 at 12:00 8°C ...
27/3/2016 at 15:00 8°C ...
27/3/2016 at 18:00 8°C ...
27/3/2016 at 21:00 6°C ...
28/3/2016 at 0:00 6°C ...
28/3/2016 at 3:00 8°C ...
28/3/2016 at 6:00 8°C ...
...
However, when viewing five days of this all at once, it is very hard to read it day by day (determine where each day starts and ends without looking at the dates) and this is not user friendly in the slightest. So, I want to add a separator line (or separator cell, depending on how it can be done) after the last forecast step of each day, to separate the string of data out into easily recognisable days. I want it to look like this:
27/3/2016 at 0:00 6°C ...
27/3/2016 at 3:00 4°C ...
27/3/2016 at 6:00 4°C ...
27/3/2016 at 9:00 7°C ...
27/3/2016 at 12:00 8°C ...
27/3/2016 at 15:00 8°C ...
27/3/2016 at 18:00 8°C ...
27/3/2016 at 21:00 6°C ...
-------------------------------
28/3/2016 at 0:00 6°C ...
28/3/2016 at 3:00 8°C ...
28/3/2016 at 6:00 8°C ...
...
What is the best way that I can do this (in a single NSTableView, put separators at select points in the dataset)? Thanks

Get COUNT of hours on 30 mins Interval

I have this below data.
Date Interval
2014-01-01 12:00 AM
2014-01-01 12:30 AM
2014-01-01 1:00 AM
2014-01-01 1:30 AM
2014-01-01 2:00 AM
2014-01-01 2:30 AM
2014-01-01 3:00 AM
2014-01-01 3:30 AM
2014-01-01 4:00 AM
2014-01-01 4:30 AM
I need to extract the hour of the interval column.
I could do it using EXTRACT('hour', Interval) which gives me the hour numbers as int.
Result will be as follows:
Date Interval HourCount
2014-01-01 12:00 AM 0
2014-01-01 12:30 AM 0
2014-01-01 1:00 AM 1
2014-01-01 1:30 AM 1
2014-01-01 2:00 AM 2
2014-01-01 2:30 AM 2
2014-01-01 3:00 AM 3
2014-01-01 3:30 AM 3
2014-01-01 4:00 AM 4
2014-01-01 4:30 AM 4
But what I'm looking for is. I need the count for every 30 mins as 1.
Example data what I'm looking for.
Date Interval HourCount
2014-01-01 12:00 AM 1
2014-01-01 12:30 AM 2
2014-01-01 1:00 AM 3
2014-01-01 1:30 AM 4
2014-01-01 2:00 AM 5
2014-01-01 2:30 AM 6
2014-01-01 3:00 AM 7
2014-01-01 3:30 AM 8
2014-01-01 4:00 AM 9
2014-01-01 4:30 AM 10
This way, in a day I'll be getting 48 intervals.
I could use ROW_NUMBER() OVER (PARTITION BY Date ORDER BY Date). But this will give me wrong count if any interval is been missed out.
Suppose if the below row is missed out.
2014-01-01 4:00 AM 9
I'll be getting 9 as the HourCount for this row.
2014-01-01 4:30 AM 9
Someone help me to get the count of hours on 30 mins interval.

Try something like:
with series as
(select interval_num, interval_num * interval '30 minutes' as interval_time
from generate_series(0,47) series(interval_num))
select *
from data_table
right join series on series.interval_time = data_table.interval

Like you wrote, you can extract the hour, what you're missing is you can also extract the minutes and just check if they are bigger or equal 30.
SELECT date_part('hour', interval) * 2 + CASE WHEN date_part('minute', interval) >= 30 THEN 1 ELSE 0 END