I would like to calculate a z-score over a bin based on the data of a rolling look-back period.
Example
Todays visitor amount during [9:30-9:35) should be z-score normalized based off the (mean, std) of the last 3 days of visitors that visited during [9:30-9:35).
My current attempts both raise InvalidOperationError. Is there a way in polars to calculate this?
import polars as pl
def z_score(col: str, over: str, alias: str):
# calculate z-score normalized `col` over `over`
return (
(pl.col(col)-pl.col(col).over(over).mean()) / pl.col(col).over(over).std()
).alias(alias)
df = pl.from_dict(
{
"timestamp": pd.date_range("2019-12-02 9:30", "2019-12-02 12:30", freq="30s").union(
pd.date_range("2019-12-03 9:30", "2019-12-03 12:30", freq="30s")
),
"visitors": [(e % 2) + 1 for e in range(722)]
}
# 5 minute bins for grouping [9:30-9:35) -> 930
).with_column(
pl.col("timestamp").dt.truncate(every="5m").dt.strftime("%H%M").cast(pl.Int32).alias("five_minute_bin")
).with_column(
pl.col("timestamp").dt.truncate(every="3d").alias("daytrunc")
)
# normalize visitor amount for each 5 min bin over the rolling 3 day window using z-score.
# not rolling but also wont work (InvalidOperationError: window expression not allowed in aggregation)
# df.with_column(
# z_score("visitors", "five_minute_bin", "normalized").over("daytrunc")
# )
# won't work either (InvalidOperationError: window expression not allowed in aggregation)
#df.groupby_rolling(index_column="daytrunc", period="3i").agg(z_score("visitors", "five_minute_bin", "normalized"))
For an example of 4 days of data with four data-points each lying in two time-bins ({0,0} - {0,1}), ({1,0} - {1,1})
Input:
Day 0: x_d0_{0,0}, x_d0_{0,1}, x_d0_{1,0}, x_d0_{1,1}
Day 1: x_d1_{0,0}, x_d1_{0,1}, x_d1_{1,0}, x_d1_{1,1}
Day 2: x_d2_{0,0}, x_d2_{0,1}, x_d2_{1,0}, x_d2_{1,1}
Day 3: x_d3_{0,0}, x_d3_{0,1}, x_d3_{1,0}, x_d3_{1,1}
Output:
Day 0: norm_x_d0_{0,0} = nan, norm_x_d0_{0,1} = nan, norm_x_d0_{1,0} = nan, norm_x_d0_{1,1} = nan
Day 1: norm_x_d1_{0,0} = nan, norm_x_d1_{0,1} = nan, norm_x_d1_{1,0} = nan, norm_x_d1_{1,1} = nan
Day 2: norm_x_d2_{0,0} = nan, norm_x_d2_{0,1} = nan, norm_x_d2_{1,0} = nan, norm_x_d2_{1,1} = nan
Day 3: norm_x_d3_{0,0} = (x_d3_{0,0} - np.mean([x_d0_{0,0}, x_d0_{0,1}, X_d1_{0,0}, ..., x_d3_{0,1}]) / np.std([x_d0_{0,0}, x_d0_{0,1}, X_d1_{0,0}, ..., x_d3_{0,1}])) , ... ,
They key here is to use over to restrict your calculations to the five minute bins and then use the rolling functions to get the rolling mean and standard deviation over days restricted by those five minute bin keys. five_minute_bin works as in your code and I believe that a truncated day_bin is necessary so that, for example, 9:33 on one day will include 9:31 both 9:34 on the same and 9:31 from 2 days ago.
days = 5
pl.DataFrame(
{
"timestamp": pl.concat(
[
pl.date_range(
datetime(2019, 12, d, 9, 30), datetime(2019, 12, d, 12, 30), "30s"
)
for d in range(2, days + 2)
]
),
"visitors": [(e % 2) + 1 for e in range(days * 361)],
}
).with_columns(
five_minute_bin=pl.col("timestamp").dt.truncate(every="5m").dt.strftime("%H%M"),
day_bin=pl.col("timestamp").dt.truncate(every="1d"),
).with_columns(
standardized_visitors=(
(
pl.col("visitors")
- pl.col("visitors").rolling_mean("3d", by="day_bin", closed="right")
)
/ pl.col("visitors").rolling_std("3d", by="day_bin", closed="right")
).over("five_minute_bin")
)
Now, that said, when trying out the code for this, I found polars doesn't handle non-unique values in the by-column in the rolling function correctly, so that the same values in the same 5-minute bin don't end up as the same standardized values. Opened bug report here: https://github.com/pola-rs/polars/issues/6691. For large amounts of real world data, this shouldn't actually matter that much, unless your data systematically differs in distribution within the 5 minute bins.
How do I plot horizontal lines using high and low of the first one minute bar after market opens at 9:30 (NYSE time)?
Possibly I would like to plot the open, high and close of 1st 1 minute bar.
Then when first 15 minutes is done, would like to plot the high and low of that bar as well.
I tried with OHLC indicator and others, but couldn't find something to tackle exactly this.
Also the below code that I got it from chatGPT, but it got couple of errors
type here
indicator("First Minute Bar High and Low", overlay=true)
var float firstBarHigh = na
var float firstBarLow = na
// Define the regular trading hours for the NYSE in Eastern Time
var tradingHoursStart = input("09:30-5", "Trading Hours")
var tradingHoursEnd = input("16:00-5", "Trading Hours")
var tradingHoursRange = time(tradingHoursStart + ":1234567-" + tradingHoursEnd + ":1234567")
// Define the desired time in Eastern Time
var timeInET = timenow("America/New_York")
if hour(timeInET) == 9 and minute(timeInET) == 30 and timeInET >= tradingHoursRange
if barstate.isfirst
firstBarHigh := high
firstBarLow := low
```plot(firstBarHigh, "First Minute High", color=color.green)
```plot(firstBarLow, "First Minute Low", color=color.red)
I am looking to take the date Friday, 08 July 2022, and convert it to 7/8/2022. I originally had a manual process to find and replace the date but if I can add it to the code it would be more efficient. The date moves to the summary tab and can be formatted there or before it moves.
Any help is appreciated, thank you!
Tea
Sub Datacleanup()
Dim c As Range, cDest As Range, lr As Long
Set cDest = Worksheets("summary").Cells(Rows.Count, "A").End(xlUp).Offset(1) 'first paste position
Set c = Worksheets("raw").Range("A1") 'start search here
lr = c.EntireColumn.Cells(Rows.Count).End(xlUp).Row 'last row of data in colA
Do While c.Row < lr
If Len(c.Value) > 0 And Application.CountA(c.EntireRow) = 1 Then
cDest.Resize(1, 3).Value = Array(c.Offset(2, 0).Value, _
c.Value, c.Offset(2, 14).Value)
Set cDest = cDest.Offset(1) ' next destination row
Set c = c.Offset(3) 'skip data block
Else
Set c = c.Offset(1) 'next row
End If
Loop
End Sub
I am trying to use asp classic to find how many working days (mon - sat) are in the month and how many are left.
any help or pointers greatly appreciated!
Here's how you can find the number of Sundays in a month without iteration. Somebody posted a JavaScript solution a few months back and I ported it to VBScript:
Function GetSundaysInMonth(intMonth, intYear)
dtmStart = DateSerial(intYear, intMonth, 1)
intDays = Day(DateAdd("m", 1, dtmStart) - 1)
GetSundaysInMonth = Int((intDays + (Weekday(dtmStart) + 5) Mod 7) / 7)
End Function
So, your total work days would just be the number of days in the month minus the number of Sundays.
Edit:
As #Lankymart pointed out in the comments, the above function gives you the number of Sundays in the month but it doesn't tell you how many are left.
Here's another version that does just that. Pass in any date and it will tell you how many Sundays are left in the month starting with that date. If you want to know how many Sundays are in a full month, just pass in the first day of the month (e.g., DateSerial(2014, 8, 1)).
Function GetSundaysRemainingInMonth(dtmStart)
intDays = Day(DateSerial(Year(dtmStart), Month(dtmStart) + 1, 1) - 1)
intDays = intDays - Day(dtmStart) + 1
GetSundaysRemainingInMonth = Int((intDays + (Weekday(dtmStart) + 5) Mod 7) / 7)
End Function
Edit 2:
#Cheran Shunmugavel was interested in some specifics about how this works. First, I just want to restate that I didn't develop this method originally. I just ported it to VBScript and tailored it to the OP's requirement (Sundays).
Imagine a February during a leap year. We have 29 days during the month. We know from the start that we have four full weeks, so each weekday will be represented at least four times. But that still leaves one addition day that's unaccounted for (29 Mod 7 = 1). How do we know if we get an extra Sunday from that one day? Well, in this case, it's pretty simple. Only if our start date is a Sunday can we count an extra Sunday for the month.
What if the month has 30 days? Then we have two extra days to account for. In that case, the start date can be a Saturday or a Sunday and we can count an extra Sunday for the month. And so it goes. So we can see that if we're X additional days within an upcoming Sunday, we can count an extra Sunday.
Let's put this in tabular form:
Addl Days Needed
Day To Count Sunday
---------- ----------------
Sunday 1
Saturday 2
Friday 3
Thursday 4
Wednesday 5
Tuesday 6
Monday 7
So what we need is a formula that we can apply to these situations so that they all result in the same value. We'll need to assign some value to each day and combine that value with the number of addition days needed for Sunday to count. Seems reasonable that if we assign an inverse value to the weekdays and add that to the number of additional days, we can get the same result.
Addl Days Needed Value Assigned
Day To Count Sunday To Weekday Sum
---------- ---------------- -------------- ---
Sunday 1 6 7
Saturday 2 5 7
Friday 3 4 7
Thursday 4 3 7
Wednesday 5 2 7
Tuesday 6 1 7
Monday 7 0 7
So, if weekday_value + addl_days = 7 then we count an extra Sunday. (We'll divide this by 7 later to give us 1 additional Sunday). But how do we assign the values we want to the weekdays? Well, VBScript's Weekday() function already does this but, unfortunately, it doesn't use the values we need by default (it uses 1 for Sunday through 7 for Saturday). We could change the way Weekday() works by using the second param, or we could just use a Mod(). This is where the + 5 Mod 7 comes in. If we take the Weekday() value and add 5, then mod that by 7, we get the values we need.
Day Weekday() +5 Mod 7
---------- --------- -- -----
Sunday 1 6 6
Saturday 7 12 5
Friday 6 11 4
Thursday 5 10 3
Wednesday 4 9 2
Tuesday 3 8 1
Monday 2 7 0
That's how the + 5 Mod 7 was determined. And, with that solved, the rest is easy(er)!
#Zam is on the right track you need to use WeekDay() function, here is a basic idea of how to script it;
<%
Dim month_start, month_end, currentdate, dayofmonth
Dim num_weekdays, num_past, num_future
Dim msg
'This can be configured how you like even use Date().
month_start = CDate("01/08/2014")
month_end = DateAdd("d", -1, DateAdd("m", 1, month_start))
msgbox(Day(month_end))
For dayofmonth = 1 To Day(month_end)
currentdate = CDate(DateAdd("d", dayofmonth, month_start))
'Only ignore Sundays
If WeekDay(currentdate) <> vbSunday Then
num_weekdays = num_weekdays + 1
If currentdate <= Date() Then
num_past = num_past + 1
Else
num_future = num_future + 1
End If
End If
Next
msg = ""
msg = msg & "Start: " & month_start & "<br />"
msg = msg & "End: " & month_end & "<br />"
msg = msg & "Number of Weekdays: " & num_weekdays & "<br />"
msg = msg & "Weekdays Past: " & num_past & "<br />"
msg = msg & "Weekdays Future: " & num_future & "<br />"
Response.Write msg
%>
How about using "The Weekday function returns a number between 1 and 7, that represents the day of the week." ?
I recently encountered below scenario in drools. I want to know how to proceed with the rule design for this.
Class Emp{
beingDate:Date
endDate:Date
}
Rule to determine annual income for the employee based on the given dates:
For dates before 3/5/2003 the hourly rate is $3.5 and annual multiplier is 2100
For dates after 3/5/2003 the hourly rate changes every year (given data) and annual multiplier is 2092.
There might be scenarios where begin date is before 3/5/2003 and end date is after 3/5/2003.
What is the best way to design rules for this scenario.
Update: added an e.g. for more clarity
If the object is
empObj={
beginDate=10/8/2001,
endDate=5/10/2005
}
The rule should give the sum of below:
3.5 * (no. of days in 2001 starting 10/8/2001) / (total no. of days in 2001) * 2100
3.5 * 2100 ==> This is for year 2002
3.5 * (no. of days in 2003 before 3/5/2003) / (total no. of days in 2003) * 2100
(2003 hourly rate) * (no. of days in 2003 after 3/5/2003) / (total no. of days in 2003) * 2092 ==> note the change in yearly multiplier..
(2004 hourly rate) * 2092
(2005 hourly rate) * (no. of days in 2005 before 5/10/2005) / (total no. of days in 2005) * 2092
One way to do this is to have one rule per year. So it would look something like this
rule "2001"
when:
e : Emp( beginDate < "01-Jan-2002" )
then:
// 1. Get the number of days worked in 2001, probably easiest to do with some Java helper method
// 2. Calculate the sum
// 3. Add the sum to some Fact, could be the same Emp fact even
end
rule "2002"
when:
e : Emp( beginDate < "01-Jan-2003" )
then:
// As with 2001
end
The rest of the rules are very similar, just change the yearly multiplier accordingly. If you decide to use the Emp object to hold the sum, add method like
class Emp {
long sum = 0
void addToSum( long value ) { sum += value }
}
And in your RHS side call the method and update the object on each rule.
Hope this helps.