pyspark get week number of month by starting week on Thursday - pyspark

Question

I need to get the weekday of a month. However the starting day of the week should be Thursday.
The data is given below
Currently my code gives the starting day as Sunday.
df = df.withColumn("Week_Number",date_format(to_date("inv_dt", "yyyy-MM-dd"), "W"))
However I want the week to start on a Thursday

Answer 1

Consider creating a User Defined Function (UDF) to handle the special case.
For example, the following UDF handles special case when a week starts other than Sunday.
The argument start_day_of_week takes an integer as isoweekday, where Monday is 1 and Sunday is 7:
from pyspark.sql.functions import udf
def week_number(date_str, start_day_of_week=7):
#udf("long")
def _week_number(date_str):
from datetime import datetime, date
d = datetime.strptime(date_str, '%Y-%m-%d') # compatible with Python2
wd_d1 = date(d.year, d.month, 1).isoweekday()
offset = (wd_d1 - start_day_of_week + 7 ) % 7
return (d.day - 1 + offset) // 7 + 1
return _week_number(date_str)
Note: code above is tested with Python3 (3.7.3) and Python (2.7.15) in PySpark (2.4.4). By default PySpark starts with Python (2.7.15). To use Python3, start PySpark with PYSPARK_PYTHON=python3 pyspark, or add export PYSPARK_PYTHON=python3 to startup script.
If the UDF only get used in Python3, one can simply use d = date.fromisoformat(date_str) and only import date from datetime.
Demo
Consider input data like:
from pyspark.sql.functions import *
inv_dt = [ '2018-09-{:02d}'.format(x) for x in range(1, 31) ]
# if with Pandas installed
import pandas as pd
df_pd = pd.DataFrame({'inv_dt':inv_dt})
df = spark.createDataFrame(df_pd)
# if without Pandas
from pyspark.sql.types import StringType
df = spark.createDataFrame(inv_dt, StringType()).withColumnRenamed('value', 'inv_dt')
df.show()
# +----------+
# | inv_dt|
# +----------+
# |2018-09-01|
# |2018-09-02|
# |2018-09-03|
# |2018-09-04|
# |2018-09-05|
# |2018-09-06|
# |2018-09-07|
# |2018-09-08|
# |2018-09-09|
# |2018-09-10|
# |2018-09-11|
# |2018-09-12|
# |2018-09-13|
# |2018-09-14|
# |2018-09-15|
# |2018-09-16|
# |2018-09-17|
# |2018-09-18|
# |2018-09-19|
# |2018-09-20|
# +----------+
# only showing top 20 rows
And call our UDF week_number just like other Spark SQL functions such as date_format.
We can compare our UDF output week_number("inv_dt", 7) with Spark builtin date_format(to_date("inv_dt", "yyyy-MM-dd"), "W").
( df
.withColumn("Day_of_Week", date_format(to_date("inv_dt", "yyyy-MM-dd"), "u"))
.withColumn("Week_Number_Sun_ref", date_format(to_date("inv_dt", "yyyy-MM-dd"), "W"))
.withColumn("Week_Number_Sun", week_number("inv_dt", 7))
.withColumn("Day_of_Week_Thu", week_number("inv_dt", 4))
.withColumn("Day_of_Week_Mon", week_number("inv_dt", 1))
).show()
# +----------+-----------+-------------------+---------------+---------------+---------------+
# | inv_dt|Day_of_Week|Week_Number_Sun_ref|Week_Number_Sun|Day_of_Week_Thu|Day_of_Week_Mon|
# +----------+-----------+-------------------+---------------+---------------+---------------+
# |2018-09-01| 6| 1| 1| 1| 1|
# |2018-09-02| 7| 2| 2| 1| 1|
# |2018-09-03| 1| 2| 2| 1| 2|
# |2018-09-04| 2| 2| 2| 1| 2|
# |2018-09-05| 3| 2| 2| 1| 2|
# |2018-09-06| 4| 2| 2| 2| 2|
# |2018-09-07| 5| 2| 2| 2| 2|
# |2018-09-08| 6| 2| 2| 2| 2|
# |2018-09-09| 7| 3| 3| 2| 2|
# |2018-09-10| 1| 3| 3| 2| 3|
# |2018-09-11| 2| 3| 3| 2| 3|
# |2018-09-12| 3| 3| 3| 2| 3|
# |2018-09-13| 4| 3| 3| 3| 3|
# |2018-09-14| 5| 3| 3| 3| 3|
# |2018-09-15| 6| 3| 3| 3| 3|
# |2018-09-16| 7| 4| 4| 3| 3|
# |2018-09-17| 1| 4| 4| 3| 4|
# |2018-09-18| 2| 4| 4| 3| 4|
# |2018-09-19| 3| 4| 4| 3| 4|
# |2018-09-20| 4| 4| 4| 4| 4|
# +----------+-----------+-------------------+---------------+---------------+---------------+
# only showing top 20 rows
#

Answer 2

The "pure" PySpark way is as follows
from pyspark.sql import functions as F
def week_number(date_str, start_day_of_week=7):
day_of_month = F.dayofmonth(date_str)
first_day = F.date_sub(date_str, day_of_month - 1)
first_day_day_of_week = F.dayofweek(first_day)
offset = (first_day_day_of_week - start_day_of_week + 7) % 7
week_num = F.floor((day_of_month - 1 + offset) / 7) + 1
return week_num
week_num_from_thurs = df.withColumn('week_num_from_thurs', week_number('inv_dt', 4))
It's 2 a.m. and I don't have a local Spark cluster or anything so this might be buggy. Credit should go to #Quar for the solution. I'm just converting Python to the Pyspark SQL API here. Variable names are deliberately verbose.