I have a problem. I have a date in String f.eg "2021-05-06", and now i need to take one day before (2021-05-05). Here I'm making date from String but I cannot take one day before. Any tips?
val date = SimpleDateFormat("dd-MM-yyyy").parse(currentDate)
val formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd")
val date = LocalDate.parse("2021-05-06", formatter).minusDays(1)
println(date)
Output:
2021-05-05
If working with LocalDate is fine you could do
var date = LocalDate.parse("2021-05-06")
date = date.minusDays(1)
By analogy with similar questions in Java (there was addition, but we can perform subtraction), we can get the following piece of code:
val date = LocalDate.parse(currentDate)
val newDate = date.minusDays(1)
First similar question
Second similar question
Related
Hdfs blob stores the json data in the below format on a daily basis. I will need to read the json data using spark.read.json() on a day wise. Ex: Today i want to read day=01 day's files and tomorrow i want to read day=02 day's files. Is there a logic i can write in Scala which auto increments the date consider month and year also. Any help would me much appreciated.
/signals/year=2019/month=08/day=01
/signals/year=2019/month=08/day=01/*****.json
/signals/year=2019/month=08/day=01/*****.json
/signals/year=2019/month=08/day=02
/signals/year=2019/month=08/day=02/*****_.json
/signals/year=2019/month=08/day=02/*****_.json
Looks like data stored in partitioned format, and for read only one date such function can be used:
def readForDate(year: Int, month: Int, day: Int): DataFrame = {
spark.read.json("/signals")
.where($"year" === year && $"month" === month && $"day" === day)
}
For use this function, take current date and split on parts, with regular Scala code, not related to Spark.
If there is any relation between current date and the date you want to process the JSON file, you can get the current date (you can add/minus any number of days) using below Scala code and use it in your Spark application as #pasha701 suggested.
scala> import java.time.format.DateTimeFormatter
scala> import java.time.LocalDateTime
scala> val dtf = DateTimeFormatter.ofPattern("dd") // you can get the Year and Month like this.
scala> val now = LocalDateTime.now()
scala> println(dtf.format(now))
02
scala> println(dtf.format(now.plusDays(2))) // Added two days on the current date
04
Just a thought: If you are using Azure's Databricks then you can run shell command in notebook to get the current day (again if there is any relation on the partition's files you are trying to fetch with the current date) using "%sh" command.
Hope this may help any of you in future. Below code helps to read the data available in blob where the files are stored inside date folders which auto increments everyday. I wanted to read the data of previous day's data so adding now.minusDays(1)
val dtf = DateTimeFormatter.ofPattern("yyyy-MM-dd")
val now = LocalDateTime.now()
val date = dtf.format(now.minusDays(1))
val currentDateHold = date.split("-").toList
val year = currentDateHold(0)
val month = currentDateHold(1)
val day = currentDateHold(2)
val path = "/signals/year="+year+"/month="+month+"/day="+day
// Read JSON data from the Azure Blob`enter code here`
var initialDF = spark.read.format("json").load(path)
I'd like to get the day from a date object as an integer. This is my code so far.
val dateString = "2015-11-24 23:23:09"
val format = new java.text.SimpleDateFormat("yyyy-MM-dd H:m:s")
val date = format.parse(dateString)
print(date) # this gives Tue Nov 24 23:23:09 CST 2015
Now, from date, I want to get the day of the month as an integer. How do I do that?
Thanks.
Just use the Java Calendar class (although I would recommend moving over to the Joda library if you are doing much serious work with dates/times):
val cal = Calendar.getInstance()
cal.setTime(date)
val dayOfMonth = cal.get(Calendar.DAY_OF_MONTH)
tl;dr
LocalDateTime.parse(
"2015-11-24 23:23:09".replace( " " , "T" )
).getDayOfMonth()
java.time
The modern approach uses the java.time classes.
Using Java syntax here as I don't know Scala. Note that java.time uses immutable objects.
Convert your string to comply with ISO 8601 standard format with a T in the middle.
String input = "2015-11-24 23:23:09".replace( " " , "T" ) ;
Parse as an LocalDateTime as your input lacks any time zone or offset-from-UTC.
LocalDateTime ldt = LocalDateTime.parse( input ) ;
Interrogate for the day of month.
int dom = ldt.getDayOfMonth() ;
Using Joda,
org.joda.time.DateTime.now().getDayOfMonth()
or equivalently,
import java.util.Date
new org.joda.time.DateTime(new Date()).getDayOfMonth
OK, I am trying to convert a date string from a format like:
2014-01-21 00:00:00
to
01/21/2014
I have tried many variations and am crashing and burning. The issue is that to test I have to create the script, export it in a process in Bonita (a BPM software), Import it and then create some cases. This all takes a long time.
Hopefully someone knows how to do this.
Also, is there a simple groovy editor out there? That would help me learn how to write groovy very quickly.
Groovy Dates have methods parse and format for converting to and from strings in various formats:
def format1 = '2014-01-21 00:00:00'
def format2 = Date.parse("yyyy-MM-dd hh:mm:ss", format1).format("dd/MM/yyyy")
assert format2 == '01/21/2014'
The format of the format strings are the same as Java's SimpleDateFormat.
String olddate='2014/01/21 00:00:00'
Date date = Date.parse("yyyy/MM/dd HH:mm:ss",olddate)
String newDate = date.format( 'MM/dd/yyyy' )
log.info newDate
I have been struggling to understand how to use datetime objects. I want to use datetime.date instances as keys in a dictionary. I then want to be able to return dates within specified ranges using datetime.delta.
My first conundrum is when I create an object to be entered into the dictionary.
class Work_day():
'''input a workday , date and hours worked'''
def __init__(self, date, hours, rate):
self.date = datetime.date()
self.hours = hours
self.rate = rate
I want self.date to be a datetime.date object but datetime.date takes 3 argument (year, month, day) so what is the correct syntax for the def_init_ argument 'date'?
Then I assume when I change how that is written in the Work_day class then I will have to modify my code when I create instances of it in the Timesheet class e.g. in add_work_day() method
class Timesheet():
'''Represent a collection of workdays'''
def __init__(self):
self.timesheet = {}
def add_work_day(self, date, hours,rate):
'''adds a record of a work day into the timesheet dictionary'''
day = Work_day(date, hours, rate)
if day.date in self.timesheet:
print("There is already an entry for this day. ")
else:
self.timesheet[day.date] = hours, rate
I've been researching the python docs and scouring books but I'm not getting it! Need some help.
I also have a method that prints a range of the workdays in the timesheet. I made it work when I subbed the date key for a simple int. here it is (in ''' ''') with a shonky attempt at a datetime delta underneath
def show_days(self):
'''shows a user defined range of dates and the total pay for that period'''
pp = pprint.PrettyPrinter()
date_from = input("From date: ")
date_to = input("To date: ")
t = self.timesheet
total = 0
'''for dates in range(date_from, date_to + 1):
if dates in t:
total += self.sum_day(dates)
pp.pprint((dates, t[dates)])
print("Total £", total)'''
date = date_start = datetime.date(date_from)
date_end = datetime.date(date_to)
while date <= date_end:
if date in t:
print(date, t[dates])
date += datetime.timedelta(days=1)
I hope someone can find the patience to talk me through this. Cheers.
If you assign the date with self.date = datetime.date(*date), then you can create a Work_day by passing a (year,month,day) tuple:
day = Work_day((2013,5,31), 8.0, 8.25)
Alternatively, if you want the input to be a date string, use datetime.strptime, an appropriate formatting string, and the date() method to get a date object:
self.date = datetime.datetime.strptime(date,'%m/%d/%Y').date()
...
date = Work_day('5/31/2013', 8.0, 8.25)
Finally, you could just pass a date object:
day = Work_day(datetime.date(2013,5,31), 8.0, 8.25)
...
self.date = date
The Timesheet class should work after any of these changes. show_days still needs some work, but I'll leave that as an exercise. Hint: Parse the input dates with strptime.
I have a mysql DB and the dates are stored in it using the sql format of yyyy-MM-dd
I am using Jcalender for gui to get my customers DOB in java swing.
How do I use it to calculate the age of the person using yearsBetween() in joda time and then convert it to an int.
Please help, a code will be really useful :)
Okay I will post the code which is causing the problem
String dob = "1965-02-03";
DateTime today = new DateTime(DateTime.now().toLocalDate().toString());
DateTime start = new DateTime(dob);
System.out.println(today);
System.out.println(dob);
Years y= Years.yearsBetween(start, today);
System.out.println(y);
why does the y return a value of P485 instead of the correct value?
Your SysOut over class Years is missing a "getYears( )".
The code should be like this:
final String dob = "1965-02-03";
final DateTime today = LocalTime.now().toDateTimeToday();
final DateTime start = DateTime.parse(dob);
System.out.println(today);
System.out.println(dob);
final Years y = Years.yearsBetween(start, today);
System.out.println(y.getYears());