creating arrays of results based upon chunks of time - postgresql

I've got an app that has lots of sensor_events being saved; I'd like to get results by a date then map those into chunks of 15 minute times... this is not the same as doing a group by in postgres as that would only return something averaged and I need the specific events...
What I'm thinking is given a day I get the beginning_of_day and split it up as 15 minute chunks as keys to a hash of arrays ie
def return_time_chunk_hash
t=Date.today
st = t.beginning_of_day
times = Hash.new
while st<t.end_of_day
times[st.to_formatted_s(:time)] = Array.new
st = st + 15.minutes
end
return times
end
And from that I would compare the sensor_events created_at date, find which bucket it belonged to and plop it in there. Once I've got it that way I know whether or not a chunk has any (.count) and if so can do all the data manipulation on the specific events.
Does this seem nutty? Is there a simpler way I'm not seeing?
Update:
I liked the way jgraft was thinking but thought it wouldn't work as I'd have to do multiple queries based upon the group column flag, but then I thought of the group_by of Enumerable so I tried something like this in the actual SensorEvent model:
def chunk
Time.at((self.created_at.to_f / 15.minutes).floor * 15.minutes).to_formatted_s(:time)
end
This allows me to get all the sensor events I need as usual (ie #se=SensorEvent.where(sensor_id: 10)) but then I could do #se.group_by(&:chunk) and I get those singular events grouped into a hash ie:
{"13:30"=>
[#<SensorEvent:0x007ffac0128438
id: 25006,
force: 0.0,
xaccel: 502.0,
yaccel: 495.0,
zaccel: 616.0,
battery: 0.0,
position: 25.0,
created_at: Thu, 18 Jun 2015 13:33:37 EDT -04:00,
updated_at: Thu, 18 Jun 2015 15:51:32 EDT -04:00,
deviceID: "D36330135FE3D36",
location: 3,
sensor_id: 10>,
#<SensorEvent:0x007ffac0128140
id: 25007,
force: 0.0,
xaccel: 502.0,
yaccel: 495.0,
zaccel: 616.0,
battery: 0.0,
position: 27.0,
created_at: Thu, 18 Jun 2015 13:39:46 EDT -04:00,
updated_at: Thu, 18 Jun 2015 15:51:32 EDT -04:00,
deviceID: "D36330135FE3D36",
location: 3,
sensor_id: 10>,
.........
The trouble is of course not every chunk of time might be created since there was no event to spawn it; also that being a hash it's not sorted in anyway:
res.keys
=> ["13:30",
"13:45",
"14:00",
"13:00",
"15:45",
"16:00",
"16:15",
"16:45",
"17:00",
"17:15",
"17:30",
"14:15",
"14:30",
I have to do calculations on the chunks of events; I might keep a master TIMECHUNKS array to compare / lookup in order...

Why not just add a column to the sensor_events table that specifies which block it belongs to? Would basically give you an array as you could do a query like:
SensorEvent.where(date: Date.today, block: 1)
and return a relation of the data in an array-esque format.
You could just add an after_create callback to the SensorEvents model that sets the block column.
class SensorEvent < ActiveRecord::Base
after_create :set_block
private
def set_block
value = ((4 * created_at.hour) + (created_at.min.to_f / 15).ceil).to_i
self.update_columns(block: value)
end

Related

Is there any formula that i can use to how to show up value (month) in between from start to end date in spreadsheet

Is there any formula that I can use to show up each month according to start & end date in spreadsheet.
Example:
Start Date:2022-07-22
End Date:2022-10-22
I expected formula to extract value something like this
Jul - Aug - Sep - Oct
I've tried formula
=IF(A2="","",IF(TEXT(B2,"MM")-TEXT(A2,"MM")>1,CONCATENATE(TEXT(A2,"MMM")&" - "&text(EDATE(A2,1),"MMM")&" - "&TEXT(B2,"MMM")),IF(TEXT(A2,"MMM")=TEXT(B2,"MMM"),TEXT(A2,"MMM"),CONCATENATE(TEXT(A2,"MMM")&" - "&TEXT(B2,"MMM"))))) but it only give me correct value if there is up to 3 month period between start & end date.
Here's a link to the sample spreadsheet
For single cell can try-
=JOIN("-",UNIQUE(INDEX(TEXT(SEQUENCE(B2-A2+1,1,A2),"mmm"))))
For spill array-
=BYROW(A2:INDEX(B2:B,MATCH(9^9,B2:B)),LAMBDA(x,JOIN("-",UNIQUE(INDEX(TEXT(SEQUENCE(INDEX(x,2)-INDEX(x,1)+1,1,INDEX(x,1)),"mmm"))))))
See your sheet.
Get the difference in dates in months using DATEDIF and get dates in each intervening month using EOMONTH+SEQUENCE and convert the end of month dates to TEXT:
Start Date
End Date
Months
2022-07-01
2022-10-30
Jul - Aug - Sep - Oct
2022-08-02
2022-08-31
Aug
2022-07-03
2022-11-01
Jul - Aug - Sep - Oct - Nov
Drag fill formula:
=ARRAYFORMULA(JOIN(" - ",TEXT(EOMONTH(A2,SEQUENCE(DATEDIF(A2,EOMONTH(B2,),"M")+1)-1),"mmm")))
Or as a self adjusting array formula:
=MAP(A2:INDEX(A:A,COUNTA(A:A)),LAMBDA(a, ARRAYFORMULA(JOIN(" - ",TEXT(EOMONTH(a,SEQUENCE(DATEDIF(a,EOMONTH(OFFSET(a,0,1),),"M")+1)-1),"mmm")))))
This should be faster and efficient than getting all the dates and filtering them out one by one, thereby reducing space and time complexity.
Use sequence(), edate() and join(), like this:
=arrayformula( map(
A2:A, B2:B,
lambda(
start, end,
if(
isdate(start) * isdate(end),
join(
" - ",
text(
edate(
start,
sequence(
12 * (year(end) - year(start)) + month(end) - month(start) + 1,
1, 0
)
),
"MMM"
)
),
iferror(1/0)
)
)
) )

Between Dates using Waterline ORM SailsJS

Goal: Return a list of items that were created between two dates.
According to this issue https://github.com/balderdashy/waterline/issues/110 there is no between function just yet. However the work around is the following:
User.find({
date: { '>': new Date('2/4/2014'), '<': new Date('2/7/2014') }
}).exec(/* ... */);
To be more exact, we don't want the hard coded dates above so we read in the input from a form submission like so:
start = new Date(req.param('yearStart') + '/' + req.param('monthStart') + '/' + req.param('dayStart'));
end = new Date(req.param('yearEnd') + '/' + req.param('monthEnd') + '/' + req.param('dayEnd'));
Printing start and end to console shows me this (different timezones for some reason)?
from: Sat Mar 01 2014 00:00:00 GMT-0500 (EST)
to: Sat Apr 30 2016 00:00:00 GMT-0400 (EDT)
However my view returns nothing every time.
While writing this question I realized the issue was that I had date instead of createdAt in my filter.
So the following works:
User.find({
createdAt: { '>': start, '<': end }
}).exec(/* ... */);
If you are wondering how to use the API blueprint query, you will need to use the toISOString() method of the Date object. For example :
http://localhost:1337/:model/?where={date: {'<=', date.toISOString()}}

Dealing with (birthday) dates and Timezone with Dojo

I have a simple widget that has:
<input class="input" id="${id}_dateOfBirth" name="dateOfBirth" data-dojo-type="dijit/form/DateTextBox" />
Note that it's a birthday. So, it's meant to stay the same, regardless of where you are when you are (timezone shouldn't happen). If you are born on the 10th of January at 3:00AM in England, and view your personal information from New York, you are meant to still see 10th of January, NOT the 9th!
I am in GMT+8 right now.
When I submit this form, this actually gets to the server when I put in 1/1/1970:
dateOfBirth: "1969-12-31T16:00:00.000Z"
Which is bad, because it's 8 hours short of the actual date.
Basically, I need a way for the DateTextBox to show the date as it came from the server, effectively ignoring the browser's timezone.
FWIW, here is my variant of UTCDateTextBox:
define([
"dojo/_base/declare",
"dijit/form/DateTextBox"
], function(declare, DateTextBox) {
function isValidDate(value) {
return value instanceof Date && isFinite(value.getTime());
}
function toUTCDate(value) {
if (isValidDate(value)) {
value = new Date(
Date.UTC(value.getFullYear(), value.getMonth(), value.getDate())
);
}
return value;
}
return declare(DateTextBox, {
_getValueAttr : function() {
return toUTCDate(this.inherited("_getValueAttr", arguments));
}
});
});
For my use case, I found that I didn't need to override _setValueAttr(). With the above implementation, when getUTCXXX(), toUTCString(), toISOString() or toJSON() are called on the date object returned from _getValueAttr(), then the correct UTC date with zeroed time elements is returned.
Hope this helps.
After much hacking and analysing Dojo's source code, I came up with this:
var UTCDateTextBox = declare( 'UTCDateTextBox', [ DateTextBox ], {
_getValueAttr: function(){
var ov = this.inherited(arguments);
if( ov ){
ov.setTime( ov.getTime() - ov.getTimezoneOffset() * 60 * 1000 );
}
return ov;
},
_setValueAttr: function( value, priorityChange, formattedValue){
var v = stamp.fromISOString( value );
if( v ){
v.setTime( v.getTime() + v.getTimezoneOffset() * 60 * 1000 );
value = v;
}
this.inherited(arguments);
}
});
Basically:
When the value is set, the timezone difference gets added. This means that if the server has 1979-12-25T00:00:00.000Z, rather than assigning Tue Dec 25 1979 08:00:00 GMT+0800 (WST), it will assign Tue Dec 25 1979 00:00:00 GMT+0800 (WST) . Basically, the date is converted locally to whatever it was in UTC.
When the value is parsed, it will be changed from Tue Dec 25 1979 00:00:00 GMT+0800 (WST) to Tue Dec 25 1979 08:00:00 GMT+0800 (WST)
The changed value is the one submitted to the server. So, it will be correct regardless of what timezone it will be edited at.
Since I only ever ever deal with dates, if the server has 1979-12-31T23:00:00Z (which is an error: for birthdays, the time is actually ignored and mustn't matter), this will happen:
When the value is set, ISO is 1979-12-31T23:00:00.000Z. So, Tue Jan 01 1980 07:00:00 GMT+0800 (WST) is changed into Mon Dec 31 1979 23:00:00 GMT+0800 (WST). This means that the right date is placed into the date textbox (31/12/1979).
When the value is parsed from the textbox, Mon Dec 31 1979 00:00:00 GMT+0800 (WST) becomes Mon Dec 31 1979 08:00:00 GMT+0800 (WST). So, the server will save 1979-12-31T00:00:00Z -- which is, again, the correct date!
If there are bettere solutions, please let me know. Frankly, I hope there are as this one feels like a bit of a cheat!

MATLAB Cut 3D array of daily data into monthly segments

I have a 1437x159x1253 large matrix (let's call it A) of daily sea ice data for a little over 2 years. I need to write a code that takes the daily data from each month and does mean(A, 3) on it. So basically, 1253 is the t in days. If I start from January, I need to do mean(A,3) of the first 31 days, then the mean(A,3) of February, the next 28 or 29 days. Because the days alternate between 31 and 30 (and 28 or 29 for February), I don't know how to write a code to do this. I can do it manually, but that would take a while.
Thanks!
You can initialize an array containing the number of days in each month, Mon = [31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31] using boolean to check whether it's a leap year (to set Mon(2) = 29). The number of days will help you index each month appropriately, using a loop like:
index=1;
for i=1:12
average = mean(A(:,:,index:(index+Mon(i)-1),3);
index = index+M(i); % Starting location of the next month
end

How can I convert a 5 digit int date and 7 digit int time to a real date?

I've come across some data where the date for today's value is 77026 and the time (as of a few minutes ago) is 4766011. FYI: today is Fri, 18 Nov 2011 12:54:46 -0600
I can't figure out how these represent a date/time, and there is no supporting documentation.
How can I convert these numbers to a date value?
Some other dates from today are:
77026 | 4765509
77026 | 4765003
77026 | 4714129
77026 | 4617107
And some dates from what is probably yesterday:
77025 | 6292509
77025 | 6238790
77025 | 4009544
Ok, with your expanded examples, it would appear the first number is a day count. That'd put this time system's epoch at
to_days(today) = 734824
734824 - 77025 = 657799
from_days(657799) = Dec 29, 1800
The time values are problematic, it looks like they're decreasing (unless you listed most recent first?), but if they are some "# of intervals since midnight", then centi-seconds could be likely. That'd give us a range of 0 - 8,640,000.
4765509 = 47655.09 seconds -> sec_to_time(47655) = 13:14:15
sec_to_time(47650.03) -> 13:14:10
sec_to_time(47141.29) -> 13:05:41
sec_to_time(46171.07) -> 12:49:31