Hi Guys I am currently working on Gsuite Admin SDK Report API. I am successfully able to send the request and getting the response.
Now, the issue is that I am not able to identify the date format returned by the Activities.list().
Here is a snippet:
"events": [
{
"type": "event_change",
"name": "create_event",
"parameters": [
{
"name": "event_id",
"value": "jdlvhwrouwovhuwhovvwuvhw"
},
{
"name": "organizer_calendar_id",
"value": "abc#xyz.com"
},
{
"name": "calendar_id",
"value": "abc#xyz.com"
},
{
"name": "target_calendar_id",
"value": "abc#xyz.com"
},
{
"name": "event_title",
"value": "test event 3"
},
{
"name": "start_time",
"intValue": "63689520600"
},
{
"name": "end_time",
"intValue": "63689524200"
},
{
"name": "user_agent",
"value": "Mozilla/5.0"
}
]
}
]
Note: Please have a look at start_time and end_time and let me know if you guys have any idea about it.
Please have a look and share some info and let me know if any other infomation is needed.
I ran into this same question when parsing google calendar logs.
The time format they use are the number of seconds since January 1st, 0001 (0001-01-01).
I never found documentation where they referenced that time format. Google uses this instead of epoch for some of their app logs.
You can find an online calculator here https://www.epochconverter.com/seconds-days-since-y0
Use the one under "Seconds Since 0001-01-01 AD" and not the one under year zero.
So your start_time of "63689520600" converts to March 30, 2019 5:30:00 AM GMT.
If you want start_time to be in epoch, you could subtract 62135596800 seconds from the number. 62135596800 converts to January 1, 1970 12:00:00 AM when counting the number of seconds since 0001-01-01. Subtracting 62135596800 from the start_time would give you the number of seconds since January 1, 1970 12:00:00 AM AKA Epoch Time.
Hope this helps.
Related
A pipeline has to trigger every December on second Friday from the end of the month.
I am trying to do this using scheduled trigger of ADF See Trigger Definition by using,
Start date of Dec 1st 2021
Recurrence of 12 months
No end date
Advanced recurrence option of weekdays with occurrance as -2 and day as Friday.
"name": "Dec_Last_But_One_Friday",
"properties": {
"annotations": [],
"runtimeState": "Stopped",
"pipelines": [
{
"pipelineReference": {
"referenceName": "pipeline_test_triggers",
"type": "PipelineReference"
}
}
],
"type": "ScheduleTrigger",
"typeProperties": {
"recurrence": {
"frequency": "Month",
"interval": 12,
"startTime": "2021-12-01T14:24:00Z",
"timeZone": "UTC",
"schedule": {
"monthlyOccurrences": [
{
"day": "Friday",
"occurrence": -2
}
]
}
Is this right way? how do I know it will be definitely trigger every year, the Second Friday from the end of the December month. Is there a way to see the future schedules in ADF?
Thank you!
Viewing future ADF schedule, this feature does not currently exist in Data Factory V2 at this time.
Due to advanced recurrence options are not perfect, we'd better check once a year.
I'm trying to add a day to my dates which look like "2020-11-20" for November, 20, 2020. However I am encountering difficulty doing that - do I need to use the offset function? The reason I am doing this is that Vega-Lite is automatically offsetting my dates back 1 day through its GMT conversion and I cannot get it to stop. Please help!
Here is an example. If you look at the timeline graph it ends at 2020-11-19, but the final date in my data is 2020-11-20, and I need to make it so 2020-11-20 is the last date on my timeline graph.
This issue comes from an unfortunate "feature" of how javascript parses dates. Here is a minimal example of the problem you're seeing (open in editor):
{
"data": {
"values": [
{"date": "2020-11-17", "value": 5},
{"date": "2020-11-18", "value": 6},
{"date": "2020-11-19", "value": 7},
{"date": "2020-11-20", "value": 8}
]
},
"mark": "bar",
"encoding": {
"x": {"field": "value", "type": "quantitative"},
"y": {
"field": "date",
"timeUnit": "yearmonthdate",
"type": "ordinal"
},
"tooltip": [
{
"field": "date",
"timeUnit": "yearmonthdate",
"type": "temporal"
}
]
}
}
Each of the days in the chart are off by one compared to the input. So why is this happening?
Well, it turns out that the Vega-Lite's renderer makes use of Javascript's built-in date parsing, and Javascript's date parsing treats inputs differently depending on how they're formatted. In particular, Javascript will parse non-standard timestamps in UTC time, but will parse full ISO-8601 timestamps in local time, a fact you can confirm in your browser's javascript console (I executed this on a computer set to PST):
> new Date('2020-11-20')
Thu Nov 19 2020 16:00:00 GMT-0800 (Pacific Standard Time)
> new Date('2020-11-20T00:00:00')
Fri Nov 20 2020 00:00:00 GMT-0800 (Pacific Standard Time)
The Vega-Lite docs recommend using UTC timeUnits and scales to work around this, but I tend to find that approach a bit clunky. Instead, I try to always specify dates in Vega-Lite via full ISO 8601 timestamps.
In your case, the best approach is probably to use a calculate transform to regularize your dates, and proceed from there. Modifying the simplified example above, it might look something like this (open in editor):
{
"data": {
"values": [
{"date": "2020-11-17", "value": 5},
{"date": "2020-11-18", "value": 6},
{"date": "2020-11-19", "value": 7},
{"date": "2020-11-20", "value": 8}
]
},
"transform": [
{"calculate": "toDate(datum.date + 'T00:00:00')", "as": "date"}
],
"mark": "bar",
"encoding": {
"x": {"field": "value", "type": "quantitative"},
"y": {
"field": "date",
"timeUnit": "yearmonthdate",
"type": "ordinal"
},
"tooltip": [
{
"field": "date",
"timeUnit": "yearmonthdate",
"type": "temporal"
}
]
}
}
I'm trying to get some insights on page I own, for example page_fans which is stated in https://developers.facebook.com/docs/graph-api/reference/v3.2/insights as having both "day" and "lifetime" periods. (This is NOT the only metric I want to get but for the sake of simplicity I'll give it as an example.)
Getting it by day works, but I only wa
nt to know the lifetime value, so instead of period=day I tried period=lifetime, and I get an empty data array.
This is the request I am sending:
{page_id}/insights?fields=&metric=page_fans&period=lifetime
Note, if I change to period=day I get a nice object:
"data": [
{
"name": "page_fans",
"period": "day",
"values": [
{
"value": 21578,
"end_time": "2019-02-24T08:00:00+0000"
},
{
"value": 21580,
"end_time": "2019-02-25T08:00:00+0000"
}
],
"title": "Lifetime Total Likes",
"description": "Lifetime: The total number of people who have liked your Page. (Unique Users)",
"id": "{page_id}/insights/page_fans/day"
}
],
Other metrics I want to get lifetime data for include:
"post_video_view_time_by_region_id",
"post_video_views_by_distribution_type",
"post_video_view_time_by_distribution_type",
"post_video_view_time_by_country_id"
According to the changelog for v3.2
Page Insights
These changes apply to v3.2+, and will apply to all versions on January 21, 2019.
Period parameters have been changed to day from lifetime for
the following metrics. The lifetime period parameter will return
an empty dataset.
page_fans
page_fans_city
page_fans_country
page_fans_gender_age
page_fans_locale
You can use /insights?metric=page_fans&date_preset=maximum
can somebody let me know how to get previous days data i.e 2017-07-28 etc from my onpremises file system if my pipleline start and end dates are
"start": "2017-07-29T00:00:00Z",
"end": "2017-08-03T00:00:00Z"
My pipeline's input is"FileSystemSource" and output is "AzureDataLakeStore". I have tried below JSON in my copy pipeline as input
"inputs": [
{
"name": "OnPremisesFileInput2"
"startTime": "Date.AddDays(SliceStart, -1)",
"endTime": "Date.AddDays(SliceEnd, -1)"
}
]
I have also tried defining "offset" in the input and output datasets and in the pipeline as follows
"availability": {
"frequency": "Day",
"interval": 1,
"offset": "-1.00:00:00",
"style": "StartOfInterval"
},
"scheduler": {
"frequency": "Day",
"interval": 1,
"offset": "-1.00:00:00",
"style": "StartOfInterval"
},
none of the above seems to be working. Request someone to help me.
I think a good strategy to do this is to think about yesterday's output as today's input. Azure Data Factory let's you run activities one after another in sequence using different data sources.
There's good documentation here
With an example like this:
Like this you can either have a temporary storage in between the two activities or use your main input data source but with a filter to get only yesterday's slice.
Your offset should be positive.
"availability": {
"frequency": "Day",
"interval": 1,
"offset": "01:00:00",
"style": "EndOfInterval"
}
In this case it will run for example on September 7th at 1:00 AM UTC and will run the slice from Sep 6th 0:00 UTC to Sept 7th UTC. Which is yesterday slice.
Your input dataset should be configured to use the SliceStart for the naming of the file
"partitionedBy": [
{
"name": "Slice",
"value": {
"type": "DateTime",
"date": SliceStart",
"format": "yyyymmdd"
}
}],
"typeProperties": {
"fileName": "{slice}.csv",
}
It would look for 20170906.csv file when executed on Sept 7th.
Facebook recently added timezone less events (https://developers.facebook.com/roadmap/#timezone-less-events) to its Developer roadmap which says
"Since this migration was originally created, we have added a timezone field to events which indicates the name of the timezone (as defined here) where the event is expected to happen. FYI, developers reading time in ISO 8601 should be supporting the full standard when reading event times. Most events return local times (no GMT offset), but in the future events likely will return other formats (namely date-only and precise)."
It works for dates in ISO 8601 format but if I get dates in epoch format I always get +7 hrs difference.
e.g.
https://graph.facebook.com/369000383135224 returns
{
"id": "369000383135224",
"owner": {
"name": "Horst Uwe Peter",
"id": "1117563687"
},
"name": "Event in Dublin time 10:25",
"start_time": "2012-05-04T10:25:00",
"end_time": "2012-05-04T11:25:00",
"timezone": "Europe/Dublin",
"location": "Dublin, Ireland",
"venue": {
"id": "110769888951990"
},
"privacy": "FRIENDS",
"updated_time": "2012-05-04T09:27:29+0000",
"type": "event"
}
and
http://graph.facebook.com/369000383135224?date_format=U returns
{
"id": "369000383135224",
"owner": {
"name": "Horst Uwe Peter",
"id": "1117563687"
},
"name": "Event in Dublin time 10:25",
"start_time": 1336152300, <== Fri, 04 May 2012 17:25:00 GMT
"end_time": 1336155900, <== Fri, 04 May 2012 18:25:00 GMT
"timezone": "Europe/Dublin",
"location": "Dublin, Ireland",
"venue": {
"id": "110769888951990"
},
"privacy": "FRIENDS",
"updated_time": 1336123649,
"type": "event"
}
and with FQL using GRAPH end point
graph.facebook.com/fql?q=SELECT eid, name, description, location, venue, start_time, end_time, update_time, creator, privacy FROM event WHERE eid = 369000383135224
{
"data": [
{
"eid": 369000383135224,
"name": "Event in Dublin time 10:25",
"description": "",
"location": "Dublin, Ireland",
"venue": {
"id": 110769888951990
},
"start_time": 1336152300, <== Fri, 04 May 2012 18:25:00 GMT
"end_time": 1336155900, <== Fri, 04 May 2012 18:25:00 GMT
"update_time": 1336123649,
"creator": 1117563687,
"privacy": "FRIENDS"
}
]
}
does that mean migration works only for ISO 8601 formatted dates? and has no affect on FQL or epoch date format?
My events on a page I administer have never returned a timezone.
What I have found is that event times entered in the frontend dialog are treated as local times in "America/Los_Angeles" (complete with the US Daylight Savings Time changes, so you'll see +6 in winter and +7 in summer) and are then converted to "UTC" for storage in the database.
For display I use the following php function to show the correct times and note on the page that the times are local to the event's location:
function fb_event_time_convert($fb_time) {
$origin_dtz = new DateTimeZone('UTC');
$remote_dtz = new DateTimeZone('America/Los_Angeles');
$fb_time_str = '#' . $fb_time;
$origin_dt = new DateTime($fb_time_str, $origin_dtz);
$remote_dt = new DateTime($fb_time_str, $remote_dtz);
$offset = $origin_dtz->getOffset($origin_dt) - $remote_dtz->getOffset($remote_dt);
return $fb_time - $offset;
}