I would like to change the time zone of airflow from UTC to my current time zone(UTC+8 / Asia/Macau), and mark the DAG will run in Macau midnight not UTC midnight ( coz run in UTC midnight it make the data create time always be come previous day not today)
However I tried many method it still not work, I tried:
change DockerFile of the airflow project
# Airflow configurations
ENV AIRFLOW_HOME=/app/project/brownian
ENV AIRFLOW__CORE__LOAD_EXAMPLES=False
ENV AIRFLOW__CORE__LOGGING_LEVEL=DEBUG
ENV AIRFLOW__CORE__DAGS_FOLDER=/app/project/brownian/brownian/dags
ENV AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL=60
ENV AIRFLOW__SCHEDULER__SCHEDULER_MAX_THREADS=3
ENV AIRFLOW__CORE__DEFAULT_TIMEZONE=Asia/Macau
ENV AIRFLOW__WEBSERVER__DEFAULT_UI_TIMEZONE=Asia/Macau
Also tried to only change the DAG time znoe
tz = pytz.timezone('Asia/Macau')
def localize_utc_tz(d):
return tz.fromutc(d)
default_args = {
"start_date":datetime(2020, 7, 20),
}
dag = DAG("scrap_sns", default_args=default_args, schedule_interval="#daily",
user_defined_filters={
'localtz': localize_utc_tz,
})
However both of them are not work, the time show in airflow still is UTC, and the DAG still run in UTC midnight not my position midnight.
And method I can try?
Or on the other hand, I just want the scheduler run in UTC+8 midnight, could I change the code/ something to achieve this target?
You can solve it by add this environment "TZ: 'Asia/Tehran' " in the &airflow-common-env part in your docker-compose.yml file :
version: '3'
x-airflow-common:
&airflow-common
build: .
environment:
&airflow-common-env
TZ: 'Asia/Tehran'
It works for me :)
Related
I have automated test running in my CI/CD build pipeline, but the time in DevOps is UTC and my assertions tests check the local time.
Is there a way to set a time zone in my build pipeline?
Yes. For example this simple BASH script run using a Microsoft Hosted Agent:
echo "checking date"
date
echo "setting date to Asia/Kolkata"
sudo timedatectl set-timezone "Asia/Kolkata"
date
The results as seen in the log:
2019-07-05T20:26:48.5992486Z checking date
2019-07-05T20:26:48.5992954Z Fri Jul 5 20:26:48 UTC 2019
2019-07-05T20:26:48.5993264Z setting date to Asia/Kolkata
2019-07-05T20:26:48.9107025Z Sat Jul 6 01:56:48 IST 2019
As you can see, you can manipulate the local time on the agent. I do not agree with the other poster that this is necessarily a bad thing to do in the context of running tests.
You put some extra code in your tests to account for the local / target time or you could add 1 line into your build agent and achieve the same thing.
It just depends, the devil is in the details. Be careful with how you handle time.
using Powershell, you can do:
Get-TimeZone
Set-TimeZone "India Standard Time"
Get-TimeZone
Asia/Kolkata and "sudo timedatectl set-timezone" is not working on mac OS vmimage. Use below task for mac.
- task: CmdLine#2
displayName: Set system Timezone to IST
inputs:
script: |
sudo systemsetup -gettimezone
sudo systemsetup -settimezone Asia/Calcutta
sudo systemsetup -gettimezone
Also you can use this command to get the list of all valid timezones
sudo systemsetup -listtimezones
Additional to other replies for Windows Powershell and Linux Bash,
Mac OSX agent such xcode or Xamarin.iOS should use the script below to set timezone:
- task: Bash#3
displayName: Set Mac OS X Timezone
inputs:
targetType: 'inline'
script: sudo systemsetup -settimezone Asia/Shanghai
If ubuntu agent is used, the following commands can be used.
List timezones: timedatectl list-timezones
Set timezone: sudo timedatectl set-timezone '<your timezone>'
there is a way to compare between the current time using ansible_date_time
with another date (aws ec2 launch time) So I will be able to know if it happened in the last hour?
I have the following var ec2_launch_time: 2018-01-04T15:57:52+00:00
I know that I can compare days with the following method (from here and here):
"{{ (ansible_date_time.iso8601[:19] | to_datetime('%Y-%m-%dT%H:%M:%S') - ec2_launch_time[:19] | to_datetime('%Y-%m-%dT%H:%M:%S')).days }}"
if I will get 0 I will know that it happened on the same day but I'm looking for a way to get true or false if the difference between the dates are 1 hour (on the same day).
there is a filter for this or elegant way to write this or only using shell (like this)?
By subtracting two dates, you get a timedelta Python object.
You can use total_seconds method to get the exact difference in seconds. Add some basic arithmetic to get the number of full hours:
---
- hosts: localhost
gather_facts: no
connection: local
vars:
date_later: 2018-01-05 08:30:00
date_earlier: 2018-01-05 06:50:00
tasks:
- debug:
msg: "{{ ( (date_later - date_earlier).total_seconds() / 3600 ) | int }}"
returns:
ok: [localhost] => {
"msg": "1"
}
Use the above expression in the condition you want.
Since the date in BusyBox is not as powerful as gnu date, I have problems to calculate the date of last saturday.
last_sat=`date +"%Y-%m-%d" -d "last saturday"`
only works fine with gnu date.
I've found something like this to calculate from Epoch
busybox date -D '%s' -d "$(( `busybox date +%s`+3*60 ))"
but my BusyBox (v1.1.0) doesn't recognize the -D argument.
Any suggestions?
For the last Saturday before today, under busybox 1.16:
date -d "UTC 1970-01-01 $(date +"%s - 86400 - %w * 86400"|xargs expr) secs"
How it works: take the current date in seconds, subtract one day, subtract one day times the number of the current weekday, then convert those seconds back to a date.
EDIT: after hacking together a build of 1.1, this works:
date -d "1970.01.01-00:00:$(date +"%s - 86400 - %w * 86400"|xargs expr)"
This working version is based on code-reading:
} else if (t = *tm_time, sscanf(t_string, "%d.%d.%d-%d:%d:%d", &t.tm_year,
&t.tm_mon, &t.tm_mday,
&t.tm_hour, &t.tm_min,
&t.tm_sec) == 6) {
t.tm_year -= 1900; /* Adjust years */
t.tm_mon -= 1; /* Adjust dates from 1-12 to 0-11 */
BusyBox's date command has been the topic of some discussion over the years. Apparently it doesn't always work as documented, and it doesn't always work the same as previous versions.
On a BB system I administer running BusyBox v1.01, I'm able to use the -d option with dates in the format MMDDhhmmYYYY.ss, and in no other format that I've tried. Luckily, output formats work as expected, presumably because date is using a proper strftime() according to comments in the source.
Here's my forward-and-reverse example:
[~] # busybox date '+%m%d%H%M%Y.%S'
090500152016.41
[~] # busybox date -d 090500152016.41
Mon Sep 5 00:15:41 EDT 2016
So .. what can we do with this? It seems that we can't do an arbitrary adjustment of seconds, as it only reads the first two digits:
[~] # busybox date -d 123119001969.65 '+%s'
65
[~] # busybox date -d 123119001969.100 '+%s'
10
Well, it turns out you can load the date fields with "invalid" numbers.
[~] # busybox date 090100002016
Thu Sep 1 00:00:00 EDT 2016
[~] # busybox date 093400002016
Wed Oct 4 00:00:00 EDT 2016
[~] # busybox date 09-200002016
Mon Aug 29 00:00:00 EDT 2016
So let's adjust the "day" field using something based on %w.
today=$(busybox date '+%m%d%H%M%Y')
last_sat=$(busybox date -d "${today:0:2}$( printf '%02d' $(( 10#${today:2:2} - 1 - $(busybox date '+%w') )) )${today:4}" '+%F')
This simply subtracts numbers in the second field (the 3rd and 4th characters of the date string). It obviously requires that your shell either be bash or understand bash-style math notation ($((...))). Math-wise, it should work as long as "last saturday" is within the same month, and it MAY work (I haven't tested it) with rollvers to the previous month (per the last test above).
Rather than jumping through these burning hoops, I recommend you just install a GNU date binary, and don't use busybox for this one binary. :-P
Good luck!
I have several servers running under centos 6.3 and I faced issue that perl module DateTime treats Europe/Moscow timezone as UTC+3
[ulan#rt-virtual ~]$ perl -MDateTime -e 'print DateTime->now()->set_time_zone("Europe/Moscow"), "\n";'
2013-12-19T11:11:38
but in fact it is UTC+4 and system tools like zdump or date work correctly
[ulan#rt-virtual ~]$ zdump Europe/Moscow
Europe/Moscow Thu Dec 19 12:11:47 2013 MSK
I updated tzdata and DateTime module but it didn't help.
How can I amend this?
Thanks.
Well, DateTime module is doing its magic by following the rules specified in the TimeZone modules specific for each timezone. For Europe/Moscow, the module's is DateTime::TimeZone::Europe::Moscow. The problem is all the files are generated automatically corresponding to the rules existing when a specific version of DateTime module is released.
In this case one very important change - Russia's stopping following DST routines in 2011 - wasn't obviously reflected in that file. So updating - either the whole module or only the relevant TimeZone part - should have fixed the issue.
You can use your systems tzfile(5), using DateTime::TimeZone::Tzfile. Not only does it perform better than DateTime::TimeZone it also removes the need to have redundant data that needs to be in sync.
$tz = DateTime::TimeZone::Tzfile->new('/etc/localtime');
$dt = DateTime->now(time_zone => $tz);
I have the following few lines in a batch file:
#echo off
echo Running dump...
"D:\wamp\bin\mysql\mysql5.5.16\bin\mysqldump" --host="localhost" --user="****" --password="****" database> "D:\wamp\backup\mysql\"back.%date:~0,2%.sql
echo Done!
Supposedly, it should create a backup file with today's day as 01, 02, 03, ..., 31, that is,
back.01.sql
back.02.sql
back.03.sql
When I run it from CMD, it actually creates it as expected, but when it is run from the scheduler it looks like:
back.Mo.sql
back.Tu.sql
back.We.sql
What date format should I use to ensure it actually created with number of the day of the month?
Working with dates and times in Windows batch is a pain.
The most robust solution is to use WMIC to get the local date and time. It returns a value in the following format:
YYYYMMDDhhmmss.dddddd-ttt
YYYY = year
MM = month
DD = day
hh = hour in 24 hour format
mm = minutes
ss = seconds
dddddd = fractional seconds
ttt = time zone (minutes difference from Greenwich Mean Time)
So you can use the following to get the current day of the month in a variable
Edit - corrected starting substring location from 7 to 6
set curDate=
for /f "skip=1" %%x in ('wmic os get localdatetime') do if not defined curDate set curDate=%%x
set day=%curDate:~6,2%
The big advantage of this technique is it will work on any Windows machine in the world, regardless of the locale settings.
Here is your code with the technique inserted
Edit - corrected starting substring location from 7 to 6
#echo off
echo Running dump...
set curDate=
for /f "skip=1" %%x in ('wmic os get localdatetime') do if not defined curDate set curDate=%%x
set "day=%curDate:~6,2%"
"D:\wamp\bin\mysql\mysql5.5.16\bin\mysqldump" --host="localhost" --user="****" --password="****" database> "D:\wamp\backup\mysql\back.%day%.sql"
echo Done!
Just setup regional settings dd.MM.yyyy for the account under which your task is run. (use control panel or registry HKCU\Control Panel\International\sShortDate for Windows XP at least.