starting from about 2-3 weeks ago ntp lost its sync and since then it loses between 40 sec to 1 minute every day,
it then syncs this difference instantly, every day at around 9 pm.
/var/log/ntp.log :
8 Jun 09:01:54 ntpd[7688]: kernel time sync status change 4001
8 Jun 09:18:57 ntpd[7688]: kernel time sync status change 0001
8 Jun 09:53:08 ntpd[7688]: kernel time sync status change 4001
8 Jun 10:10:14 ntpd[7688]: kernel time sync status change 0001
8 Jun 12:26:45 ntpd[7688]: kernel time sync status change 4001
8 Jun 12:43:51 ntpd[7688]: kernel time sync status change 0001
8 Jun 14:43:19 ntpd[7688]: kernel time sync status change 4001
8 Jun 15:00:23 ntpd[7688]: kernel time sync status change 0001
8 Jun 16:25:45 ntpd[7688]: kernel time sync status change 4001
8 Jun 16:42:49 ntpd[7688]: kernel time sync status change 0001
8 Jun 17:16:59 ntpd[7688]: kernel time sync status change 4001
8 Jun 17:34:05 ntpd[7688]: kernel time sync status change 0001
8 Jun 19:16:30 ntpd[7688]: kernel time sync status change 4001
8 Jun 19:33:33 ntpd[7688]: kernel time sync status change 0001
8 Jun 20:59:28 ntpd[7688]: synchronized to LOCAL(0), stratum 10
8 Jun 21:17:43 ntpd[7688]: synchronized to 172.16.33.74, stratum 3
**8 Jun 21:33:19 ntpd[7688]: time reset -33.667071 s**
8 Jun 21:37:36 ntpd[7688]: synchronized to LOCAL(0), stratum 10
8 Jun 21:38:40 ntpd[7688]: synchronized to 172.16.33.74, stratum 3
9 Jun 10:56:38 ntpd[7688]: kernel time sync status change 4001
9 Jun 11:13:41 ntpd[7688]: kernel time sync status change 0001
9 Jun 12:04:56 ntpd[7688]: kernel time sync status change 4001
9 Jun 13:47:20 ntpd[7688]: kernel time sync status change 0001
9 Jun 14:21:31 ntpd[7688]: kernel time sync status change 4001
9 Jun 14:38:37 ntpd[7688]: kernel time sync status change 0001
9 Jun 20:11:40 ntpd[7688]: kernel time sync status change 4001
9 Jun 20:45:48 ntpd[7688]: kernel time sync status change 0001
9 Jun 21:03:40 ntpd[7688]: synchronized to LOCAL(0), stratum 10
9 Jun 21:21:35 ntpd[7688]: synchronized to 172.16.33.74, stratum 3
**9 Jun 21:54:16 ntpd[7688]: time reset -35.209243 s**
9 Jun 21:57:54 ntpd[7688]: synchronized to LOCAL(0), stratum 10
9 Jun 21:59:00 ntpd[7688]: synchronized to 172.16.33.74, stratum 3
10 Jun 20:57:04 ntpd[7688]: synchronized to LOCAL(0), stratum 10
10 Jun 21:03:59 ntpd[7688]: synchronized to 172.16.33.74, stratum 3
**10 Jun 21:18:25 ntpd[7688]: time reset -36.757260 s**
10 Jun 21:21:45 ntpd[7688]: synchronized to LOCAL(0), stratum 10
10 Jun 21:23:52 ntpd[7688]: synchronized to 172.16.33.74, stratum 3
11 Jun 21:07:25 ntpd[7688]: synchronized to LOCAL(0), stratum 10
11 Jun 21:24:25 ntpd[7688]: synchronized to 172.16.33.74, stratum 3
**11 Jun 21:40:52 ntpd[7688]: time reset -38.305589 s**
11 Jun 21:44:29 ntpd[7688]: synchronized to LOCAL(0), stratum 10
11 Jun 21:45:33 ntpd[7688]: synchronized to 172.16.33.74, stratum 3
12 Jun 08:37:43 ntpd[7688]: kernel time sync status change 4001
12 Jun 08:50:34 ntpd[7688]: kernel time sync status change 0001
12 Jun 21:01:10 ntpd[7688]: synchronized to LOCAL(0), stratum 10
12 Jun 21:07:53 ntpd[7688]: synchronized to 172.16.33.74, stratum 3
**12 Jun 21:22:13 ntpd[7688]: time reset -39.854883 s**
12 Jun 21:26:19 ntpd[7688]: synchronized to LOCAL(0), stratum 10
12 Jun 21:27:23 ntpd[7688]: synchronized to 172.16.33.74, stratum 3
13 Jun 02:26:51 ntpd[7688]: kernel time sync status change 4001
13 Jun 04:43:29 ntpd[7688]: kernel time sync status change 0001
13 Jun 05:34:43 ntpd[7688]: kernel time sync status change 4001
13 Jun 06:25:57 ntpd[7688]: kernel time sync status change 0001
13 Jun 07:34:16 ntpd[7688]: kernel time sync status change 4001
13 Jun 07:51:21 ntpd[7688]: kernel time sync status change 0001
13 Jun 13:18:34 ntpd[7688]: ntpd exiting on signal 15
13 Jun 13:21:51 ntpd[6771]: synchronized to LOCAL(0), stratum 10
13 Jun 13:21:51 ntpd[6771]: kernel time sync status change 0001
13 Jun 13:23:57 ntpd[6771]: synchronized to 172.16.33.74, stratum 3
13 Jun 21:10:58 ntpd[6771]: synchronized to LOCAL(0), stratum 10
13 Jun 21:27:09 ntpd[6771]: synchronized to 172.16.33.74, stratum 3
**13 Jun 22:00:28 ntpd[6771]: time reset -41.396780 s**
13 Jun 22:00:28 ntpd[6771]: kernel time sync status change 4001
13 Jun 22:04:38 ntpd[6771]: synchronized to LOCAL(0), stratum 10
13 Jun 22:04:38 ntpd[6771]: kernel time sync status change 0001
13 Jun 22:05:44 ntpd[6771]: synchronized to 172.16.33.74, stratum 3
14 Jun 04:09:19 ntpd[6771]: kernel time sync status change 4001
14 Jun 04:43:27 ntpd[6771]: kernel time sync status change 0001
14 Jun 06:08:50 ntpd[6771]: kernel time sync status change 4001
14 Jun 06:25:55 ntpd[6771]: kernel time sync status change 0001
14 Jun 07:00:05 ntpd[6771]: kernel time sync status change 4001
14 Jun 07:51:19 ntpd[6771]: kernel time sync status change 0001
14 Jun 08:25:28 ntpd[6771]: kernel time sync status change 4001
14 Jun 08:42:31 ntpd[6771]: kernel time sync status change 0001
14 Jun 10:59:05 ntpd[6771]: kernel time sync status change 4001
14 Jun 11:16:11 ntpd[6771]: kernel time sync status change 0001
14 Jun 12:24:27 ntpd[6771]: kernel time sync status change 4001
14 Jun 12:33:00 ntpd[6771]: kernel time sync status change 0001
14 Jun 16:40:51 ntpd[6771]: kernel time sync status change 4001
14 Jun 16:57:55 ntpd[6771]: kernel time sync status change 0001
14 Jun 17:49:11 ntpd[6771]: kernel time sync status change 4001
14 Jun 18:57:29 ntpd[6771]: kernel time sync status change 0001
14 Jun 19:31:38 ntpd[6771]: kernel time sync status change 4001
14 Jun 19:40:10 ntpd[6771]: kernel time sync status change 0001
14 Jun 20:56:52 ntpd[6771]: synchronized to LOCAL(0), stratum 10
14 Jun 21:03:37 ntpd[6771]: synchronized to 172.16.33.74, stratum 3
**14 Jun 21:16:53 ntpd[6771]: time reset -42.943002 s**
14 Jun 21:20:34 ntpd[6771]: synchronized to LOCAL(0), stratum 10
14 Jun 21:21:39 ntpd[6771]: synchronized to 172.16.33.74, stratum 3
15 Jun 04:57:42 ntpd[6771]: kernel time sync status change 4001
15 Jun 05:14:47 ntpd[6771]: kernel time sync status change 0001
15 Jun 06:23:10 ntpd[6771]: kernel time sync status change 4001
15 Jun 06:57:17 ntpd[6771]: kernel time sync status change 0001
15 Jun 09:13:47 ntpd[6771]: kernel time sync status change 4001
15 Jun 11:13:19 ntpd[6771]: kernel time sync status change 0001
15 Jun 12:04:32 ntpd[6771]: kernel time sync status change 4001
15 Jun 12:21:38 ntpd[6771]: kernel time sync status change 0001
15 Jun 13:12:54 ntpd[6771]: kernel time sync status change 4001
15 Jun 13:55:37 ntpd[6771]: kernel time sync status change 0001
15 Jun 15:55:11 ntpd[6771]: kernel time sync status change 4001
15 Jun 16:12:14 ntpd[6771]: kernel time sync status change 0001
15 Jun 20:59:18 ntpd[6771]: synchronized to LOCAL(0), stratum 10
15 Jun 21:21:16 ntpd[6771]: synchronized to 172.16.33.74, stratum 3
15 Jun 21:54:18 ntpd[6771]: time reset -44.484272 s
15 Jun 21:58:27 ntpd[6771]: synchronized to LOCAL(0), stratum 10
15 Jun 21:59:30 ntpd[6771]: synchronized to 172.16.33.74, stratum 3
16 Jun 03:53:30 ntpd[6771]: kernel time sync status change 4001
16 Jun 04:44:45 ntpd[6771]: kernel time sync status change 0001
16 Jun 05:53:03 ntpd[6771]: kernel time sync status change 4001
16 Jun 07:01:18 ntpd[6771]: kernel time sync status change 0001
16 Jun 08:26:43 ntpd[6771]: kernel time sync status change 4001
16 Jun 08:43:49 ntpd[6771]: kernel time sync status change 0001
16 Jun 09:35:06 ntpd[6771]: kernel time sync status change 4001
16 Jun 09:52:09 ntpd[6771]: kernel time sync status change 0001
16 Jun 10:26:19 ntpd[6771]: kernel time sync status change 4001
16 Jun 11:00:27 ntpd[6771]: kernel time sync status change 0001
16 Jun 11:34:37 ntpd[6771]: kernel time sync status change 4001
16 Jun 11:51:41 ntpd[6771]: kernel time sync status change 0001
16 Jun 12:42:59 ntpd[6771]: kernel time sync status change 4001
16 Jun 13:00:03 ntpd[6771]: kernel time sync status change 0001
16 Jun 14:25:25 ntpd[6771]: kernel time sync status change 4001
16 Jun 15:16:39 ntpd[6771]: kernel time sync status change 0001
16 Jun 16:24:57 ntpd[6771]: kernel time sync status change 4001
16 Jun 17:16:09 ntpd[6771]: kernel time sync status change 0001
16 Jun 18:58:35 ntpd[6771]: kernel time sync status change 4001
16 Jun 19:15:40 ntpd[6771]: kernel time sync status change 0001
16 Jun 20:59:36 ntpd[6771]: synchronized to LOCAL(0), stratum 10
16 Jun 21:16:46 ntpd[6771]: synchronized to 172.16.33.74, stratum 3
16 Jun 21:49:33 ntpd[6771]: time reset -46.023214 s
16 Jun 21:52:53 ntpd[6771]: synchronized to LOCAL(0), stratum 10
16 Jun 21:53:57 ntpd[6771]: synchronized to 172.16.33.74, stratum 3
17 Jun 03:12:33 ntpd[6771]: kernel time sync status change 4001
17 Jun 03:29:36 ntpd[6771]: kernel time sync status change 0001
17 Jun 04:20:49 ntpd[6771]: kernel time sync status change 4001
17 Jun 05:12:01 ntpd[6771]: kernel time sync status change 0001
17 Jun 07:28:37 ntpd[6771]: kernel time sync status change 4001
17 Jun 08:02:43 ntpd[6771]: kernel time sync status change 0001
17 Jun 09:28:07 ntpd[6771]: kernel time sync status change 4001
17 Jun 10:02:13 ntpd[6771]: kernel time sync status change 0001
17 Jun 11:10:31 ntpd[6771]: kernel time sync status change 4001
17 Jun 11:27:36 ntpd[6771]: kernel time sync status change 0001
17 Jun 17:01:37 ntpd[6771]: kernel time sync status change 4001
17 Jun 17:18:42 ntpd[6771]: kernel time sync status change 0001
17 Jun 20:09:32 ntpd[6771]: kernel time sync status change 4001
17 Jun 21:01:48 ntpd[6771]: synchronized to LOCAL(0), stratum 10
17 Jun 21:01:48 ntpd[6771]: kernel time sync status change 0001
17 Jun 21:19:34 ntpd[6771]: synchronized to 172.16.33.74, stratum 3
17 Jun 21:52:12 ntpd[6771]: time reset -47.573538 s
17 Jun 21:56:07 ntpd[6771]: synchronized to LOCAL(0), stratum 10
17 Jun 21:57:11 ntpd[6771]: synchronized to 172.16.33.74, stratum 3
18 Jun 20:59:25 ntpd[6771]: synchronized to LOCAL(0), stratum 10
18 Jun 21:24:16 ntpd[6771]: synchronized to 172.16.33.74, stratum 3
18 Jun 21:40:42 ntpd[6771]: time reset -49.122802 s
18 Jun 21:44:59 ntpd[6771]: synchronized to LOCAL(0), stratum 10
18 Jun 21:46:03 ntpd[6771]: synchronized to 172.16.33.74, stratum 3
19 Jun 09:30:18 ntpd[6771]: kernel time sync status change 4001
19 Jun 11:12:46 ntpd[6771]: kernel time sync status change 0001
19 Jun 11:46:55 ntpd[6771]: kernel time sync status change 4001
19 Jun 12:38:10 ntpd[6771]: kernel time sync status change 0001
19 Jun 14:03:33 ntpd[6771]: kernel time sync status change 4001
19 Jun 14:20:36 ntpd[6771]: kernel time sync status change 0001
19 Jun 15:11:48 ntpd[6771]: kernel time sync status change 4001
19 Jun 15:45:55 ntpd[6771]: kernel time sync status change 0001
19 Jun 16:20:04 ntpd[6771]: kernel time sync status change 4001
19 Jun 16:37:08 ntpd[6771]: kernel time sync status change 0001
19 Jun 17:11:19 ntpd[6771]: kernel time sync status change 4001
19 Jun 17:28:24 ntpd[6771]: kernel time sync status change 0001
19 Jun 19:36:29 ntpd[6771]: kernel time sync status change 4001
19 Jun 19:53:33 ntpd[6771]: kernel time sync status change 0001
19 Jun 21:02:58 ntpd[6771]: synchronized to LOCAL(0), stratum 10
19 Jun 21:20:53 ntpd[6771]: synchronized to 172.16.33.74, stratum 3
19 Jun 21:37:16 ntpd[6771]: time reset -50.669270 s
19 Jun 21:41:34 ntpd[6771]: synchronized to LOCAL(0), stratum 10
19 Jun 21:42:07 ntpd[6771]: synchronized to 172.16.33.74, stratum 3
at this point i set up a cron job to run ntpdate every hour:
20 Jun 17:57:39 ntpdate[22583]: adjust time server 172.16.33.74 offset -0.000421 sec
20 Jun 18:01:01 ntpdate[22623]: adjust time server 172.16.33.74 offset 0.000238 sec
20 Jun 19:01:01 ntpdate[23046]: adjust time server 172.16.33.74 offset 0.000753 sec
20 Jun 20:01:02 ntpdate[23471]: adjust time server 172.16.33.74 offset 0.000574 sec
20 Jun 21:00:09 ntpdate[23890]: step time server 172.16.33.74 offset -52.211231 sec
20 Jun 21:01:01 ntpdate[23909]: adjust time server 172.16.33.74 offset 0.000033 sec
20 Jun 22:01:01 ntpdate[24332]: adjust time server 172.16.33.74 offset 0.000870 sec
20 Jun 23:01:02 ntpdate[24757]: adjust time server 172.16.33.74 offset 0.000499 sec
21 Jun 00:01:01 ntpdate[25188]: adjust time server 172.16.33.74 offset 0.004997 sec
21 Jun 01:01:01 ntpdate[25611]: adjust time server 172.16.33.74 offset -0.000058 sec
21 Jun 02:01:01 ntpdate[26035]: adjust time server 172.16.33.74 offset 0.001489 sec
21 Jun 03:01:01 ntpdate[26460]: adjust time server 172.16.33.74 offset 0.000491 sec
21 Jun 04:01:01 ntpdate[27712]: adjust time server 172.16.33.74 offset 0.000730 sec
21 Jun 05:01:01 ntpdate[28270]: adjust time server 172.16.33.74 offset 0.000841 sec
21 Jun 06:01:01 ntpdate[28753]: adjust time server 172.16.33.74 offset 0.000405 sec
21 Jun 07:01:01 ntpdate[29177]: adjust time server 172.16.33.74 offset 0.000474 sec
21 Jun 08:01:01 ntpdate[29601]: adjust time server 172.16.33.74 offset 0.000583 sec
21 Jun 09:01:01 ntpdate[30024]: adjust time server 172.16.33.74 offset 0.000660 sec
21 Jun 10:01:01 ntpdate[30448]: adjust time server 172.16.33.74 offset 0.000469 sec
21 Jun 11:01:01 ntpdate[30872]: adjust time server 172.16.33.74 offset 0.000605 sec
21 Jun 12:01:01 ntpdate[31298]: adjust time server 172.16.33.74 offset 0.000159 sec
21 Jun 13:01:01 ntpdate[31721]: adjust time server 172.16.33.74 offset 0.000539 sec
21 Jun 14:01:01 ntpdate[32146]: adjust time server 172.16.33.74 offset 0.000530 sec
21 Jun 15:01:01 ntpdate[32570]: adjust time server 172.16.33.74 offset 0.000560 sec
21 Jun 16:01:01 ntpdate[583]: adjust time server 172.16.33.74 offset 0.000820 sec
21 Jun 17:01:01 ntpdate[1020]: adjust time server 172.16.33.74 offset 0.000512 sec
21 Jun 18:01:01 ntpdate[1445]: adjust time server 172.16.33.74 offset 0.000239 sec
21 Jun 19:01:01 ntpdate[1870]: adjust time server 172.16.33.74 offset 0.000607 sec
21 Jun 20:01:01 ntpdate[2294]: adjust time server 172.16.33.74 offset 0.000717 sec
21 Jun 21:01:02 ntpdate[2743]: adjust time server 172.16.33.74 offset 0.000965 sec
21 Jun 22:00:07 ntpdate[3174]: step time server 172.16.33.74 offset -53.812615 sec
21 Jun 22:01:01 ntpdate[3196]: adjust time server 172.16.33.74 offset 0.000024 sec
21 Jun 23:01:01 ntpdate[3641]: adjust time server 172.16.33.74 offset 0.000496 sec
22 Jun 00:01:01 ntpdate[4133]: adjust time server 172.16.33.74 offset 0.000920 sec
22 Jun 01:01:01 ntpdate[4557]: adjust time server 172.16.33.74 offset 0.000914 sec
22 Jun 02:01:01 ntpdate[4980]: adjust time server 172.16.33.74 offset 0.000702 sec
22 Jun 03:01:01 ntpdate[5404]: adjust time server 172.16.33.74 offset 0.000663 sec
22 Jun 04:01:01 ntpdate[6631]: adjust time server 172.16.33.74 offset 0.000608 sec
22 Jun 05:01:01 ntpdate[7189]: adjust time server 172.16.33.74 offset 0.000650 sec
22 Jun 06:01:01 ntpdate[7613]: adjust time server 172.16.33.74 offset 0.000895 sec
22 Jun 07:01:01 ntpdate[8037]: adjust time server 172.16.33.74 offset 0.000843 sec
22 Jun 08:01:01 ntpdate[8462]: adjust time server 172.16.33.74 offset 0.000559 sec
22 Jun 09:01:01 ntpdate[8885]: adjust time server 172.16.33.74 offset 0.000595 sec
22 Jun 10:01:01 ntpdate[9309]: adjust time server 172.16.33.74 offset 0.000617 sec
22 Jun 11:01:01 ntpdate[9734]: adjust time server 172.16.33.74 offset 0.000527 sec
22 Jun 12:01:01 ntpdate[10158]: adjust time server 172.16.33.74 offset 0.000546 sec
22 Jun 13:01:01 ntpdate[10582]: adjust time server 172.16.33.74 offset 0.000663 sec
22 Jun 14:01:01 ntpdate[11006]: adjust time server 172.16.33.74 offset 0.000790 sec
22 Jun 15:01:01 ntpdate[11429]: adjust time server 172.16.33.74 offset 0.000599 sec
22 Jun 16:01:01 ntpdate[11854]: adjust time server 172.16.33.74 offset 0.001093 sec
22 Jun 17:01:02 ntpdate[12277]: adjust time server 172.16.33.74 offset 0.000444 sec
22 Jun 18:01:01 ntpdate[12702]: adjust time server 172.16.33.74 offset 0.001054 sec
22 Jun 19:01:01 ntpdate[13126]: adjust time server 172.16.33.74 offset 0.000751 sec
22 Jun 20:01:01 ntpdate[13550]: adjust time server 172.16.33.74 offset 0.000832 sec
22 Jun 21:01:01 ntpdate[13973]: adjust time server 172.16.33.74 offset 0.001551 sec
22 Jun 22:00:06 ntpdate[14398]: step time server 172.16.33.74 offset -55.338569 sec
22 Jun 22:01:01 ntpdate[14418]: adjust time server 172.16.33.74 offset 0.000044 sec
22 Jun 23:01:01 ntpdate[14845]: adjust time server 172.16.33.74 offset 0.000949 sec
23 Jun 00:01:01 ntpdate[15276]: adjust time server 172.16.33.74 offset 0.000699 sec
23 Jun 01:01:01 ntpdate[15700]: adjust time server 172.16.33.74 offset 0.001248 sec
23 Jun 02:01:01 ntpdate[16123]: adjust time server 172.16.33.74 offset 0.001125 sec
23 Jun 03:01:02 ntpdate[16547]: adjust time server 172.16.33.74 offset -0.000017 sec
23 Jun 04:01:01 ntpdate[17812]: adjust time server 172.16.33.74 offset 0.000511 sec
23 Jun 05:01:01 ntpdate[18371]: adjust time server 172.16.33.74 offset 0.000460 sec
23 Jun 06:01:01 ntpdate[18795]: adjust time server 172.16.33.74 offset 0.000756 sec
23 Jun 07:01:01 ntpdate[19219]: adjust time server 172.16.33.74 offset 0.000756 sec
23 Jun 08:01:01 ntpdate[19643]: adjust time server 172.16.33.74 offset 0.000925 sec
23 Jun 09:01:01 ntpdate[20066]: adjust time server 172.16.33.74 offset 0.000770 sec
23 Jun 10:01:01 ntpdate[20491]: adjust time server 172.16.33.74 offset 0.000929 sec
23 Jun 11:01:01 ntpdate[20915]: adjust time server 172.16.33.74 offset 0.001033 sec
23 Jun 12:01:01 ntpdate[21339]: adjust time server 172.16.33.74 offset 0.000177 sec
23 Jun 13:01:01 ntpdate[21763]: adjust time server 172.16.33.74 offset 0.000831 sec
23 Jun 14:01:01 ntpdate[22187]: adjust time server 172.16.33.74 offset 0.000671 sec
23 Jun 15:01:01 ntpdate[22611]: adjust time server 172.16.33.74 offset 0.000742 sec
23 Jun 16:01:01 ntpdate[23034]: adjust time server 172.16.33.74 offset 0.000978 sec
23 Jun 17:01:01 ntpdate[23459]: adjust time server 172.16.33.74 offset 0.000954 sec
23 Jun 18:01:02 ntpdate[23882]: adjust time server 172.16.33.74 offset 0.000546 sec
23 Jun 19:01:01 ntpdate[24307]: adjust time server 172.16.33.74 offset 0.001156 sec
23 Jun 20:01:01 ntpdate[24731]: adjust time server 172.16.33.74 offset 0.000255 sec
23 Jun 21:01:02 ntpdate[25155]: adjust time server 172.16.33.74 offset 0.001095 sec
23 Jun 22:00:04 ntpdate[25579]: step time server 172.16.33.74 offset -56.876308 sec
23 Jun 22:01:01 ntpdate[25600]: adjust time server 172.16.33.74 offset 0.000056 sec
23 Jun 23:01:01 ntpdate[26019]: adjust time server 172.16.33.74 offset 0.001729 sec
24 Jun 00:01:02 ntpdate[26450]: adjust time server 172.16.33.74 offset 0.000364 sec
then added cron every 5 minutes:
24 Jun 22:00:01 ntpdate[5200]: adjust time server 172.16.33.74 offset 0.000874 sec
24 Jun 22:01:01 ntpdate[5217]: adjust time server 172.16.33.74 offset -0.000417 sec
24 Jun 22:05:01 ntpdate[5250]: adjust time server 172.16.33.74 offset 0.000289 sec
24 Jun 22:10:01 ntpdate[5288]: adjust time server 172.16.33.74 offset -0.000060 sec
**24 Jun 22:14:02 ntpdate[5326]: step time server 172.16.33.74 offset -58.428188 sec**
24 Jun 22:15:01 ntpdate[5332]: adjust time server 172.16.33.74 offset 0.000055 sec
**24 Jun 22:19:02 ntpdate[5372]: step time server 172.16.33.74 offset -58.458807 sec**
24 Jun 22:20:02 ntpdate[5380]: adjust time server 172.16.33.74 offset 0.000030 sec
24 Jun 22:25:01 ntpdate[5416]: adjust time server 172.16.33.74 offset 0.000078 sec
24 Jun 22:30:01 ntpdate[5454]: adjust time server 172.16.33.74 offset 0.000039 sec
24 Jun 22:35:01 ntpdate[5490]: adjust time server 172.16.33.74 offset 0.000058 sec
24 Jun 22:40:01 ntpdate[5530]: adjust time server 172.16.33.74 offset 0.000038 sec
24 Jun 22:45:01 ntpdate[5566]: adjust time server 172.16.33.74 offset 0.000055 sec
24 Jun 22:50:01 ntpdate[5604]: adjust time server 172.16.33.74 offset 0.000039 sec
24 Jun 22:55:01 ntpdate[5640]: adjust time server 172.16.33.74 offset 0.000060 sec
24 Jun 23:01:01 ntpdate[5693]: adjust time server 172.16.33.74 offset 0.000043 sec
there are two times there the clock skips by almost 1 minute.
already checked other log files on the server and there is literally no data writen to any log other then ntp.log at this time.
there are no backup tasks etc. running at this time.
the server is a VMWARE virtual machine, and time synchronization with the host has been disabled.
server os is openfiler 2.3
uname -a
Linux xxxxxxxxxxxxxxxxxxx 2.6.29.6-0.23.smp.gcc3.4.x86.i686 #1 SMP Mon Feb 8 16:28:15 GMT 2010 i686 i686 i386 GNU/Linux
Related
Problem
I have a Rails 7 app deployed on render.com, and it doesn't get a lot of traffic (maybe once per day). However, when a few requests do come in, everything seems to running fine for a moment until Puma seems to barf. The incoming requests are from Twilio for a voice call, and the call eventually errors with "We're sorry, an application error has occurred. Goodbye". It seems like something about a "timed out" worker happens, then the worker boots, and whammo! a flood of "Completed 2XX OK" and "Kredis Connected to shared" lines come crashing through like they've been pent up the entire time. THEN, nearly a day later without any outside requests coming in, several log lines about Out-of-sync worker list, no 78 worker come through. My Puma config file is unchanged from what ships with Rails.
Questions
Where might I go look for the offending code? What tools could help me decipher why a Puma worker is timing out? Could it have something to do with how I'm using Redis via Kredis in my app?
Workaround
To get around this issue, I've started to occasionally redeploy my latest commit and that seems to help. I'm not certain, but it seems like inactivity causes Puma to become discombobulated.
Log output
Here's what the offending lines in my log file look like:
... a few requests that complete 200 OK ...
Sep 13 05:53:15 PM [70] ! Terminating timed out worker (worker failed to check in within 60 seconds): 90
... a couple more normal log lines and then ...
Sep 13 05:53:16 PM [70] - Worker 3 (PID: 134) booted in 0.04s, phase: 0
... some more normal log lines and then ...
Sep 13 05:53:16 PM I, [2022-09-13T22:53:16.593713 #74] INFO -- : [595ad8e5-fa3a-45a3-8c5b-a506e6c94b69] Completed 204 No Content in 110ms (Allocations: 13681)
Sep 13 05:53:16 PM I, [2022-09-13T22:53:16.425579 #86] INFO -- : [f1a64c71-8048-4032-8bf6-2e68aa1fa7ba] Completed 204 No Content in 2ms (Allocations: 541)
Sep 13 05:53:16 PM I, [2022-09-13T22:53:16.595408 #86] INFO -- : [68d19bd9-2286-4f75-a982-5fa3e864d6ac] Completed 200 OK in 105ms (Views: 0.2ms | Allocations: 1592)
Sep 13 05:53:16 PM I, [2022-09-13T22:53:16.614951 #76] INFO -- : [e883350f-9a26-4d3d-8f1c-4853285aa71a] Kredis (10.6ms) Connected to shared
Sep 13 05:53:16 PM I, [2022-09-13T22:53:16.615787 #76] INFO -- : [fbcd8730-1514-4af5-9332-0bdf0c89fc2d] Kredis (17.2ms) Connected to shared
Sep 13 05:53:16 PM I, [2022-09-13T22:53:16.705926 #86] INFO -- : [1f67a177-38f2-4bf5-bd03-1c59a3edb3a4] Kredis (224.1ms) Connected to shared
Sep 13 05:53:16 PM I, [2022-09-13T22:53:16.958386 #76] INFO -- : [e883350f-9a26-4d3d-8f1c-4853285aa71a] Completed 200 OK in 472ms (ActiveRecord: 213.1ms | Allocations: 32402)
Sep 13 05:53:17 PM I, [2022-09-13T22:53:17.034211 #86] INFO -- : [1f67a177-38f2-4bf5-bd03-1c59a3edb3a4] Completed 200 OK in 606ms (ActiveRecord: 256.6ms | Allocations: 17832)
Sep 13 05:53:17 PM I, [2022-09-13T22:53:17.136231 #76] INFO -- : [fbcd8730-1514-4af5-9332-0bdf0c89fc2d] Completed 200 OK in 654ms (ActiveRecord: 88.0ms | Allocations: 37385)
... literally a day later without any other activity ...
Sep 14 05:02:29 AM [69] ! Terminating timed out worker (worker failed to check in within 60 seconds): 78
Sep 14 05:02:31 AM [69] ! Out-of-sync worker list, no 78 worker
Sep 14 05:02:31 AM [69] ! Out-of-sync worker list, no 78 worker
Sep 14 05:02:31 AM [69] ! Out-of-sync worker list, no 78 worker
Sep 14 05:02:31 AM [69] ! Out-of-sync worker list, no 78 worker
Sep 14 05:02:31 AM [69] ! Out-of-sync worker list, no 78 worker
Sep 14 05:02:31 AM [69] ! Out-of-sync worker list, no 78 worker
Sep 14 05:02:31 AM [69] - Worker 1 (PID: 132) booted in 0.03s, phase: 0
We use a redis cache in out kubernetes cluster which stops working really randomly. It's a Standalone version based on this image: bitnami/redis:6.0.15
As custom parameters we use:
MASTER true
REDIS_AOF_ENABLED no
Every time when the redis stop working I see the following logs:
Jul 5 13:30:27 redis-0 redis 1:M 05 Jul 2022 11:30:27.060 * 10000 changes in 60 seconds. Saving...
Jul 5 13:30:27 redis-0 redis 1:M 05 Jul 2022 11:30:27.090 * Background saving started by pid 364
Jul 5 13:31:34 redis-0 redis 364:C 05 Jul 2022 11:31:34.307 * DB saved on disk
Jul 5 13:31:34 redis-0 redis 364:C 05 Jul 2022 11:31:34.341 * RDB: 431 MB of memory used by copy-on-write
Jul 5 13:31:34 redis-0 redis 1:M 05 Jul 2022 11:31:34.488 * Background saving terminated with success
Jul 5 13:32:35 redis-0 redis 1:M 05 Jul 2022 11:32:35.022 * 10000 changes in 60 seconds. Saving...
Jul 5 13:32:35 redis-0 redis 1:M 05 Jul 2022 11:32:35.052 * Background saving started by pid 365
-----
Jul 5 13:32:40 redis-0 redis 1:S 05 Jul 2022 11:32:40.436 * Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
Jul 5 13:32:40 redis-0 redis 1:S 05 Jul 2022 11:32:40.436 * REPLICAOF 178.20.40.200:8886 enabled (user request from 'id=71457 addr=10.0.16.46:14072 fd=12 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=47 qbuf-free=32721 argv-mem=24 obl=0 oll=0 omem=0 tot-mem=61488 events=r cmd=slaveof user=default')
Jul 5 13:32:41 redis-0 redis 1:S 05 Jul 2022 11:32:41.316 * Connecting to MASTER 178.20.40.200:8886
Jul 5 13:32:41 redis-0 redis 1:S 05 Jul 2022 11:32:41.316 * MASTER <-> REPLICA sync started
Jul 5 13:32:41 redis-0 redis 1:S 05 Jul 2022 11:32:41.362 * Non blocking connect for SYNC fired the event.
Jul 5 13:32:41 redis-0 redis Error 1:S 05 Jul 2022 11:32:41.409 # Error reply to PING from master: '-Reading from master: Connection reset by peer'
Jul 5 13:32:42 redis-0 redis 1:S 05 Jul 2022 11:32:42.316 * Connecting to MASTER 178.20.40.200:8886
Jul 5 13:32:42 redis-0 redis 1:S 05 Jul 2022 11:32:42.317 * MASTER <-> REPLICA sync started
Jul 5 13:32:42 redis-0 redis 1:S 05 Jul 2022 11:32:42.366 * Non blocking connect for SYNC fired the event.
Jul 5 13:32:42 redis-0 redis Error 1:S 05 Jul 2022 11:32:42.415 # Error reply to PING from master: '-Reading from master: Connection reset by peer'
Jul 5 13:32:43 redis-0 redis 1:S 05 Jul 2022 11:32:43.317 * Connecting to MASTER 178.20.40.200:8886
Jul 5 13:32:43 redis-0 redis 1:S 05 Jul 2022 11:32:43.317 * MASTER <-> REPLICA sync started
Jul 5 13:32:43 redis-0 redis 1:S 05 Jul 2022 11:32:43.366 * Non blocking connect for SYNC fired the event.
Jul 5 13:32:43 redis-0 redis Error 1:S 05 Jul 2022 11:32:43.416 # Error reply to PING from master: '-Reading from master: Connection reset by peer'
Jul 5 13:32:44 redis-0 redis 1:S 05 Jul 2022 11:32:44.320 * Connecting to MASTER 178.20.40.200:8886
Jul 5 13:32:44 redis-0 redis 1:S 05 Jul 2022 11:32:44.320 * MASTER <-> REPLICA sync started
Jul 5 13:32:44 redis-0 redis 1:S 05 Jul 2022 11:32:44.370 * Non blocking connect for SYNC fired the event.
Then I see that the queue increase, but I need to kill the pod to restart redis otherwise it will not work anymore.
next: GET 6126674261995698486,
inst: 1,
qu: 0, // queue => waiting operations
qs: 17,
aw: False,
rs: ReadAsync,
ws: Idle,
in: 0, // bytes waiting from input stream
in-pipe: 0,
out-pipe: 0,
serverEndpoint: redis.default.svc.cluster.local:6379,
mc: 1/1/0,
mgr: 10 of 10 available, // tread pool
clientName: production-9bbd94544-nlmv7,
IOCP: (Busy=0,Free=1000,Min=5,Max=1000), // no busy threads
WORKER: (Busy=14,Free=32753,Min=256,Max=32767),
v: 2.2.4.27433```
```Timeout performing GET (3000ms),
next: 2865582319381864083,
inst: 0,
qu: 0,
qs: 333,
aw: False,
rs: ReadAsync,
ws: Idle,
in: 0,
in-pipe: 0,
out-pipe: 0,
serverEndpoint: redis.default.svc.cluster.local:6379,
mc: 1/1/0,
mgr: 10 of 10 available,
clientName: production-58c7874fd8-tdcpz,
IOCP: (Busy=0,Free=1000,Min=1,Max=1000),
WORKER: (Busy=3,Free=32764,Min=256,Max=32767),
v: 2.2.4.27433
next: GET 6126674261995698486,
inst: 47,
qu: 0,
qs: 21368,
aw: False,
rs: ReadAsync,
ws: Idle,
in: 0,
in-pipe: 0,
out-pipe: 0,
serverEndpoint: redis.default.svc.cluster.local:6379,
mc: 1/1/0,
mgr: 10 of 10 available,
clientName: production-9bbd94544-nlmv7,
IOCP: (Busy=0,Free=1000,Min=5,Max=1000),
WORKER: (Busy=162,Free=32605,Min=256,Max=32767),
v: 2.2.4.27433```
Has anyone an idea?
Thank you.
Can you check if the redis service is running? kubectl get service/redis. It seems like the service is unable to receive traffic, which is possible if there are no pods to receive it.
I just upgraded my MapR Cluster and I am trying to start Streamsets. However I get the following Error:
Exception in thread "main" java.lang.ExceptionInInitializerError: Expected exactly 1 stage lib jar but found 0 with name streamsets-datacollector-mapr_6_0-mep4-lib Jun 12 09:48:56 BGDTEST5 streamsets[24543]: at com.streamsets.pipeline.SDCClassLoader.bringStageAndProtoLibsToFront(SDCClassLoader.java:186) Jun 12 09:48:56 BGDTEST5 streamsets[24543]: at com.streamsets.pipeline.SDCClassLoader.getOrderedURLsForClassLoader(SDCClassLoader.java:404) Jun 12 09:48:56 BGDTEST5 streamsets[24543]: at com.streamsets.pipeline.SDCClassLoader.(SDCClassLoader.java:119) Jun 12 09:48:56 BGDTEST5 streamsets[24543]: at com.streamsets.pipeline.SDCClassLoader.getStageClassLoader(SDCClassLoader.java:389) Jun 12 09:48:56 BGDTEST5 streamsets[24543]: at com.streamsets.pipeline.SDCClassLoader.getStageClassLoader(SDCClassLoader.java:383) Jun 12 09:48:56 BGDTEST5 streamsets[24543]: at com.streamsets.pipeline.BootstrapMain.main(BootstrapMain.java:291) Jun 12 09:48:56 BGDTEST5 systemd[1]: sdc.service: main process exited, code=exited, status=1/FAILURE Jun 12 09:48:56 BGDTEST5 systemd[1]: Unit sdc.service entered failed state. Jun 12 09:48:56 BGDTEST5 systemd[1]: sdc.service failed.
I can see the stage-lib in question in the streamsets-libs directory. This used to work seamlessly before. What am I doing wrong?
My released app (Hidden London - Free App) has been reported as crashing on iPod touch 2nd and 3rd Generation. It works without issue on iPhone and iPad.
The crash log is as follows:
Wed Apr 13 09:03:40 unknown kernel[0] <Debug>: launchd[908] Builtin profile: container (sandbox)
Wed Apr 13 09:03:40 unknown kernel[0] <Debug>: launchd[908] Container: /private/var/mobile/Applications/E509FC07-A742-4483-95B1-7393B3F129B9 [69] (sandbox)
Wed Apr 13 09:03:40 unknown sandboxd[911] <Notice>: HiddenLondon(908) deny file-write-data /private/var/mobile/Applications/E509FC07-A742-4483-95B1-7393B3F129B9/HiddenLondon.app/HiddenLondon.sqlite
Wed Apr 13 09:03:40 unknown HiddenLondon[908] <Warning>: Not purchased: com.Beutelkind.HiddenLondon.ExtraPlaces
Wed Apr 13 09:03:40 unknown HiddenLondon[908] <Warning>: Not purchased: com.Beutelkind.HiddenLondon.RemoveAds
Wed Apr 13 09:03:42 unknown securityd[910] <Error>: OCSPResponse: now + maxAge > latestNextUpdate, using latestNextUpdate
Wed Apr 13 09:03:42 unknown com.apple.SpringBoard[24] <Notice>: CoreAnimation: timed out fence 500
Wed Apr 13 09:04:00 unknown HiddenLondon[908] <Warning>: ADBannerView: WARNING A banner view (0x159470) has an ad but may be obscured. This message is only printed once per banner view.
Wed Apr 13 09:04:31 unknown configd[22] <Notice>: jetsam: kernel memory event (90), free: 323, active: 1250, inactive: 1494, purgeable: 0, wired: 7841
Wed Apr 13 09:04:31 unknown configd[22] <Notice>: jetsam: kernel termination snapshot being created
Wed Apr 13 09:04:31 unknown com.apple.launchd[1] <Notice>: (UIKitApplication:com.apple.AdSheetPhone[0xabc0]) Exited: Killed
Wed Apr 13 09:04:31 unknown com.apple.launchd[1] <Notice>: (UIKitApplication:com.Beutelkind.HiddenLondon[0x2fef]) Exited: Killed
Wed Apr 13 09:04:31 unknown SpringBoard[24] <Warning>: Received memory warning. Level=1
Wed Apr 13 09:04:31 unknown com.apple.launchd[1] <Notice>: (UIKitApplication:com.apple.mobilemail[0x736e]) Exited: Killed
Wed Apr 13 09:04:31 unknown SpringBoard[24] <Warning>: Received memory warning. Level=1
Wed Apr 13 09:04:32 unknown SpringBoard[24] <Warning>: Application 'AdSheet' exited abnormally with signal 9: Killed
Wed Apr 13 09:04:32 unknown SpringBoard[24] <Warning>: Application 'HiddenLondon' exited abnormally with signal 9: Killed
Wed Apr 13 09:04:32 unknown SpringBoard[24] <Warning>: Application 'Mail' exited abnormally with signal 9: Killed
Wed Apr 13 09:04:32 unknown SpringBoard[24] <Warning>: Memory level is not normal (56%). Delaying auto-relaunch of 'Mail' for 30 seconds.
Wed Apr 13 09:04:32 unknown SpringBoard[24] <Warning>: Received memory warning. Level=1
Wed Apr 13 09:04:33 unknown ReportCrash[917] <Error>: Saved crashreport to /Library/Logs/CrashReporter/LowMemory-2011-04-13-090433.plist using uid: 0 gid: 0, synthetic_euid: 0 egid: 0
Could anyone help me start to troubleshoot why this might be happening on the iPod Touch?
The hint to why your app is crashing is Received memory warning. Level=1. You're out of memory. This could be the amount of RAM your app is using for data structures, but I've more typically seen this type of error when you run out of graphics memory - i.e. the memory used to composite views, cache images, etc. The total amount of RAM on gen 1 and 2 iPod touches is only 128MB. Gen 3 and 4 bumped that to 256MB.
The memory related info in your log starts with jetsam: kernel memory event (90), free: 323, active: 1250, inactive: 1494, purgeable: 0, wired: 7841, which is a "low memory" warning message. You then see a whole bunch of apps being terminated by the OS (Exited: Killed and exited abnormally with signal 9: Killed). This is the documented behaviour of iOS in low memory conditions - it kills apps until it has enough free memory.
I believe that your application is using a lot of memory, and as a result it gets killed by the operating system.
I would like to know one thing. I hope you people can guide me well.
I am creating an application which needs some audio alerts. I am using AVAudioPlayer for this purpose.
I also set the Audio session like this (in my main delegate class):
//to play from sleep
OSStatus result = AudioSessionInitialize(NULL, NULL, interruptionListenerCallback, self);
UInt32 category = kAudioSessionCategory_MediaPlayback;
result = AudioSessionSetProperty(kAudioSessionProperty_AudioCategory, sizeof(category), category);
AudioSessionSetActive(YES);
I am able to play music if my screen is locked forcefully, but I am not able to play mp3 files if the iPhone go into sleep mode automatically.
In general after 10 minutes I am not getting any music from my application. Could anyone tell me what I am missing? Please guide me if you know about my problem.
EDIT 1
I am currently using kAudioSessionCategory_MediaPlayback.
I written it in my code, i pasted it here.
EDIT 2
I want to start a audio play back from a sleeping iphone.
Please help me if you know.
EDIT 3
idleTimerDisabled , consume my battery ?
I am creating a alarm so this application can run up to 10 hours.
It will disable light on screen?
then i can use it.
I already seen some application which running on iPhone which play music from sleep mode.
Let me try this, thank you for the reply.
I found the answer already on SO in this thread here and also in this thread on SO.
What's happening is that the iPhone goes into deep sleep mode after 10 minutes (after the screen is locked) where it stops your NSTimer's and goes into lower power mode. Even when you have the category set to MediaPlayback it will do this unless you are actually playing a sound. The solution given is to play a "silent" sound file periodically to prevent it from going into deep sleep mode.
Here's what I see in the console log at the time deep sleep occurs (the Warnings are from my clock app):
Thu Dec 24 09:25:09 unknown Clock[16346] <Warning>: ClockDigital itemInterval=0 curInterval=0 needsUpdate=YES
Thu Dec 24 09:25:09 unknown Clock[16346] <Warning>: BatteryIcon itemInterval=6 curInterval=0 needsUpdate=NO
Thu Dec 24 09:25:09 unknown CommCenter[28] <Notice>: Telling CSI to go low power.
Thu Dec 24 09:25:09 unknown CommCenter[28] <Notice>: CSI can enter low power, so now telling to do so.
Thu Dec 24 09:25:09 unknown CommCenter[28] <Notice>: Will sleep. Heard from CSI in 0.00501698 seconds
Thu Dec 24 09:25:10 unknown kernel[0] <Debug>: AppleSynopsysOTGCore::sleepWakeNotification: Sysmtem Going to sleep
Thu Dec 24 09:25:10 unknown kernel[0] <Debug>: AirPort: Disabled AppleBCMWLAN (link 2, sys 1, user 1)
Thu Dec 24 09:25:10 unknown kernel[0] <Debug>: AppleBCMWLAN::setPOWER() [kernel_task]: Setting power state to 0
Thu Dec 24 09:25:10 unknown kernel[0] <Debug>: AppleMultitouchN1SPI: disabled power
Thu Dec 24 09:25:10 unknown kernel[0] <Debug>: AppleBCMWLAN Left BSS: # 0xcf3cc800, BSSID = 00:0f:a3:1c:9f:0c, rssi = -53, rate = 54 (100%), channel = 7, encryption = 0x2, ap = 1, failures = 0, age = 26, ssid[ 8] = "mggm.ap1"
Thu Dec 24 09:25:10 unknown kernel[0] <Debug>: AirPort: Link Down on en0
Thu Dec 24 09:25:10 unknown kernel[0] <Debug>: AppleBCMWLAN::powerOff Ready to power off
Thu Dec 24 09:25:10 unknown kernel[0] <Debug>: AppleBCMWLAN::setPowerStateGated() : Powering Off and sleeping
Thu Dec 24 09:25:10 unknown kernel[0] <Debug>: AppleBCMWLAN::powerOff Ready to power off
Thu Dec 24 09:25:10 unknown configd[22] <Error>: WiFi:[283368310.000857]: Unable to dispatch message to client dataaccessd (0x10000004)
Thu Dec 24 09:25:10 unknown configd[22] <Error>: WiFi:[283368310.013365]: Unable to dispatch message to client apsd (0x10000004)
Thu Dec 24 09:25:10 unknown Clock[16346] <Warning>: ClockVC: adjusted interval: 0.993291
Thu Dec 24 09:25:10 unknown configd[22] <Error>: WiFi:[283368310.018146]: Error initiating scan request: 82
Thu Dec 24 09:25:10 unknown kernel[0] <Debug>: AppleBCMWLAN::setPOWER(): IOKit power off. Discarding request.
Thu Dec 24 09:25:10 unknown Clock[16346] <Warning>: CalendarDay itemInterval=5 curInterval=0 needsUpdate=NO
Thu Dec 24 09:25:10 unknown Clock[16346] <Warning>: BatteryState itemInterval=6 curInterval=0 needsUpdate=NO
Thu Dec 24 09:25:10 unknown Clock[16346] <Warning>: MindfulnessBell itemInterval=2 curInterval=0 needsUpdate=NO
Thu Dec 24 09:25:10 unknown Clock[16346] <Warning>: ClockDigital itemInterval=0 curInterval=0 needsUpdate=YES
Thu Dec 24 09:25:10 unknown Clock[16346] <Warning>: BatteryIcon itemInterval=6 curInterval=0 needsUpdate=NO
Thu Dec 24 09:25:10 unknown kernel[0] <Debug>: System Sleep
You need to use a different Audio Session Category. You can check the documentation here.
I think the one you should use is kAudioSessionCategory_MediaPlayback.
To prevent your app from being silenced after n minutes, you might also need to set the UIApplication's idleTimerDisabled property to YES.
You should try this
[[AVAudioSession sharedInstance] setDelegate: self];
[[AVAudioSession sharedInstance] setCategory:AVAudioSessionCategoryPlayback error:nil];
[[AVAudioSession sharedInstance] setActive: YES error: nil];
referenced from a tutorial at http://www.mindyourcode.com/ios/iphone/how-to-play-audio-in-iphone-sleep-mode/