Why is Akka spinningup a new thread for processing each message? - scala
Logs:
02:10:23.594 [scala-execution-context-global-3347] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2033. Backoff: 5000 ms]
02:10:23.594 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:10:29.704 [scala-execution-context-global-3348] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2034. Backoff: 5000 ms]
02:10:29.705 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:10:35.784 [scala-execution-context-global-3349] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2035. Backoff: 5000 ms]
02:10:35.784 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:10:41.914 [scala-execution-context-global-3350] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2036. Backoff: 5000 ms]
02:10:41.914 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:10:48.074 [scala-execution-context-global-3351] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2037. Backoff: 5000 ms]
02:10:48.074 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:10:54.194 [scala-execution-context-global-3352] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2038. Backoff: 5000 ms]
02:10:54.194 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:11:00.284 [scala-execution-context-global-3353] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2039. Backoff: 5000 ms]
02:11:00.284 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:11:06.405 [scala-execution-context-global-3354] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2040. Backoff: 5000 ms]
02:11:06.405 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:11:12.505 [scala-execution-context-global-3355] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2041. Backoff: 5000 ms]
02:11:12.505 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:11:18.594 [scala-execution-context-global-3356] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2042. Backoff: 5000 ms]
02:11:18.594 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:11:24.674 [scala-execution-context-global-3357] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2043. Backoff: 5000 ms]
02:11:24.674 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:11:30.744 [scala-execution-context-global-3358] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2044. Backoff: 5000 ms]
02:11:30.744 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:11:36.854 [scala-execution-context-global-3359] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2045. Backoff: 5000 ms]
02:11:36.854 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:11:42.974 [scala-execution-context-global-3360] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2046. Backoff: 5000 ms]
02:11:42.974 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:11:49.074 [scala-execution-context-global-3361] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2047. Backoff: 5000 ms]
02:11:49.074 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:11:55.194 [scala-execution-context-global-3362] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2048. Backoff: 5000 ms]
02:11:55.194 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:12:01.254 [scala-execution-context-global-3363] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2049. Backoff: 5000 ms]
02:12:01.254 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:12:07.344 [scala-execution-context-global-3364] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2050. Backoff: 5000 ms]
02:12:07.347 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:12:13.444 [scala-execution-context-global-3365] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2051. Backoff: 5000 ms]
02:12:13.444 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:12:19.544 [scala-execution-context-global-3366] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2052. Backoff: 5000 ms]
02:12:19.544 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:12:25.624 [scala-execution-context-global-3367] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2053. Backoff: 5000 ms]
02:12:25.624 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:12:31.744 [scala-execution-context-global-3368] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2054. Backoff: 5000 ms]
02:12:31.744 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:12:37.824 [scala-execution-context-global-3369] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2055. Backoff: 5000 ms]
02:12:37.824 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:12:43.894 [scala-execution-context-global-3370] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2056. Backoff: 5000 ms]
02:12:43.894 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:12:49.974 [scala-execution-context-global-3371] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2057. Backoff: 5000 ms]
02:12:49.974 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:12:56.064 [scala-execution-context-global-3372] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2058. Backoff: 5000 ms]
02:12:56.065 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:13:02.134 [scala-execution-context-global-3373] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2059. Backoff: 5000 ms]
02:13:02.135 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:13:08.244 [scala-execution-context-global-3374] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2060. Backoff: 5000 ms]
02:13:08.244 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:13:14.344 [scala-execution-context-global-3375] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2061. Backoff: 5000 ms]
02:13:14.345 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:13:20.445 [scala-execution-context-global-3376] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2062. Backoff: 5000 ms]
02:13:20.445 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:13:26.524 [scala-execution-context-global-3377] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2063. Backoff: 5000 ms]
02:13:26.525 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:13:32.595 [scala-execution-context-global-3378] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2064. Backoff: 5000 ms]
02:13:32.597 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:13:38.685 [scala-execution-context-global-3379] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2065. Backoff: 5000 ms]
02:13:38.685 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:13:44.804 [scala-execution-context-global-3380] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2066. Backoff: 5000 ms]
02:13:44.804 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:13:50.874 [scala-execution-context-global-3381] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2067. Backoff: 5000 ms]
02:13:50.875 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:13:56.954 [scala-execution-context-global-3382] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2068. Backoff: 5000 ms]
02:13:56.954 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:14:03.039 [scala-execution-context-global-3383] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2069. Backoff: 5000 ms]
02:14:03.039 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:14:09.124 [scala-execution-context-global-3384] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2070. Backoff: 5000 ms]
02:14:09.124 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:14:15.204 [scala-execution-context-global-3385] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2071. Backoff: 5000 ms]
02:14:15.204 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:14:21.284 [scala-execution-context-global-3386] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2072. Backoff: 5000 ms]
02:14:21.284 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:14:27.404 [scala-execution-context-global-3387] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2073. Backoff: 5000 ms]
02:14:27.404 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:14:33.504 [scala-execution-context-global-3388] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2074. Backoff: 5000 ms]
02:14:33.504 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:14:39.584 [scala-execution-context-global-3389] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2075. Backoff: 5000 ms]
02:14:39.585 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:14:45.695 [scala-execution-context-global-3390] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2076. Backoff: 5000 ms]
02:14:45.695 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:14:51.814 [scala-execution-context-global-3391] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2077. Backoff: 5000 ms]
02:14:51.815 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:14:57.924 [scala-execution-context-global-3392] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2078. Backoff: 5000 ms]
02:14:57.925 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:15:03.994 [scala-execution-context-global-3393] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2079. Backoff: 5000 ms]
02:15:03.997 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:15:10.124 [scala-execution-context-global-3394] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2080. Backoff: 5000 ms]
02:15:10.124 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:15:16.244 [scala-execution-context-global-3395] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2081. Backoff: 5000 ms]
02:15:16.244 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:15:22.345 [scala-execution-context-global-3396] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2082. Backoff: 5000 ms]
02:15:22.345 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:15:28.424 [scala-execution-context-global-3397] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2083. Backoff: 5000 ms]
02:15:28.424 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:15:34.524 [scala-execution-context-global-3398] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2084. Backoff: 5000 ms]
02:15:34.524 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:15:40.644 [scala-execution-context-global-3399] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2085. Backoff: 5000 ms]
02:15:40.644 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:15:46.744 [scala-execution-context-global-3400] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2086. Backoff: 5000 ms]
02:15:46.744 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:15:52.844 [scala-execution-context-global-3401] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2087. Backoff: 5000 ms]
02:15:52.846 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:15:58.944 [scala-execution-context-global-3402] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2088. Backoff: 5000 ms]
02:15:58.944 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:16:05.064 [scala-execution-context-global-3403] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2089. Backoff: 5000 ms]
02:16:05.064 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:16:11.195 [scala-execution-context-global-3404] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2090. Backoff: 5000 ms]
02:16:11.195 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:16:17.334 [scala-execution-context-global-3405] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2091. Backoff: 5000 ms]
02:16:17.334 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:16:23.434 [scala-execution-context-global-3406] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2092. Backoff: 5000 ms]
02:16:23.434 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:16:29.534 [scala-execution-context-global-3407] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2093. Backoff: 5000 ms]
02:16:29.535 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:16:35.624 [scala-execution-context-global-3408] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2094. Backoff: 5000 ms]
02:16:35.624 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:16:41.724 [scala-execution-context-global-3409] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2095. Backoff: 5000 ms]
02:16:41.725 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:16:47.834 [scala-execution-context-global-3410] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2096. Backoff: 5000 ms]
02:16:47.834 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:16:53.964 [scala-execution-context-global-3411] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2097. Backoff: 5000 ms]
02:16:53.964 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:17:00.074 [scala-execution-context-global-3412] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2098. Backoff: 5000 ms]
02:17:00.075 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:17:06.184 [scala-execution-context-global-3413] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2099. Backoff: 5000 ms]
02:17:06.184 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:17:12.274 [scala-execution-context-global-3414] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2100. Backoff: 5000 ms]
02:17:12.274 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:17:18.374 [scala-execution-context-global-3415] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2101. Backoff: 5000 ms]
02:17:18.374 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:17:24.464 [scala-execution-context-global-3416] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2102. Backoff: 5000 ms]
02:17:24.464 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:17:30.554 [scala-execution-context-global-3417] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2103. Backoff: 5000 ms]
02:17:30.554 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:17:36.664 [scala-execution-context-global-3418] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2104. Backoff: 5000 ms]
02:17:36.664 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:17:42.774 [scala-execution-context-global-3419] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2105. Backoff: 5000 ms]
02:17:42.776 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:17:48.914 [scala-execution-context-global-3420] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2106. Backoff: 5000 ms]
02:17:48.914 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:17:55.014 [scala-execution-context-global-3421] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2107. Backoff: 5000 ms]
02:17:55.014 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:18:01.144 [scala-execution-context-global-3422] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2108. Backoff: 5000 ms]
02:18:01.144 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:18:07.234 [scala-execution-context-global-3423] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2109. Backoff: 5000 ms]
02:18:07.234 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:18:13.334 [scala-execution-context-global-3424] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2110. Backoff: 5000 ms]
02:18:13.335 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:18:19.424 [scala-execution-context-global-3425] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2111. Backoff: 5000 ms]
02:18:19.425 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:18:25.564 [scala-execution-context-global-3426] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2112. Backoff: 5000 ms]
02:18:25.564 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:18:31.654 [scala-execution-context-global-3427] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2113. Backoff: 5000 ms]
02:18:31.654 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:18:37.744 [scala-execution-context-global-3428] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2114. Backoff: 5000 ms]
02:18:37.744 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:18:43.825 [scala-execution-context-global-3429] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2115. Backoff: 5000 ms]
02:18:43.825 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:18:49.954 [scala-execution-context-global-3430] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2116. Backoff: 5000 ms]
02:18:49.954 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
02:18:56.104 [scala-execution-context-global-3431] INFO c.d.i.service.SchedulerActor - SEARCH scheduled from scheduler[0 Messages in Queue. Empty Queries: 2117. Backoff: 5000 ms]
02:18:56.104 [my-service-akka.actor.default-dispatcher-8] INFO c.d.i.service.SchedulerActor - Searching
Code:
override def receive: Receive = {
case Search =>
logger.info("Searching")
val work = getWork()
val workLeft = getWorkLeft()
self ! ScheduleNextSearch(workLeft)
self ! Work(work)
case ScheduleNextSearch(workLeft) =>
if (workLeft == 0) noResultQueries += 1 else noResultQueries=0
val backoff = if(noResultQueries < 100) 100 else 5000
cancellable = Some(context.system.scheduler.scheduleOnce(backoff.milli) {
logger.info(s"SEARCH scheduled from scheduler[${workLeft} Messages in Queue. Empty SQS Queries: ${noResultQueries}. Backoff: ${backoff} ms]")
self ! Search
)}
}
Issue:
I have two search related messages in my actor
One searches a Queue for any Jobs to process and Delegates them to some other code to run
The second decides how long to wait before checking the Queue again
I don't understand why the second message always gets logged in a new thread?
scala-execution-context-global-3350
scala-execution-context-global-3351
scala-execution-context-global-3352
scala-execution-context-global-3353
The first one is always logged by the same thread
my-service-akka.actor.default-dispatcher-8
Related
can flink on k8s use pv using NFS and pvc as the high avalibility storageDir
I want to deploy Flink 1.12.0 on k8s with HA mode, and I don't want to deploy the HDFS cluster, and I have an NFS so that I am created a PV that use NFS as the backend storage, and I created a PVC for Deployment mount.
this is my FLINK configMap
kubernetes.cluster-id: mta-flink
high-availability: org.apache.flink.kubernetes.highavailability. KubernetesHaServicesFactory
high-availability.storageDir: file:///opt/flink/nfs/ha
and this is my jobmanager yaml file:
volumeMounts:
name: flink-config-volume
mountPath: /opt/flink/conf
name: flink-nfs
mountPath: /opt/flink/nfs
securityContext:
runAsUser: 9999 # refers to user flink from official flink image, change if necessary
#fsGroup: 9999
volumes:
name: flink-config-volume
configMap:
name: mta-flink-config
items:
key: flink-conf.yaml
path: flink-conf.yaml
key: log4j-console.properties
path: log4j-console.properties
name: flink-nfs
persistentVolumeClaim:
claimName: mta-flink-nfs-pvc
It can be deployed successfully, but if I browser the jobmanager:8081 website, I get the result below:
{"errors": ["Service temporarily unavailable due to an ongoing leader election. Please refresh."]}
is the PVC can be used as high-availability.storageDir? if it's can be used, how can I fix this error?
the logs of jobmanager shows that the leader is elected already:
2020-12-29T06:45:54.177850394Z 2020-12-29 14:45:54,177 DEBUG io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector [] - Leader election started
2020-12-29T06:45:54.177855303Z 2020-12-29 14:45:54,177 DEBUG io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector [] - Attempting to acquire leader lease 'ConfigMapLock: default - mta-flink-resourcemanager-leader (6f6479c6-86cc-4d62-84f9-37ff968bd0e5)'...
2020-12-29T06:45:54.178668055Z 2020-12-29 14:45:54,178 DEBUG io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager [] - WebSocket successfully opened
2020-12-29T06:45:54.178895963Z 2020-12-29 14:45:54,178 INFO org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService [] - Starting DefaultLeaderRetrievalService with KubernetesLeaderRetrievalDriver{configMapName='mta-flink-resourcemanager-leader'}.
2020-12-29T06:45:54.179327491Z 2020-12-29 14:45:54,179 DEBUG io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager [] - Connecting websocket ... io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager#6d303498
2020-12-29T06:45:54.230081993Z 2020-12-29 14:45:54,229 DEBUG io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager [] - WebSocket successfully opened
2020-12-29T06:45:54.230202329Z 2020-12-29 14:45:54,230 INFO org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService [] - Starting DefaultLeaderRetrievalService with KubernetesLeaderRetrievalDriver{configMapName='mta-flink-dispatcher-leader'}.
2020-12-29T06:45:54.230219281Z 2020-12-29 14:45:54,229 DEBUG io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager [] - WebSocket successfully opened
2020-12-29T06:45:54.230353912Z 2020-12-29 14:45:54,230 INFO org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] - Starting DefaultLeaderElectionService with KubernetesLeaderElectionDriver{configMapName='mta-flink-resourcemanager-leader'}.
2020-12-29T06:45:54.237004177Z 2020-12-29 14:45:54,236 DEBUG io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector [] - Leader changed from null to 6f6479c6-86cc-4d62-84f9-37ff968bd0e5
2020-12-29T06:45:54.237024655Z 2020-12-29 14:45:54,236 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - New leader elected 6f6479c6-86cc-4d62-84f9-37ff968bd0e5 for mta-flink-restserver-leader.
2020-12-29T06:45:54.237027811Z 2020-12-29 14:45:54,236 DEBUG io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector [] - Successfully Acquired leader lease 'ConfigMapLock: default - mta-flink-restserver-leader (6f6479c6-86cc-4d62-84f9-37ff968bd0e5)'
2020-12-29T06:45:54.237297376Z 2020-12-29 14:45:54,237 DEBUG org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] - Grant leadership to contender http://mta-flink-jobmanager:8081 with session ID 9587e13f-322f-4cd5-9fff-b4941462be0f.
2020-12-29T06:45:54.237353551Z 2020-12-29 14:45:54,237 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint [] - http://mta-flink-jobmanager:8081 was granted leadership with leaderSessionID=9587e13f-322f-4cd5-9fff-b4941462be0f
2020-12-29T06:45:54.237440354Z 2020-12-29 14:45:54,237 DEBUG org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] - Confirm leader session ID 9587e13f-322f-4cd5-9fff-b4941462be0f for leader http://mta-flink-jobmanager:8081.
2020-12-29T06:45:54.254555127Z 2020-12-29 14:45:54,254 DEBUG io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector [] - Leader changed from null to 6f6479c6-86cc-4d62-84f9-37ff968bd0e5
2020-12-29T06:45:54.254588299Z 2020-12-29 14:45:54,254 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - New leader elected 6f6479c6-86cc-4d62-84f9-37ff968bd0e5 for mta-flink-resourcemanager-leader.
2020-12-29T06:45:54.254628053Z 2020-12-29 14:45:54,254 DEBUG io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector [] - Successfully Acquired leader lease 'ConfigMapLock: default - mta-flink-resourcemanager-leader (6f6479c6-86cc-4d62-84f9-37ff968bd0e5)'
2020-12-29T06:45:54.254871569Z 2020-12-29 14:45:54,254 DEBUG org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] - Grant leadership to contender LeaderContender: StandaloneResourceManager with session ID b1730dc6-0f94-49f4-b519-56917f3027b7.
2020-12-29T06:45:54.256608291Z 2020-12-29 14:45:54,256 DEBUG io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector [] - Attempting to renew leader lease 'ConfigMapLock: default - mta-flink-resourcemanager-leader (6f6479c6-86cc-4d62-84f9-37ff968bd0e5)'...
2020-12-29T06:45:54.259155793Z 2020-12-29 14:45:54,258 DEBUG io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector [] - Leader changed from null to 6f6479c6-86cc-4d62-84f9-37ff968bd0e5
2020-12-29T06:45:54.259176091Z 2020-12-29 14:45:54,258 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - New leader elected 6f6479c6-86cc-4d62-84f9-37ff968bd0e5 for mta-flink-dispatcher-leader.
2020-12-29T06:45:54.25918096Z 2020-12-29 14:45:54,259 DEBUG io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector [] - Successfully Acquired leader lease 'ConfigMapLock: default - mta-flink-dispatcher-leader (6f6479c6-86cc-4d62-84f9-37ff968bd0e5)'
2020-12-29T06:45:54.259362149Z 2020-12-29 14:45:54,259 DEBUG org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] - Grant leadership to contender LeaderContender: DefaultDispatcherRunner with session ID fbbaa883-69f6-43df-9ca0-c646bc1baad1.
2020-12-29T06:45:54.260301799Z 2020-12-29 14:45:54,260 DEBUG org.apache.flink.runtime.dispatcher.runner.DefaultDispatcherRunner [] - Create new DispatcherLeaderProcess with leader session id fbbaa883-69f6-43df-9ca0-c646bc1baad1.
2020-12-29T06:45:54.266724597Z 2020-12-29 14:45:54,266 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - Start SessionDispatcherLeaderProcess.
2020-12-29T06:45:54.267718418Z 2020-12-29 14:45:54,267 DEBUG io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector [] - Attempting to renew leader lease 'ConfigMapLock: default - mta-flink-dispatcher-leader (6f6479c6-86cc-4d62-84f9-37ff968bd0e5)'...
2020-12-29T06:45:54.26786349Z 2020-12-29 14:45:54,267 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - Recover all persisted job graphs.
2020-12-29T06:45:54.267976912Z 2020-12-29 14:45:54,267 DEBUG org.apache.flink.runtime.jobmanager.DefaultJobGraphStore [] - Retrieving all stored job ids from KubernetesStateHandleStore{configMapName='mta-flink-dispatcher-leader'}.
2020-12-29T06:45:54.277681598Z 2020-12-29 14:45:54,277 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - ResourceManager akka.tcp://flink#mta-flink-jobmanager:6123/user/rpc/resourcemanager_0 was granted leadership with fencing token b51956917f3027b7b1730dc60f9449f4
2020-12-29T06:45:54.280411279Z 2020-12-29 14:45:54,280 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl [] - Starting the SlotManager.
2020-12-29T06:45:54.281367931Z 2020-12-29 14:45:54,281 DEBUG org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver [] - Successfully wrote leader information: Leader=http://mta-flink-jobmanager:8081, session ID=9587e13f-322f-4cd5-9fff-b4941462be0f.
2020-12-29T06:45:54.281528772Z 2020-12-29 14:45:54,281 DEBUG io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector [] - Attempting to renew leader lease 'ConfigMapLock: default - mta-flink-restserver-leader (6f6479c6-86cc-4d62-84f9-37ff968bd0e5)'...
2020-12-29T06:45:54.286191344Z 2020-12-29 14:45:54,286 DEBUG org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Trigger heartbeat request.
2020-12-29T06:45:54.286304807Z 2020-12-29 14:45:54,286 DEBUG org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Trigger heartbeat request.
2020-12-29T06:45:54.286438227Z 2020-12-29 14:45:54,286 DEBUG org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] - Confirm leader session ID b1730dc6-0f94-49f4-b519-56917f3027b7 for leader akka.tcp://flink#mta-flink-jobmanager:6123/user/rpc/resourcemanager_0.
2020-12-29T06:45:54.309361096Z 2020-12-29 14:45:54,309 DEBUG org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver [] - Successfully wrote leader information: Leader=akka.tcp://flink#mta-flink-jobmanager:6123/user/rpc/resourcemanager_0, session ID=b1730dc6-0f94-49f4-b519-56917f3027b7.
2020-12-29T06:45:54.320673232Z 2020-12-29 14:45:54,320 INFO org.apache.flink.runtime.jobmanager.DefaultJobGraphStore [] - Retrieved job ids [] from KubernetesStateHandleStore{configMapName='mta-flink-dispatcher-leader'}
2020-12-29T06:45:54.3206989Z 2020-12-29 14:45:54,320 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - Successfully recovered 0 persisted job graphs.
2020-12-29T06:45:54.324829616Z 2020-12-29 14:45:54,324 DEBUG org.apache.flink.runtime.rpc.akka.SupervisorActor [] - Starting FencedAkkaRpcActor with name dispatcher_1.
2020-12-29T06:45:54.325343659Z 2020-12-29 14:45:54,325 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/rpc/dispatcher_1 .
2020-12-29T06:45:54.33778039Z 2020-12-29 14:45:54,337 DEBUG org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] - Confirm leader session ID fbbaa883-69f6-43df-9ca0-c646bc1baad1 for leader akka.tcp://flink#mta-flink-jobmanager:6123/user/rpc/dispatcher_1.
2020-12-29T06:45:54.36249763Z 2020-12-29 14:45:54,362 DEBUG org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver [] - Successfully wrote leader information: Leader=akka.tcp://flink#mta-flink-jobmanager:6123/user/rpc/dispatcher_1, session ID=fbbaa883-69f6-43df-9ca0-c646bc1baad1.
2020-12-29T06:46:04.298366262Z 2020-12-29 14:46:04,297 DEBUG org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Trigger heartbeat request.
2020-12-29T06:46:04.298442695Z 2020-12-29 14:46:04,298 DEBUG org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Trigger heartbeat request.
2020-12-29T06:46:14.318174464Z 2020-12-29 14:46:14,317 DEBUG org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Trigger heartbeat request.
2020-12-29T06:46:14.318256849Z 2020-12-29 14:46:14,318 DEBUG org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Trigger heartbeat request.
2020-12-29T06:46:24.337694477Z 2020-12-29 14:46:24,337 DEBUG org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Trigger heartbeat request.
2020-12-29T06:46:24.337816516Z 2020-12-29 14:46:24,337 DEBUG org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Trigger heartbeat request.
2020-12-29T06:46:26.044624193Z 2020-12-29 14:46:26,044 DEBUG org.apache.flink.shaded.netty4.io.netty.buffer.AbstractByteBuf [] - -Dorg.apache.flink.shaded.netty4.io.netty.buffer.checkAccessible: true
Not able to install zookeeper in windows
When i am executing zkServer.cmd via below command: D:\zookeeper\zookeeper-3.4.14\bin>call "C:\Program Files\Java\jdk-13.0.2"\bin\java "-Dzookeeper.log.dir=D:\zookeeper\zookeeper-3.4.14\bin\.." "-Dzookeeper.root.logger=INFO,CONSOLE" -cp "D:\zookeeper\zookeeper-3.4.14\bin\..\build\classes;D:\zookeeper\zookeeper-3.4.14\bin\..\build\lib\*;D:\zookeeper\zookeeper-3.4.14\bin\..\*;D:\zookeeper\zookeeper-3.4.14\bin\..\lib\*;D:\zookeeper\zookeeper-3.4.14\bin\..\conf" org.apache.zookeeper.server.quorum.QuorumPeerMain "D:\zookeeper\zookeeper-3.4.14\bin\..\conf\zoo.cfg" Output: 2020-02-22 22:07:42,378 [myid:] - INFO [main:QuorumPeerConfig#136] - Reading configuration from: D:\zookeeper\zookeeper-3.4.14\bin\..\conf\zoo.cfg 2020-02-22 22:07:42,384 [myid:] - INFO [main:DatadirCleanupManager#78] - autopurge.snapRetainCount set to 3 2020-02-22 22:07:42,384 [myid:] - INFO [main:DatadirCleanupManager#79] - autopurge.purgeInterval set to 0 2020-02-22 22:07:42,385 [myid:] - INFO [main:DatadirCleanupManager#101] - Purge task is not scheduled. 2020-02-22 22:07:42,387 [myid:] - WARN [main:QuorumPeerMain#116] - Either no config or no quorum defined in config, running in standalone mode 2020-02-22 22:07:42,487 [myid:] - INFO [main:QuorumPeerConfig#136] - Reading configuration from: D:\zookeeper\zookeeper-3.4.14\bin\..\conf\zoo.cfg 2020-02-22 22:07:42,487 [myid:] - INFO [main:ZooKeeperServerMain#98] - Starting server 2020-02-22 22:07:42,512 [myid:] - INFO [main:Environment#100] - Server environment:zookeeper.version=3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT 2020-02-22 22:07:42,512 [myid:] - INFO [main:Environment#100] - Server environment:host.name=LAPTOP-78H8U6PB 2020-02-22 22:07:42,517 [myid:] - INFO [main:Environment#100] - Server environment:java.version=13.0.2 2020-02-22 22:07:42,518 [myid:] - INFO [main:Environment#100] - Server environment:java.vendor=Oracle Corporation 2020-02-22 22:07:42,519 [myid:] - INFO [main:Environment#100] - Server environment:java.home=C:\Program Files\Java\jdk-13.0.2 2020-02-22 22:07:42,519 [myid:] - INFO [main:Environment#100] - Server environment:java.class.path=D:\zookeeper\zookeeper-3.4.14\bin\..\build\classes;D:\zookeeper\zookeeper-3.4.14\bin\..\build\lib\*;D:\zookeeper\zookeeper-3.4.14\bin\..\zookeeper-3.4.14.jar;D:\zookeeper\zookeeper-3.4.14\bin\..\lib\audience-annotations-0.5.0.jar;D:\zookeeper\zookeeper-3.4.14\bin\..\lib\jline-0.9.94.jar;D:\zookeeper\zookeeper-3.4.14\bin\..\lib\log4j-1.2.17.jar;D:\zookeeper\zookeeper-3.4.14\bin\..\lib\netty-3.10.6.Final.jar;D:\zookeeper\zookeeper-3.4.14\bin\..\lib\slf4j-api-1.7.25.jar;D:\zookeeper\zookeeper-3.4.14\bin\..\lib\slf4j-log4j12-1.7.25.jar;D:\zookeeper\zookeeper-3.4.14\bin\..\conf 2020-02-22 22:07:42,520 [myid:] - INFO [main:Environment#100] - Server environment:java.library.path=C:\Program Files\Java\jdk-13.0.2\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\Program Files\Java\jdk-13.0.2\bin;. 2020-02-22 22:07:42,520 [myid:] - INFO [main:Environment#100] - Server environment:java.io.tmpdir=C:\Users\erdiv\AppData\Local\Temp\ 2020-02-22 22:07:42,521 [myid:] - INFO [main:Environment#100] - Server environment:java.compiler=<NA> 2020-02-22 22:07:42,523 [myid:] - INFO [main:Environment#100] - Server environment:os.name=Windows 10 2020-02-22 22:07:42,524 [myid:] - INFO [main:Environment#100] - Server environment:os.arch=amd64 2020-02-22 22:07:42,525 [myid:] - INFO [main:Environment#100] - Server environment:os.version=10.0 2020-02-22 22:07:42,525 [myid:] - INFO [main:Environment#100] - Server environment:user.name=erdiv 2020-02-22 22:07:42,526 [myid:] - INFO [main:Environment#100] - Server environment:user.home=C:\Users\erdiv 2020-02-22 22:07:42,526 [myid:] - INFO [main:Environment#100] - Server environment:user.dir=D:\zookeeper\zookeeper-3.4.14\bin 2020-02-22 22:07:42,534 [myid:] - INFO [main:ZooKeeperServer#836] - tickTime set to 2000 2020-02-22 22:07:42,534 [myid:] - INFO [main:ZooKeeperServer#845] - minSessionTimeout set to -1 2020-02-22 22:07:42,534 [myid:] - INFO [main:ZooKeeperServer#854] - maxSessionTimeout set to -1 2020-02-22 22:07:42,962 [myid:] - INFO [main:ServerCnxnFactory#117] - Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory 2020-02-22 22:07:42,964 [myid:] - INFO [main:NIOServerCnxnFactory#89] - binding to port 0.0.0.0/0.0.0.0:2181 the process gets stuck and zookeeper is not started Also, when i do ./zkServer.cmd start, following is the output: D:\zookeeper\zookeeper-3.4.14\bin>call "C:\Program Files\Java\jdk-13.0.2"\bin\java "-Dzookeeper.log.dir=D:\zookeeper\zookeeper-3.4.14\bin\.." "-Dzookeeper.root.logger=INFO,CONSOLE" -cp "D:\zookeeper\zookeeper-3.4.14\bin\..\build\classes;D:\zookeeper\zookeeper-3.4.14\bin\..\build\lib\*;D:\zookeeper\zookeeper-3.4.14\bin\..\*;D:\zookeeper\zookeeper-3.4.14\bin\..\lib\*;D:\zookeeper\zookeeper-3.4.14\bin\..\conf" org.apache.zookeeper.server.quorum.QuorumPeerMain "D:\zookeeper\zookeeper-3.4.14\bin\..\conf\zoo.cfg" start 2020-02-22 22:14:58,220 [myid:] - INFO [main:DatadirCleanupManager#78] - autopurge.snapRetainCount set to 3 2020-02-22 22:14:58,223 [myid:] - INFO [main:DatadirCleanupManager#79] - autopurge.purgeInterval set to 0 2020-02-22 22:14:58,225 [myid:] - INFO [main:DatadirCleanupManager#101] - Purge task is not scheduled. 2020-02-22 22:14:58,232 [myid:] - WARN [main:QuorumPeerMain#116] - Either no config or no quorum defined in config, running in standalone mode 2020-02-22 22:14:58,338 [myid:] - ERROR [main:ZooKeeperServerMain#57] - Invalid arguments, exiting abnormally java.lang.NumberFormatException: For input string: "D:\zookeeper\zookeeper-3.4.14\bin\..\conf\zoo.cfg" at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:68) at java.base/java.lang.Integer.parseInt(Integer.java:658) at java.base/java.lang.Integer.parseInt(Integer.java:776) at org.apache.zookeeper.server.ServerConfig.parse(ServerConfig.java:61) at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:86) at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:55) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:119) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81) 2020-02-22 22:14:58,340 [myid:] - INFO [main:ZooKeeperServerMain#58] - Usage: ZooKeeperServerMain configfile | port datadir [ticktime] [maxcnxns] Usage: ZooKeeperServerMain configfile | port datadir [ticktime] [maxcnxns]
the process gets stuck and zookeeper is not started It's not stuck. It started and is waiting on connections By the way, Java 13 might not work with Zookeeper 3.4.x
zookeeper issue - taking 15 minutes to recover if leader is killed
i m trying to implement Kafka with zookeeper in my network but i am facing a weird issue with zookeeper. i have looked around google and realized that many other users reported such issue but no one posted any proper solution for this . My current setup has 3 different zookeeper nodes (32 GB ram dedicated boxes) The issue is if i kill zookeeper leader, both the remaining follower nodes also goes down and they do not recover for at-least next 15-20 minutes. All i am getting in the zookeeper logs is "notification timeouts" without any explanation Here is my zookeeper config file tickTime=2000 initLimit=10 syncLimit=5 maxClientCnxns=100 maxSessionTimeout=50000 dataDir=/var/lib/zookeeper clientPort=2181 autopurge.snapRetainCount=100 autopurge.purgeInterval=1 preAllocSize=131072 snapCount=3000000 server.1=zo1:2888:3888 server.2=zo2:2888:3888 server.3=zo3:2888:3888 in my /etc/hosts file i have mapped zo1 , zo2 , zo3 to their ip address. Note:i have also tested by setting current node ip to 0.0.0.0 it doesn't makes any difference. just few minutes ago i tested it and again it failed to recover. As i have three node cluster zo1 , zo2 and zo3 . zo3 was the leader and zo1 and zo2 were followers. after i killed zo3 node . it took approx 13 minutes to recover automatically . i got the following logs in zo1 and zo2 . Log for zo1. tail /var/lib/zookeeper/zookeeper.out -n 10000 | grep 'QuorumPeer' 2019-01-02 10:25:50,848 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FollowerZooKeeperServer#140] - Shutting down 2019-01-02 10:25:50,848 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer#505] - shutting down 2019-01-02 10:25:50,848 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FollowerRequestProcessor#107] - Shutting down 2019-01-02 10:25:50,848 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:CommitProcessor#184] - Shutting down 2019-01-02 10:25:50,848 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FinalRequestProcessor#402] - shutdown of request processor complete 2019-01-02 10:25:50,849 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:SyncRequestProcessor#208] - Shutting down 2019-01-02 10:25:50,849 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumPeer#865] - LOOKING 2019-01-02 10:25:50,850 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#818] - New election. My id = 1, proposed zxid=0x2d00035c8e 2019-01-02 10:25:51,057 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 400 2019-01-02 10:25:51,458 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 800 2019-01-02 10:25:52,259 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 1600 2019-01-02 10:25:53,859 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 3200 2019-01-02 10:25:57,060 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 6400 2019-01-02 10:26:03,461 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 12800 2019-01-02 10:26:16,262 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 25600 2019-01-02 10:26:41,862 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 51200 2019-01-02 10:27:33,063 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:28:33,065 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:29:33,066 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:30:33,066 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:31:33,067 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:32:33,068 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:33:33,069 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:34:33,069 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:35:33,070 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:36:33,071 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:37:33,071 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:38:33,072 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:39:33,073 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:40:33,074 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:41:33,075 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:42:33,076 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:43:33,076 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:43:33,082 [myid:1] - INFO [WorkerSender[myid=1]:QuorumPeer$QuorumServer#167] - Resolved hostname: zo3 to address: zo3/144.76.xxx.xxx 2019-01-02 10:43:33,091 [myid:1] - INFO [WorkerSender[myid=1]:QuorumPeer$QuorumServer#167] - Resolved hostname: zo3 to address: zo3/144.76.xxx.xxx 2019-01-02 10:43:33,290 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumPeer#935] - FOLLOWING 2019-01-02 10:43:33,290 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer#173] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 50000 datadir /var/lib/zookeeper/version-2 snapdir /var/lib/zookeeper/version-2 2019-01-02 10:43:33,291 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Follower#64] - FOLLOWING - LEADER ELECTION TOOK - 1062441 2019-01-02 10:43:33,291 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumPeer$QuorumServer#167] - Resolved hostname: zo2 to address: zo2/88.198.35.34 2019-01-02 10:43:33,294 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Learner#237] - Unexpected exception, tries=0, connecting to zo2/88.198.35.34:2888 at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:937) 2019-01-02 10:43:34,468 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Learner#332] - Getting a diff from the leader 0x2d00035c8e 2019-01-02 10:43:35,120 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer#687] - Established session 0x2680a49e3dc0013 with negotiated timeout 6000 for client /5.9.xxx.xxx:36664 2019-01-02 10:43:35,244 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer#687] - Established session 0x1680a49b6b90011 with negotiated timeout 30000 for client /5.9.xxx.xxx:36668 2019-01-02 10:43:35,625 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Follower#118] - Got zxid 0x2e00000001 expected 0x1 Logs from node zo2 , which became leader later 2019-01-02 10:25:50,852 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn#1044] - Closed socket connection for client /5.9.xxx.xxx:21218 which had sessionid 0x2680a49e3dc0012 2019-01-02 10:25:50,852 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FollowerZooKeeperServer#140] - Shutting down 2019-01-02 10:25:50,853 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer#505] - shutting down 2019-01-02 10:25:50,853 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FollowerRequestProcessor#107] - Shutting down 2019-01-02 10:25:50,854 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:CommitProcessor#184] - Shutting down 2019-01-02 10:25:50,854 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FinalRequestProcessor#402] - shutdown of request processor complete 2019-01-02 10:25:50,856 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:SyncRequestProcessor#208] - Shutting down 2019-01-02 10:25:50,857 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:QuorumPeer#865] - LOOKING 2019-01-02 10:25:50,858 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#818] - New election. My id = 2, proposed zxid=0x2d00035c8e 2019-01-02 10:25:51,061 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 400 2019-01-02 10:25:51,462 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 800 2019-01-02 10:25:52,283 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 1600 2019-01-02 10:25:53,884 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 3200 2019-01-02 10:25:57,084 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 6400 2019-01-02 10:26:03,485 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 12800 2019-01-02 10:26:16,286 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 25600 2019-01-02 10:26:41,887 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 51200 2019-01-02 10:27:33,087 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:28:33,088 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:29:33,089 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:30:33,090 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:31:33,091 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:32:33,092 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:33:33,092 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:34:33,093 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:35:33,094 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:36:33,095 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:37:33,095 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:38:33,096 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:39:33,097 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:40:33,098 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:41:33,099 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:42:33,100 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 60000 2019-01-02 10:43:33,293 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:QuorumPeer#947] - LEADING 2019-01-02 10:43:33,299 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Leader#62] - TCP NoDelay set to: true 2019-01-02 10:43:33,301 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer#173] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 50000 datadir /var/lib/zookeeper/version-2 snapdir /var/lib/zookeeper/version-2 2019-01-02 10:43:33,301 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Leader#371] - LEADING - LEADER ELECTION TOOK - 1062443 2019-01-02 10:43:34,307 [myid:2] - INFO [LearnerHandler-/144.76.120.143:64542:LearnerHandler#346] - Follower sid: 1 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer#33d2c290 2019-01-02 10:43:34,509 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Leader#961] - Have quorum of supporters, sids: [ 1,2 ]; starting up and setting last processed zxid: 0x2e00000000 As you can see all i am getting is continuous timeouts in the log without any explanation. Been testing it since more then a week still cant find any solution for this. I would be very grateful if somebody could point me in right direction. Thanks
We faced the same issue and found that zookeeper leader election happened after 15 minutes. we bypassed this 15 min idle time by setting tcpKeepAlive=true in zookeeper properties.
No election can happen when the cluster size is 2. If you intend to run a HA Zookeeper cluster, increase your zookeeper count to 5. Also, Zookeeper doesn't need 32GB to run effectively. Check out: https://docs.confluent.io/current/zookeeper/deployment.html#multi-node-setup for cluster information and https://docs.confluent.io/current/zookeeper/deployment.html#jvm for JVM sizing.
TO ANYONE WHO FACE THE SAME ISSUE - the reason for this was zookeeper being idle. if you the zookeeper nodes becomes idle due to no activity (data transaction) between the nodes then internally the zookeeper nodes stops talking to each other . and if any zookeeper nodes goes down the other nodes will not know until they do their periodic ping after 10-15 minutes.
Zookeeper running or not in relation to standard port 2181 usage?
CLOUDERA QUICKSTART 5.13 as follows. I am not sure whether zookeeper out of the box is running or not, and if so, then if it would work reliably? I got this when trying to run zookeeper from within the from kafka supplied version that I downloaded, in standalone mode: [2018-06-17 00:49:32,847] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory) [2018-06-17 00:49:32,854] ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.ZooKeeperServerMain) java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) netstat on the vm reveals: [cloudera#quickstart kafka_2.11-1.1.0]$ netstat -an | grep 2181 tcp 0 0 0.0.0.0:2181 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:2181 127.0.0.1:49718 ESTABLISHED tcp 0 0 127.0.0.1:49707 127.0.0.1:2181 ESTABLISHED tcp 0 0 127.0.0.1:2181 127.0.0.1:49707 ESTABLISHED tcp 0 0 127.0.0.1:49697 127.0.0.1:2181 ESTABLISHED tcp 0 0 10.0.2.15:49065 10.0.2.15:2181 ESTABLISHED tcp 0 0 127.0.0.1:49718 127.0.0.1:2181 ESTABLISHED tcp 0 0 127.0.0.1:49706 127.0.0.1:2181 ESTABLISHED tcp 0 0 127.0.0.1:49714 127.0.0.1:2181 ESTABLISHED tcp 0 0 10.0.2.15:2181 10.0.2.15:49060 ESTABLISHED tcp 0 0 10.0.2.15:2181 10.0.2.15:49065 ESTABLISHED tcp 0 0 127.0.0.1:2181 127.0.0.1:49701 ESTABLISHED tcp 0 0 127.0.0.1:2181 127.0.0.1:49714 ESTABLISHED tcp 0 0 127.0.0.1:2181 127.0.0.1:49706 ESTABLISHED tcp 0 0 10.0.2.15:49060 10.0.2.15:2181 ESTABLISHED tcp 0 0 127.0.0.1:49701 127.0.0.1:2181 ESTABLISHED tcp 0 0 127.0.0.1:2181 127.0.0.1:49697 ESTABLISHED sudo jps when executed shows QuorumPeerMain - which I think is zookeeper these days(?): 8196 5559 SecondaryNameNode 7116 HistoryServer 5831 NodeManager 5290 DataNode 10995 Jps 5216 QuorumPeerMain 6449 ThriftServer 6587 RunJar 7068 Bootstrap 5384 JournalNode 7879 Bootstrap 6317 RESTServer 7237 HRegionServer 5687 Bootstrap 6061 ResourceManager 8124 Bootstrap 8153 5479 NameNode 5745 JobHistoryServer 6699 RunJar 6158 HMaster Not sure what to make of it as got the below when starting zookeeper from cloudera install. Do I have zookeeper working? No such process to kill means? [cloudera#quickstart kafka_2.11-1.1.0]$ sudo /usr/lib/zookeeper/bin/zkServer.sh start JMX enabled by default Using config: /usr/lib/zookeeper/bin/../conf/zoo.cfg Starting zookeeper ... STARTED [cloudera#quickstart kafka_2.11-1.1.0]$ sudo /usr/lib/zookeeper/bin/zkServer.sh stop JMX enabled by default Using config: /usr/lib/zookeeper/bin/../conf/zoo.cfg Stopping zookeeper ... /usr/lib/zookeeper/bin/zkServer.sh: line 162: kill: (11140) - No such process STOPPED Basically, I find the observations not adding up with standard descriptions.
As your error states that you have an
Address already in use
you should figure out (as beny23 said) which process occupies that port:
lsof -i :2181
Once you have a zookeeper running you should see logs like:
Validating environment
ZK_REPLICAS=1
MY_ID=1
ZK_LOG_LEVEL=INFO
ZK_DATA_DIR=/var/lib/zookeeper/data
ZK_DATA_LOG_DIR=/var/lib/zookeeper/log
ZK_LOG_DIR=/var/log/zookeeper
ZK_CLIENT_PORT=2181
ZK_SERVER_PORT=2888
ZK_ELECTION_PORT=3888
ZK_TICK_TIME=2000
ZK_INIT_LIMIT=10
ZK_SYNC_LIMIT=2000
ZK_MAX_CLIENT_CNXNS=60
ZK_MIN_SESSION_TIMEOUT= 4000
ZK_MAX_SESSION_TIMEOUT= 40000
ZK_HEAP_SIZE=1G
ZK_SNAP_RETAIN_COUNT=3
ZK_PURGE_INTERVAL=0
ENSEMBLE
server.1=zookeeper-0.zookeeper.default.svc.cluster.local:2888:3888
Environment validation successful
Creating ZooKeeper configuration
Wrote ZooKeeper configuration file to /etc/zookeeper/zoo.cfg
Creating ZooKeeper log4j configuration
Wrote log4j configuration to /etc/zookeeper/log4j.properties
Creating ZooKeeper data directories and setting permissions
Created ZooKeeper data directories and set permissions in /var/lib/zookeeper/data
Creating JVM configuration file
Wrote JVM configuration to /etc/zookeeper/java.env
ZooKeeper JMX enabled by default
Using config: /etc/zookeeper/zoo.cfg
2018-06-18 07:52:43,747 [myid:] - INFO [main:QuorumPeerConfig#136] - Reading configuration from: /etc/zookeeper/zoo.cfg
2018-06-18 07:52:43,752 [myid:] - INFO [main:DatadirCleanupManager#78] - autopurge.snapRetainCount set to 3
2018-06-18 07:52:43,752 [myid:] - INFO [main:DatadirCleanupManager#79] - autopurge.purgeInterval set to 0
2018-06-18 07:52:43,752 [myid:] - INFO [main:DatadirCleanupManager#101] - Purge task is not scheduled.
2018-06-18 07:52:43,752 [myid:] - WARN [main:QuorumPeerMain#116] - Either no config or no quorum defined in config, running in standalone mode
2018-06-18 07:52:43,764 [myid:] - INFO [main:QuorumPeerConfig#136] - Reading configuration from: /etc/zookeeper/zoo.cfg
2018-06-18 07:52:43,764 [myid:] - INFO [main:ZooKeeperServerMain#98] - Starting server
2018-06-18 07:52:43,771 [myid:] - INFO [main:Environment#100] - Server environment:zookeeper.version=3.4.11-37e277162d567b55a07d1755f0b31c32e93c01a0, built on 11/01/2017 18:06 GMT
2018-06-18 07:52:43,771 [myid:] - INFO [main:Environment#100] - Server environment:host.name=zookeeper-0.zookeeper.default.svc.cluster.local
2018-06-18 07:52:43,771 [myid:] - INFO [main:Environment#100] - Server environment:java.version=1.8.0_151
2018-06-18 07:52:43,772 [myid:] - INFO [main:Environment#100] - Server environment:java.vendor=Oracle Corporation
2018-06-18 07:52:43,772 [myid:] - INFO [main:Environment#100] - Server environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre
2018-06-18 07:52:43,772 [myid:] - INFO [main:Environment#100] - Server environment:java.class.path=/usr/bin/../build/classes:/usr/bin/../build/lib/*.jar:/usr/bin/../share/zookeeper/zookeeper-3.4.11.jar:/usr/bin/../share/zookeeper/slf4j-log4j12-1.6.1.jar:/usr/bin/../share/zookeeper/slf4j-api-1.6.1.jar:/usr/bin/../share/zookeeper/netty-3.10.5.Final.jar:/usr/bin/../share/zookeeper/log4j-1.2.16.jar:/usr/bin/../share/zookeeper/jline-0.9.94.jar:/usr/bin/../share/zookeeper/audience-annotations-0.5.0.jar:/usr/bin/../src/java/lib/*.jar:/etc/zookeeper:
2018-06-18 07:52:43,772 [myid:] - INFO [main:Environment#100] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2018-06-18 07:52:43,772 [myid:] - INFO [main:Environment#100] - Server environment:java.io.tmpdir=/tmp
2018-06-18 07:52:43,772 [myid:] - INFO [main:Environment#100] - Server environment:java.compiler=<NA>
2018-06-18 07:52:43,773 [myid:] - INFO [main:Environment#100] - Server environment:os.name=Linux
2018-06-18 07:52:43,773 [myid:] - INFO [main:Environment#100] - Server environment:os.arch=amd64
2018-06-18 07:52:43,773 [myid:] - INFO [main:Environment#100] - Server environment:os.version=4.9.64
2018-06-18 07:52:43,773 [myid:] - INFO [main:Environment#100] - Server environment:user.name=zookeeper
2018-06-18 07:52:43,774 [myid:] - INFO [main:Environment#100] - Server environment:user.home=/home/zookeeper
2018-06-18 07:52:43,774 [myid:] - INFO [main:Environment#100] - Server environment:user.dir=/usr/bin
2018-06-18 07:52:43,778 [myid:] - INFO [main:ZooKeeperServer#825] - tickTime set to 2000
2018-06-18 07:52:43,778 [myid:] - INFO [main:ZooKeeperServer#834] - minSessionTimeout set to 4000
2018-06-18 07:52:43,778 [myid:] - INFO [main:ZooKeeperServer#843] - maxSessionTimeout set to 40000
2018-06-18 07:52:43,785 [myid:] - INFO [main:ServerCnxnFactory#117] - Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory
2018-06-18 07:52:43,788 [myid:] - INFO [main:NIOServerCnxnFactory#89] - binding to port 0.0.0.0/0.0.0.0:2181
2018-06-18 07:52:56,088 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory#215] - Accepted socket connection from /127.0.0.1:40634
2018-06-18 07:52:56,093 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ServerCnxn#324] - The list of known four letter word commands is : [{1936881266=srvr, 1937006964=stat, 2003003491=wchc, 1685417328=dump, 1668445044=crst, 1936880500=srst, 1701738089=envi, 1668247142=conf, 2003003507=wchs, 2003003504=wchp, 1668247155=cons, 1835955314=mntr, 1769173615=isro, 1920298859=ruok, 1735683435=gtmk, 1937010027=stmk}]
In your config you should have define the server name, I have this line (I deployed zookeeper in a kubernetes cluster)
server.1=zookeeper-0.zookeeper.default.svc.cluster.local:2888:3888
Anyway you need to declare it (in your zoo.cfg) or pass it as variable at startup.
unable to start zookeeper server
I was doing setup zookeeper and while setting zookeeper server I am getting below out put. Server is not starting. .10\bin>call "C:\Program Files\Java\jre1.8.0_111"\bin\java "-Dzookeeper.log.dir=C:\zookeeper-3.4.10\b :43,637 [myid:] - INFO [main:QuorumPeerConfig#134] - Reading configuration from: C:\zookeeper-3.4.10 :43,651 [myid:] - INFO [main:DatadirCleanupManager#78] - autopurge.snapRetainCount set to 3 :43,651 [myid:] - INFO [main:DatadirCleanupManager#79] - autopurge.purgeInterval set to 0 :43,651 [myid:] - INFO [main:DatadirCleanupManager#101] - Purge task is not scheduled. :43,654 [myid:] - WARN [main:QuorumPeerMain#113] - Either no config or no quorum defined in config, :43,790 [myid:] - INFO [main:QuorumPeerConfig#134] - Reading configuration from: C:\zookeeper-3.4.10 :43,792 [myid:] - INFO [main:ZooKeeperServerMain#96] - Starting server :43,824 [myid:] - INFO [main:Environment#100] - Server environment:zookeeper.version=3.4.10-39d3a4f2 :43,824 [myid:] - INFO [main:Environment#100] - Server environment:host.name=P4A-508SD1Q.dir.svc.acc :43,824 [myid:] - INFO [main:Environment#100] - Server environment:java.version=1.8.0_111 :43,824 [myid:] - INFO [main:Environment#100] - Server environment:java.vendor=Oracle Corporation :43,824 [myid:] - INFO [main:Environment#100] - Server environment:java.home=C:\Program Files\Java\j :43,825 [myid:] - INFO [main:Environment#100] - Server environment:java.class.path=C:\zookeeper-3.4. :43,825 [myid:] - INFO [main:Environment#100] - Server environment:java.library.path=C:\Program File :43,826 [myid:] - INFO [main:Environment#100] - Server environment:java.io.tmpdir=C:\Users\SHASHI~1. :43,826 [myid:] - INFO [main:Environment#100] - Server environment:java.compiler=<NA> :43,829 [myid:] - INFO [main:Environment#100] - Server environment:os.name=Windows 7 :43,830 [myid:] - INFO [main:Environment#100] - Server environment:os.arch=amd64 :43,830 [myid:] - INFO [main:Environment#100] - Server environment:os.version=6.1 :43,830 [myid:] - INFO [main:Environment#100] - Server environment:user.name=shashibhusan.mohanta :43,831 [myid:] - INFO [main:Environment#100] - Server environment:user.home=C:\Users\shashibhusan.m :43,832 [myid:] - INFO [main:Environment#100] - Server environment:user.dir=C:\zookeeper-3.4.10\bin :43,852 [myid:] - INFO [main:ZooKeeperServer#829] - tickTime set to 2000 :43,855 [myid:] - INFO [main:ZooKeeperServer#838] - minSessionTimeout set to -1 :43,856 [myid:] - INFO [main:ZooKeeperServer#847] - maxSessionTimeout set to -1 :44,194 [myid:] - INFO [main:NIOServerCnxnFactory#89] - binding to port 0.0.0.0/0.0.0.0:2181 I am not understanding what is the maxSessionTimeout set to -1 what does that mean and how to fix please help.
That means, you haven't specifically set those values and zookeeper will be using default values for maxSessionTimeout and minSessionTimeout which are 20 and 2 times the tickTime respectively. Read the zookeeper minimum configuration docs.