I have installed jenkins on kubernetes (EKS). It's been running for over a year now. i'v noticed that when you run parallell jobs, it just runs one job at a time. I cannot remember if it ever ran two jobs in parallel.
The pod works after the first one finishes.
Second jobs logs that get stuck :
jenkins-master logs:
2023-02-02 09:14:00.995+0000 [id=962] INFO hudson.slaves.NodeProvisioner#update: devops-iac-prod-eu-west-1-1-master-28-71sh2-hhnlk-46vsg provisioning successfully completed. We have now 3 computer(s)
[jenkins-0][jenkins] 2023-02-02 09:14:22.133+0000 [id=961] INFO o.c.j.p.k.KubernetesLauncher#launch: Created Pod: kubernetes jenkins/devops-iac-prod-eu-west-1-2-master-18-9vk02-n88q-qxlv5
[jenkins-0][config-reload] [2023-02-02 09:14:22] Working on ADDED configmap jenkins/jenkins-jenkins-jcasc-config
[jenkins-0][config-reload] [2023-02-02 09:14:22] Contents of jcasc-default-config.yaml haven't changed. Not overwriting existing file
[jenkins-0][jenkins] 2023-02-02 09:14:24.818+0000 [id=961] INFO o.c.j.p.k.KubernetesLauncher#launch: Pod is running: kubernetes jenkins/devops-iac-prod-eu-west-1-2-master-18-9vk02-n88q-qxlv5
[jenkins-0][jenkins] 2023-02-02 09:14:24.942+0000 [id=982] INFO h.TcpSlaveAgentListener$ConnectionHandler#run: Accepted JNLP4-connect connection #8 from /10.20.41.123:54556
[jenkins-0][jenkins] 2023-02-02 09:14:31.008+0000 [id=974] INFO o.c.j.p.k.KubernetesLauncher#launch: Created Pod: kubernetes jenkins/devops-iac-prod-eu-west-1-1-master-28-71sh2-hhnlk-46vsg
[jenkins-0][jenkins] 2023-02-02 09:14:33.287+0000 [id=974] INFO o.c.j.p.k.KubernetesLauncher#launch: Pod is running: kubernetes jenkins/devops-iac-prod-eu-west-1-1-master-28-71sh2-hhnlk-46vsg
[jenkins-0][jenkins] 2023-02-02 09:14:43.168+0000 [id=995] INFO o.c.j.p.k.p.ContainerExecDecorator$1#doLaunch: Created process inside pod: [devops-iac-prod-eu-west-1-2-master-18-9vk02-n88q-qxlv5], container: [awsterraform][384 ms]
[jenkins-0][jenkins] 2023-02-02 09:15:03.536+0000 [id=974] INFO o.c.j.p.k.KubernetesLauncher#launch: Waiting for agent to connect (30/1,000): devops-iac-prod-eu-west-1-1-master-28-71sh2-hhnlk-46vsg
[jenkins-0][jenkins] 2023-02-02 09:15:19.081+0000 [id=1023] INFO o.c.j.p.k.p.ContainerExecDecorator$1#doLaunch: Created process inside pod: [devops-iac-prod-eu-west-1-2-master-18-9vk02-n88q-qxlv5], container: [awsterraform][312 ms]
[jenkins-0][jenkins] 2023-02-02 09:15:21.728+0000 [id=1027] INFO o.c.j.p.k.p.ContainerExecDecorator$1#doLaunch: Created process inside pod: [devops-iac-prod-eu-west-1-2-master-18-9vk02-n88q-qxlv5], container: [awsterraform][303 ms]
[jenkins-0][jenkins] 2023-02-02 09:15:25.390+0000 [id=996] INFO o.c.j.p.k.p.ContainerExecDecorator$1#doLaunch: Created process inside pod: [devops-iac-prod-eu-west-1-2-master-18-9vk02-n88q-qxlv5], container: [awsterraform][323 ms]
[jenkins-0][config-reload] [2023-02-02 09:15:27] Working on ADDED configmap jenkins/jenkins-jenkins-jcasc-config
[jenkins-0][config-reload] [2023-02-02 09:15:27] Contents of jcasc-default-config.yaml haven't changed. Not overwriting existing file
[jenkins-0][jenkins] 2023-02-02 09:15:33.771+0000 [id=974] INFO o.c.j.p.k.KubernetesLauncher#launch: Waiting for agent to connect (60/1,000): devops-iac-prod-eu-west-1-1-master-28-71sh2-hhnlk-46vsg
[jenkins-0][jenkins] 2023-02-02 09:15:51.476+0000 [id=994] INFO o.c.j.p.k.p.ContainerExecDecorator$1#doLaunch: Created process inside pod: [devops-iac-prod-eu-west-1-2-master-18-9vk02-n88q-qxlv5], container: [awsterraform][285 ms]
[jenkins-0][jenkins] 2023-02-02 09:16:04.013+0000 [id=974] INFO o.c.j.p.k.KubernetesLauncher#launch: Waiting for agent to connect (90/1,000): devops-iac-prod-eu-west-1-1-master-28-71sh2-hhnlk-46vsg
[jenkins-0][config-reload] [2023-02-02 09:16:32] Working on ADDED configmap jenkins/jenkins-jenkins-jcasc-config
[jenkins-0][config-reload] [2023-02-02 09:16:32] Contents of jcasc-default-config.yaml haven't changed. Not overwriting existing file
[jenkins-0][jenkins] 2023-02-02 09:16:34.209+0000 [id=974] INFO o.c.j.p.k.KubernetesLauncher#launch: Waiting for agent to connect (120/1,000): devops-iac-prod-eu-west-1-1-master-28-71sh2-hhnlk-46vsg
[jenkins-0][jenkins] 2023-02-02 09:17:04.399+0000 [id=974] INFO o.c.j.p.k.KubernetesLauncher#launch: Waiting for agent to connect (150/1,000): devops-iac-prod-eu-west-1-1-master-28-71sh2-hhnlk-46vsg
iac-prod-eu-west-1-1-master-28-71sh2-hhnlk-46vsg (stuck agent)
Feb 02, 2023 9:14:32 AM hudson.remoting.jnlp.Main createEngine
INFO: Setting up agent: devops-iac-prod-eu-west-1-nyxgib-master-28-71sh2-hhnlk-46vsg
Feb 02, 2023 9:14:33 AM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Feb 02, 2023 9:14:33 AM hudson.remoting.Engine startEngine
INFO: Using Remoting version: 4.11
Feb 02, 2023 9:14:33 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using /home/jenkins/agent/remoting as a remoting work directory
Feb 02, 2023 9:14:33 AM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to /home/jenkins/agent/remoting
Feb 02, 2023 9:14:33 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://jenkins.jenkins.svc.cluster.local:80/]
Feb 02, 2023 9:14:33 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
Feb 02, 2023 9:14:33 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting TCP connection tunneling is enabled. Skipping the TCP Agent Listener Port availability check
Feb 02, 2023 9:14:33 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
Agent address: jenkins-agent.jenkins.svc.cluster.local
Agent port: 50000
Identity: fe:1f:fb:8a:cf:ca:ad:1e:55:a3:a0:b1:94:5a:36:2b
Feb 02, 2023 9:14:33 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Feb 02, 2023 9:14:33 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins-agent.jenkins.svc.cluster.local:50000
Feb 02, 2023 9:16:53 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins-agent.jenkins.svc.cluster.local:50000 (retrying:2)
java.io.IOException: Failed to connect to jenkins-agent.jenkins.svc.cluster.local:50000
at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:248)
at hudson.remoting.Engine.connectTcp(Engine.java:880)
at hudson.remoting.Engine.innerRun(Engine.java:757)
at hudson.remoting.Engine.run(Engine.java:540)
Caused by: java.net.ConnectException: Connection timed out
at java.base/sun.nio.ch.Net.connect0(Native Method)
at java.base/sun.nio.ch.Net.connect(Unknown Source)
at java.base/sun.nio.ch.Net.connect(Unknown Source)
at java.base/sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
at java.base/java.nio.channels.SocketChannel.open(Unknown Source)
at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:206)
... 3 more
Feb 02, 2023 9:19:13 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins-agent.jenkins.svc.cluster.local:50000 (retrying:3)
java.io.IOException: Failed to connect to jenkins-agent.jenkins.svc.cluster.local:50000
at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:248)
at hudson.remoting.Engine.connectTcp(Engine.java:880)
at hudson.remoting.Engine.innerRun(Engine.java:757)
at hudson.remoting.Engine.run(Engine.java:540)
Caused by: java.net.ConnectException: Connection timed out
at java.base/sun.nio.ch.Net.connect0(Native Method)
at java.base/sun.nio.ch.Net.connect(Unknown Source)
at java.base/sun.nio.ch.Net.connect(Unknown Source)
at java.base/sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
at java.base/java.nio.channels.SocketChannel.open(Unknown Source)
at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:206)
... 3 more
Related
Environment:
Jenkins - Version 2.235.5 (LTS)
Kubernetes Jenkins Plugin - 1.27.7
K8s-Master & Worker Server Details:-
OS - CentOS Linux release 7.9.2009 (Core) & Kernal - 3.10.0-1160.6.1.el7.x86_64
Docker Version - 19.03.12
Client Version: v1.19.0
Server Version: v1.19.0
I am trying to mount a hostPathVolume through Jenkins Kubernetes-plugin podTemplate and when the pod gets kicked off via Jenkins master the path of my K8s worker not getting mounted on K8s pods(Jenkins Agent).
I have the below scripted pipeline.
podTemplate(label: 'mypod', containers: [
containerTemplate(name: 'git', image: 'ubuntu/16.04', ttyEnabled: true, command: 'cat'),
containerTemplate(name: 'maven', image: 'ubuntu/16.04', command: 'cat', ttyEnabled: true)
],
volumes: [
hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock'),
]
) {
node('mypod') {
stage('Check running containers') {
container('docker') {
// example to show you can run docker commands when you mount the socket
sh 'hostname'
sh 'hostname -i'
sh 'docker ps'
}
}
..... further stages....
K8s cluster Jenkins agent pod log
# kubectl logs -f jenkins_job-10-b04wk-dn4gf-llpbp -c jnlp
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main createEngine
INFO: Setting up agent: jenkins_job-10-b04wk-dn4gf-llpbp
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Dec 05, 2020 5:41:16 AM hudson.remoting.Engine startEngine
INFO: Using Remoting version: 4.3
Dec 05, 2020 5:41:16 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using /home/jenkins/agent/remoting as a remoting work directory
Dec 05, 2020 5:41:16 AM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to /home/jenkins/agent/remoting
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://jenkins-server/jenkins/]
Dec 05, 2020 5:41:16 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
Agent address: jenkins-server
Agent port: 40003
Identity: 23:66:a9:70:0b:f2:03:e0:47:44:0d:ee:7e:ce:8c:31
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins-server:40003
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: 23:66:a9:70:0b:f2:03:e0:47:44:0d:ee:7e:ce:8c:31
Dec 05, 2020 5:41:17 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
Dec 05, 2020 5:41:21 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave$SlaveDisconnector call
INFO: Disabled slave engine reconnects.
Dec 05, 2020 5:41:21 AM hudson.util.ProcessTree get
WARNING: Error while determining if vetoers exist
hudson.remoting.RequestAbortedException: hudson.remoting.ChannelClosedException: Channel "unknown": Protocol stack cannot write data anymore. It is not open for write
at hudson.remoting.Request.abort(Request.java:340)
at hudson.remoting.Channel.terminate(Channel.java:1081)
at hudson.remoting.Channel.close(Channel.java:1482)
at hudson.remoting.Channel.close(Channel.java:1446)
at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1313)
at hudson.remoting.Channel$1.handle(Channel.java:606)
at hudson.remoting.AbstractByteBufferCommandTransport.processCommand(AbstractByteBufferCommandTransport.java:203)
at hudson.remoting.AbstractByteBufferCommandTransport.receive(AbstractByteBufferCommandTransport.java:189)
at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onRead(ChannelApplicationLayer.java:187)
at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecv(ApplicationLayer.java:206)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:668)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processRead(SSLEngineFilterLayer.java:369)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecv(SSLEngineFilterLayer.java:117)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:668)
at org.jenkinsci.remoting.protocol.NetworkLayer.onRead(NetworkLayer.java:136)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$2200(BIONetworkLayer.java:48)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:283)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:117)
at java.lang.Thread.run(Thread.java:748)
Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection to jenkins-server/10.20.30.35:40003
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1788)
at hudson.remoting.Request.call(Request.java:202)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:288)
at com.sun.proxy.$Proxy5.fetch3(Unknown Source)
at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:211)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
at jenkins.agents.AgentComputerUtil.getChannelToMaster(AgentComputerUtil.java:48)
at hudson.util.ProcessTree.get(ProcessTree.java:432)
at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1100)
at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1091)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:369)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 4 more
Caused by: hudson.remoting.ChannelClosedException: Channel "unknown": Protocol stack cannot write data anymore. It is not open for write
at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.write(ChannelApplicationLayer.java:305)
at hudson.remoting.AbstractByteBufferCommandTransport.write(AbstractByteBufferCommandTransport.java:304)
at hudson.remoting.Channel.send(Channel.java:764)
at hudson.remoting.Channel.close(Channel.java:1479)
... 18 more
Dec 05, 2020 5:41:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Terminated
Dec 05, 2020 5:41:21 AM hudson.util.ProcessTree get
WARNING: Failed to load winp. Reverting to the default
java.lang.NoClassDefFoundError: hudson/util/ProcessTree$Linux$LinuxProcess
at hudson.util.ProcessTree.get(ProcessTree.java:451)
at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1100)
at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1091)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:369)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:117)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: hudson.util.ProcessTree$Linux$LinuxProcess
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:173)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
... 12 more
Dec 05, 2020 5:41:21 AM hudson.remoting.Request$2 run
INFO: Failed to send back a reply to the request hudson.remoting.Request$2#1affe1cf: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel#157a1c18:JNLP4-connect connection to jenkins-server/10.20.30.35:40003": channel is already closed
openjdk version "1.8.0_272" installed in (Jenkins master, K8s Master and K8s-Worker) systems, Also have restarted all three systems however still no success.
I expect the volume to be mounted in my pod, I have tried with K8s v1.19.4 and Kubernetes Jenkins Plugin - 1.27.0. Not sure what is causing the problem on CentOS-7.9. Whereas the same is working perfectly on CentOS - 8.2.2004 (Core) & 4.18.0-193.14.2.el8_2.x86_64 kernal of K8s master host's native OS.
I have upgraded my docker image from Ubuntu 16.04.6 LTS - 3.10.0-1160.6.1.el7.x86_64 to Ubuntu 16.04.7 to 3.10.0-1160.6.1.el7.x86_64 still didn't work. But it seems Kubernetes on CentOS-7.9 & with Kernal-3.10.0-1160.6.1.el7.x86_64 is having kernel specific issue. So i have upgraded my native host's OS to CentOS-8.2 with - 4.18.0-193.28.1.el8_2.x86_64 kernel, Now my pipeline build working fine.
I've inherited a platform which runs Zookeeper and Solr. The main problem at first was the zoo.cfg and Zookeeper had old setup so DNS named resolution wasn't working.
I've already fixed. I also fixed the issue of the myid value in /var/lib/zookeeper thanks to this thread (someone left a string value there...)
Zookeeper - three nodes and nothing but errors
Now, the output of logs is different:
2017-08-28 14:24:19,368 [myid:3] - WARN [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager#400] - Cannot open channel to 2 at election address zookeeper-test-2/10.240.102.89:3888
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:426)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:843)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:822)
2017-08-28 14:24:19,368 [myid:3] - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:QuorumPeer$QuorumServer#149] - Resolved hostname: zookeeper-test-2 to address: zookeeper-test-2/10.240.102.89
2017-08-28 14:24:19,368 [myid:3] - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 400
and If I run this script from one EC2 inside the VPC I can see is almost working:
+
+ SOLR
+ Online Nodes: 2
+ solrcloud-test-2 [ Connection to 8983 succeeded ]
+ solrcloud-test-1 [ Connection to 8983 succeeded ]
-------------
+ ELB not responding properly. HTTP response: 503
+
+ ZOOKEEPER
+ Online Instances: 3
+ zookeeper-test-3 [ Connection to 2181 succeeded ]
+ zookeeper-test-2 [ Connection to 2181 succeeded ]
+ zookeeper-test-1 [ Connection to 2181 succeeded ]
+ Minimal Configured: 3
+ Cluster Status: UP
as you can see even the Solr Ec2 seems to be online the ELB doesn't detect the path of the Solr.
I checked the parameters of how Tomcat was bringing up the Solr and I detected again a misconfiguration related to the hostnames so I fixed and restarted Tomcat, however my ELB health check still does not detect the url.
This check is setup as
HTTP:8983/solr/
I do a netstat I can see the port up and listening
[root#solrcloud-test-2 ~]# netstat -lnp |grep 8983
tcp 0 0 :::8983 :::* LISTEN 3338/java
Moreover another thing I noticed is even I restarted tomcat the logs of serviced are freezed in the past 25th of August!
ERROR - 2017-08-25 17:10:25.369; org.apache.solr.common.SolrException; null:org.apache.solr.common.SolrException: java.net.UnknownHostException: zookeeper-1: unknown error
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:139)
at org.apache.solr.cloud.ZkController.<init>(ZkController.java:207)
at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:152)
at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:67)
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:216)
at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:189)
at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:136)
at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:279)
at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:260)
at org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:105)
at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4939)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5633)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1092)
at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1984)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.UnknownHostException: zookeeper-1: unknown error
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:61)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380)
at org.apache.solr.common.cloud.SolrZooKeeper.<init>(SolrZooKeeper.java:41)
at org.apache.solr.common.cloud.DefaultConnectionStrategy.connect(DefaultConnectionStrategy.java:37)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:114)
... 22 more
INFO - 2017-08-25 17:10:25.370; org.apache.solr.servlet.SolrDispatchFilter; SolrDispatchFilter.init() done
ERROR - 2017-08-25 17:10:25.525; org.apache.solr.core.CoreContainer; CoreContainer was not shutdown prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!! instance=650576553
But in the other hand the catalina.out
INFO: Deploying web application archive /var/lib/tomcat7/webapps/solr.war
Aug 29, 2017 8:39:26 AM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Aug 29, 2017 8:39:26 AM org.apache.catalina.core.StandardContext startInternal
SEVERE: One or more Filters failed to start. Full details will be found in the appropriate container log file
Aug 29, 2017 8:39:26 AM org.apache.catalina.core.StandardContext startInternal
SEVERE: Context [/solr] startup failed due to previous errors
Aug 29, 2017 8:39:26 AM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deployment of web application archive /var/lib/tomcat7/webapps/solr.war has finished in 3,447 ms
Aug 29, 2017 8:39:26 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-8983"]
Aug 29, 2017 8:39:26 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-8443"]
Aug 29, 2017 8:39:26 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["ajp-bio-8009"]
Aug 29, 2017 8:39:26 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 3579 ms
So maybe my problem here is I don't know how exactly restart the Solr component? Is really my first work with this software so apologies if my questions are totally noob :)
UPDATE:
I found new info in the logs. Now I think I'm closer to the root issue. However I don't understand the problem, some files missing?
Aug 29, 2017 9:12:11 AM org.apache.catalina.core.StandardContext filterStart
SEVERE: Exception starting filter SolrRequestFilter
java.lang.NoClassDefFoundError: Failed to initialize Apache Solr: Could not find necessary SLF4j logging jars. If using Jetty, the SLF4j logging jars need to go in the jetty lib/ext directory. For other containers, the corresponding directory should be used. For more information, see: http://wiki.apache.org/solr/SolrLogging
Thanks
Wow, apologies people. Maybe I shot the question very fast. The setup was completely misconfigured so basically to sum up, Zookeeper cluster was partially running because service was up and running but it was lack of service. So Solr was unable to connect to them. Once I was able to bring up Zookeeper to good status Solr could connect to it.
thanks anyway
I have deployed a 4 node Couchbase cluster using Google GKE.
The master node exposes ports 8091, 8093 to the Loadbaancer.
When connecting to the Loadbalancer IP (external) via a Java app to insert data, I get the timeout error with this stack:
Apr 04, 2017 3:32:15 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
WARNING: [null][ViewEndpoint]: Socket connect took longer than specified timeout.
Apr 04, 2017 3:32:15 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
WARNING: [null][KeyValueEndpoint]: Socket connect took longer than specified timeout.
Apr 04, 2017 3:32:15 PM com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise notifyListener0
WARNING: An exception was thrown by com.couchbase.client.core.endpoint.AbstractEndpoint$2.operationComplete()
rx.exceptions.OnErrorNotImplementedException: connection timed out: /10.4.0.3:8093
at rx.Observable$26.onError(Observable.java:7955)
at rx.observers.SafeSubscriber._onError(SafeSubscriber.java:159)
at rx.observers.SafeSubscriber.onError(SafeSubscriber.java:120)
at rx.internal.operators.OperatorMap$1.onError(OperatorMap.java:48)
What's puzzling is that the stack shows 10.4.0.3:8093 which is actually the the IP of the docker container.
Appreciate all suggestions.
Have you checked the firewall rules for the master node and the workers? You need to allow ingress for the ports you have set up.
See this answer
I currently have a JBoss instance running with standalone-ha.xml as the config file. JBoss 7.1.1 Final. I have deployed a basic HelloBean inside a .jar called ClusterTestEJB inside an .ear called ClusterTest.
HelloBean:
package com.sample;
import javax.ejb.Remote;
import javax.ejb.Stateless;
import org.jboss.ejb3.annotation.Clustered;
#Stateless
#Clustered
#Remote
public class HelloBean implements HelloBeanItf {
int counter = 0;
public String doSomething() {
return "HelloBean Called";
}
}
The HelloBeanItf is as follows:
package com.sample;
public interface HelloBeanItf {
public String doSomething();
}
I got the code from a tutorial I was following.
So I have everything seemingly working. I am now trying to write a java client to invoke that EJB. Here is that code:
import java.util.Properties;
import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import com.sample.HelloBeanItf;
public class Client {
public static void main(String[] args){
Properties p = new Properties();
p.put(Context.INITIAL_CONTEXT_FACTORY, "org.jboss.naming.remote.client.InitialContextFactory");
p.put("jboss.naming.client.ejb.context", true);
p.put(Context.SECURITY_PRINCIPAL, "user");
p.put(Context.SECURITY_CREDENTIALS, "password");
p.put(Context.PROVIDER_URL, "remote://mynode:4447");
InitialContext ctx;
try {
ctx = new InitialContext(p);
} catch (NamingException e) {
// TODO Auto-generated catch block
e.printStackTrace();
return;
}
HelloBeanItf ejb = null;
try {
ejb = (HelloBeanItf) ctx.lookup("java:ClusterTest/ClusterTestEJB//HelloBean!com.sample.HelloBeanItf");
} catch (NamingException e) {
// TODO Auto-generated catch block
e.printStackTrace();
return;
}
for (int ii=0;ii<10;ii++) {
ejb.doSomething();
}
System.out.println("Terminated");
}
}
Here is client side log output:
Bad level value for property: java.util.logging.ConsoleHandler.level
Apr 03, 2014 10:49:30 PM org.jboss.logging.LoggerProviders find
DEBUG: Logging Provider: org.jboss.logging.JDKLoggerProvider
Apr 03, 2014 10:49:30 PM org.jboss.naming.remote.client.InitialContextFactory findClientProperties
DEBUG: Looking for jboss-naming-client.properties using classloader sun.misc.Launcher$AppClassLoader#6921dcac
Apr 03, 2014 10:49:30 PM org.jboss.naming.remote.client.InitialContextFactory getOptionMapFromProperties
DEBUG: jboss.naming.client.endpoint.create.options. has the following options {}
Apr 03, 2014 10:49:30 PM org.jboss.naming.remote.client.InitialContextFactory getOptionMapFromProperties
DEBUG: jboss.naming.client.remote.connectionprovider.create.options. has the following options {}
Apr 03, 2014 10:49:30 PM org.xnio.Xnio <clinit>
INFO: XNIO Version 3.0.3.GA
Apr 03, 2014 10:49:30 PM org.xnio.nio.NioXnio <clinit>
INFO: XNIO NIO Implementation Version 3.0.3.GA
Apr 03, 2014 10:49:30 PM org.jboss.remoting3.EndpointImpl <clinit>
INFO: JBoss Remoting version 3.2.3.GA
Apr 03, 2014 10:49:30 PM org.xnio.nio.WorkerThread run
DEBUG: Started channel thread 'Remoting "config-based-naming-client-endpoint" read-1', selector sun.nio.ch.WindowsSelectorImpl#3a151cc3
Apr 03, 2014 10:49:30 PM org.xnio.nio.WorkerThread run
DEBUG: Started channel thread 'Remoting "config-based-naming-client-endpoint" write-1', selector sun.nio.ch.WindowsSelectorImpl#42d2821b
Apr 03, 2014 10:49:30 PM org.jboss.naming.remote.client.InitialContextFactory getOptionMapFromProperties
DEBUG: jboss.naming.client.connect.options. has the following options {}
Apr 03, 2014 10:49:32 PM org.jboss.ejb.client.EJBClientPropertiesLoader loadEJBClientProperties
DEBUG: Looking for jboss-ejb-client.properties using classloader sun.misc.Launcher$AppClassLoader#6921dcac
Apr 03, 2014 10:49:32 PM org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector <init>
DEBUG: EJB client context org.jboss.ejb.client.EJBClientContext#51bc1897 will have no EJB receivers associated with it since there was no EJB client configuration available to create the receivers
Apr 03, 2014 10:49:32 PM org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver$1 handleDone
DEBUG: Channel Channel ID 828f699b (outbound) of Remoting connection 0d818ff0 to node1-w7.wv-lab.mentorg.com/139.181.88.6:4447 opened for context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext#7c572f78, receiver=Remoting connection EJB receiver [connection=Remoting connection <6981170d>,channel=jboss.ejb,nodename=node1]} Waiting for version handshake message from server
Apr 03, 2014 10:49:32 PM org.jboss.ejb.client.remoting.VersionReceiver handleMessage
INFO: Received server version 1 and marshalling strategies [river]
Apr 03, 2014 10:49:32 PM org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver associate
INFO: Successful version handshake completed for receiver context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext#7c572f78, receiver=Remoting connection EJB receiver [connection=Remoting connection <6981170d>,channel=jboss.ejb,nodename=node1]} on channel Channel ID 828f699b (outbound) of Remoting connection 0d818ff0 to node1-w7.wv-lab.mentorg.com/139.181.88.6:4447
Apr 03, 2014 10:49:32 PM org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver modulesAvailable
DEBUG: Received module availability report for 3 modules
Apr 03, 2014 10:49:32 PM org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver modulesAvailable
DEBUG: Registering module EJBModuleIdentifier{appName='', moduleName='ClusterWebApp', distinctName=''} availability for receiver context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext#7c572f78, receiver=Remoting connection EJB receiver [connection=Remoting connection <6981170d>,channel=jboss.ejb,nodename=node1]}
Apr 03, 2014 10:49:32 PM org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver modulesAvailable
DEBUG: Registering module EJBModuleIdentifier{appName='ClusterTest', moduleName='ClusterTest', distinctName=''} availability for receiver context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext#7c572f78, receiver=Remoting connection EJB receiver [connection=Remoting connection <6981170d>,channel=jboss.ejb,nodename=node1]}
Apr 03, 2014 10:49:32 PM org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver modulesAvailable
DEBUG: Registering module EJBModuleIdentifier{appName='ClusterTest', moduleName='ClusterTestEJB', distinctName=''} availability for receiver context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext#7c572f78, receiver=Remoting connection EJB receiver [connection=Remoting connection <6981170d>,channel=jboss.ejb,nodename=node1]}
Apr 03, 2014 10:49:32 PM org.jboss.ejb.client.remoting.ClusterNode resolveDestination
DEBUG: Checking for a match of client address /fe80:0:0:0:0:5efe:ac1e:5010%14 with client mapping ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='139.181.88.6', destinationPort=4447}
Apr 03, 2014 10:49:32 PM org.jboss.ejb.client.remoting.ClusterNode resolveDestination
DEBUG: Client mapping ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='139.181.88.6', destinationPort=4447} matches client address /fe80:0:0:0:0:5efe:ac1e:5010%14
Apr 03, 2014 10:49:32 PM org.jboss.ejb.client.remoting.ClusterTopologyMessageHandler processMessage
DEBUG: Received a cluster node(s) addition message, for cluster named ejb with 1 nodes [ClusterNode{clusterName='ejb', nodeName='node1', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='139.181.88.6', destinationPort=4447}], resolvedDestination=[Destination address=139.181.88.6, destination port=4447]}]
Apr 03, 2014 10:49:32 PM org.jboss.ejb.client.EJBClientContext getEJBReceiver
DEBUG: org.jboss.ejb.client.RandomDeploymentNodeSelector#5254e73f deployment node selector selected node1 node for appname=ClusterTest,modulename=ClusterTestEJB,distinctname=
Apr 03, 2014 10:49:33 PM org.jboss.ejb.client.EJBClientContext getEJBReceiver
DEBUG: org.jboss.ejb.client.RandomDeploymentNodeSelector#5254e73f deployment node selector selected node1 node for appname=ClusterTest,modulename=ClusterTestEJB,distinctname=
Apr 03, 2014 10:49:33 PM org.jboss.ejb.client.EJBClientContext getEJBReceiver
DEBUG: org.jboss.ejb.client.RandomDeploymentNodeSelector#5254e73f deployment node selector selected node1 node for appname=ClusterTest,modulename=ClusterTestEJB,distinctname=
Apr 03, 2014 10:49:33 PM org.jboss.remoting3.remote.ClientConnectionOpenListener$Authentication$1 run
DEBUG: Client authentication failed for mechanism DIGEST-MD5: javax.security.sasl.SaslException: DIGEST-MD5: Cannot perform callback to acquire realm, authentication ID or password [Caused by javax.security.auth.callback.UnsupportedCallbackException]
Apr 03, 2014 10:49:33 PM org.jboss.ejb.client.EJBClientContext getEJBReceiver
DEBUG: org.jboss.ejb.client.RandomDeploymentNodeSelector#5254e73f deployment node selector selected node1 node for appname=ClusterTest,modulename=ClusterTestEJB,distinctname=
Apr 03, 2014 10:49:33 PM org.jboss.remoting3.remote.RemoteConnection handleException
ERROR: JBREM000200: Remote connection failed: javax.security.sasl.SaslException: Authentication failed: all available authentication mechanisms failed
Apr 03, 2014 10:49:33 PM org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager getEJBReceiver
INFO: Could not create a connection for cluster node ClusterNode{clusterName='ejb', nodeName='node1', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='139.181.88.6', destinationPort=4447}], resolvedDestination=[Destination address=139.181.88.6, destination port=4447]} in cluster ejb
java.lang.RuntimeException: javax.security.sasl.SaslException: Authentication failed: all available authentication mechanisms failed
at org.jboss.ejb.client.remoting.IoFutureHelper.get(IoFutureHelper.java:91)
at org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager.getEJBReceiver(RemotingConnectionClusterNodeManager.java:117)
at org.jboss.ejb.client.ClusterContext$EJBReceiverAssociationTask.run(ClusterContext.java:333)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: javax.security.sasl.SaslException: Authentication failed: all available authentication mechanisms failed
at org.jboss.remoting3.remote.ClientConnectionOpenListener$Capabilities.handleEvent(ClientConnectionOpenListener.java:365)
at org.jboss.remoting3.remote.ClientConnectionOpenListener$Capabilities.handleEvent(ClientConnectionOpenListener.java:214)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:189)
at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:103)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
at org.xnio.nio.NioHandle.run(NioHandle.java:90)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:184)
at ...asynchronous invocation...(Unknown Source)
at org.jboss.remoting3.EndpointImpl.doConnect(EndpointImpl.java:270)
at org.jboss.remoting3.EndpointImpl.doConnect(EndpointImpl.java:251)
at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:349)
at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:333)
at org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager.getEJBReceiver(RemotingConnectionClusterNodeManager.java:115)
... 6 more
Apr 03, 2014 10:49:33 PM org.jboss.ejb.client.EJBClientContext getEJBReceiver
DEBUG: org.jboss.ejb.client.RandomDeploymentNodeSelector#5254e73f deployment node selector selected node1 node for appname=ClusterTest,modulename=ClusterTestEJB,distinctname=
Apr 03, 2014 10:49:33 PM org.jboss.ejb.client.EJBClientContext getEJBReceiver
DEBUG: org.jboss.ejb.client.RandomDeploymentNodeSelector#5254e73f deployment node selector selected node1 node for appname=ClusterTest,modulename=ClusterTestEJB,distinctname=
Apr 03, 2014 10:49:33 PM org.jboss.ejb.client.EJBClientContext getEJBReceiver
DEBUG: org.jboss.ejb.client.RandomDeploymentNodeSelector#5254e73f deployment node selector selected node1 node for appname=ClusterTest,modulename=ClusterTestEJB,distinctname=
Apr 03, 2014 10:49:34 PM org.jboss.ejb.client.EJBClientContext getEJBReceiver
DEBUG: org.jboss.ejb.client.RandomDeploymentNodeSelector#5254e73f deployment node selector selected node1 node for appname=ClusterTest,modulename=ClusterTestEJB,distinctname=
Apr 03, 2014 10:49:34 PM org.jboss.ejb.client.EJBClientContext getEJBReceiver
DEBUG: org.jboss.ejb.client.RandomDeploymentNodeSelector#5254e73f deployment node selector selected node1 node for appname=ClusterTest,modulename=ClusterTestEJB,distinctname=
Apr 03, 2014 10:49:34 PM org.jboss.ejb.client.EJBClientContext getEJBReceiver
DEBUG: org.jboss.ejb.client.RandomDeploymentNodeSelector#5254e73f deployment node selector selected node1 node for appname=ClusterTest,modulename=ClusterTestEJB,distinctname=
Terminated
I have tried the following:
Removing the remoting security realm, removing the username and password. I am not sure what else to try at this point.
First time poster, so let me know if more details are needed or if you have questions.
Thanks for the help!
For the record, the inability to execute the EJB was fixed by upgrading from 7.1.1. to 7.1.3. After the upgrade, I was still getting the SASL exception, but the EJB was executing successfully.
The SASL exception was resolved by switching to using the ejb-client instead of the remote-naming libraries and adding clustering properties. Remote naming does not do load balancing anyway, so the switch was necessary for me.
Hi i am not able to start tomcat 7 server from eclipse.
When i give start from eclipse i get the following error log.
Jan 11, 2013 10:10:27 AM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: C:\Program Files\Java\jre7\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:/Program Files/Java/jre7/bin/client;C:/Program Files/Java/jre7/bin;C:/Program Files/Java/jre7/lib/i386;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Users\Jayant\Desktop\data\adt-bundle-windows-x86\eclipse;;.
Jan 11, 2013 10:10:28 AM org.apache.coyote.AbstractProtocolHandler init
INFO: Initializing ProtocolHandler ["http-bio-8080"]
Jan 11, 2013 10:10:28 AM org.apache.coyote.AbstractProtocolHandler init
INFO: Initializing ProtocolHandler ["ajp-bio-8009"]
Jan 11, 2013 10:10:28 AM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 1165 ms
Jan 11, 2013 10:10:28 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
Jan 11, 2013 10:10:28 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.12
Jan 11, 2013 10:10:28 AM org.apache.coyote.AbstractProtocolHandler start
INFO: Starting ProtocolHandler ["http-bio-8080"]
Jan 11, 2013 10:10:28 AM org.apache.coyote.AbstractProtocolHandler start
INFO: Starting ProtocolHandler ["ajp-bio-8009"]
Jan 11, 2013 10:10:28 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 507 ms
Jan 11, 2013 10:10:28 AM org.apache.catalina.core.StandardServer await
SEVERE: StandardServer.await: create[localhost:8005]:
java.net.BindException: Address already in use: JVM_Bind
at java.net.DualStackPlainSocketImpl.bind0(Native Method)
at java.net.DualStackPlainSocketImpl.socketBind(Unknown Source)
at java.net.AbstractPlainSocketImpl.bind(Unknown Source)
at java.net.PlainSocketImpl.bind(Unknown Source)
at java.net.ServerSocket.bind(Unknown Source)
at java.net.ServerSocket.<init>(Unknown Source)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:422)
at org.apache.catalina.startup.Catalina.await(Catalina.java:707)
at org.apache.catalina.startup.Catalina.start(Catalina.java:653)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:303)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:431)
Jan 11, 2013 10:10:28 AM org.apache.coyote.AbstractProtocolHandler pause
INFO: Pausing ProtocolHandler ["http-bio-8080"]
Jan 11, 2013 10:10:29 AM org.apache.coyote.AbstractProtocolHandler pause
INFO: Pausing ProtocolHandler ["ajp-bio-8009"]
Jan 11, 2013 10:10:30 AM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Jan 11, 2013 10:10:30 AM org.apache.coyote.AbstractProtocolHandler stop
INFO: Stopping ProtocolHandler ["http-bio-8080"]
Jan 11, 2013 10:10:30 AM org.apache.coyote.AbstractProtocolHandler stop
INFO: Stopping ProtocolHandler ["ajp-bio-8009"]
Now i am getting the following error.
Jan 11, 2013 10:18:59 AM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: C:\Program Files\Java\jre7\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:/Program Files/Java/jre7/bin/client;C:/Program Files/Java/jre7/bin;C:/Program Files/Java/jre7/lib/i386;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Users\Jayant\Desktop\data\adt-bundle-windows-x86\eclipse;;.
Jan 11, 2013 10:19:00 AM org.apache.coyote.AbstractProtocolHandler init
INFO: Initializing ProtocolHandler ["http-bio-8080"]
Jan 11, 2013 10:19:00 AM org.apache.coyote.AbstractProtocolHandler init
INFO: Initializing ProtocolHandler ["ajp-bio-8009"]
Jan 11, 2013 10:19:00 AM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 872 ms
Jan 11, 2013 10:19:00 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
Jan 11, 2013 10:19:00 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.12
Jan 11, 2013 10:19:00 AM org.apache.coyote.AbstractProtocolHandler start
INFO: Starting ProtocolHandler ["http-bio-8080"]
Jan 11, 2013 10:19:00 AM org.apache.coyote.AbstractProtocolHandler start
INFO: Starting ProtocolHandler ["ajp-bio-8009"]
Jan 11, 2013 10:19:00 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 535 ms
Jan 11, 2013 10:20:13 AM org.apache.catalina.core.StandardServer await
WARNING: StandardServer.await: Invalid command 'GET / HTTP/1.1' received
Jan 11, 2013 10:20:19 AM org.apache.catalina.core.StandardServer await
WARNING: StandardServer.await: Invalid command '' received
Go to server.xml file and change
<Server port="8005" shutdown="SHUTDOWN">
to
<Server port="8006" shutdown="SHUTDOWN">
The admin port is for receiving admin commands,and the HTTP port is for receiving HTTP requests. You must use different ports. What happened is: the HTTP connector could not start, because the server already opened that port. Your browser/client/whatever sent the request to the admin port , the server does not understand HTTP and logs this error.
Try Reinstalling Tomcat to another port.
In Windows 10 port 8005 is already reserved (for something). Change to for example 8089 (or any free port) in tomcat/conf/server.xml
use:
pkill -f tomcat
to get rid of any existing or un-responsive tomcat instances or processes.
start tomcat.
I was having this problem (port 8005 already being used) and the root cause was due to an existing tomcat process already running on my machine. So, what I needed to do was basically kill the existing process and start up tomcat again.
The common mistake is to use the same Shutdown and Connector port in your server.xml configuration file.
These ports should be different and here is the right example:
<Server port="8005" shutdown="SHUTDOWN">
<Connector port="8983" protocol="HTTP/1.1"
By default Tomcat listens on port 8005 for SHUTDOWN command and it should be always different to Connector port.
If your port is still already in use, try sudo lsof -i:8005 for finding why.
If you are using linux system, then type ps -ef|grep tomcat on terminal. Then find the process id(PID). then type kill -9 on the terminal. Now , start the tomcat server.
2 ways to solve this issue:
1> Go the tomcat server installation directory and open the server.xml file, check if the
Connector and server ports are different. If not make them different ports and then check if nothing is run on these ports
2> Open command prompt from start and type netstat -ano | findstr 8010 to check if the port are being used. if they are used then you get the
TCP 0.0.0.0:8005 0.0.0.0:0 LISTENING 4
TCP [::]:8005 [::]:0 LISTENING 4
then
a> kill the process using the process id (4 in my case) from the task manager
If they are not being used then you see nothing in cmd
If you have multiple apps, check if any of the other apps use the same server port defined in the server.xml file. This was the case for me.
For me (as I was using tomcat server on VS Code).
Tomcat provides a interface called "configure tomcat", there are a options below service status like start, stop etc. The reason I think your terminal showing error is due to internally tomcat is already using that port so you can stop the port from there and start your tomcat server from your IDE terminal.
For detail description and If you OS is windows, search Services and inside you can see the option 'Apache Tomcat'
for people using IntelliJ
Goto Run --> Edit configurations
In the configuration that exists for Tomcat server, change the Admin port
What you can do is rather then killing process you can goto tomcat directory/conf/server.xml, Here you can change shutdown port(8005) to some other port. Also change default port (8080) and other to Different one. So you can run several instance of your Tomcat on single Machine. Restart Your Server (if still issue persists) Recreate it in Eclipse.