Any good suggestion for Monitoring JBOSS 7 in Production ? I would also like to configure alerts based on certain condition. Of course , It has to be Open source.
Thanks.
You can use standard JConsole that comes with JBoss dependencies added. It's used to monitor your servers state and mbeans, it's very useful.
To test it on localhost start your server and then run the JConsole from your server/bin directory and select JBoss in the Local process selection.
To use it on "remote" server, start your server on "REMOTE_HOST" and then run JConsole from an JBoss/bin directory and connect with the followin string
service:jmx:jmx-remoting://REMOTE_HOST_NAME:9999 (or the port you use) and enter the username and password.
Secondly, for more detailed info of objects creation, memory leaks, CPU% (profiling) there is another one as:
http://jbossprofiler.jboss.org/
You can try to use free open source APM like scouter.
It shows very useful realtime performance information of every request.
And also you can set a threshold of resources and can make plugin for alerting to external.
https://github.com/scouter-project/scouter
JBoss7 need to set module option.
-Djboss.modules.system.pkgs=~~~,scouter
Related
I have setup the apache cloudstack on CentOS 6.8 machine following quick installation guide. The management server and KVM are setup on the same machine. The management server is running without problems. I was able to add zone, pod, cluster, primary and secondary storage from the web interface. But when I tried to add an instance it is not showing any templates in the second stage as you can see in the screenshot
However, I am able to see two templates under Templates link in web UI.
But when I select the template and navigate to Zone tab, I see Timeout waiting for response from storage host and Ready field shows no.
When I check the management server logs, it seems there is an error when cloudstack tries to mount secondary storage for use. The below segment from cloudstack-management.log file describes this error.
2017-03-09 23:26:43,207 DEBUG [c.c.a.t.Request] (AgentManager-Handler-
14:null) (logid:) Seq 2-7686800138991304712: Processing: { Ans: , MgmtId:
279278805450918, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":
{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
GetRootDir for nfs://172.16.10.2/export/secondary failed due to
com.cloud.utils.exception.CloudRuntimeException: Unable to mount
172.16.10.2:/export/secondary at /mnt/SecStorage/6e26529d-c659-3053-8acb-
817a77b6cfc6 due to mount.nfs: Connection timed out\n\tat
org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.getRootDir(Nf
sSecondaryStorageResource.java:2080)\n\tat
org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.execute(NfsSe
condaryStorageResource.java:1829)\n\tat
org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.executeReques
t(NfsSecondaryStorageResource.java:265)\n\tat
com.cloud.agent.Agent.processRequest(Agent.java:525)\n\tat
com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:833)\n\tat
com.cloud.utils.nio.Task.call(Task.java:83)\n\tat
com.cloud.utils.nio.Task.call(Task.java:29)\n\tat
java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\
n\tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\
n\tat java.lang.Thread.run(Thread.java:745)\n","wait":0}}] }
Can anyone please guide me how to resolve this issue? I have been trying to figure it out for some hours now and don't know how to proceed further.
Edit 1: Please note that my LAN address was 10.103.72.50 which I assume is not /24 address. I tried to give CentOs a static IP by making the following settings in ifcg-eth0 file
DEVICE=eth0
HWADDR=52:54:00:B9:A6:C0
NM_CONTROLLED=no
ONBOOT=yes
BOOTPROTO=none
IPADDR=172.16.10.2
NETMASK=255.255.255.0
GATEWAY=172.16.10.1
DNS1=8.8.8.8
DNS2=8.8.4.4
But doing this would stop my internet. As a workaround, I reverted these changes and installed all the packages first. Then I changed the IP to static by the same configuration settings as above and ran the cloudstack management. Everything worked fine untill I bumped into this template thing. Please help me figure out what might have went wrong
I know I'm late, but for people trying out in the future, here it goes:
I hope you have successfully added a host as mentioned in Quick Install Guide before you changed your IP to static as it autoconfigures VLANs for different traffic and creates two bridges - generally with names 'cloud' or 'cloudbr'. Cloudstack uses the Secondary Storage System VM for doing all the storage-related operations in each Zone and Cluster. What seems to be the problem is that secondary storage system vm (SSVM) is not able to communicate with the management server at port 8250. If not, try manually mounting the NFS server's mount points in the SSVM shell. You can ssh into the SSVM using the below command:
ssh -i /var/cloudstack/management/.ssh/id_rsa -p 3922 root#<Private or Link local Ip address of SSVM>
I suggest you run the /usr/local/cloud/systemvm/ssvm-check.sh after doing ssh into the secondary storage system VM (assuming it is running) and has it's private, public and link local IP address. If that doesn't help you much, take a look at the secondary storage troubleshooting docs at Cloudstack.
I would further recommend, if anyone in future runs into similar issues, check if the SSVM is running and is in "Up" state in the System VMs section of Infrastructure tab and that you are able to open up a console session of it from the browser. If that is working go on to run the ssvm-check.sh script mentioned above which systematically checks each and every point of operation that SSVM executes. Even if console session cannot be opened up, you can still ssh using the link local IP address of SSVM which can be accessed by opening up details of SSVM and than execute the script. If it says, it cannot communicate with Management Server at port 8250, I recommend you check the iptables rules of management server and make sure all traffic is allowed at port 8250. A custom command to check the same is nc -v <mngmnt-server-ip> 8250. You can do a simple search and learn how to add port 8250 in your iptables rules if that is not opened. Next, you mentioned you used CentOS 6.8, so it probably uses older versions of nfs, so execute exportfs -a in your NFS server to make sure all the NFS shares are properly exported and there are no errors. I would recommend that you wait for the downloading status of CentOS 5.5 no GUI kvm template to be complete and its Ready status shown as 'Yes' before you start importing your own templates and ISOs to execute on VMs. Finally, if your ssvm-check.sh script shows everything is good and the download still does not start, you can run the command: service cloud restart and actually check if the service has gotten a PID using service cloud status as the older versions of system vm templates sometimes need us to manually start the cloud service using service cloud start even after the restart command. Restarting the cloud service in SSVM triggers the restart of downloading of all remaining templates and ISOs. Side note: the system VMs uses a Debian kernel if you want to do some more troubleshooting. Hope this helps.
I have three managed servers running on a Weblogic domain. Now I need to configured node manager so that I can stop and start each of the managed servers.
My question is do I need to define a separate 'Machine' and 'Node Manager port' for each of the managed servers? Or can a single 'Machine' and "Node Manager port' combination be used to start/stop multiple managed servers
Thanks in advance
Yes it is possible but the configuration depends on how your Machines are distributed across your hosts on whether you need to use different ports etc. Oracle provides quite a detailed tutorial on this here. The contents of which is too much to replicate into SO.
I recommend you follow the tutorial and then post any specific questions you may have as a new question.
Step 1: Start weblogic server and open weblogic console in browser and login with correct credential.
Step 2: Expand Environment and click on Machine link
Step 3: Click on New to add new machine and give the machine name. Then click on next.
Step 4: Enter Listen Address (server IP) and port on which node manage will run. And then click Finish.
Please use below link for more detail view
https://fi-sm.com/blog/how-to-add-new-server-on-admin-server-in-web-logic-12c-server/
Hello I saw some 11g tutorials with a Google search:
http://saltnlight5.blogspot.com/2014/04/how-to-setup-remote-debug-with-weblogic.html
http://java.dzone.com/articles/how-setup-remote-debug
They both specify adding something like:
JAVA_OPTIONS="$JAVA_OPTIONS -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=y"
Two questions I have is:
One, at my work we have domains/domainName/bin/setDomainEnv.sh and in the same "domainName" directory another folder called servers, with a bunch of our servers on it. I am not an expert in infrastructure but I am looking for a way to remotely debug an application on Weblogic through Eclipse that is located on one of the servers. What is the best way to sync up the one server and its application with Eclipse rather than effecting all of the severs (unless I am misunderstanding).
Second, is there not a way to set this up through the Weblogic Administration Console? I tried looking around for anything intuitive, but nothing stood out.
If you are starting your servers with NodeManager, you should be able to add the arguments to "Arguments" in the Weblogic Admin Console under:
Environment -> Servers -> Your Server -> Configuration (tab) -> Server Start (sub-tab)
setDomainEnv - set debugFlag="true" and every server you start will listen on the port specified, allowing remote debug connections.
Inside of Eclipse, select, Run > Debug Configuration and then select Remote Java Application Choose the host your server runs on with the port from setDomainEnv
This will affect all servers, so if you have two running on the same host you will have a port conflict. You can do something simple like putting an if statement into setDomainEnv so the debug is only turned on for a specific SERVER_NAME
<domain>/servers - this is the folder where weblogic caches security information, application information, startup params, etc for each server. You shouldn't be manipulating this directly on the file system
Another thing that may be of interest to you it the Oracle Enterprise Pack for Eclipse (OEPE). It allows you to start/stop servers directly from Eclipse as well as additional debug support.
Sample from my setDomainEnv.sh:
debugFlag="true"
export debugFlag
JAVA_DEBUG=""
export JAVA_DEBUG
if [ "${debugFlag}" = "true" ] ; then
JAVA_DEBUG="-Xdebug -Xnoagent -Xrunjdwp:transport=dt_socket,address=8453,server=y,suspend=n -Djava.compiler=NONE"
export JAVA_DEBUG
JAVA_OPTIONS="${JAVA_OPTIONS} ${enableHotswapFlag} -ea -da:com.bea... -da:javelin... -da:weblogic... -ea:com.bea.wli... -ea:com.bea.broker... -ea:com.bea.sbconsole..."
export JAVA_OPTIONS
fi
does any one know how to deactivate the automatic clustering in a JBoss 5.1.0?
we have a JBoss running on each developer machine and because we are all in the same network, they do an auto clustering. The problem could be solved if each of us could get its own multicast ip, but the network hardware is not capable of that.
Isn't there a switch in jboss to deactivate this?
Under Eclipse under Windows, you can run the server using the following JVM property (see Open Launch Configuration) :
-Djboss.partition.name=${env_var:COMPUTERNAME}
This way each of the developer machine will have its own cluster (with a single server if you run only one server). Under Linux, you will need to replace COMPUTERNAME by HOSTNAME.
If you run JBoss AS from the command line, you would use something like -Djboss.partition.name=%COMPUTERNAME% under Windows (not tested).
Note that using -Djgroups.udp.ip_ttl=0 (as proposed in another answer) has the following drawbacks:
server startup is slower (4 minutes instead of 1 minute in my case);
there are a lot of NAKACK warn/error logs;
the JGroups UDP multicast is limited to the local machine which could conflict with other applications based on JGroupds UDP;
other servers on the same machine with the same configuration will be in the same cluster, which may not be desired.
You can use different multicast or partition name to avoid conflict.
However, if you want to disable clustering in "production" or "all" configuration , you need to do following actions:
Remove
farm/
deploy-hasingleton/
deploy/cluster/
In deploy/messaging/*-persistence-service.xml, change Clustered to false:
<attribute name="Clustered>false</attribute>
and remove
<depends optional-attribute-name="ChannelFactoryName">jboss.jgroups:service=ChannelFactory</depends>
In conf/bootstrap/profile.xml, replace
<bean name="BootstrapProfileFactory" class="org.jboss.system.server.profileservice.StaticClusteredProfileFactory">
with
<bean name="BootstrapProfileFactory" class="org.jboss.system.server.profileservice.repository.StaticProfileFactory">
and remove the "farmURIs" property a few lines below that.
Replace deploy/httpha-invoker.sar with http-invoker.sar from the default profile
In the deployers/clustering-deployer-jboss-beans.xml, comment out WebAppClusteringDependencyDeployer.
In SOA-P, if you are removing clustering, you will need to take a few additional steps.
Copy the server/default/deploy/jbpm.esb/hibernate.cfg.xml to server//deploy/jbpm.esb/hibernate.cfg.xml
Remove server//deploy/riftsaw* and cp -R server/default/deploy/riftsaw* server//deploy/
You can do this by setting the TTL (time-to-live) on the multicast packets to zero. Clustering will still be enabled, but none of the JBoss servers running on the developer machines will be able to locate each other.
When starting JBoss, set the jgroups.udp.ip_ttl system property, e.g.
-Djgroups.udp.ip_ttl=0
You'll need to hack that into the JBoss startup script, most likely.
We upgraded from JBoss 4 (and JDK 5) to JBoss 5 (and JDK 6). The problem is that the start time has gone from 1.5 minutes (on JBoss 4) to more than 4 minutes.
18:53:35,444 INFO [ServerImpl] JBoss (Microcontainer) [5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221053)] Started in 3m:9s:262ms
It seems like the component that is taking JBoss the longest time to initialize is the JMX
18:50:41,926 INFO [LogNotificationListener] Adding notification listener for logging mbean "jboss.system:service=Logging,type=Log4jService" to server org.jboss.mx.server.MBeanServerImpl#1adc122[ defaultDomain='jboss' ]
18:52:38,797 INFO [JMXConnectorServerService] JMX Connector server: service:jmx:rmi://lharel2/jndi/rmi://lharel2:1090/jmxconnector
From the DEBUG server log, I get these lines at the problematic time:
2009-12-18 18:51:00,886 DEBUG [org.jboss.deployment.MappedReferenceMetaDataResolverDeployer] (main) vfsfile:/C:/QC/Views/QCDev/jboss-5.1.0.GA/server/default/deploy/jmx-console.war/ endpoint mappings:
2009-12-18 18:51:00,886 DEBUG [org.jboss.deployment.MappedReferenceMetaDataResolverDeployer] (main) Processing unit=jmx-console.war, structure: jmx-console.war
2009-12-18 18:52:35,209 DEBUG [org.jboss.deployment.OptAnnotationMetaDataDeployer] (main) Deployment is metadata-complete, skipping annotation processing, ejbJarMetaData=null, jbossWebMetaData=org.jboss.metadata.web.spec.Web23MetaData#1f, jbossClientMetaData=null, metaDataCompleteIsDefault=false
There is no EJB in the project.
The memory settings are:
-Xms128m -Xmx512m -XX:MaxPermSize=256m
Do you have any idea how JBoss start time can be improved?
Update: so far no luck, I tried shreeni's suggestion (changed the scanning xmls). The server is not running in debug mode so MicSim's suggestion is not relevant
A shot into the blue sky without more information
Network timeouts: 1,5 minutes of delay when deploying jmx-console.war may indicate a network timeout (e.g. 3 x 30 seconds). Try to start JBoss and bind it to a specific IP address using the -b command line argument or the jboss.bind.address system property. Also, try to make sure your host and DNS resolution settings on your system are correct.
JMX is also using RMI and you may want to set the RMI server host name as system property. On some Linux distributions, RMI has problems with looking up the correct hostname and jmx-console.war may try to connect to the 'wrong localhost'. The system property is java.rmi.server.hostname
System tracing: If that does not help, you may want to use strace to start the java process, so you can see the point where the system hangs (if it really does hang due to a network timeout or similar).
That is an awfully big gap in the logs. I suggest changing the log configuration to log everything at DEBUG level, rather than INFO. This will generate an awful lot more log entries, but hopefully will help you narrow it down.
The easiest way to do this is to set the -Djboss.server.log.threshold=DEBUG system property when you start JBoss
I had an issue like this but I found a good improvement by setting the initial and max heap size to same values, I mean:
-Xms512m -Xmx512m
With this, I improved from 4 to 2 minutes the starting time.
Your suspicion about the jmx-console can be misleading. There may be other components doing work in the background unrelated to the jmx-console. In my experience, we had an issue where a small war file appeared to take 3 minutes to load! It was innocent. The culprit was partly due to an EAR file with many wars and jars.
While I'm no expert, I would suggest the following:
Try turning up the logging to TRACE. By doing this, I witnessed one of the deployers (EJBDeployer, I think) unnecessarily scanning WARs in one of our EARs. I then manually disabled the scanning of those WAR files.
Run wireshark during the startup. I discovered some war files were hanging while waiting for a response from an external DTD request. Those websites were either now non-existent or would not properly serve the DTD files to java-based programs. I could speed it up by either having the programmers use local DTD files or mirroring those DTDs locally and having /etc/host loopback locally.
You could refer to this link to avoid unnecessary annotation scanning which could speedup your server start
See http://community.jboss.org/wiki/jboss5xtuningslimming especially the "Tuning" part.
I'm using JBoss 5.1.0 with a Macbook pro (2.26ghz 4gb) without applications it start in 54s
15:00:26,449 INFO [ServerImpl] JBoss (Microcontainer) [5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221634)] Started in 54s:720ms
I made a new configuration based on the "default". The JMS dataosurce points to a Postgres database instead of the "Hypersonic Database" (in memory database)
I suppose you are starting in debug mode. This mode can be up to 3 times slower than normal mode.
But there might be also a problem when switching from JDK5 to JDK6. I found this solution here on the net:
I've solved that. It's a debugging issue. I've changed my debug settings from:
wrapper.java.additional.26=-Xdebug
wrapper.java.additional.27=-Xnoagent
wrapper.java.additional.28=-Djava.compiler=NONE
wrapper.java.additional.27=-Xrunjdwp:transport=dt_socket,address=7199,server=y,suspend=n
to:
wrapper.java.additional.26=-Xdebug
wrapper.java.additional.27=-Xrunjdwp:transport=dt_socket,address=7199,server=y,suspend=n
and JBoss becomes fast again.
Hope this helps.
Do you need the JMX console application? Pragmatic thing would be to un-deploy it from the server, you could still use the jconsole or jvisualvm for basically the same thing.
Turn off annotation scanning and other features you don't need https://community.jboss.org/wiki/jboss5xtuningslimming
When you start/stop JBoss from eclipse, it does not clean up the tmp & work folders correctly. Setup an External Tool configuration and run a batch file to delete everything in tmp & work folders before each startup.
I was able to speedup running the "default" profile from 15/20 minutes to 5 minutes.