I have a problem configuring Zookeeper to work with zoodiscovery mode centralized and replicated. The guide at http://wiki.eclipse.org/Zookeeper_Based_ECF_Discovery is a little mysterious about that.
I'm working on Windows XP SP3, Java JDK 1.6, Eclipse STS 2.7.2, org.eclipse.osgi 3.7 and a proxied network.
NOTE: Using the standalone configuration mode gives no problem. I use -Dzoodiscovery.flavor.standalone=192.168.23.21:3030;clientPort=3031 on the server and -Dzoodiscovery.flavor.standalone=192.168.23.28:3031;clientPort=3030 and it works nicely.
I will split the question in multiple parts:
1) In a setup with a (one) central server on 192.168.23.28, multiple clients. The clients will both publish and consume services.
I launch the server as:
java -Dzoodiscovery.dataDir=name -Dzoodiscovery.flavor=zoodiscovery.flavor.centralized=192.168.23.28 -jar org.eclipse.osgi.jar -console -consoleLog -clean -configuration c:\temp\osgiserver\configuration
I can see the ZooDiscovery> Discovery Service Activated.
When I launch the clients (in the example there's only one) as:
java -Dzoodiscovery.autoStart=true -Dzoodiscovery.flavor=zoodiscovery.flavor.centralized=192.168.23.28 -jar org.eclipse.osgi.jar -console -consoleLog -clean -configuration c:\temp\osgiclient\configuration
I can see ZooDiscovery> Discovery Service Activated. but then INFO - Attempting connection to server: /192.168.23.28 which goes on and on never succeeding.
I have to start server and clients by configuring Zookeeper from command line, I cannot insert those parameters inside the bundles. I have tried setting the -Dzoodiscovery.clientPort=8888 on the server (8888 is available) and then -Dzoodiscovery.flavor=zoodiscovery.flavor.centralized=192.168.23.28:8888 on the client, but still it changes nothing.
How do I configure such a setup?
2) Plus I'd like to know if it's possible, using centralized, to have multiple central servers talking between them or if I'd have to use the replicated mode.
3) Which leads to.. how do I configure server and clients to use replicated mode by passing VM command line arguments?
4) In replicated mode, if I add a new Zookeeper instance later on, will I have to stop and reconfigure the existing Zookeeper instances to work with the new one or is it sufficient to configure the new one to work with the existing ones?
Thank you very much,
cheers
Related
I am not sure if I can get help for this on here, but I thought it was worth a try.
I have 3 node cluster on AWS, I am running MAPR M3 , I installed Storm, Kafka and Divolte-collector and Cassandra. I would like try some of the clickstream examples and I am running into an issue with the tcp-consumer example. Also being quite new to java and distributed processing I have some clarification questions. Again I am not quite sure where to post this because I feel like this is divolte-collector specific and I also have some gaps in my understanding of the javadoc concept and the building and running of jar files; but I figured someone could point me to some resources or help with some clarifications. I can't get the json string to appear in the console running netcat socket listening for clicks:
Divolte tcp-kafka-consumer example
Everything works until the netcat part step 7 and my knowledge gap is with step 6.
Step 1: install and configure Divolte Collector
Install works and hello world click collections is promising :-)
Step 2: download, unpack and run Kafka
# In one terminal session
cd kafka_2.10-0.8.1.1/bin
./zookeeper-server-start.sh ../config/zookeeper.properties
# Leave Zookeeper running and in another terminal session, do:
cd kafka_2.10-0.8.1.1/bin
./kafka-server-start.sh ../config/server.properties
No erros plus tested kafka examples so seems to working as well
Step 3: start Divolte Collector
Go into the bin directory of your installation and run:
cd divolte-collector-0.2/bin
./divolte-collector
Step 3 no hitch, can test default divole-collector test page
Step 4: host your Javadoc files
Setup a HTTP server that serves the Javadoc files that you generated or downloaded for the examples. If you have Python installed, you can use this:
cd <your-javadoc-directory>
python -m SimpleHTTPServer
Ok so I can reach the javadoc pages
Step 5: listen on TCP port 1234
nc -kl 1234
Note: when using netcat (nc) as TCP server, make sure that you configure the Kafka consumer to use only 1 thread, because nc won't handle multiple incoming connections.
Tested netcat by opening port and sending messages so I figured I don't have any port issues on AWS.
Step 6: run the example
cd divolte-examples/tcp-kafka-consumer
mvn clean package
java -jar target/tcp-kafka-consumer-*-jar-with-dependencies.jar
Note: for this to work, you need to have the avro-schema project installed into your local Maven repository.
I installed the avro-schema with mvn clean install in avro project that comes with the examples. as per instructions here
Step 7: click around and check that you see events being flushed to the console where you run netcat
When you click around the Javadoc pages, you console should show events in JSON format similar to this:
I don't see the clicks in my netcat window :(
Investigating the issue I viewed the console and network tabs using chrome developer tools it seems divolte is running, but I am not sure how to dig further. This is the console view. Any ideas or pointers?
Thanks anyways
Initializing Divolte.
divolte.js:140 Divolte base URL detected http://ec2-x-x-x-x.us-west-x.compute.amazonaws.com:8290/
divolte.js:280 Divolte party/session/pageview identifiers ["0:i6i3g0jy:nxGMDVdU9~f1wF3RGqwmCKKICn4d1Sb9", "0:i6qx4rmi:IXc1i6Qcr17pespL5lIlQZql956XOqzk", "0:6ZIHf9BHzVt_vVNj76KFjKmknXJixquh"]
divolte.js:307 Module initialized. Object {partyId: "0:i6i3g0jy:nxGMDVdU9~f1wF3RGqwmCKKICn4d1Sb9", sessionId: "0:i6qx4rmi:IXc1i6Qcr17pespL5lIlQZql956XOqzk", pageViewId: "0:6ZIHf9BHzVt_vVNj76KFjKmknXJixquh", isNewPartyId: false, isFirstInSession: false…}
divolte.js:21 Signalling event: pageView 0:6ZIHf9BHzVt_vVNj76KFjKmknXJixquh0
allclasses-frame.html:9 GET http://ec2-x-x-x-x.us-west-x.compute.amazonaws.com:8000/resources/fonts/dejavu.css
overview-summary.html:200 GET http://localhost:8290/divolte.js net::ERR_CONNECTION_REFUSED
(Intro: I work on Divolte Collector)
It seems that you are running the example on an AWS instance somewhere. If you are using the pre-packaged JavaDoc files that come with the examples, they have hard-coded the divolte location as http://localhost:8290/divolte.js. So if you are running somewhere other than localhost, you should probably create your own JavaDoc for the example, using the correct hostname for the Divolte Collector server.
You can do so using this command. Be sure to run it from the directory where you source tree is rooted. And of course change localhost for the hostname where you are running the collector.
javadoc -d YOUR_OUTPUT_DIRECTORY \
-bottom '<script src="//localhost:8290/divolte.js" defer async></script>' \
-subpackages .
As an alternative, you could also just try to run the examples locally first (possibly in a virtual machine, if you are on a Windows machine).
It doesn't seem there is anything MapR specific with the issue that you are seeing so far. The Kafka based examples and pipeline should work in any environment that has the required components installed. This doesn't touch MapR-FS or anything else MapR specific. Writing to the distributed filesystem is another story.
We don't compile Divolte Collector against MapR Hadoop currently, but incidentally I have given it a run on the MapR sandbox VM. When installing from the RPM distribution, create a /etc/divolte/divolte-env.sh with the following env var setting:
HADOOP_CONF_DIR=/usr/share/divolte/lib/guava-18.0.jar:/usr/share/divolte/lib/avro-1.7.7.jar:$(hadoop classpath)
Obviously this is a bit of a hack to get around classpath peculiarities and we hope to provide a distribution compiled against MapR that works out of the box in the future.
Also, you need Java 8 to run Divolte. If you install this from the Oracle RPM, add the proper JAVA_HOME to divolte-env.sh as well, e.g.:
JAVA_HOME=/usr/java/jdk1.8.0_31
With these settings I'm able to run the server and collect Avro files on MapR FS, create a external Hive table on those files and run a query.
Hello I saw some 11g tutorials with a Google search:
http://saltnlight5.blogspot.com/2014/04/how-to-setup-remote-debug-with-weblogic.html
http://java.dzone.com/articles/how-setup-remote-debug
They both specify adding something like:
JAVA_OPTIONS="$JAVA_OPTIONS -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=y"
Two questions I have is:
One, at my work we have domains/domainName/bin/setDomainEnv.sh and in the same "domainName" directory another folder called servers, with a bunch of our servers on it. I am not an expert in infrastructure but I am looking for a way to remotely debug an application on Weblogic through Eclipse that is located on one of the servers. What is the best way to sync up the one server and its application with Eclipse rather than effecting all of the severs (unless I am misunderstanding).
Second, is there not a way to set this up through the Weblogic Administration Console? I tried looking around for anything intuitive, but nothing stood out.
If you are starting your servers with NodeManager, you should be able to add the arguments to "Arguments" in the Weblogic Admin Console under:
Environment -> Servers -> Your Server -> Configuration (tab) -> Server Start (sub-tab)
setDomainEnv - set debugFlag="true" and every server you start will listen on the port specified, allowing remote debug connections.
Inside of Eclipse, select, Run > Debug Configuration and then select Remote Java Application Choose the host your server runs on with the port from setDomainEnv
This will affect all servers, so if you have two running on the same host you will have a port conflict. You can do something simple like putting an if statement into setDomainEnv so the debug is only turned on for a specific SERVER_NAME
<domain>/servers - this is the folder where weblogic caches security information, application information, startup params, etc for each server. You shouldn't be manipulating this directly on the file system
Another thing that may be of interest to you it the Oracle Enterprise Pack for Eclipse (OEPE). It allows you to start/stop servers directly from Eclipse as well as additional debug support.
Sample from my setDomainEnv.sh:
debugFlag="true"
export debugFlag
JAVA_DEBUG=""
export JAVA_DEBUG
if [ "${debugFlag}" = "true" ] ; then
JAVA_DEBUG="-Xdebug -Xnoagent -Xrunjdwp:transport=dt_socket,address=8453,server=y,suspend=n -Djava.compiler=NONE"
export JAVA_DEBUG
JAVA_OPTIONS="${JAVA_OPTIONS} ${enableHotswapFlag} -ea -da:com.bea... -da:javelin... -da:weblogic... -ea:com.bea.wli... -ea:com.bea.broker... -ea:com.bea.sbconsole..."
export JAVA_OPTIONS
fi
Any good suggestion for Monitoring JBOSS 7 in Production ? I would also like to configure alerts based on certain condition. Of course , It has to be Open source.
Thanks.
You can use standard JConsole that comes with JBoss dependencies added. It's used to monitor your servers state and mbeans, it's very useful.
To test it on localhost start your server and then run the JConsole from your server/bin directory and select JBoss in the Local process selection.
To use it on "remote" server, start your server on "REMOTE_HOST" and then run JConsole from an JBoss/bin directory and connect with the followin string
service:jmx:jmx-remoting://REMOTE_HOST_NAME:9999 (or the port you use) and enter the username and password.
Secondly, for more detailed info of objects creation, memory leaks, CPU% (profiling) there is another one as:
http://jbossprofiler.jboss.org/
You can try to use free open source APM like scouter.
It shows very useful realtime performance information of every request.
And also you can set a threshold of resources and can make plugin for alerting to external.
https://github.com/scouter-project/scouter
JBoss7 need to set module option.
-Djboss.modules.system.pkgs=~~~,scouter
does any one know how to deactivate the automatic clustering in a JBoss 5.1.0?
we have a JBoss running on each developer machine and because we are all in the same network, they do an auto clustering. The problem could be solved if each of us could get its own multicast ip, but the network hardware is not capable of that.
Isn't there a switch in jboss to deactivate this?
Under Eclipse under Windows, you can run the server using the following JVM property (see Open Launch Configuration) :
-Djboss.partition.name=${env_var:COMPUTERNAME}
This way each of the developer machine will have its own cluster (with a single server if you run only one server). Under Linux, you will need to replace COMPUTERNAME by HOSTNAME.
If you run JBoss AS from the command line, you would use something like -Djboss.partition.name=%COMPUTERNAME% under Windows (not tested).
Note that using -Djgroups.udp.ip_ttl=0 (as proposed in another answer) has the following drawbacks:
server startup is slower (4 minutes instead of 1 minute in my case);
there are a lot of NAKACK warn/error logs;
the JGroups UDP multicast is limited to the local machine which could conflict with other applications based on JGroupds UDP;
other servers on the same machine with the same configuration will be in the same cluster, which may not be desired.
You can use different multicast or partition name to avoid conflict.
However, if you want to disable clustering in "production" or "all" configuration , you need to do following actions:
Remove
farm/
deploy-hasingleton/
deploy/cluster/
In deploy/messaging/*-persistence-service.xml, change Clustered to false:
<attribute name="Clustered>false</attribute>
and remove
<depends optional-attribute-name="ChannelFactoryName">jboss.jgroups:service=ChannelFactory</depends>
In conf/bootstrap/profile.xml, replace
<bean name="BootstrapProfileFactory" class="org.jboss.system.server.profileservice.StaticClusteredProfileFactory">
with
<bean name="BootstrapProfileFactory" class="org.jboss.system.server.profileservice.repository.StaticProfileFactory">
and remove the "farmURIs" property a few lines below that.
Replace deploy/httpha-invoker.sar with http-invoker.sar from the default profile
In the deployers/clustering-deployer-jboss-beans.xml, comment out WebAppClusteringDependencyDeployer.
In SOA-P, if you are removing clustering, you will need to take a few additional steps.
Copy the server/default/deploy/jbpm.esb/hibernate.cfg.xml to server//deploy/jbpm.esb/hibernate.cfg.xml
Remove server//deploy/riftsaw* and cp -R server/default/deploy/riftsaw* server//deploy/
You can do this by setting the TTL (time-to-live) on the multicast packets to zero. Clustering will still be enabled, but none of the JBoss servers running on the developer machines will be able to locate each other.
When starting JBoss, set the jgroups.udp.ip_ttl system property, e.g.
-Djgroups.udp.ip_ttl=0
You'll need to hack that into the JBoss startup script, most likely.
We upgraded from JBoss 4 (and JDK 5) to JBoss 5 (and JDK 6). The problem is that the start time has gone from 1.5 minutes (on JBoss 4) to more than 4 minutes.
18:53:35,444 INFO [ServerImpl] JBoss (Microcontainer) [5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221053)] Started in 3m:9s:262ms
It seems like the component that is taking JBoss the longest time to initialize is the JMX
18:50:41,926 INFO [LogNotificationListener] Adding notification listener for logging mbean "jboss.system:service=Logging,type=Log4jService" to server org.jboss.mx.server.MBeanServerImpl#1adc122[ defaultDomain='jboss' ]
18:52:38,797 INFO [JMXConnectorServerService] JMX Connector server: service:jmx:rmi://lharel2/jndi/rmi://lharel2:1090/jmxconnector
From the DEBUG server log, I get these lines at the problematic time:
2009-12-18 18:51:00,886 DEBUG [org.jboss.deployment.MappedReferenceMetaDataResolverDeployer] (main) vfsfile:/C:/QC/Views/QCDev/jboss-5.1.0.GA/server/default/deploy/jmx-console.war/ endpoint mappings:
2009-12-18 18:51:00,886 DEBUG [org.jboss.deployment.MappedReferenceMetaDataResolverDeployer] (main) Processing unit=jmx-console.war, structure: jmx-console.war
2009-12-18 18:52:35,209 DEBUG [org.jboss.deployment.OptAnnotationMetaDataDeployer] (main) Deployment is metadata-complete, skipping annotation processing, ejbJarMetaData=null, jbossWebMetaData=org.jboss.metadata.web.spec.Web23MetaData#1f, jbossClientMetaData=null, metaDataCompleteIsDefault=false
There is no EJB in the project.
The memory settings are:
-Xms128m -Xmx512m -XX:MaxPermSize=256m
Do you have any idea how JBoss start time can be improved?
Update: so far no luck, I tried shreeni's suggestion (changed the scanning xmls). The server is not running in debug mode so MicSim's suggestion is not relevant
A shot into the blue sky without more information
Network timeouts: 1,5 minutes of delay when deploying jmx-console.war may indicate a network timeout (e.g. 3 x 30 seconds). Try to start JBoss and bind it to a specific IP address using the -b command line argument or the jboss.bind.address system property. Also, try to make sure your host and DNS resolution settings on your system are correct.
JMX is also using RMI and you may want to set the RMI server host name as system property. On some Linux distributions, RMI has problems with looking up the correct hostname and jmx-console.war may try to connect to the 'wrong localhost'. The system property is java.rmi.server.hostname
System tracing: If that does not help, you may want to use strace to start the java process, so you can see the point where the system hangs (if it really does hang due to a network timeout or similar).
That is an awfully big gap in the logs. I suggest changing the log configuration to log everything at DEBUG level, rather than INFO. This will generate an awful lot more log entries, but hopefully will help you narrow it down.
The easiest way to do this is to set the -Djboss.server.log.threshold=DEBUG system property when you start JBoss
I had an issue like this but I found a good improvement by setting the initial and max heap size to same values, I mean:
-Xms512m -Xmx512m
With this, I improved from 4 to 2 minutes the starting time.
Your suspicion about the jmx-console can be misleading. There may be other components doing work in the background unrelated to the jmx-console. In my experience, we had an issue where a small war file appeared to take 3 minutes to load! It was innocent. The culprit was partly due to an EAR file with many wars and jars.
While I'm no expert, I would suggest the following:
Try turning up the logging to TRACE. By doing this, I witnessed one of the deployers (EJBDeployer, I think) unnecessarily scanning WARs in one of our EARs. I then manually disabled the scanning of those WAR files.
Run wireshark during the startup. I discovered some war files were hanging while waiting for a response from an external DTD request. Those websites were either now non-existent or would not properly serve the DTD files to java-based programs. I could speed it up by either having the programmers use local DTD files or mirroring those DTDs locally and having /etc/host loopback locally.
You could refer to this link to avoid unnecessary annotation scanning which could speedup your server start
See http://community.jboss.org/wiki/jboss5xtuningslimming especially the "Tuning" part.
I'm using JBoss 5.1.0 with a Macbook pro (2.26ghz 4gb) without applications it start in 54s
15:00:26,449 INFO [ServerImpl] JBoss (Microcontainer) [5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221634)] Started in 54s:720ms
I made a new configuration based on the "default". The JMS dataosurce points to a Postgres database instead of the "Hypersonic Database" (in memory database)
I suppose you are starting in debug mode. This mode can be up to 3 times slower than normal mode.
But there might be also a problem when switching from JDK5 to JDK6. I found this solution here on the net:
I've solved that. It's a debugging issue. I've changed my debug settings from:
wrapper.java.additional.26=-Xdebug
wrapper.java.additional.27=-Xnoagent
wrapper.java.additional.28=-Djava.compiler=NONE
wrapper.java.additional.27=-Xrunjdwp:transport=dt_socket,address=7199,server=y,suspend=n
to:
wrapper.java.additional.26=-Xdebug
wrapper.java.additional.27=-Xrunjdwp:transport=dt_socket,address=7199,server=y,suspend=n
and JBoss becomes fast again.
Hope this helps.
Do you need the JMX console application? Pragmatic thing would be to un-deploy it from the server, you could still use the jconsole or jvisualvm for basically the same thing.
Turn off annotation scanning and other features you don't need https://community.jboss.org/wiki/jboss5xtuningslimming
When you start/stop JBoss from eclipse, it does not clean up the tmp & work folders correctly. Setup an External Tool configuration and run a batch file to delete everything in tmp & work folders before each startup.
I was able to speedup running the "default" profile from 15/20 minutes to 5 minutes.