JMeter - XMPP Authentication - xmpp

I am building a test plan to test XMPP with JMeter. But I always meet an error when I send a authentication string to server even the authentication string is correct. Does anybody have the same problem or know how to fix this issue? Thanks.
2014/07/04 10:23:22 INFO - jmeter.engine.StandardJMeterEngine: Running the test!
2014/07/04 10:23:22 INFO - jmeter.gui.util.JMeterMenuBar: setRunning(true,*local*)
2014/07/04 10:23:22 INFO - jmeter.engine.StandardJMeterEngine: Starting ThreadGroup: 1 : Thread Group
2014/07/04 10:23:22 INFO - jmeter.engine.StandardJMeterEngine: Starting 1 threads for group Thread Group.
2014/07/04 10:23:22 INFO - jmeter.engine.StandardJMeterEngine: Thread will continue on error
2014/07/04 10:23:22 INFO - jmeter.threads.ThreadGroup: Starting thread group number 1 threads 1 ramp-up 1 perThread 1000.0 delayedStart=false
2014/07/04 10:23:22 INFO - jmeter.threads.ThreadGroup: Started thread group number 1
2014/07/04 10:23:22 INFO - jmeter.engine.StandardJMeterEngine: All thread groups have been started
2014/07/04 10:23:22 INFO - jmeter.threads.JMeterThread: Thread started: Thread Group 1-1
2014/07/04 10:23:22 ERROR - ru.yandex.jmeter.XMPPClientImpl: Error reading data java.lang.RuntimeException: Retries more than 1000, aborting read
at ru.yandex.jmeter.XMPPClientImpl.read(XMPPClientImpl.java:116)
at org.apache.jmeter.protocol.tcp.sampler.TCPSampler.sample(TCPSampler.java:414)
at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:428)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:256)
at java.lang.Thread.run(Unknown Source)
2014/07/04 10:23:22 ERROR - jmeter.protocol.tcp.sampler.TCPSampler: java.lang.RuntimeException: Error reading data
at ru.yandex.jmeter.XMPPClientImpl.read(XMPPClientImpl.java:152)
at org.apache.jmeter.protocol.tcp.sampler.TCPSampler.sample(TCPSampler.java:414)
at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:428)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:256)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.RuntimeException: Retries more than 1000, aborting read
at ru.yandex.jmeter.XMPPClientImpl.read(XMPPClientImpl.java:116)
... 4 more
2014/07/04 10:23:22 INFO - jmeter.threads.JMeterThread: Thread finished: Thread Group 1-1
2014/07/04 10:23:22 INFO - jmeter.engine.StandardJMeterEngine: Notifying test listeners of end of test
2014/07/04 10:23:22 INFO - jmeter.gui.util.JMeterMenuBar: setRunning(false,*local*)

I had tried using XMPPClientImpl plugin but always got the same error ("Retries more than 1000, aborting read"), so I decided to leave it and write my own code.
I use a BeanShell Sampler in which I run the following code (using smack library) to connect to XMPP server.
String CLASS_PATH = "C:/JMeter/apache-jmeter-2.13/lib/ext/smack/";
addClassPath(CLASS_PATH + "smack-android-extensions-4.1.3.jar");
addClassPath(CLASS_PATH + "smack-tcp-4.1.3.jar");
addClassPath(CLASS_PATH + "smack-android-4.1.3.jar");
// explicitly import every class you need
import org.jivesoftware.smack.XMPPConnection;
import org.jivesoftware.smack.ConnectionConfiguration;
import org.jivesoftware.smack.ConnectionListener;
import org.jivesoftware.smack.tcp.XMPPTCPConnection;
import org.jivesoftware.smack.tcp.XMPPTCPConnectionConfiguration;
import org.jivesoftware.smack.SmackException;
import org.jivesoftware.smack.XMPPException;
import org.jivesoftware.smack.XMPPException.XMPPErrorException;
String jabberId = "...";
String jabberPass = "...";
String SERVER_ADDRESS = "...";
int PORT = 5222; // or any other port
XMPPTCPConnection getConnection() {
XMPPTCPConnectionConfiguration config =
XMPPTCPConnectionConfiguration.builder()
.setUsernameAndPassword(jabberId, jabberPass)
.setHost(SERVER_ADDRESS)
.setServiceName(SERVER_ADDRESS)
.setPort(DEFAULT_PORT)
// .setSecurityMode(ConnectionConfiguration.SecurityMode.disabled)
.setSendPresence(true)
// .setDebuggerEnabled(YouMe.DEBUG)
.build();
XMPPTCPConnection con = new XMPPTCPConnection(config);
int REPLY_TIMEOUT = 50000; // 50 seconds, but can be shorter
con.setPacketReplyTimeout(REPLY_TIMEOUT);
return con;
}
Don't forget to add the smack path (e.g. C:\JMeter\apache-jmeter-2.13\lib\ext\smack) to the library field (under "Add directory or jar classpath") in the Test Plan node of your test plan.
To connect -
con = getConnection();
con.connect();
To login -
con.login(jabberId, jabberPass);
You can also add connection listener -
ConnectionListener listener = new ConnectionListener() {
public void connected(XMPPConnection xmppConnection) {
// run main code incl. the login code
runMain();
}
public void authenticated(XMPPConnection xmppConnection, boolean resumed) {
}
public void connectionClosed() {
}
public void connectionClosedOnError(Exception e) {
}
public void reconnectingIn(int i) {
}
public void reconnectionSuccessful() {
}
public void reconnectionFailed(Exception e) {
}
};
con.addConnectionListener(listener);
// connect
con.connect();
runMain() {
con.login(jabberId, jabberPass);
// ...
}

Related

Pi4J v2 SPI w/ GPIO Pin

I'm trying to use Pi4J to talk to an SPI device where the CS pin is a GPIO pin (GPIO 5) and not one of the mapped CS pins. I don't see how to configure it this way in any of the examples or javadocs. I think it would be somewhere on my SpiConfig line.
// Initialize Pi4J with an auto context
// An auto context includes AUTO-DETECT BINDINGS enabled
// which will load all detected Pi4J extension libraries
// (Platforms and Providers) in the class path
var pi4j = Pi4J.newAutoContext();
// create SPI config
SpiConfig config = Spi.newConfigBuilder(pi4j).id("thermocouple-1")
.name("Thermocouple 1").bus(SpiBus.BUS_1)
.chipSelect(SpiChipSelect.CS_0).build();
// get a SPI I/O provider from the Pi4J context
SpiProvider spiProvider = pi4j.provider("pigpio-spi");
// use try-with-resources to auto-close SPI when complete
try (Spi spi = spiProvider.create(config)) {
byte data[] = new byte[]{0, 0, 0, 0};
It looks like this is possible with v1 of the library, but it's not the same in v2.
I ended up posting on the pi4j project on GitHub and through those conversations worked out what I needed. Here is a cross post from there.
Well looking at the DAC8552 example was the key. Here is a working
version that asks the device for it's ID (register 0xD0) and gives the
correct answer for BME280 (0x60). Thanks for the assist. Here is the
working code (needs to be cleaned up, but it's a basic start).
package com.pi4j.example.spi;
import com.pi4j.Pi4J;
import com.pi4j.context.Context;
import com.pi4j.io.gpio.digital.DigitalOutput;
import com.pi4j.io.gpio.digital.DigitalState;
import com.pi4j.io.spi.Spi;
import com.pi4j.io.spi.SpiBus;
import com.pi4j.io.spi.SpiChipSelect;
import com.pi4j.io.spi.SpiMode;
import com.pi4j.util.Console;
import com.pi4j.util.StringUtil;
public class BmeTest {
private static final SpiBus spiBus = SpiBus.BUS_0;
public static void main(String[] args) throws Exception {
final var console = new Console();
// print program title/header
console.title("<-- The Pi4J Project -->",
"Basic SPI Communications Example");
Context pi4j = Pi4J.newAutoContext();
var spiConfig = Spi.newConfigBuilder(pi4j)
.id("SPI" + spiBus + "_BME280")
.name("BME280")
.bus(spiBus)
.chipSelect(SpiChipSelect.CS_0) // not used
.mode(SpiMode.MODE_0)
.provider("pigpio-spi")
.build();
// required all configs
var outputConfig = DigitalOutput.newConfigBuilder(pi4j)
.id("CS_GPIO5")
.name("CS GPIO5")
.address(5)
.shutdown(DigitalState.HIGH)
.initial(DigitalState.HIGH)
.provider("pigpio-digital-output");
DigitalOutput csGpio = null;
try {
csGpio = pi4j.create(outputConfig);
} catch (Exception e) {
e.printStackTrace();
console.println("create DigOut DRDY failed");
System.exit(202);
}
// use try-with-resources to auto-close SPI when complete
try (var spi = pi4j.create(spiConfig)) {
byte data[] = new byte[] { (byte) (0x80 | 0xD0), (byte) 0x00 };
csGpio.high();
csGpio.low();
spi.transfer(data, data);
csGpio.high();
// take a breath to allow time for the SPI
// data to get updated in the SPI device
Thread.sleep(100);
// read data back from the SPI channel
console.println("--------------------------------------");
console.println("SPI [READ] :");
console.println(" [BYTES] 0x" + StringUtil.toHexString(data));
console.println(" [STRING] " + new String(data));
console.println("--------------------------------------");
}
// shutdown Pi4J
console.println("ATTEMPTING TO SHUTDOWN/TERMINATE THIS PROGRAM");
pi4j.shutdown();
}
}
With the following output:
[main] INFO com.pi4j.util.Console - ************************************************************
[main] INFO com.pi4j.util.Console - ************************************************************
[main] INFO com.pi4j.util.Console -
[main] INFO com.pi4j.util.Console - <-- The Pi4J Project -->
[main] INFO com.pi4j.util.Console - Basic SPI Communications Example
[main] INFO com.pi4j.util.Console -
[main] INFO com.pi4j.util.Console - ************************************************************
[main] INFO com.pi4j.util.Console - ************************************************************
[main] INFO com.pi4j.util.Console -
[main] INFO com.pi4j.Pi4J - New auto context
[main] INFO com.pi4j.Pi4J - New context builder
[main] INFO com.pi4j.platform.impl.DefaultRuntimePlatforms - adding platform to managed platform map [id=raspberrypi; name=RaspberryPi Platform; priority=5; class=com.pi4j.plugin.raspberrypi.platform.RaspberryPiPlatform]
[main] INFO com.pi4j.util.Console - --------------------------------------
[main] INFO com.pi4j.util.Console - SPI [READ] :
[main] INFO com.pi4j.util.Console - [BYTES] 0xFF 60
[main] INFO com.pi4j.util.Console - [STRING] �`
[main] INFO com.pi4j.util.Console - --------------------------------------
[main] INFO com.pi4j.util.Console - ATTEMPTING TO SHUTDOWN/TERMINATE THIS PROGRAM

java.net.SocketException: Connection reset since TLSv1.2 protocol

I'm no more able to connect to api.softlayer.com
I'm calling the rest API from a WebSphere application portal 8.5 (java7) using Apache (HTTP-client-4.5.3.jar)
The coding is
HttpClient client = HttpClientBuilder.create().build();
HttpResponse response = client.execute(request);
The error is
17:44:00.997 [main] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection request: [route: {s}->https://api.softlayer.com:443][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 20]
17:44:01.044 [main] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection leased: [id: 0][route: {s}->https://api.softlayer.com:443][total kept alive: 0; route allocated: 1 of 2; total allocated: 1 of 20]
17:44:01.044 [main] DEBUG o.a.h.impl.execchain.MainClientExec - Opening connection {s}->https://api.softlayer.com:443
17:44:01.060 [main] DEBUG o.a.h.i.c.DefaultHttpClientConnectionOperator - Connecting to api.softlayer.com/66.228.119.120:443
17:44:01.060 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - Connecting socket to api.softlayer.com/66.228.119.120:443 with timeout 0
17:44:01.606 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - Enabled protocols: [TLSv1]
17:44:01.606 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - Enabled cipher suites:[TLS_EMPTY_RENEGOTIATION_INFO_SCSV, SSL_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, SSL_ECDHE_RSA_WITH_AES_128_CBC_SHA256, SSL_RSA_WITH_AES_128_CBC_SHA256, SSL_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, SSL_ECDH_RSA_WITH_AES_128_CBC_SHA256, SSL_DHE_RSA_WITH_AES_128_CBC_SHA256, SSL_DHE_DSS_WITH_AES_128_CBC_SHA256, SSL_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, SSL_ECDHE_RSA_WITH_AES_128_CBC_SHA, SSL_RSA_WITH_AES_128_CBC_SHA, SSL_ECDH_ECDSA_WITH_AES_128_CBC_SHA, SSL_ECDH_RSA_WITH_AES_128_CBC_SHA, SSL_DHE_RSA_WITH_AES_128_CBC_SHA, SSL_DHE_DSS_WITH_AES_128_CBC_SHA, SSL_ECDHE_ECDSA_WITH_RC4_128_SHA, SSL_ECDHE_RSA_WITH_RC4_128_SHA, SSL_RSA_WITH_RC4_128_SHA, SSL_ECDH_ECDSA_WITH_RC4_128_SHA, SSL_ECDH_RSA_WITH_RC4_128_SHA, SSL_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, SSL_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, SSL_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, SSL_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_RC4_128_MD5]
17:44:01.606 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - Starting handshake
17:44:01.747 [main] DEBUG o.a.h.i.c.DefaultManagedHttpClientConnection - http-outgoing-0: Shutdown connection
17:44:01.747 [main] DEBUG o.a.h.impl.execchain.MainClientExec - Connection discarded
17:44:01.747 [main] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection released: [id: 0][route: {s}->https://api.softlayer.com:443][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 20]
17:44:01.747 [main] INFO o.a.http.impl.execchain.RetryExec - I/O exception (java.net.SocketException) caught when processing request to {s}->https://api.softlayer.com:443: Connection reset
17:44:01.747 [main] DEBUG o.a.http.impl.execchain.RetryExec - Connection reset
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:207) ~[na:1.7.0]
at java.net.SocketInputStream.read(SocketInputStream.java:133) ~[na:1.7.0]
at com.ibm.jsse2.a.a(a.java:110) ~[na:7.0 build_20131216]
at com.ibm.jsse2.a.a(a.java:141) ~[na:7.0 build_20131216]
at com.ibm.jsse2.qc.a(qc.java:691) ~[na:7.0 build_20131216]
at com.ibm.jsse2.qc.h(qc.java:266) ~[na:7.0 build_20131216]
at com.ibm.jsse2.qc.a(qc.java:770) ~[na:7.0 build_20131216]
at com.ibm.jsse2.qc.startHandshake(qc.java:476) ~[na:7.0 build_20131216]
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) ~[httpclient-4.5.3.jar:4.5.3]
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) ~[httpclient-4.5.3.jar:4.5.3]
The solution is to enable the enable TLSv1.2
import javax.net.ssl.SSLContext;
import org.apache.http.client.HttpClient;
import org.apache.http.conn.ssl.NoopHostnameVerifier;
import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.ssl.SSLContexts;
public class HttpClientFactory {
private static CloseableHttpClient client;
public static HttpClient getHttpsClient() throws Exception {
if (client != null) {
return client;
}
SSLContext sslContext = SSLContexts.createDefault();
SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(sslContext, new String[] { "TLSv1.2"}, null, new NoopHostnameVerifier());
client = HttpClients.custom().setSSLSocketFactory(sslsf).build();
return client;
}
}
Softlayer's servers only accept TLSv1.2 connections you must make sure that your code is only performing connections with that protocol which it seems your code is not doing
17:44:01.606 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory -
Enabled protocols: [TLSv1]

Curator Leader election give connection refused error

I implemented curator leader election example which is given in this site
Instead of having number of curator clients I added only one curator client as follows
public void selectLeader() {
CuratorFramework client = null;
try {
client = CuratorFrameworkFactory.newClient("localhost:2181", new ExponentialBackoffRetry(1000, 3));
LeaderSelectorService service = new LeaderSelectorService(client, "/leaderSelections", "LeaderElector");
client.start();
Thread.sleep(10000);
service.start();
} catch (Exception e) {
System.out.println("error"+e);
}finally
{
System.out.println("Shutting down...");
// CloseableUtils.closeQuietly(client);
}
}
public class LeaderSelectorService extends LeaderSelectorListenerAdapter implements Closeable {
private final String name;
private final LeaderSelector leaderSelector;
public LeaderSelectorService(CuratorFramework client, String path, String name) {
this.name = name;
// create a leader selector using the given path for management
// all participants in a given leader selection must use the same path
// ExampleClient here is also a LeaderSelectorListener but this isn't required
leaderSelector = new LeaderSelector(client, path, this);
// for most cases you will want your instance to requeue when it relinquishes leadership
leaderSelector.autoRequeue();
}
public void start() throws IOException
{
// the selection for this instance doesn't start until the leader selector is started
// leader selection is done in the background so this call to leaderSelector.start() returns immediately
leaderSelector.start();
}
#Override
public void takeLeadership(CuratorFramework arg0) throws Exception {
// we are now the leader. This method should not return until we want to relinquish leadership
final int waitSeconds = (int)(5 * Math.random()) + 1;
System.out.println(name + " is now the leader. Waiting " + waitSeconds + " seconds...");
//System.out.println(name + " has been leader " + leaderCount.getAndIncrement() + " time(s) before.");
try
{
Thread.sleep(TimeUnit.SECONDS.toMillis(waitSeconds));
}
catch ( InterruptedException e )
{
System.err.println(name + " was interrupted.");
Thread.currentThread().interrupt();
}
finally
{
System.out.println(name + " relinquishing leadership.\n");
}
}
#Override
public void close() throws IOException {
leaderSelector.close();
}
}
I have only one zookeeper instance and I am using Zookeeper 3.4.6, curator-framework 4.0.0 and curator-recipes 4.0.0.
when I start the client, it connects to zookeeper and in the log I can see "State change : connected" message.
Then I wait 10s and start leader election which gives me below error repeatedly.
2017-09-06 09:34:22.727 INFO 1228 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Unable to read additional data from server sessionid 0x15e555a719d0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-09-06 09:34:22.830 INFO 1228 --- [c-1-EventThread] o.a.c.f.state.ConnectionStateManager : State change: SUSPENDED
2017-09-06 09:34:23.302 INFO 1228 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-09-06 09:34:23.303 INFO 1228 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Socket connection established, initiating session, client: /127.0.0.1:49594, server: localhost/127.0.0.1:2181
2017-09-06 09:34:23.305 INFO 1228 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x15e555a719d0000, negotiated timeout = 120000
2017-09-06 09:34:23.305 INFO 1228 --- [c-1-EventThread] o.a.c.f.state.ConnectionStateManager : State change: RECONNECTED
2017-09-06 09:34:23.310 WARN 1228 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Session 0x15e555a719d0000 for server localhost/127.0.0.1:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_131]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:1.8.0_131]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_131]
at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.8.0_131]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[na:1.8.0_131]
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:75) ~[zookeeper-3.5.3-beta.jar:3.5.3-beta-8ce24f9e675cbefffb8f21a47e06b42864475a60]
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:363) ~[zookeeper-3.5.3-beta.jar:3.5.3-beta-8ce24f9e675cbefffb8f21a47e06b42864475a60]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214) ~[zookeeper-3.5.3-beta.jar:3.5.3-beta-8ce24f9e675cbefffb8f21a47e06b42864475a60]
after some time it started to give me below error message.
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) ~[zookeeper-3.5.3-beta.jar:3.5.3-beta-8ce24f9e675cbefffb8f21a47e06b42864475a60]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:831) [curator-framework-4.0.0.jar:4.0.0]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:623) [curator-framework-4.0.0.jar:4.0.0]
at org.apache.curator.framework.imps.WatcherRemovalFacade.processBackgroundOperation(WatcherRemovalFacade.java:152) [curator-framework-4.0.0.jar:4.0.0]
at org.apache.curator.framework.imps.GetConfigBuilderImpl$2.processResult(GetConfigBuilderImpl.java:222) [curator-framework-4.0.0.jar:4.0.0]
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:590) [zookeeper-3.5.3-beta.jar:3.5.3-beta-8ce24f9e675cbefffb8f21a47e06b42864475a60]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:499) [zookeeper-3.5.3-beta.jar:3.5.3-beta-8ce24f9e675cbefffb8f21a47e06b42864475a60]
2017-09-06 09:34:31.897 INFO 1228 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-09-06 09:34:31.898 INFO 1228 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Socket connection established, initiating session, client: /127.0.0.1:49611, server: localhost/127.0.0.1:2181
2017-09-06 09:34:31.899 INFO 1228 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x15e555a719d0000, negotiated timeout = 120000
2017-09-06 09:34:31.899 INFO 1228 --- [c-1-EventThread] o.a.c.f.state.ConnectionStateManager : State change: RECONNECTED
2017-09-06 09:34:31.907 WARN 1228 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Session 0x15e555a719d0000 for server localhost/127.0.0.1:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Xid out of order. Got Xid 41 with err -6 expected Xid 40 for a packet with details: clientPath:/leaderSelections serverPath:/leaderSelections finished:false header:: 40,12 replyHeader:: 0,0,-4 request:: '/leaderSelections,F response:: v{}
at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:892) ~[zookeeper-3.5.3-beta.jar:3.5.3-beta-8ce24f9e675cbefffb8f21a47e06b42864475a60]
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:101) ~[zookeeper-3.5.3-beta.jar:3.5.3-beta-8ce24f9e675cbefffb8f21a47e06b42864475a60]
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:363) ~[zookeeper-3.5.3-beta.jar:3.5.3-beta-8ce24f9e675cbefffb8f21a47e06b42864475a60]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214) ~[zookeeper-3.5.3-beta.jar:3.5.3-beta-8ce24f9e675cbefffb8f21a47e06b42864475a60]
I tried several solution in the internet but non got succeeded. Does anybody know the root cause of this issue.
I have fixed this issue. There was version number mismatch between zookeeper version and curator version. I used curator version 4.0.0 with zookeeper 3.4.6. According to apache curator site
Curator 4.0.0 - compatible with ZooKeeper 3.5.x. I changed my curator version to 2.8.0

Standalone Spark Task Hangs When Inserting Into DB

I have a standalone spark 1.4.1 job that runs on a Red Hat box I submit via spark-submit that sometimes hangs during insertion of data from an RDD. I have auto-commit on the connection turned off and commit the transactions in batches of insertions. What the logs show me before it hangs:
16/03/25 14:00:05 INFO Executor: Finished task 3.0 in stage 138.0 (TID 915). 1847 bytes result sent to driver
16/03/25 14:00:05 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] received message AkkaMessage(StatusUpdate(915,FINISHED,java.nio.HeapByteBuffer[pos=0 lim=1847 cap=1
16/03/25 14:00:05 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: Received RPC message: AkkaMessage(StatusUpdate(915,FINISHED,java.nio.HeapByteBuffer[pos=0 lim=1847 cap=1847
16/03/25 14:00:05 DEBUG TaskSchedulerImpl: parentName: , name: TaskSet_138, runningTasks: 1
16/03/25 14:00:05 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] handled message (0.118 ms) AkkaMessage(StatusUpdate(915,FINISHED,java.nio.HeapByteBuffer[pos=621 li
16/03/25 14:00:05 INFO TaskSetManager: Finished task 3.0 in stage 138.0 (TID 915) in 7407 ms on localhost (23/24)
16/03/25 14:00:05 TRACE DAGScheduler: Checking for newly runnable parent stages
16/03/25 14:00:05 TRACE DAGScheduler: running: Set(ResultStage 138)
16/03/25 14:00:05 TRACE DAGScheduler: waiting: Set()
16/03/25 14:00:05 TRACE DAGScheduler: failed: Set()
16/03/25 14:00:10 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] received message AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;#7ed52306,BlockManagerId(driver, local
16/03/25 14:00:10 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: Received RPC message: AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;#7ed52306,BlockManagerId(driver, localhos
16/03/25 14:00:10 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] handled message (0.099 ms) AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;#7ed52306,BlockManagerId(dri
And then it just repeats the last 3 lines with this intermittently:
16/03/25 14:01:04 TRACE HeartbeatReceiver: Checking for hosts with no recent heartbeats in HeartbeatReceiver.
The kicker is that I can't take a look at the web UI due to some firewall issues on these machines. What I noticed is that this issue was more prevalent when I was inserting with batches of 1000 than with 100. This is the scala code that looks to be the culprit.
//records should have up to INSERT_BATCH_SIZE entries
private def insertStuff(records: Seq[(String, (String, Stuff1, Stuff2, Stuff3))]) {
if (!records.isEmpty) {
//get statement used for insertion (instantiated in an array of statements)
val stmt = stuffInsertArray(//stuff)
logger.info("Starting insertions on stuff" + table + " for " + time + " with " + records.length + " records")
try {
records.foreach(record => {
//get vals from record
...
//perform sanity checks
if (//validate stuff)
{
//log stuff because it didn't validate
}
else
{
stmt.setInt(1, //stuff)
stmt.setLong(2, //stuff)
...
stmt.addBatch()
}
})
//check if connection is still valid
if (!connInsert.isValid(VALIDATE_CONNECTION_TIMEOUT))
{
logger.error("Insertion connection is not valid while inserting stuff.")
throw new RuntimeException(s"Insertion connection not valid while inserting stuff.")
}
logger.debug("Stuff insertion executing batch...")
stmt.executeBatch()
logger.debug("Stuff insertion execution complete. Committing...")
//commit insert batch. Either INSERT_BATCH_SIZE insertions planned or the last batch to be done
insertCommit() //this does the commit and resets some counters
logger.debug("stuff insertion commit complete.")
} catch {
case e: Exception => throw new RuntimeException(s"insertStuff exception ${e.getMessage}")
}
}
}
And here's how it gets called:
//stuffData is an RDD
stuffData.foreachPartition(recordIt => {
//new instance of the object of whose member function we're currently in
val obj = new Obj(clusterInfo)
recordIt.grouped(INSERT_BATCH_SIZE).foreach(records => obj.insertStuff(records))
})
All the extra logging and connection checking I put in just to isolate the issue but since I write for every batch of insertions, the logs get convoluted. If I serialize the insertions, the issue still persists. Any idea why the last task (out of 24) doesn't finish? Thanks.

Kafka consumer java code not working

I have Just started with Kafka, I am able to produce and consume data through command prompt and also produce data through Java code even from remote server.
But I am trying this simple consumer Java code it is not working.
public class Simpleconsumer {
private final ConsumerConnector consumer;
private final String topic;
public Simpleconsumer(String topic) {
Properties props = new Properties();
props.put("zookeeper.connect", "127.0.0.1:2181");
props.put("group.id", "topic1");
props.put("auto.offset.reset", "smallest");
consumer = Consumer.createJavaConsumerConnector(new ConsumerConfig(props));
this.topic = topic;
}
public void testConsumer() {
try{
Map<String, Integer> topicCount = new HashMap<String, Integer>();
topicCount.put(topic, 1);
Map<String, List<KafkaStream<byte[], byte[]>>> consumerStreams = consumer.createMessageStreams(topicCount);
List<KafkaStream<byte[], byte[]>> streams = consumerStreams.get(topic);
System.out.println("start.......");
for (final KafkaStream stream : streams) {
ConsumerIterator<byte[], byte[]> it = stream.iterator();
System.out.println("iterate.......");
while (it.hasNext()) {
System.out.println("Message from Single Topic: " + new String(it.next().message()));
}
}
System.out.println("end.......");
if (consumer != null) {
consumer.shutdown();
}
}
catch(Exception e)
{
System.out.println(e);
}
}
public static void main(String[] args) {
// String topic = args[0];
Simpleconsumer simpleHLConsumer = new Simpleconsumer("topic1");
simpleHLConsumer.testConsumer();
}
}
Output is :-
log4j:WARN No appenders could be found for logger (kafka.utils.VerifiableProperties).
log4j:WARN Please initialize the log4j system properly.
start.......
iterate.......
There is no error, the programme doesn't terminate or give any output!!
Log of zookeeper
2016-02-18 17:31:31,790 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory#197] - Accepted socket connection from /127.0.0.1:33338
2016-02-18 17:31:31,793 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer#868] - Client attempting to establish new session at /127.0.0.1:33338
2016-02-18 17:31:31,821 [myid:] - INFO [SyncThread:0:ZooKeeperServer#617] - Established session 0x152f4265b0b0009 with negotiated timeout 6000 for client /127.0.0.1:33338
2016-02-18 17:31:31,891 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor#645] - Got user-level KeeperException when processing sessionid:0x152f4265b0b0009 type:create cxid:0x1 zxid:0x718 txntype:-1 reqpath:n/a Error Path:/consumers Error:KeeperErrorCode = NodeExists for /consumers
2016-02-18 17:31:31,892 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor#645] - Got user-level KeeperException when processing sessionid:0x152f4265b0b0009 type:create cxid:0x2 zxid:0x719 txntype:-1 reqpath:n/a Error Path:/consumers/artinew Error:KeeperErrorCode = NodeExists for /consumers/artinew
2016-02-18 17:31:31,892 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor#645] - Got user-level KeeperException when processing sessionid:0x152f4265b0b0009 type:create cxid:0x3 zxid:0x71a txntype:-1 reqpath:n/a Error Path:/consumers/artinew/ids Error:KeeperErrorCode = NodeExists for /consumers/artinew/ids
2016-02-18 17:31:32,000 [myid:] - INFO [SessionTracker:ZooKeeperServer#347] - Expiring session 0x152f4265b0b0008, timeout of 6000ms exceeded
2016-02-18 17:31:32,000 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor#494] - Processed session termination for sessionid: 0x152f4265b0b0008
2016-02-18 17:31:32,002 [myid:] - INFO [SyncThread:0:NIOServerCnxn#1007] - Closed socket connection for client /127.0.0.1:33337 which had sessionid 0x152f4265b0b0
I am getting this in Kafka console in infinite loop. please explain
[2016-02-17 20:50:08,594] INFO Closing socket connection to /xx.xx.xx.xx. (kafka.network.Processor)
[2016-02-17 20:50:08,174] INFO Closing socket connection to /xx.xx.xx.xx. (kafka.network.Processor)
[2016-02-17 20:50:08,385] INFO Closing socket connection to /xx.xx.xx.xx. (kafka.network.Processor)
[2016-02-17 20:50:08,760] INFO Closing socket connection to /xx.xx.xx.xx. (kafka.network.Processor)
i created the topic in the following manner
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 5 --topic topic1
i am able to consume it in command prompt using
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic topic1
Not able to understand what is the issue.
Try localhost instead of 127.0.0.1 in code to make sure local resolution is working fine.