How to publish data to using MQTT - scala

I used this docker image to install Mosquitto MQTT.
Now it's running and showing the following message in the terminal:
1515680808: mosquitto version 1.4.14 (build date Mon, 10 Jul 2017 23:48:43 +0100) starting
1515680808: Config loaded from /mqtt/config/mosquitto.conf.
1515680808: Opening websockets listen socket on port 9001.
1515680808: Opening ipv4 listen socket on port 1883.
1515680808: Opening ipv6 listen socket on port 1883.
Then I created a simple Maven project:
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-mqtt_2.11</artifactId>
<version>1.6.3</version>
</dependency>
I tried to publish some data to a topic using the code shown below. I point to localhost:1883 as the MqttBrokerUrl and a topic test. However, I get this error:
Exception in thread "main" java.lang.NullPointerException at
org.eclipse.paho.client.mqttv3.MqttConnectOptions.validateURI(MqttConnectOptions.java:457)
at
org.eclipse.paho.client.mqttv3.MqttAsyncClient.(MqttAsyncClient.java:273)
at
org.eclipse.paho.client.mqttv3.MqttAsyncClient.(MqttAsyncClient.java:167)
at
org.eclipse.paho.client.mqttv3.MqttClient.(MqttClient.java:224)
at org.test.MQTTPublisher$.main(MQTTPublisher.scala:37) at
org.test.MQTTPublisher.main(MQTTPublisher.scala)
Code:
package org.test
import org.apache.log4j.{Level, Logger}
import org.eclipse.paho.client.mqttv3._
import org.eclipse.paho.client.mqttv3.persist.MemoryPersistence
import org.apache.spark.storage.StorageLevel
import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.streaming.mqtt._
import org.apache.spark.SparkConf
object MQTTPublisher {
def main(args: Array[String]) {
if (args.length < 2) {
System.err.println("Usage: MQTTPublisher <MqttBrokerUrl> <topic>")
System.exit(1)
}
// Set logging level if log4j not configured (override by adding log4j.properties to classpath)
if (!Logger.getRootLogger.getAllAppenders.hasMoreElements) {
Logger.getRootLogger.setLevel(Level.WARN)
}
val Seq(brokerUrl, topic) = args.toSeq
var client: MqttClient = null
try {
val persistence = new MemoryPersistence()
client = new MqttClient("localhost:1883", MqttClient.generateClientId(), persistence)
client.connect()
val msgtopic = client.getTopic(topic)
val msgContent = "test test test"
val message = new MqttMessage(msgContent.getBytes("utf-8"))
while (true) {
try {
msgtopic.publish(message)
println(s"Published data. topic: ${msgtopic.getName()}; Message: $message")
} catch {
case e: MqttException if e.getReasonCode == MqttException.REASON_CODE_MAX_INFLIGHT =>
Thread.sleep(10)
println("Queue is full, wait for to consume data from the message queue")
}
}
} catch {
case e: MqttException => println("Exception Caught: " + e)
} finally {
if (client != null) {
client.disconnect()
}
}
}
}

The MqttClient() constructor takes a URI.
What you have provided is just a hostname and port number (localhost:1883), it's missing a protocol section which should be tcp:// (which is what the library is expecting and getting null back. This really should throw a better error.)
You need to change the line to be
client = new MqttClient("tcp://localhost:1883", MqttClient.generateClientId(), persistence);

I think you are giving the wrong Url i.e you are not specifying the protocol over which it has to connect that is my hunch.
Try changing the url to :
tcp://localhost:1883
I think it would work ! Rest all seems fine to me.
For a working example See this : https://github.com/shiv4nsh/scala-mqtt-client-rasberrypi-starter-kit/blob/master/src/main/scala/com/knoldus/MQTTPublisher.scala

Related

java.lang.NoClassDefFoundError: org/apache/flink/streaming/api/scala/StreamExecutionEnvironment

package com.knoldus
import org.apache.flink.api.java.utils.ParameterTool
import org.apache.flink.streaming.api.scala._
import org.apache.flink.streaming.api.windowing.time.Time
object SocketWindowWordCount {
def main(args: Array[String]) : Unit = {
var hostname: String = "localhost"
var port: Int = 9000
try {
val params = ParameterTool.fromArgs(args)
hostname = if (params.has("hostname")) params.get("hostname") else "localhost"
port = params.getInt("port")
} catch {
case e: Exception => {
System.err.println("No port specified. Please run 'SocketWindowWordCount " +
"--hostname <hostname> --port <port>', where hostname (localhost by default) and port " +
"is the address of the text server")
System.err.println("To start a simple text server, run 'netcat -l <port>' " +
"and type the input text into the command line")
return
}
}
// get the execution environment
val env: StreamExecutionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment
// get input data by connecting to the socket
val text: DataStream[String] = env.socketTextStream(hostname, port, '\n')
// parse the data, group it, window it, and aggregate the counts
val windowCounts = text
.flatMap { w => w.split("\\s") }
.map { w => WordWithCount(w, 1) }
.keyBy("word")
.timeWindow(Time.seconds(5))
.sum("count")
// print the results with a single thread, rather than in parallel
windowCounts.print().setParallelism(1)
env.execute("Socket Window WordCount")
}
/** Data type for words with count */
case class WordWithCount(word: String, count: Long)
}
In the end while running this code on Intellij i am getting this error
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/flink/streaming/api/scala/StreamExecutionEnvironment$
at com.knoldus.SocketWindowWordCount$.main(SocketWindowWordCount.scala:43)
at com.knoldus.SocketWindowWordCount.main(SocketWindowWordCount.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.flink.streaming.api.scala.StreamExecutionEnvironment$
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
... 2 more
Select menu item "Run" => "Edit Configurations...",
then in the "Build and run" section select "Modify options" => Java => Add dependencies with "Provided" scope to classpath in your local configuration.
In this way you don't have to remove the <scope>provided</scope>.
I resolved this by removing the <scope>provided</scope> which was present in the maven import.

Publishing messages to ActiveMQ using Gatling

I have been using Gatling to publish messages to ActiveMq server. I get "java.lang.SecurityException: Invalid username: null or empty" tho I use valid username and password. Here is my test code and the exception were thrown. Any inputs on how to fix this will be of help.
import io.gatling.core.Predef.Simulation
import io.gatling.core.Predef._
import io.gatling.jms.Predef._
import io.gatling.core.config.Credentials
import org.apache.activemq.ActiveMQConnectionFactory
import org.apache.activemq.jndi.ActiveMQInitialContextFactory
import javax.jms._
class WebProducer extends Simulation{
val jmsUsername:String="userName"
val jmsPwd:String="Password"
val jmsConfig = jms
.connectionFactoryName("ConnectionFactory")
.url("ssl://message01-dev.platform.net:61617")
.credentials(jmsUsername,jmsPwd)
.disableAnonymousConnect
.contextFactory(classOf[org.apache.activemq.jndi.ActiveMQInitialContextFactory].getName)
.listenerCount(1)
.usePersistentDeliveryMode
.receiveTimeout(6000)
val scn = scenario("JMS DSL test").repeat(1) {
exec(jms("req reply testing").
reqreply
.queue("YourJMSQueueName")
.replyQueue("YourJMSQueueName")
.textMessage("payload To be posted")
.property("company_id", "1234598776665")
.property("event_type","EntityCreate")
.property("event_target_entity_type","Account")
)
}
setUp(scn.inject(atOnceUsers(1)))
.protocols(jmsConfig)
}
Following is the exception was thrown :
java.lang.SecurityException: Invalid username: null or empty
Ok, I got this working adding two things :
- .disableAnonymousConnect after .credentials(jmsUsername,jmsPwd)
- .replyQueue(jmsQueueName) after .queue(jmsQueueName)
I edited the above code to reflect the same.
Happy Gatling !

Debugging a standalone jetty server - how to specify single threaded mode?

I have successfully created a standalone Scalatra / Jetty server, using the official instructions from Scalatra ( http://www.scalatra.org/2.3/guides/deployment/standalone.html )
I am debugging it under Ensime, and would like to limit the number of threads handling messages to a single one - so that single-stepping through the servlet methods will be easier.
I used this code to achieve it:
package ...
import org.eclipse.jetty.server.Server
import org.eclipse.jetty.servlet.{DefaultServlet, ServletContextHandler}
import org.eclipse.jetty.webapp.WebAppContext
import org.scalatra.servlet.ScalatraListener
import org.eclipse.jetty.util.thread.QueuedThreadPool
import org.eclipse.jetty.server.ServerConnector
object JettyLauncher {
def main(args: Array[String]) {
val port =
if (System.getenv("PORT") != null)
System.getenv("PORT").toInt
else
4080
// DEBUGGING MODE BEGINS
val threadPool = new QueuedThreadPool()
threadPool.setMaxThreads(8)
val server = new Server(threadPool)
val connector = new ServerConnector(server)
connector.setPort(port)
server.setConnectors(Array(connector))
// DEBUGGING MODE ENDS
val context = new WebAppContext()
context setContextPath "/"
context.setResourceBase("src/main/webapp")
context.addEventListener(new ScalatraListener)
context.addServlet(classOf[DefaultServlet], "/")
server.setHandler(context)
server.start
server.join
}
}
It works fine - except for one minor detail...
I can't tell Jetty to use 1 thread - the minimum value is 8!
If I do, this is what happens:
$ sbt assembly
...
$ java -jar ./target/scala-2.11/CurrentVersions-assembly-0.1.0-SNAPSHOT.jar
18:13:27.059 [main] INFO org.eclipse.jetty.util.log - Logging initialized #41ms
18:13:27.206 [main] INFO org.eclipse.jetty.server.Server - jetty-9.1.z-SNAPSHOT
18:13:27.220 [main] WARN o.e.j.u.component.AbstractLifeCycle - FAILED org.eclipse.jetty.server.Server#1ac539f: java.lang.IllegalStateException: Insufficient max threads in ThreadPool: max=1 < needed=8
java.lang.IllegalStateException: Insufficient max threads in ThreadPool: max=1 < needed=8
...which is why you see setMaxThreads(8) instead of setMaxThreads(1) in my code above.
Any ideas why this happens?
The reason is that the size of the threadpool also depends on th enumber of connectors you've got defined. If you look at the source code of the jetty server you'll see this:
// check size of thread pool
SizedThreadPool pool = getBean(SizedThreadPool.class);
int max=pool==null?-1:pool.getMaxThreads();
int selectors=0;
int acceptors=0;
if (mex.size()==0)
{
for (Connector connector : _connectors)
{
if (connector instanceof AbstractConnector)
acceptors+=((AbstractConnector)connector).getAcceptors();
if (connector instanceof ServerConnector)
selectors+=((ServerConnector)connector).getSelectorManager().getSelectorCount();
}
}
int needed=1+selectors+acceptors;
if (max>0 && needed>max)
throw new IllegalStateException(String.format("Insufficient threads: max=%d < needed(acceptors=%d + selectors=%d + request=1)",max,acceptors,selectors));
So the minimum with a single serverconnector is 2. It looks like you've got a couple of other default connectors or selectors running.

AKKA remote (with SSL) can't find keystore/truststore files on classpath

I'm trying to configure AKKA SSL connection to use my keystore and trustore files, and I want it to be able to find them on a classpath.
I tried to set application.conf to:
...
remote.netty.ssl = {
enable = on
key-store = "keystore"
key-store-password = "passwd"
trust-store = "truststore"
trust-store-password = "passwd"
protocol = "TLSv1"
random-number-generator = "AES128CounterSecureRNG"
enabled-algorithms = ["TLS_RSA_WITH_AES_128_CBC_SHA"]
}
...
This works fine if keystore and trustore files are in the current directory. In my application these files get packaged into WAR and JAR archives, and because of that I'd like to read them from the classpath.
I tried to use getResource("keystore") in application.conf as described here without any luck. Config reads it literally as a string.
I also tried to parse String conf and force it to read the value:
val conf: Config = ConfigFactory parseString (s"""
...
"${getClass.getClassLoader.getResource("keystore").getPath}"
...""")
In this case it finds proper path on the classpath as file://some_dir/target/scala-2.10/server_2.10-1.1-one-jar.jar!/main/server_2.10-1.1.jar!/keystore which is exactly where it's located (in the jar). However, underlying Netty SSL transport can't find the file given this path, and I get:
Oct 03, 2013 1:02:48 PM org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink
WARNING: Failed to initialize an accepted socket.
45a13eb9-6cb1-46a7-a789-e48da9997f0fakka.remote.RemoteTransportException: Server SSL connection could not be established because key store could not be loaded
at akka.remote.netty.NettySSLSupport$.constructServerContext$1(NettySSLSupport.scala:113)
at akka.remote.netty.NettySSLSupport$.initializeServerSSL(NettySSLSupport.scala:130)
at akka.remote.netty.NettySSLSupport$.apply(NettySSLSupport.scala:27)
at akka.remote.netty.NettyRemoteTransport$PipelineFactory$.defaultStack(NettyRemoteSupport.scala:74)
at akka.remote.netty.NettyRemoteTransport$PipelineFactory$$anon$1.getPipeline(NettyRemoteSupport.scala:67)
at akka.remote.netty.NettyRemoteTransport$PipelineFactory$$anon$1.getPipeline(NettyRemoteSupport.scala:67)
at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.registerAcceptedChannel(NioServerSocketPipelineSink.java:277)
at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:242)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.io.FileNotFoundException: file:/some_dir/server/target/scala-2.10/server_2.10-1.1-one-jar.jar!/main/server_2.10-1.1.jar!/keystore (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileInputStream.<init>(FileInputStream.java:97)
at akka.remote.netty.NettySSLSupport$.constructServerContext$1(NettySSLSupport.scala:118)
... 10 more
I wonder if there is any way to configure this in AKKA without implementing custom SSL transport. Maybe I should configure Netty in the code?
Obviously I can hardcode the path or read it from an environment variable, but I would prefer a more flexible classpath solution.
I decided to look at the akka.remote.netty.NettySSLSupport at the code where exception is thrown from, and here is the code:
def initializeServerSSL(settings: NettySettings, log: LoggingAdapter): SslHandler = {
log.debug("Server SSL is enabled, initialising ...")
def constructServerContext(settings: NettySettings, log: LoggingAdapter, keyStorePath: String, keyStorePassword: String, protocol: String): Option[SSLContext] =
try {
val rng = initializeCustomSecureRandom(settings.SSLRandomNumberGenerator, settings.SSLRandomSource, log)
val factory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm)
factory.init({
val keyStore = KeyStore.getInstance(KeyStore.getDefaultType)
val fin = new FileInputStream(keyStorePath)
try keyStore.load(fin, keyStorePassword.toCharArray) finally fin.close()
keyStore
}, keyStorePassword.toCharArray)
Option(SSLContext.getInstance(protocol)) map { ctx ⇒ ctx.init(factory.getKeyManagers, null, rng); ctx }
} catch {
case e: FileNotFoundException ⇒ throw new RemoteTransportException("Server SSL connection could not be established because key store could not be loaded", e)
case e: IOException ⇒ throw new RemoteTransportException("Server SSL connection could not be established because: " + e.getMessage, e)
case e: GeneralSecurityException ⇒ throw new RemoteTransportException("Server SSL connection could not be established because SSL context could not be constructed", e)
}
It looks like it must be a plain filename (String) because that's what FileInputStream takes.
Any suggestions would be welcome!
I also got stuck in similar issue and was getting similar errors. In my case I was trying to hit an https server with self-signed certificates using akka-http, with following code I was able to get through:
val trustStoreConfig = TrustStoreConfig(None, Some("/etc/Project/keystore/my.cer")).withStoreType("PEM")
val trustManagerConfig = TrustManagerConfig().withTrustStoreConfigs(List(trustStoreConfig))
val badSslConfig = AkkaSSLConfig().mapSettings(s => s.withLoose(s.loose
.withAcceptAnyCertificate(true)
.withDisableHostnameVerification(true)
).withTrustManagerConfig(trustManagerConfig))
val badCtx = Http().createClientHttpsContext(badSslConfig)
Http().superPool[RequestTracker](badCtx)(httpMat)
At the time of writing this question there was no way to do it AFAIK. I'm closing this question but I welcome updates if new versions provide such functionality or if there are other ways to do that.

Scalate ResourceNotFoundException in Scalatra

I'm trying the following based on scalatra-sbt.g8:
class FooWeb extends ScalatraServlet with ScalateSupport {
beforeAll { contentType = "text/html" }
get("/") {
templateEngine.layout("/WEB-INF/scalate/templates/hello-scalate.jade")
}
}
but I'm getting the following exception (even though the file exists) - any clues?
Could not load resource: [/WEB-INF/scalate/templates/hello-scalate.jade]; are you sure it's within [null]?
org.fusesource.scalate.util.ResourceNotFoundException: Could not load resource: [/WEB-INF/scalate/templates/hello-scalate.jade]; are you sure it's within [null]?
FWIW, the innermost exception is coming from org.mortbay.jetty.handler.ContextHandler.getResource line 1142: _baseResource==null.
Got an answer from the scalatra mailing list. The problem was that I was starting the Jetty server with:
import org.mortbay.jetty.Server
import org.mortbay.jetty.servlet.{Context,ServletHolder}
val server = new Server(8080)
val root = new Context(server, "/", Context.SESSIONS)
root.addServlet(new ServletHolder(new FooWeb()), "/*")
server.start()
I needed to insert this before start():
root.setResourceBase("src/main/webapp")