client.publish("topic", payload,qos=2) not publishing data onto the broker whereas the same is working for qos=0 and 1.I have not defined persistence, it is set to default value.
import paho.mqtt.publish as publish
import paho.mqtt.client as mqtt
from random import *
import time
MQTT_IP='192.168.0.23'
MQTT_PORT=1883
global client
import datetime
def on_connect(client, userdata, flags, rc,qos):
if rc==0:
print("MQTT CONNECTION ESTABLISHED")
print(str(client)+'\n'+str(userdata))
def on_message(client, userdata, msg):
print("Message arrived from "+str(client))
print(msg.topic+" "+str(msg.payload))
def MQTT_CONNECTION():
global client
global MQTT_CLIENT_CONNECTED
try:
print("IN mqtt connection")
client = mqtt.Client()
client.connect(MQTT_IP,MQTT_PORT)
client.on_connect = on_connect
client.on_message = on_message
MQTT_CLIENT_CONNECTED=True
except Exception as error:
print("ERROR IN MQTT CONNECTION",error)
MQTT_CLIENT_CONNECTED=False
def publish():
global client
client.publish("1/MB/EM/3/21/IB",2,qos=2,retain=True)
if __name__=="__main__":
MQTT_CONNECTION()
publish()
I have removed all other functionality, for you all to understand better.
You need to start the MQTT client network loop. This is to handle the multi step handshake for QOS 2 messaging.
The python Paho client has 3 ways to run the loop.
You can start the loop on a background thread by calling client.loop_start()
You can start the loop on the current thread, which will then block forever with client.loop_forever()
You can run the tasks that the loop perform within your own loop by calling client.loop() at regular intervals
Which one you choose depends on what else your code is doing, but I suspect that starting the loop on a background thread is probably the best.
Related
I have a load test where three sets of users create something and a different set of users perform some actions on them.
What is the recommended way to co-ordinate this behaviour in Gatling?
I'm currently using an object which contains a LinkedBlockingQueue which the "producers" put the ID and consumers take, see below.
However, it causes the test to hang after ~20s (targeting 1tps).
I've also tried using poll with a timeout, but instead of hanging the poll almost always fails (after 30s) or causes a hang if the timeout is larger (1m+).
This seems to be because all the threads are blocked waiting for something from the queue so isn't compatible with the way Gatling tests run (i.e. not 1 thread per user). Is there a non-blocking way to wait in the Gatling DSL?
Producer.scala
// ...
scenario("Produce stuff")
.exec(/* HTTP call which extracts an ID*/)
.exec(session => Queue.ids.put(session("my-id").as[String])
// ...
Consumer.scala
// ...
scenario("Consume stuff")
.exec(session => session.set("my-id", Queue.ids.take()))
.exec(/* HTTP call which users ID*/)
// ...
Queue.scala
object Queue {
val ids = new LinkedBlockingQueue[String]()
}
As an alternative I've tried to use the application functionality but it seems a harder problem to ensure that each user picks a unique item from the app.
Acknowledging this is all a hack, my current solution in Consumer.scala is:
doIf(_ => Queue.ids.size() < MIN_COUNT)(
pause(30) // wait for 30s if queue is initially too small
)
.doWhile(_ => Queue.ids.size() >= MIN_COUNT)(
exec(session => session.set("my-id", Queue.ids.take()))
.exec(...)
.pause(30)
)
In my project, I created UserRepositoryActor which create their own router with 10 UserRepositoryWorkerActor instances as routee, see hierarchy below:
As you see, if any error occur while fetching data from database, it will occur at worker.
Once I want to fetch user from database, I send message to UserRepositoryActor with this command:
val resultFuture = userRepository ? FindUserById(1)
and I set 10 seconds for timeout.
In case of network connection has problem, UserRepositoryWorkerActor immediately get ConnectionException from underlying database driver and then (what I think) router will restart current worker and send FindUserById(1) command to other worker that available and resultFuture will get AskTimeoutException after 10 seconds passed. Then some time later, once connection back to normal, UserRepositoryWorkerActor successfully fetch data from database and then try to send result back to the caller and found that resultFuture was timed out.
I want to propagate error from UserRepositoryWorkerActor up to the caller immediately after exception occur, so that will prevent resultFuture to wait for 10 seconds and stop UserRepositoryWorkerActor to try to fetch data again and again.
How can I do that?
By the way, if you have any suggestions to my current design, please suggest me. I'm very new to Akka.
Your assumption about Router resending the message is wrong. Router has already passed the message to routee and it doesnt have it any more.
As far as ConnectionException is concerned, you could wrap in a scala.util.Try and send response to sender(). Something like,
Try(SomeDAO.getSomeObjectById(id)) match {
case Success(s) => sender() ! s
case Failure(e) => sender() ! e
}
You design looks correct. Having a router allows you to distribute work and also to limit number of concurrent workers accessing the database.
Option 1
You can make your router watch its children and act accordingly when they are terminated. For example (taken from here):
import akka.routing.{ ActorRefRoutee, RoundRobinRoutingLogic, Router }
class Master extends Actor {
var router = {
val routees = Vector.fill(5) {
val r = context.actorOf(Props[Worker])
context watch r
ActorRefRoutee(r)
}
Router(RoundRobinRoutingLogic(), routees)
}
def receive = {
case w: Work =>
router.route(w, sender())
case Terminated(a) =>
router = router.removeRoutee(a)
val r = context.actorOf(Props[Worker])
context watch r
router = router.addRoutee(r)
}
}
In your case you can send some sort of a failed message from the repository actor to the client. Repository actor can maintain a map of worker ref to request id to know which request failed when worker terminates. It can also record the time between the start of the request and actor termination to decide whether it's worth retrying it with another worker.
Option 2
Simply catch all non-fatal exceptions in your worker actor and reply with appropriate success/failed messages. This is much simpler but you might still want to restart the worker to make sure it's in a good state.
p.s. Router will not restart failed workers, neither it will try to resend messages to them by default. You can take a look at supervisor strategy and Option 1 above on how to achieve that.
I'm running a TCP server on linux using reactor.listenTCP.
All works well for hours (or days), then something happens and clients can connect but they are immediately disconnected by the server.
There is nothing in my protocol implementation that commands a disconnect.
There is nothing in the client implementations that would cause such a disconnect.
The server intermittently sends some data to all users that it finds in the user list using sendLine(data).
I'm wondering if I am meant to perform some operation when a client disconnects that I've missed amd this is causing socket issues.
class MsgHandler(Protocol):
def __init__(self, factory):
self.factory = factory
def connectionMade(self):
self.factory.users.append(self)
def connectionLost(self, reason):
self.factory.users.remove(self)
def dataReceived(self, data):
pass
class outputTcpServerFactory(Factory):
def __init__(self, users):
self.users = users
def buildProtocol(self, addr):
return MsgHandler(self)
class Mainloop(object):
def __init__(self):
self.users = []
s = reactor.listenTCP(port, outputTcpServerFactory(self.users))
reactor.run()
---> Elsewhere in the mainLoop class is code that periodically runs:
for client in self.users:
client.sendLine(data)
You should always enable logging so that you can see unhandled exceptions that the application encounters. These usually make it obvious why something is broken.
I am new in Akka actors and I am doing some tests. Suppose I have actors performing long running tasks like following:
override def receive = {
case email: Email => /*Future*/ {
Thread sleep 3000
}
}
I ran a stress test (remote actos on another machine in network) and I receive the following error:
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://EmailSystem#192.168.1.6:5000]
Caused by: akka.remote.transport.AkkaProtocolException: No response from remote. Handshake timed out
How can I configure this to don't get this error again? Should I use a future in the receive method instead of normal code (as on comment above)? What is the impact of doing that?
It's a really bad idea to have an actor that blocks for a long time like that, since it cannot respond to messages and additionally the akka default threadpool is one thread per core of your computer so you might also be stopping other actors from processing any messages.
Fork that blocking job on a separate execution context/thread pool instead (and make sure to limit how many threads there is in that threadpool). You can then notify the actor using pipeTo:
import akka.pattern.pipe
case email: Email =>
val futureEmail = Future {
... send email and then ...
EmailSent()
}
futureEmail pipeTo sender
I need to write a client that continuously polls a web server for commands. A response from the server indicates that a command is available (in which case the response contains the command) or an instruction that no command is available, and you should fire off a new request for incoming commands.
I'm trying to figure out how to do it using spray-client and Akka, and I can think of ways to do it, but none of them look like they're the idiomatic way to get it done. So the question is:
what's the most sensible way to have a couple of threads poll the same web server for incoming commands and hand the commands off to an actor?
This example uses spray-client, scala futures, and Akka scheduler.
Implementation varies depending on desired behavior (execute many requests in parallel at the same time, execute in different intervals, send responses to one actor to process one response at a time, send responses to many actors to process in parallel... etc).
This particular example shows how execute many requests in parallel at the same time, and then do something with each result as it completes, without waiting for any other requests that were fired off at the same time to complete.
The code below will execute two HTTP requests every 5 seconds to 0.0.0.0:9000/helloWorld and 0.0.0.0:9000/goodbyeWorld in parallel.
Tested in Scala 2.10, Spray 1.1-M7, and Akka 2.1.2:
Actual scheduling code that handles periodic job execution:
// Schedule a periodic task to occur every 5 seconds, starting as soon
// as this schedule is registered
system.scheduler.schedule(initialDelay = 0 seconds, interval = 5 seconds) {
val paths = Seq("helloWorld", "goodbyeWorld")
// perform an HTTP request to 0.0.0.0:9000/helloWorld and
// 0.0.0.0:9000/goodbyeWorld
// in parallel (possibly, depending on available cpu and cores)
val retrievedData = Future.traverse(paths) { path =>
val response = fetch(path)
printResponse(response)
response
}
}
Helper methods / boilerplate setup:
// Helper method to fetch the body of an HTTP endpoint as a string
def fetch(path: String): Future[String] = {
pipeline(HttpRequest(method = GET, uri = s"/$path"))
}
// Helper method for printing a future'd string asynchronously
def printResponse(response: Future[String]) {
// Alternatively, do response.onComplete {...}
for (res <- response) {
println(res)
}
}
// Spray client boilerplate
val ioBridge = IOExtension(system).ioBridge()
val httpClient = system.actorOf(Props(new HttpClient(ioBridge)))
// Register a "gateway" to a particular host for HTTP requests
// (0.0.0.0:9000 in this case)
val conduit = system.actorOf(
props = Props(new HttpConduit(httpClient, "0.0.0.0", 9000)),
name = "http-conduit"
)
// Create a simple pipeline to deserialize the request body into a string
val pipeline: HttpRequest => Future[String] = {
sendReceive(conduit) ~> unmarshal[String]
}
Some notes:
Future.traverse is used for running futures in parallel (ignores order). Using a for comprehension on a list of futures will execute one future at a time, waiting for each to complete.
// Executes `oneThing`, executes `andThenAnother` when `oneThing` is complete,
// then executes `finally` when `andThenAnother` completes.
for {
oneThing <- future1
andThenAnother <- future2
finally <- future3
} yield (...)
system will need to be replaced with your actual Akka actor system.
system.scheduler.schedule in this case is executing an arbitrary block of code every 5 seconds -- there is also an overloaded version for scheduling messages to be sent to an actorRef.
system.scheduler.schedule(
initialDelay = 0 seconds,
frequency = 30 minutes,
receiver = rssPoller, // an actorRef
message = "doit" // the message to send to the actorRef
)
For your particular case, printResponse can be replaced with an actor send instead: anActorRef ! response.
The code sample doesn't take into account failures -- a good place to handle failures would be in the printResponse (or equivalent) method, by using a Future onComplete callback: response.onComplete {...}
Perhaps obvious, but spray-client can be replaced with another http client, just replace the fetch method and accompanying spray code.
Update: Full running code example is here:
git clone the repo, checkout the specified commit sha, $ sbt run, navigate to 0.0.0.0:9000, and watch the code in the console where sbt run was executed -- it should print Hello World!\n'Goodbye World! OR Goodbye World!\nHelloWorld! (order is potentially random because of parallel Future.traverse execution).
You can use HTML5 Server-Sent Events. It is implemented in many Scala frameworks. For example in xitrum code looks like:
class SSE extends Controller {
def sse = GET("/sse") {
addConnectionClosedListener {
// The connection has been closed
// Unsubscribe from events, release resources etc.
}
future {
respondEventSource("command1")
//...
respondEventSource("command2")
//...
}
}
SSE is pretty simple and can be used in any software not only in browser.
Akka integrated in xitrum and we use it in similar system. But it uses netty for async server it is also good for processing thousands of request in 10-15 threads.
So in this way your client will keep connection with server and reconnect when connection will be broken.