I'm writing a Google pubsub client using the Java API, the client is written in Scala. The problem of this code is that it's not idiomatic in scala with the use of null and the while true loop
val receiver = new MessageReceiver() {
// React to each received message
// If there are any
override def receiveMessage(message: PubsubMessage, consumer: AckReplyConsumer): Unit = { // handle incoming message, then ack/nack the received message
System.out.println("Id : " + message.getMessageId)
System.out.println("Data : " + message.getData.toStringUtf8)
throw new RuntimeException("This is just an exception")
consumer.ack()
}
}
var subscriber: ApiService = null
try { // Create a subscriber for "my-subscription-id" bound to the message receiver
var subscriber = Subscriber.newBuilder(subscriptionName, receiver).build
subscriber.startAsync
// ...
} finally {
// stop receiving messages
if (subscriber != null) subscriber.stopAsync()
}
while (true) {
Thread.sleep(1000)
}
How do I tranform this code in order to use scala's Future or cats IO
Have you considered using Lightbend's Alpakka Google Cloud Pub/Sub connector?
https://developer.lightbend.com/docs/alpakka/current/google-cloud-pub-sub.html
It works very well and is idiomatic
There is a Pub/Sub Scala client in cats style - https://github.com/hyjay/fs2-google-cloud-pubsub
Disclaimer: I'm the author.
Related
I am new to Akka Streams and gRPC, I am trying to build an endpoint where client sends a single request and the server sends multiple responses.
This is my protobuf
syntax = "proto3";
option java_multiple_files = true;
option java_package = "customer.service.proto";
service CustomerService {
rpc CreateCustomer(CustomerRequest) returns (stream CustomerResponse) {}
}
message CustomerRequest {
string customerId = 1;
string customerName = 2;
}
message CustomerResponse {
enum Status {
No_Customer = 0;
Creating_Customer = 1;
Customer_Created = 2;
}
string customerId = 1;
Status status = 2;
}
I am trying to achieve this by sending customer request then the server will first check and respond No_Customer then it will send Creating_Customer and finally server will say Customer_Created.
I have no idea where to start for it implementation, looked for hours but still clueless, I will be very thankful if anyone can point me in the right direction.
The place to start is the Akka gRPC documentation and, in particular, the service WalkThrough. It is pretty straightforward to get the samples working in a clean project.
The relevant server sample method is this:
override def itKeepsReplying(in: HelloRequest): Source[HelloReply, NotUsed] = {
println(s"sayHello to ${in.name} with stream of chars...")
Source(s"Hello, ${in.name}".toList).map(character => HelloReply(character.toString))
}
The problem is now to create a Source that returns the right results, but that depends on how you are planning to implement the server so it is difficult to answer. Check the Akka Streams documentation for various options.
The client code is simpler, just call runForeach on the Source that gets returned by CreateCustomer as in the sample:
def runStreamingReplyExample(): Unit = {
val responseStream = client.itKeepsReplying(HelloRequest("Alice"))
val done: Future[Done] =
responseStream.runForeach(reply => println(s"got streaming reply: ${reply.message}"))
done.onComplete {
case Success(_) =>
println("streamingReply done")
case Failure(e) =>
println(s"Error streamingReply: $e")
}
}
I'm using MassTransit with Reactive Extensions to stream messages from the queue in batches. Since the behaviour isn't the same as a normal consumer I need to be able to send a message to the error queue if it fails an x number of times.
I've looked through the MassTransit source code and posted on the google groups and can't find an anwser.
Is this available on the ConsumeContext interface? Or is this even possible?
Here is my code. I've removed some of it to make it simpler.
_busControl = Bus.Factory.CreateUsingRabbitMq(cfg =>
{
var host = cfg.Host(new Uri("rabbitmq://localhost/"), h =>
{
h.Username("guest");
h.Password("guest");
});
cfg.UseInMemoryScheduler();
cfg.ReceiveEndpoint(host, "customer_update_queue", e =>
{
var _observer = new ObservableObserver<ConsumeContext<Customer>>();
_observer.Buffer(TimeSpan.FromMilliseconds(1000)).Subscribe(OnNext);
e.Observer(_observer);
});
});
private void OnNext(IList<ConsumeContext<Customer>> messages)
{
foreach (var consumeContext in messages)
{
Console.WriteLine("Content: " + consumeContext.Message.Content);
if (consumeContext.Message.RetryCount > 3)
{
// I want to be able to send to the error queue
consumeContext.SendToErrorQueue()
}
}
}
I've found a work around by using the RabbitMQ client mixed with MassTransit. Since I can't throw an exception when using an Observable and therefore no error queue is created. I create it manually using the RabbitMQ client like below.
ConnectionFactory factory = new ConnectionFactory();
factory.HostName = "localhost";
factory.UserName = "guest";
factory.Password = "guest";
using (IConnection connection = factory.CreateConnection())
{
using (IModel model = connection.CreateModel())
{
string exchangeName = "customer_update_queue_error";
string queueName = "customer_update_queue_error";
string routingKey = "";
model.ExchangeDeclare(exchangeName, ExchangeType.Fanout);
model.QueueDeclare(queueName, false, false, false, null);
model.QueueBind(queueName, exchangeName, routingKey);
}
}
The send part is to send it directly to the message queue if it fails an x amount of times like so.
consumeContext.Send(new Uri("rabbitmq://localhost/customer_update_queue_error"), consumeContext.Message);
Hopefully the batch feature will be implemented soon and I can use that instead.
https://github.com/MassTransit/MassTransit/issues/800
I want to implement an client app that first send an request to server then wait for its reply(similar to http)
My client process may be
val topic = async.topic[ByteVector]
val client = topic.subscribe
Here is the api
trait Client {
val incoming = tcp.connect(...)(client)
val reqBus = topic.pubsh()
def ask(req: ByteVector): Task[Throwable \/ ByteVector] = {
(tcp.writes(req).flatMap(_ => tcp.reads(1024))).to(reqBus)
???
}
}
Then, how to implement the remain part of ask ?
Usually, the implementation is done with publishing the message via sink and then awaiting some sort of reply on some source, like your topic.
Actually we have a lot of idioms of this in our code :
def reqRply[I,O,O2](src:Process[Task,I],sink:Sink[Task,I],reply:Process[Task,O])(pf: PartialFunction[O,O2]):Process[Task,O2] = {
merge.mergeN(Process(reply, (src to sink).drain)).collectFirst(pf)
}
Essentially this first hooks to reply stream to await any resulting O confirming our request sent. Then we publish message I and consult pf for any incoming O to be eventually translated to O2 and then terminate.
I create a thrift server using Finagle like this
val server = Thrift.serveIface(bindAddr(), new MyService[Future] {
def myRPCFuction() {}
})
But, I found that the maximum number of concurrent requests is five( why 5? when more than 5, the server just ignore the excessed ones.) I look through the doc of Finagle really hard (http://twitter.github.io/finagle/guide/Protocols.html#thrift-and-scrooge), but find nothing hint to configure the max-request-limit.
How to config the maximum concurrent request num of Finagle? Thanks
I've solved this problem by myself and I share it here to help others who may run into the same case. Because I m a thrift user before and in Thrift when you return from the RPC function you return the values back to calling client. While in Finagle only when you use Future.value() you return the value to client. And when use Finagle, you should totally use the asynchronous way, that's to say you had better not sleep or do some other RPC synchronously in the RPC function.
/* THIS is BAD */
val server = Thrift.serveIface(bindAddr(), new MyService[Future] {
def myRPCFuction() {
val rpcFuture = rpcClient.callOtherRpc() // call other rpc which return a future
val result = Await.result(rpcFuture, TwitterDuration(rpcTimeoutSec()*1000, MILLISECONDS))
Future.value(result)
}
})
/* This is GOOD */
val server = Thrift.serveIface(bindAddr(), new MyService[Future] {
def myRPCFuction() {
val rpcFuture = rpcClient.callOtherRpc() // call other rpc which return a future
rpcFuture onSuccess { // do you job when success (you can return to client using Future.value) }
rpcFuture onFailure { // do your job when fail }
}
})
Then, can get a satisfactory concurrency. Hope it helps others who have the same issue.
my app lists hosts, and the list is dynamic and changing. it is based on Akka actors and Server Sent Events.
when a new client connects, they need to get the current list to display. but, i don't want to push the list to all clients every time a new one connects. so, followed the realtime elastic search example and emulated unicast by creating an (Enumerator, Channel) per Connect() and giving it an UUID. when i need to broadcast i will map over all and update them, with the intent of being able to do unicast to clients (and there should be very few of those).
my problem is - how do i get the new client its UUID so it can use it? the flow i am looking for is:
- client asks for EventStream
- server creates a new (Enumerator, channel) with a UUID, and returns Enumerator and UUID to client
- client asks for table using uuid
- server pushes table only on channel corresponding to the uuid
so, how would the client know about the UUID? had it been web socket, sending the request should have had the desired result, as it would have reached its own channel. but in SSE the client -> server is done on a different channel. any solutions to that?
code snippets:
case class Connected(uuid: UUID, enumerator: Enumerator[ JsValue ] )
trait MyActor extends Actor{
var channelMap = new HashMap[UUID,(Enumerator[JsValue], Channel[JsValue])]
def connect() = {
val con = Concurrent.broadcast[JsValue]
val uuid = UUID.randomUUID()
channelMap += (uuid -> con)
Connected(uuid, con._1)
}
...
}
object HostsActor extends MyActor {
...
override def receive = {
case Connect => {
sender ! connect
}
...
}
object Actors {
def hostsStream = {
getStream(getActor("hosts", Props (HostsActor)))
}
def getActor(actorPath: String, actorProps : Props): Future[ActorRef] = {
/* some regular code to create a new actor if the path does not exist, or return the existing one else */
}
def getStream(far: Future[ActorRef]) = {
far flatMap {ar =>
(ar ? Connect).mapTo[Connected].map { stream =>
stream
}
}
}
...
}
object AppController extends Controller {
def getHostsStream = Action.async {
Actors.hostsStream map { ac =>
************************************
** how do i use the UUID here?? **
************************************
Ok.feed(ac.enumerator &> EventSource()).as("text/event-stream")
}
}
I managed to solve it by asynchronously pushing the uuid after returning the channel, with some time in between:
override def receive = {
case Connect => {
val con = connect()
sender ! con
import scala.concurrent.ExecutionContext.Implicits.global
context.system.scheduler.scheduleOnce(0.1 seconds){
unicast(
con.uuid,
JsObject (
Seq (
"uuid" -> JsString(con.uuid.toString)
)
)
)
}
}
this achieved its goal - the client got the UUID and was able to cache and use it to push a getHostsList to the server:
#stream = new EventSource("/streams/hosts")
#stream.addEventListener "message", (event) =>
data = JSON.parse(event.data)
if data.uuid
#uuid = data.uuid
$.ajax
type: 'POST',
url: "/streams/hosts/" + #uuid + "/sendlist"
success: (data) ->
console.log("sent hosts request to server successfully")
error: () ->
console.log("failed sending hosts request to server")
else
****************************
* *
* handle parsing hosts *
* *
* *
****************************
#view.render()
while this works, i must say i don't like it. introducing an artificial delay so the client can get the channel and start listening (i tried with no delay, and the client didn't get the uuid) is dangerous, as it might still miss if the system get busier, but making it too long hurts the reactivity aspect.
if anyone has a solution in which this can be done synchronically - having the uuid returned as part of the original eventSource request - i would be more than happy to demote my solution.