I am trying to run my webapp that is based on http4s on jetty. The http4s library provides AsyncHttp4sServlet[IO] class to extend and I use as follows:
class UserSvcServlet
extends AsyncHttp4sServlet[IO](
service = UserSvcServer.start,
servletIo = BlockingServletIo(4096, Blocker.liftExecutorService(Executors.newCachedThreadPool())),
serviceErrorHandler = DefaultServiceErrorHandler
)
As you can see on the service property, I provide my http service that has the following implementation:
def start[F[_]: ConcurrentEffect: ContextShift: Sync: Timer]: HttpApp[F] =
for {
a <- EnvironmentLoader.db.load[F].map(create_transactor)
b <- EnvironmentLoader.cors.load[F].map(origin)
http = Logger.httpApp(true, true)(CORS(UserRoutes(UserProgram(LiveUserQuery.make(a))).routes, b).orNotFound)
} yield http
The start method should return HttpApp[F] but unfortunately the for block returns F[Http[F]]. However at the end the F will be an IO type.
Here is the definition of HttpApp[F]:
type Http[F[_], G[_]] = Kleisli[F, Request[G], Response[G]]
The both EnvironmentLoader.db.load[F].map(create_transactor) and EnvironmentLoader.cors.load[F].map(origin) is in the context of F and they load environment variables. For loading environment variables, I use the library https://cir.is/.
I know, it is not possible to get Http[F] out of the context F. Do I have here to restructure the code to make it work?
Update
One possible workaround would be:
class UserSvcServlet
extends AsyncHttp4sServlet[IO](
service = UserSvcServer.start[IO].unsafeRunSync(),
servletIo = BlockingServletIo(4096, Blocker.liftExecutorService(Executors.newCachedThreadPool())),
serviceErrorHandler = DefaultServiceErrorHandler
)
It is ugly but it works.
Yes, you have to.
When you have F[Http[F]] you are supposed to compose it into your IO program:
for {
dep1 <- createDependency1
dep1 <- createDependency2
http <- createHttp // : F[HttpApp[F]] = F[Request[F] => F[Response[F]]]
// or "use side effects to create a function that
// will take side-effecting request to create
// recipe for side-effecting response" (e.g.
// reading from stream to server and returning
// a stream from server)
dep4 = needHttp(http)
...
} yield ...
In your code a change from F[HttpApp[F]] to HttpApp[F] seems to literally require just changing:
http = ...
into
http <- ...
which kind of suggest, that you are not paying enough attention to what your code does.
I am trying to use aiohttp in one of my projects and struggling to figure out how to create a persistent aiohttp.ClientSession object. I have gone through the official aiohttp documentation but did not find it help in this context.
I have looked through other online forums and noticed that a lot has changed ever since aiohttp was created. In some examples on github, the aiohttp author is shown to be creating a ClientSession outside a coroutine functions (i.e. class Session: def __init__(self): self.session = aiohttp.ClientSession()). I also found that one should not create a ClientSession outside coroutine.
I have tried the following:
class Session:
def __init__(self):
self._session = None
async def create_session(self):
self._session = aiohttp.ClientSession()
async fetch(self, url):
if self._session is None:
await self.create_session()
async with self._session.get(url) as resp:
return await resp.text()
I am getting a lot of warning about UnclosedSession and connector. I also frequently get SSLError. I also noticed that 2 out of three calls gets hung and I have to CTRL+C to kill it.
With requests I can simply initialize the session object in __init__, but it's not as simple as this with aiohttp.
I do not see any issues if I use the following (which is what I see as example all over the place) but unfortunately here I end up creating ClientSession with every request.
def fetch(url):
async with aiohttp.ClientSession() as session:
async with session.get(url) as resp:
return await resp.text()
I can wrap aiohttp.ClientSession() in another function and use that as context-manager, but then too I would end up creating a new session object every time I call the wrapper function. I am trying to figure how to save a aiohttp.ClientSession in class namespace and reuse it.
Any help would be greatly appreciated.
Here is working example:
from aiohttp import ClientSession, TCPConnector
import asyncio
class CS:
_cs: ClientSession
def __init__(self):
self._cs = ClientSession(connector=TCPConnector(verify_ssl=False))
async def get(self, url):
async with self._cs.get(url) as resp:
return await resp.text()
async def close(self):
await self._cs.close()
async def func():
cs = CS()
print(await cs.get('https://google.com'))
await cs.close() # you must close session
loop = asyncio.get_event_loop()
loop.run_until_complete(func())
You can do it.
I implemented a way to share session when writing Django programs (using asgi).Use pid to mark the session of different processes, which is convenient for django to call between different processes.
After actual testing, I can directly call the shared session.
Django 3.2
uvicorn
aiohttp.py
import os
import asyncio
import aiohttp
import logging
session_list = {}
logger = logging.getLogger(__name__)
class Req:
#property
def set_session(self):
try:
loop = asyncio.get_running_loop()
except:
loop = asyncio.get_event_loop()
asyncio.set_event_loop(loop)
session = aiohttp.ClientSession(loop=loop)
session_list.update({os.getpid(): session})
return session
def __init__(self):
if session_list.get(os.getpid()):
self.session = session_list.get(os.getpid())
else:
self.session = self.set_session
async def test(self):
if session_list:
session = session_list.get(os.getpid())
if session and session.closed:
session_list.pop(os.getpid())
session = self.set_session
else:
session = self.set_session
if not session or session.loop.is_running():
session = self.set_session
logger.warning("session abnormal")
result = await session.get("http://httpbing.org/get")
print(result.status)
req = Req()
views.py
from django.http import HttpResponse
from django.shortcuts import render # noqa
from django.views.generic import View
from django.utils.decorators import classonlymethod
import asyncio
class TTT(View):
#classonlymethod
def as_view(cls, **initkwargs):
view = super().as_view(**initkwargs)
view._is_coroutine = asyncio.coroutines._is_coroutine
return view
async def get(self, request):
await req.test()
return HttpResponse("ok")
I am making a Play web-socket app. When a client connects, I want to send a welcome message.
The code I use is below:
package controllers
import play.api._
import play.api.mvc._
import play.api.libs.iteratee.Concurrent
import play.api.libs.iteratee.Iteratee
import play.api.libs.concurrent.Execution.Implicits.defaultContext
object Test extends Controller {
def index = WebSocket.using[String] { _ =>
val (out,channel) = Concurrent.broadcast[String]
channel.push("Welcome to MyWebSocket")
val in = Iteratee.foreach[String] {
_ match {
case any => channel.push(any)
}
}
(in, out)
}
}
The code works fine when a client sends a message and the server has to respond to it. However, the initial welcome message Welcome to MyWebSocket is not sent. How can I fix this code?
[EDIT]
I kind of figured out the problem, but not a solution yet. The problem probably occurs because the websocket is not yet initialized when the welcome message is being pushed. I modified the code and replaced:
channel.push("Welcome to MyWebSocket")
with
val a = scala.concurrent.Future {
Thread.sleep(1000)
channel.push("Welcome to MyWebSocket")
}
After this I get the expected results (welcome message received by client). I think using the above approach (Thread.sleep and Future) is not the right way to solve this problem, so other solutions are welcome. It could also be a problem with the client side code which takes a while to initialize the socket. I used Firefox and echo web-socket test for the client.
You can use WebSocket.acceptWithActor helper method (have a look at this) and in actor body make something like
out ! "Welcome to MyWebSocket"
It works nicely.
I need to make some consuming calculations on the server side (such as DB querying and data analisys). And the results need to be printed in browser. For these purpose I send Future result from server to client (to load web page immediately and gradually print future results from server). For example, on the server side
import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global
def futureResult = Future {
val cc = ConsumingCalculations();
"some result"
}
on the client side
#import scala.concurrent.ExecutionContext.Implicits.global
#main{
#futureResult.onSuccess{ case res =>
#println("This line is printed in console: "+res);
<div>Any html code is NOT printed in browser</div>
}
Future result is NOT posted
}
In server consol we have: "This line is printed in console: some result"
But in the browser we have only: "Future result is NOT posted"
Play 2.1, scala 2.10 are currently used. What's may be wrong, are there any idea?
A future cannot be sent on client side, it must be resolved on server side before displaying to the client.
The classic exemple is to map the result of your future in your controller
def myAction = Action {
Async {
futureResult.map(result =>
Ok(views.html.myView(result))
)
}
}
And in your template, use the result, not the future.
In gtk applications all execution is taking place inside the gtk_main function. And other graphical frame works have similar event loops like app.exec for QT and clutter_main for Clutter. However ZeroMQ is based on the assumption that there is an while (1) ... loop that it is inserted into (see for instance here for examples).
How do you combine those two execution strategies?
I am currently wanting to use zeromq in a clutter application written in C, so I would of course like direct answers to that, but please add answers for other variants as well.
The proper way to combine zmq and gtk or clutter is to connect the file-descriptor of the zmq queue to the main event loop. The fd can be retrieved by using
int fd;
size_t sizeof_fd = sizeof(fd);
if(zmq_getsockopt(socket, ZMQ_FD, &fd, &sizeof_fd))
perror("retrieving zmq fd");
Connecting it to the main loop is the matter of using io_add_watch:
GIOChannel* channel = g_io_channel_unix_new(fd);
g_io_add_watch(channel, G_IO_IN|G_IO_ERR|G_IO_HUP, callback_func, NULL);
In the callback function, it is necessary to first check if there is really stuff to read, before reading. Otherwise, the function might block waiting for IO.
gboolean callback_func(GIOChannel *source, GIOCondition condition,gpointer data)
{
uint32_t status;
size_t sizeof_status = sizeof(status);
while (1){
if (zmq_getsockopt(socket, ZMQ_EVENTS, &status, &sizeof_status)) {
perror("retrieving event status");
return 0; // this just removes the callback, but probably
// different error handling should be implemented
}
if (status & ZMQ_POLLIN == 0) {
break;
}
// retrieve one message here
}
return 1; // keep the callback active
}
Please note: this is not actually tested, I did a translation from Python+Clutter, which is what I use, but I'm pretty sure that it'll work.
For reference, below is full Python+Clutter code which actually works.
import sys
from gi.repository import Clutter, GObject
import zmq
def Stage():
"A Stage with a red spinning rectangle"
stage = Clutter.Stage()
stage.set_size(400, 400)
rect = Clutter.Rectangle()
color = Clutter.Color()
color.from_string('red')
rect.set_color(color)
rect.set_size(100, 100)
rect.set_position(150, 150)
timeline = Clutter.Timeline.new(3000)
timeline.set_loop(True)
alpha = Clutter.Alpha.new_full(timeline, Clutter.AnimationMode.EASE_IN_OUT_SINE)
rotate_behaviour = Clutter.BehaviourRotate.new(
alpha,
Clutter.RotateAxis.Z_AXIS,
Clutter.RotateDirection.CW,
0.0, 359.0)
rotate_behaviour.apply(rect)
timeline.start()
stage.add_actor(rect)
stage.show_all()
stage.connect('destroy', lambda stage: Clutter.main_quit())
return stage, rotate_behaviour
def Socket(address):
ctx = zmq.Context()
sock = ctx.socket(zmq.SUB)
sock.setsockopt(zmq.SUBSCRIBE, "")
sock.connect(address)
return sock
def zmq_callback(queue, condition, sock):
print 'zmq_callback', queue, condition, sock
while sock.getsockopt(zmq.EVENTS) & zmq.POLLIN:
observed = sock.recv()
print observed
return True
def main():
res, args = Clutter.init(sys.argv)
if res != Clutter.InitError.SUCCESS:
return 1
stage, rotate_behaviour = Stage()
sock = Socket(sys.argv[2])
zmq_fd = sock.getsockopt(zmq.FD)
GObject.io_add_watch(zmq_fd,
GObject.IO_IN|GObject.IO_ERR|GObject.IO_HUP,
zmq_callback, sock)
return Clutter.main()
if __name__ == '__main__':
sys.exit(main())
It sounds like the ZeroMQ code wants simply to be executed over and over again as often as possible. The simplest way is to put the ZeroMQ code into an idle function or timeout function, and use non-blocking versions of the functions if they exist.
For Clutter, you would use clutter_threads_add_idle() or clutter_threads_add_timeout(). For GTK, you would use g_idle_add() or g_timeout_add().
The more difficult, but possibly better, way is to create a separate thread for the ZeroMQ code using g_thread_create(), and just use the while(1) construction with blocking functions as they suggest. If you do that, you will also have to find some way for the threads to communicate with each other - GLib's mutexes and async queues usually do fine.
I found that there is a QT integration library called Zeromqt. Looking at the source, the core of the integration is the following:
ZmqSocket::ZmqSocket(int type, QObject *parent) : QObject(parent)
{
...
notifier_ = new QSocketNotifier(fd, QSocketNotifier::Read, this);
connect(notifier_, SIGNAL(activated(int)), this, SLOT(activity()));
}
...
void ZmqSocket::activity()
{
uint32_t flags;
size_t size = sizeof(flags);
if(!getOpt(ZMQ_EVENTS, &flags, &size)) {
qWarning("Error reading ZMQ_EVENTS in ZMQSocket::activity");
return;
}
if(flags & ZMQ_POLLIN) {
emit readyRead();
}
if(flags & ZMQ_POLLOUT) {
emit readyWrite();
}
...
}
Hence, it is relying on QT's integrated socket handling and Clutter will not have something similar.
You can get a file descriptor for 0MQ socket (ZMQ_FD option) and integrate that with your event loop. I presume gtk has some mechanism for handling sockets.
This an example in Python, using the PyQt4. It's derived from a working application.
import zmq
from PyQt4 import QtCore, QtGui
class QZmqSocketNotifier( QtCore.QSocketNotifier ):
""" Provides Qt event notifier for ZMQ socket events """
def __init__( self, zmq_sock, event_type, parent=None ):
"""
Parameters:
----------
zmq_sock : zmq.Socket
The ZMQ socket to listen on. Must already be connected or bound to a socket address.
event_type : QtSocketNotifier.Type
Event type to listen for, as described in documentation for QtSocketNotifier.
"""
super( QZmqSocketNotifier, self ).__init__( zmq_sock.getsockopt(zmq.FD), event_type, parent )
class Server(QtGui.QFrame):
def __init__(self, topics, port, mainwindow, parent=None):
super(Server, self).__init__(parent)
self._PORT = port
# Create notifier to handle ZMQ socket events coming from client
self._zmq_context = zmq.Context()
self._zmq_sock = self._zmq_context.socket( zmq.SUB )
self._zmq_sock.bind( "tcp://*:" + self._PORT )
for topic in topics:
self._zmq_sock.setsockopt( zmq.SUBSCRIBE, topic )
self._zmq_notifier = QZmqSocketNotifier( self._zmq_sock, QtCore.QSocketNotifier.Read )
# connect signals and slots
self._zmq_notifier.activated.connect( self._onZmqMsgRecv )
mainwindow.quit.connect( self._onQuit )
#QtCore.pyqtSlot()
def _onZmqMsgRecv():
self._test_info_notifier.setEnabled(False)
# Verify that there's data in the stream
sock_status = self._zmq_sock.getsockopt( zmq.EVENTS )
if sock_status == zmq.POLLIN:
msg = self._zmq_sock.recv_multipart()
topic = msg[0]
callback = self._topic_map[ topic ]
callback( msg )
self._zmq_notifier.setEnabled(True)
self._zmq_sock.getsockopt(zmq.EVENTS)
def _onQuit(self):
self._zmq_notifier.activated.disconnect( self._onZmqMsgRecv )
self._zmq_notifier.setEnabled(False)
del self._zmq_notifier
self._zmq_context.destroy(0)
Disabling and then re-enabling the notifier in _on_ZmqMsgRecv is per the documentation for QSocketNotifier.
The final call to getsockopt is for some reason necessary. Otherwise, the notifier stops working after the first event. I was actually going to post a new question for this. Does anyone know why this is needed?
Note that if you don't destroy the notifier before the ZMQ context, you'll probably get an error like this when you quit the application:
QSocketNotifier: Invalid socket 16 and type 'Read', disabling...