I am using zeromq PUB/SUB pattern, a simple example is as follow:
def test_pubber(self):
context = zmq.Context()
socket = context.socket(zmq.PUB)
socket.bind(ZMQ_PUB_ADDR)
for i in range(2):
socket.send("10001 end")
time.sleep(1)
so this is ok, but if I do not sleep a while, it doesn't flush the message:
for i in range(2):
socket.send("10001 end")
how to flush the message?
Related
I am trying to create a websocket server that receives input from user, and at the same time read from kafka and continuously send data to user. I am having trouble getting both to run concurrently.
async def sender(websocket):
consumer = KafkaConsumer(...) # read from kafka
for message in consumer:
msg = json.loads(message.value.decode())
await websocket.send(msg)
async def handle_client(websocket):
asyncio.create_task(sender(websocket))
while True:
print("Waiting for client input")
try:
msg = await websocket.recv()
# do something with msg
async def main():
async with websockets.serve(handle_client, "localhost", 8765):
await asyncio.Future() # run forever
if __name__ == "__main__":
asyncio.run(main())
They seem to be blocking. I think the issuer is because sender() blocks it, it never sleeps?
I need to set timeout on python's socket recv method. How to do it?
The typical approach is to use select() to wait until data is available or until the timeout occurs. Only call recv() when data is actually available. To be safe, we also set the socket to non-blocking mode to guarantee that recv() will never block indefinitely. select() can also be used to wait on more than one socket at a time.
import select
mysocket.setblocking(0)
ready = select.select([mysocket], [], [], timeout_in_seconds)
if ready[0]:
data = mysocket.recv(4096)
If you have a lot of open file descriptors, poll() is a more efficient alternative to select().
Another option is to set a timeout for all operations on the socket using socket.settimeout(), but I see that you've explicitly rejected that solution in another answer.
there's socket.settimeout()
As mentioned both select.select() and socket.settimeout() will work.
Note you might need to call settimeout twice for your needs, e.g.
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(("",0))
sock.listen(1)
# accept can throw socket.timeout
sock.settimeout(5.0)
conn, addr = sock.accept()
# recv can throw socket.timeout
conn.settimeout(5.0)
conn.recv(1024)
You could set timeout before receiving the response and after having received the response set it back to None:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(5.0)
data = sock.recv(1024)
sock.settimeout(None)
The timeout that you are looking for is the connection socket's timeout not the primary socket's, if you implement the server side. In other words, there is another timeout for the connection socket object, which is the output of socket.accept() method. Therefore:
sock.listen(1)
connection, client_address = sock.accept()
connection.settimeout(5) # This is the one that affects recv() method.
connection.gettimeout() # This should result 5
sock.gettimeout() # This outputs None when not set previously, if I remember correctly.
If you implement the client side, it would be simple.
sock.connect(server_address)
sock.settimeout(3)
Got a bit confused from the top answers so I've wrote a small gist with examples for better understanding.
Option #1 - socket.settimeout()
Will raise an exception in case the sock.recv() waits for more than the defined timeout.
import socket
sock = socket.create_connection(('neverssl.com', 80))
timeout_seconds = 2
sock.settimeout(timeout_seconds)
sock.send(b'GET / HTTP/1.1\r\nHost: neverssl.com\r\n\r\n')
data = sock.recv(4096)
data = sock.recv(4096) # <- will raise a socket.timeout exception here
Option #2 - select.select()
Waits until data is sent until the timeout is reached. I've tweaked Daniel's answer so it will raise an exception
import select
import socket
def recv_timeout(sock, bytes_to_read, timeout_seconds):
sock.setblocking(0)
ready = select.select([sock], [], [], timeout_seconds)
if ready[0]:
return sock.recv(bytes_to_read)
raise socket.timeout()
sock = socket.create_connection(('neverssl.com', 80))
timeout_seconds = 2
sock.send(b'GET / HTTP/1.1\r\nHost: neverssl.com\r\n\r\n')
data = recv_timeout(sock, 4096, timeout_seconds)
data = recv_timeout(sock, 4096, timeout_seconds) # <- will raise a socket.timeout exception here
You can use socket.settimeout() which accepts a integer argument representing number of seconds. For example, socket.settimeout(1) will set the timeout to 1 second
try this it uses the underlying C.
timeval = struct.pack('ll', 2, 100)
s.setsockopt(socket.SOL_SOCKET, socket.SO_RCVTIMEO, timeval)
As mentioned in previous replies, you can use something like: .settimeout()
For example:
import socket
s = socket.socket()
s.settimeout(1) # Sets the socket to timeout after 1 second of no activity
host, port = "somehost", 4444
s.connect((host, port))
s.send("Hello World!\r\n")
try:
rec = s.recv(100) # try to receive 100 bytes
except socket.timeout: # fail after 1 second of no activity
print("Didn't receive data! [Timeout]")
finally:
s.close()
I hope this helps!!
#! /usr/bin/python3.6
# -*- coding: utf-8 -*-
import socket
import time
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
s.settimeout(5)
PORT = 10801
s.bind(('', PORT))
print('Listening for broadcast at ', s.getsockname())
BUFFER_SIZE = 4096
while True:
try:
data, address = s.recvfrom(BUFFER_SIZE)
except socket.timeout:
print("Didn't receive data! [Timeout 5s]")
continue
Shout out to: https://boltons.readthedocs.io/en/latest/socketutils.html
It provides a buffered socket, this provides a lot of very useful functionality such as:
.recv_until() #recv until occurrence of bytes
.recv_closed() #recv until close
.peek() #peek at buffer but don't pop values
.settimeout() #configure timeout (including recv timeout)
Problem
When I restart/complete/STOP stream the old Consumer does not Die/Shutdown:
[INFO ] a.a.RepointableActorRef -
Message [akka.kafka.KafkaConsumerActor$Internal$Stop$]
from Actor[akka://ufo-sightings/deadLetters]
to Actor[akka://ufo-sightings/system/kafka-consumer-1#1896610594]
was not delivered. [1] dead letters encountered.
Description
I'm building a service that receives a message from Kafka topic and sends the message to an external service via HTTP request.
A connection with the external service can be broken, and my service needs to retry the request.
Additionally, if there is an error in the Stream, entire stream needs to restart.
Finally, sometimes I don't need the stream and its corresponding Kafka-consumer and I would like to shut down the entire stream
So I have a Stream:
Consumer.committableSource(customizedSettings, subscriptions)
.flatMapConcat(sourceFunction)
.toMat(Sink.ignore)
.run
Http request is sent in sourceFunction
I followed new Kafka Consumer Restart instructions in the new documentation
RestartSource.withBackoff(
minBackoff = 20.seconds,
maxBackoff = 5.minutes,
randomFactor = 0.2 ) { () =>
Consumer.committableSource(customizedSettings, subscriptions)
.watchTermination() {
case (consumerControl, streamComplete) =>
logger.info(s" Started Watching Kafka consumer id = ${consumer.id} termination: is shutdown: ${consumerControl.isShutdown}, is f completed: ${streamComplete.isCompleted}")
consumerControl.isShutdown.map(_ => logger.info(s"Shutdown of consumer finally happened id = ${consumer.id} at ${DateTime.now}"))
streamComplete
.flatMap { _ =>
consumerControl.shutdown().map(_ -> logger.info(s"3.consumer id = ${consumer.id} SHUTDOWN at ${DateTime.now} GRACEFULLY:CLOSED FROM UPSTREAM"))
}
.recoverWith {
case _ =>
consumerControl.shutdown().map(_ -> logger.info(s"3.consumer id = ${consumer.id} SHUTDOWN at ${DateTime.now} ERROR:CLOSED FROM UPSTREAM"))
}
}
.flatMapConcat(sourceFunction)
}
.viaMat(KillSwitches.single)(Keep.right)
.toMat(Sink.ignore)(Keep.left)
.run
There is an issue opened that discusses this non-terminating Consumer in a complex Akka-stream, but there is no solution yet.
Is there a workaround that forces the Kafka Consumer termination
How about wrapping the consumer in an Actor and registering a KillSwitch, see: https://doc.akka.io/docs/akka/2.5/stream/stream-dynamic.html#dynamic-stream-handling
Then in the Actor postStop method you can terminate the stream.
By wrapping the Actor in a BackoffSupervisor, you get the exponential backoff.
Example actor: https://github.com/tradecloud/kafka-akka-extension/blob/master/src/main/scala/nl/tradecloud/kafka/KafkaSubscriberActor.scala#L27
I am working with an asyncio.Protocol server where the purpose is for the client to call the server, but wait until the server has responded and data is returned before stopping the client loop.
Based on the asyncio doc Echo Client and Server here: https://docs.python.org/3/library/asyncio-protocol.html#protocol-example-tcp-echo-server-and-client , results of transport.write(...) are returned immediately when called.
Through experience, calling loop.run_until_complete(coroutine) fails with RuntimeError: Event loop is running.
Running asyncio.sleep(n) in the data_received() method of the server doesn't have any effect either.
yield from asyncio.sleep(n) and yield from asyncio.async(asyncio.sleep(n)) in data_received() both hang the server.
My question is, how do I get my client to wait for the server to write a response before giving back control?
I guess to never use transport/protocol pair directly.
asyncio has Streams API for high-level programming.
Client code can look like:
#asyncio.coroutine
def communicate():
reader, writer = yield from asyncio.open_connection(HOST, PORT)
writer.write(b'data')
yield from writer.drain()
answer = yield from reader.read()
# process answer, maybe send new data back to server and wait for answer again
writer.close()
You don't have to change the client code.
echo-client.py
#!/usr/bin/env python3.4
import asyncio
class EchoClient(asyncio.Protocol):
message = 'Client Echo'
def connection_made(self, transport):
transport.write(self.message.encode())
print('data sent: {}'.format(self.message))
def data_received(self, data):
print('data received: {}'.format(data.decode()))
def connection_lost(self, exc):
print('server closed the connection')
asyncio.get_event_loop().stop()
loop = asyncio.get_event_loop()
coro = loop.create_connection(EchoClient, '127.0.0.1', 8888)
loop.run_until_complete(coro)
loop.run_forever()
loop.close()
The trick is to place your code (including self.transport methods) into a coroutine and use the wait_for() method, with the yield from statement in front of the statements that require their values returned, or ones which take a while to complete:
echo-server.py
#!/usr/bin/env python3.4
import asyncio
class EchoServer(asyncio.Protocol):
def connection_made(self, transport):
peername = transport.get_extra_info('peername')
print('connection from {}'.format(peername))
self.transport = transport
def data_received(self, data):
print('data received: {}'.format(data.decode()))
fut = asyncio.async(self.sleeper())
result = asyncio.wait_for(fut, 60)
#asyncio.coroutine
def sleeper(self):
yield from asyncio.sleep(2)
self.transport.write("Hello World".encode())
self.transport.close()
loop = asyncio.get_event_loop()
coro = loop.create_server(EchoServer, '127.0.0.1', 8888)
server = loop.run_until_complete(coro)
print('serving on {}'.format(server.sockets[0].getsockname()))
try:
loop.run_forever()
except KeyboardInterrupt:
print("exit")
finally:
server.close()
loop.close()
Call echo-server.py and then echo-client.py, the client will wait 2 seconds as determined by asyncio.sleep, then stop.
I've been working on a client for this chat server but I am running into a bit of a challenge. The server uses Python's 3.4RC1 asyncio module.
Behavior:
My client connects. My second client connects. Either can send messages to the server BUT, the server is not broadcasting them as it should in a normal public chat room.
User1: Hello. Presses Enter.
User2 does not see it.
User2: Anyone there? Presses Enter.
User2 sees User1: Hello. and User2: Anyone there?
Just... strange. Not sure what I'm missing.
Here are the files. Give it a try.
Server:
from socket import socket, SO_REUSEADDR, SOL_SOCKET
from asyncio import Task, coroutine, get_event_loop
class Peer(object):
def __init__(self, server, sock, name):
self.loop = server.loop
self.name = name
self._sock = sock
self._server = server
Task(self._peer_handler())
def send(self, data):
return self.loop.sock_send(self._sock, data.encode('utf-8'))
#coroutine
def _peer_handler(self):
try:
yield from self._peer_loop()
except IOError:
pass
finally:
self._server.remove(self)
#coroutine
def _peer_loop(self):
while True:
buf = yield from self.loop.sock_recv(self._sock, 1024)
if buf == b'':
break
self._server.broadcast('%s: %s' % (self.name, buf.decode('utf-8')))
class Server(object):
def __init__(self, loop, port):
self.loop = loop
self._serv_sock = socket()
self._serv_sock.setblocking(0)
self._serv_sock.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
self._serv_sock.bind(('',port))
self._serv_sock.listen(5)
self._peers = []
Task(self._server())
def remove(self, peer):
self._peers.remove(peer)
self.broadcast('Peer %s quit!' % (peer.name,))
def broadcast(self, message):
for peer in self._peers:
peer.send(message)
#coroutine
def _server(self):
while True:
peer_sock, peer_name = yield from self.loop.sock_accept(self._serv_sock)
peer_sock.setblocking(0)
peer = Peer(self, peer_sock, peer_name)
self._peers.append(peer)
self.broadcast('Peer %s connected!' % (peer.name,))
def main():
loop = get_event_loop()
Server(loop, 1234)
loop.run_forever()
if __name__ == '__main__':
main()
Client:
# import socket
from socket import *
# form socket import socket, bind, listen, recv, send
HOST = 'localhost' #localhost / 192.168.1.1
# LAN - 192.168.1.1
PORT = 1234
s = socket(AF_INET, SOCK_STREAM)# 98% of all socket programming will use AF_INET and SOCK_STREAM
s.connect((HOST, PORT))
while True:
message = input("Your Message: ")
encoded_msg = message.encode('utf-8')
s.send(encoded_msg)
print('Awaiting Reply..')
reply = s.recv(1024)
decoded_reply = reply.decode('utf-8')
decoded_reply = repr(decoded_reply)
print('Received ', decoded_reply)
s.close()
Here's the non threaded server code I wrote. works great but ONLY between 2 people. How could this code be updated to broadcast every message received to all clients connected?
# import socket
from socket import *
# form socket import socket, bind, listen, recv, send
HOST = 'localhost' #localhost / 192.168.1.1
# LAN - 192.168.1.1
PORT = 1234
s = socket(AF_INET, SOCK_STREAM) # 98% of all socket programming will use AF_INET and SOCK_STREAM
s.bind((HOST, PORT))
s.listen(5) # how many connections it can receive at one time
conn, addr = s.accept() # accept the connection
print('Connected by', addr) # print the address of the person connected
while True:
data = conn.recv(1024)
decoded_data = data.decode('utf-8')
data = repr(decoded_data)
print('Received ', decoded_data)
reply = input("Reply: ")
encoded_reply = reply.encode('utf-8')
conn.sendall(encoded_reply)
print('Server Started')
conn.close()
Okay, let’s think about what your client does. You ask for a message to send, blocking for user input. Then you send that message and receive whatever there is at the server. Afterwards, you block again, waiting for another message.
So when client A sends a text, client B is likely blocking for user input. As such, B won’t actually check if the server sent anything. It will only display what’s there after you have sent something.
Obviously, in a chat, you don’t want to block on user input. You want to continue receiving new messages from the server even if the user isn’t sending messages. So you need to separate those, and run both asynchronously.
I haven’t really done much with asyncio yet, so I don’t really know if this can be nicely done with it, but you essentially just need to put the reading and sending into two separate concurrent tasks, e.g. using threads or concurrent.futures.
A quick example of what you could do, using threading:
from socket import *
from threading import Thread
HOST = 'localhost'
PORT = 1234
s = socket(AF_INET, SOCK_STREAM)
s.connect((HOST, PORT))
def keepReading ():
try:
while True:
reply = s.recv(1024).decode()
print('Received ', reply)
except ConnectionAbortedError:
pass
t = Thread(target=keepReading)
t.start()
try:
while True:
message = input('')
s.send(message.encode())
except EOFError:
pass
finally:
s.close()