I have an application that will be streaming data in from an IRC server in Python. Currently, I'm creating a socket object in Python that connects to the IRC server. Before I can receive the data, I send the socket authentication initially and may have to periodically send other messages throughout the streaming process. The problem is, I want to stream the messages in with Spark.
I've read that you can create CustomReceivers to do something like this, but I can't find any information on creating one in Python.
Now, I have two possible solutions that I could implement, both of which I feel are sub-optimal.
1) Create a Python socket that connects to IRC and parse/save the data directly without using Spark.
2) Create a Python socket that connects to IRC, send that data to another local Python socket, and have Spark connect directly to that socket and stream from there.
Am I trying to solve the wrong problem with the wrong tool or is there a way to send data to a socket in Spark, namely, in Python?
Related
I am trying to find a way to connect Google pubsub directly over a socket connection written in C language. Is their a way to do that? I will be publishing JSON and/or Binary messages over the socket. Is their a specific message format that needs to be followed.
I am using the gorilla websockets package to make a server to handle web-socket connections from clients. I am using the sockets to send a json array of ints. I was wondering if there is a way to verify if all the data got transferred to the client without corruption. Any help would be great, Thanks
We are in the process in moving away from a python based environment to scala play framework one and I have to write a buffered publisher subscriber which will interface with another system written in python. The goal here is to write a publisher which would take messages from a buffer/queue and send them to a server over a tcp socket and keep the socket opened so we can reuse it later.
Our old python publisher created a socket.socket(socket.AF_INET, socket.SOCK_STREAM) socket which would never close, and worked with a scheduler to take messages off the queue and send them over the streaming socket.
I'm interested in using akka streaming io, but I don't know how to create a streaming socket that works with a scheduler periodically looking at a queue and sending message to a server.
If anyone has a code sample that could illustrate how this could be done in scala/play framework/akka streaming io that would be awesome.
Thanking you in advance
Francis
It sounds like you want Akka TCP:
http://doc.akka.io/docs/akka/current/scala/io-tcp.html
currently I have a server and a client console application in c++ using winsock. In my situation the client is able to send a message to the server, which listens and it can aswer, but I need to make them to be able to always recieve and send messages to one another.
My problem is I can not use threads, I need to achieve this goal without them. How can this be done by sockets and loops? Any hint is welcome.
Thank you
If you cannot use threads, you have to do send and recv in the same loop. Rather than doing it blindly it's better to use select function to check whether is there any incoming massages from any of the sockets or is it possible to write (send) data in to the socket. This is a reference to study about select function.
I'm trying to create an app that is able to send a .pdf-file directly to a printer from an iPhone. I'd like to create a raw socket connection. I should be able to open a stream to a specific IP-adress and a port. Then I'd like to throw that .PDF-file into the stream so the printer (or my server on the computer) receives it.
I've made a stream already using ftp. Ofcourse, printers don't handle ftp-protocols. That is why I want to send the data in a raw stream to the device's port.
Any idea how I can accomplish this?
CocoaAsyncSocket is really easy to use and has good documentation. It is an objective-c wrapper around lower level socket primitives. It sounds like all you need is to write data out to a socket. If so then it is the easiest way to go.