Flink can read a socket stream, can it read http requests? how?
// socket example
DataStream<XXX> socketStream = env
.socketTextStream("localhost", 9999)
.map(...);
There's an open JIRA ticket for creating an HTTP sink connector for Flink, but I've seen no discussion about creating a source connector.
Moreover, it's not clear this is a good idea. Flink's approach to fault tolerance requires sources that can be rewound and replayed, so it works best with input sources that behave like message queues. I would suggest buffering the incoming http requests in a distributed log.
For an example, look at how DriveTribe uses Flink to power their website on the data Artisans blog and on YouTube.
I write one custom http source. please ref OneHourHttpTextStreamFunction. you need create a fat jar to include apache httpserver classes if you want run my code.
package org.apache.flink.streaming.examples.http;
import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.common.functions.ReduceFunction;
import org.apache.flink.api.java.utils.ParameterTool;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.source.SourceFunction;
import org.apache.flink.streaming.api.windowing.time.Time;
import org.apache.flink.streaming.examples.socket.SocketWindowWordCount.WordWithCount;
import org.apache.flink.util.Collector;
import org.apache.http.HttpException;
import org.apache.http.HttpRequest;
import org.apache.http.HttpResponse;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.bootstrap.HttpServer;
import org.apache.http.impl.bootstrap.ServerBootstrap;
import org.apache.http.protocol.HttpContext;
import org.apache.http.protocol.HttpRequestHandler;
import java.io.IOException;
import java.util.concurrent.TimeUnit;
import static org.apache.flink.util.Preconditions.checkArgument;
import static org.apache.flink.util.Preconditions.checkNotNull;
public class HttpRequestCount {
public static void main(String[] args) throws Exception {
// the host and the port to connect to
final String path;
final int port;
try {
final ParameterTool params = ParameterTool.fromArgs(args);
path = params.has("path") ? params.get("path") : "*";
port = params.getInt("port");
} catch (Exception e) {
System.err.println("No port specified. Please run 'SocketWindowWordCount "
+ "--path <hostname> --port <port>', where path (* by default) "
+ "and port is the address of the text server");
System.err.println("To start a simple text server, run 'netcat -l <port>' and "
+ "type the input text into the command line");
return;
}
// get the execution environment
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
// get input data by connecting to the socket
DataStream<String> text = env.addSource(new OneHourHttpTextStreamFunction(path, port));
// parse the data, group it, window it, and aggregate the counts
DataStream<WordWithCount> windowCounts = text
.flatMap(new FlatMapFunction<String, WordWithCount>() {
#Override
public void flatMap(String value, Collector<WordWithCount> out) {
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
for (String word : value.split("\\s")) {
out.collect(new WordWithCount(word, 1L));
}
}
})
.keyBy("word").timeWindow(Time.seconds(5))
.reduce(new ReduceFunction<WordWithCount>() {
#Override
public WordWithCount reduce(WordWithCount a, WordWithCount b) {
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return new WordWithCount(a.word, a.count + b.count);
}
});
// print the results with a single thread, rather than in parallel
windowCounts.print().setParallelism(1);
env.execute("Http Request Count");
}
}
class OneHourHttpTextStreamFunction implements SourceFunction<String> {
private static final long serialVersionUID = 1L;
private final String path;
private final int port;
private transient HttpServer server;
public OneHourHttpTextStreamFunction(String path, int port) {
checkArgument(port > 0 && port < 65536, "port is out of range");
this.path = checkNotNull(path, "path must not be null");
this.port = port;
}
#Override
public void run(SourceContext<String> ctx) throws Exception {
server = ServerBootstrap.bootstrap().setListenerPort(port).registerHandler(path, new HttpRequestHandler(){
#Override
public void handle(HttpRequest req, HttpResponse rep, HttpContext context) throws HttpException, IOException {
ctx.collect(req.getRequestLine().getUri());
rep.setStatusCode(200);
rep.setEntity(new StringEntity("OK"));
}
}).create();
server.start();
server.awaitTermination(1, TimeUnit.HOURS);
}
#Override
public void cancel() {
server.stop();
}
}
Leave you comment, if you want the demo jar.
Related
I'm writing a simple tcp server for short connection.
I have two vertical.
One handling the tcp request, get the eventBus and send the message to it.
Another vertical just consume and reply it.
The question is.Tcp client can't receive the result.I know the eventBus.request() is async.But don't know how to control it,just let the socket waiting for the result.
main
package org.example;
import io.vertx.core.DeploymentOptions;
import io.vertx.core.Vertx;
import io.vertx.core.VertxOptions;
public class Application {
public static void main(String[] args) {
VertxOptions options = new VertxOptions();
options.setEventLoopPoolSize(1);
Vertx vertx = Vertx.vertx(options);
DeploymentOptions ruleOptions = new DeploymentOptions();
ruleOptions.setWorker(true);
ruleOptions.setInstances(1);
vertx.deployVerticle("org.example.RuleVertical", ruleOptions);
DeploymentOptions serverOptions = new DeploymentOptions();
serverOptions.setInstances(1);
vertx.deployVerticle("org.example.ServerVertical", serverOptions);
}
}
server vertical
package org.example;
import io.vertx.core.AbstractVerticle;
import io.vertx.core.eventbus.EventBus;
import io.vertx.core.net.NetServer;
public class ServerVertical extends AbstractVerticle {
#Override
public void start() throws Exception {
EventBus eventBus = vertx.eventBus();
NetServer netServer = vertx.createNetServer();
netServer.connectHandler(socket -> {
socket.handler(buffer -> {
eventBus.request("message.id", buffer.toString(), reply -> {
System.out.println("Server reply: " + reply.result().body());
socket.write(reply.result().body().toString()).onFailure(event -> event.printStackTrace());
});
});
});
netServer.listen(1234);
}
}
rule vertical
package org.example;
import io.vertx.core.AbstractVerticle;
import io.vertx.core.eventbus.EventBus;
public class RuleVertical extends AbstractVerticle {
#Override
public void start() throws Exception {
EventBus eventBus = vertx.eventBus();
eventBus.consumer("message.id", message -> {
try {
Thread.sleep(2000);
message.reply("hello");
} catch (InterruptedException e) {
e.printStackTrace();
}
});
}
}
client
package org.example;
import java.io.*;
import java.net.Socket;
public class Client {
public static void main(String[] args) throws IOException {
Socket socket = new Socket("127.0.0.1", 1234);
OutputStream outputStream = socket.getOutputStream();
PrintWriter printWriter = new PrintWriter(outputStream);
System.out.println("Request: 1234");
printWriter.write("1234");
printWriter.flush();
socket.shutdownOutput();
InputStream inputStream = socket.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream));
String line = null;
StringBuilder sb = new StringBuilder();
while ((line = reader.readLine()) != null) {
sb.append(line);
}
System.out.println("Response: " + sb);
reader.close();
printWriter.close();
inputStream.close();
outputStream.close();
socket.close();
}
}
Server log
Server reply: hello
io.netty.channel.StacklessClosedChannelException
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(Object, ChannelPromise)(Unknown Source)
Client log
Request: 1234
Response:
I've just started looking at netty for some projects and have been able to get some simple client and server examples running that use INET and unix domain sockets to send messages back and forth. I've also been able to send datagram packets over INET sockets. But I have a need to send datagram packets over UNIX domain sockets. Is this supported in netty? If so, could someone point me at documentation or an example? I suspect this is not supported given that the DatagramPacket explicitly takes InetSocketAddress. If not supported, would it be feasible to add this to netty?
Is this supported in netty?
Yes. Below is a simple example I wrote.
import io.netty.bootstrap.Bootstrap;
import io.netty.bootstrap.ServerBootstrap;
import io.netty.buffer.ByteBuf;
import io.netty.channel.*;
import io.netty.channel.epoll.EpollDomainSocketChannel;
import io.netty.channel.epoll.EpollEventLoopGroup;
import io.netty.channel.epoll.EpollServerDomainSocketChannel;
import io.netty.channel.unix.DomainSocketAddress;
/**
* #author louyl
*/
public class App {
public static void main(String[] args) throws Exception {
String sockPath = "/tmp/echo.sock";
final ServerBootstrap bootstrap = new ServerBootstrap();
EventLoopGroup serverBossEventLoopGroup = new EpollEventLoopGroup();
EventLoopGroup serverWorkerEventLoopGroup = new EpollEventLoopGroup();
bootstrap.group(serverBossEventLoopGroup, serverWorkerEventLoopGroup)
.localAddress(new DomainSocketAddress(sockPath))
.channel(EpollServerDomainSocketChannel.class)
.childHandler(
new ChannelInitializer<Channel>() {
#Override
protected void initChannel(final Channel channel) throws Exception {
channel.pipeline().addLast(
new ChannelInboundHandlerAdapter() {
#Override
public void channelActive(final ChannelHandlerContext ctx) throws Exception {
final ByteBuf buff = ctx.alloc().buffer();
buff.writeBytes("This is a test".getBytes());
ctx.writeAndFlush(buff).addListeners(new ChannelFutureListener() {
#Override
public void operationComplete(ChannelFuture future) {
future.channel().close();
future.channel().parent().close();
}
});
}
}
);
}
}
);
final ChannelFuture serverFuture = bootstrap.bind().sync();
final Bootstrap bootstrapClient = new Bootstrap();
EventLoopGroup clientEventLoop = new EpollEventLoopGroup();
bootstrapClient.group(clientEventLoop)
.channel(EpollDomainSocketChannel.class)
.handler(new ChannelInitializer<Channel>() {
#Override
protected void initChannel(final Channel channel) throws Exception {
channel.pipeline().addLast(
new ChannelInboundHandlerAdapter() {
#Override
public void channelRead(final ChannelHandlerContext ctx, final Object msg) throws Exception {
final ByteBuf buff = (ByteBuf) msg;
try {
byte[] bytes = new byte[buff.readableBytes()];
buff.getBytes(0, bytes);
System.out.println(new String(bytes));
} finally {
buff.release();
}
ctx.close();
}
#Override
public void exceptionCaught(final ChannelHandlerContext ctx, final Throwable cause) throws Exception {
System.out.println("Error occur when reading from Unix domain socket: " + cause.getMessage());
ctx.close();
}
}
);
}
}
);
final ChannelFuture clientFuture = bootstrapClient.connect(new DomainSocketAddress(sockPath)).sync();
clientFuture.channel().closeFuture().sync();
serverFuture.channel().closeFuture().sync();
serverBossEventLoopGroup.shutdownGracefully();
serverWorkerEventLoopGroup.shutdownGracefully();
clientEventLoop.shutdownGracefully();
}
}
I'm new to socket programming and Netty framework. I was trying to modify the Echo Server example so that the message is not sent from client as soon as a message is received, but a call from another thread would trigger the client send a message to the server.
The problem is, the server does not get the message unless the client sends it from readChannel or MessageReceived or channelActive which are where the server is specified with a parameter (ChannelHandlerContext). I couldn't manage to find a way to save the server channel and send a message later and repeatedly.
Here's my Client Handler code;
import io.netty.channel.ChannelHandlerAdapter;
import io.netty.channel.ChannelHandlerContext;
public class EchoClientHandler extends ChannelHandlerAdapter {
ChannelHandlerContext server;
#Override
public void channelActive(ChannelHandlerContext ctx) {
this.server = ctx;
}
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
// ctx.write(msg); //not
}
#Override
public void channelReadComplete(ChannelHandlerContext ctx) throws Exception {
//ctx.flush();
}
public void externalcall(String msg) throws Exception {
if(server!=null){
server.writeAndFlush("[" + "] " + msg + '\n');
}
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
// Close the connection when an exception is raised.
ctx.close();
}
}
When Client creates the handler, it also creates a thread with a "SourceGenerator" object which gets the handler as parameter so as to call the externalcall() method.
import io.netty.bootstrap.Bootstrap;
import io.netty.channel.ChannelFuture;
import io.netty.channel.ChannelInitializer;
import io.netty.channel.ChannelOption;
import io.netty.channel.EventLoopGroup;
import io.netty.channel.nio.NioEventLoopGroup;
import io.netty.channel.socket.SocketChannel;
import io.netty.channel.socket.nio.NioSocketChannel;
/**
* Sends one message when a connection is open and echoes back any received
* data to the server. Simply put, the echo client initiates the ping-pong
* traffic between the echo client and server by sending the first message to
* the server.
*/
public class EchoClient {
private final String host;
private final int port;
public EchoClient(String host, int port, int firstMessageSize) {
this.host = host;
this.port = port;
}
public void run() throws Exception {
// Configure the client.
EventLoopGroup group = new NioEventLoopGroup();
final EchoClientHandler x = new EchoClientHandler();
SourceGenerator sg = new SourceGenerator(x);
new Thread(sg).start();
try {
Bootstrap b = new Bootstrap();
b.group(group)
.channel(NioSocketChannel.class)
.option(ChannelOption.TCP_NODELAY, true)
.handler(new ChannelInitializer<SocketChannel>() {
#Override
public void initChannel(SocketChannel ch) throws Exception {
ch.pipeline().addLast(x);
}
});
// Start the client.
ChannelFuture f = b.connect(host, port).sync();
// Wait until the connection is closed.
f.channel().closeFuture().sync();
} finally {
// Shut down the event loop to terminate all threads.
group.shutdownGracefully();
}
}
public static void main(String[] args) throws Exception {
// Print usage if no argument is specified.
if (args.length < 2 || args.length > 3) {
System.err.println(
"Usage: " + EchoClient.class.getSimpleName() +
" <host> <port> [<first message size>]");
return;
}
// Parse options.
final String host = args[0];
final int port = Integer.parseInt(args[1]);
final int firstMessageSize;
if (args.length == 3) {
firstMessageSize = Integer.parseInt(args[2]);
} else {
firstMessageSize = 256;
}
new EchoClient(host, port, firstMessageSize).run();
}
}
and the SourceGenerator class;
public class SourceGenerator implements Runnable {
public String dat;
public EchoClientHandler asd;
public SourceGenerator(EchoClientHandler x) {
asd = x;
System.out.println("initialized source generator");
dat = "";
}
#Override
public void run() {
try{
while(true){
Thread.sleep(2000);
dat += "a";
asd.externalcall(dat);
System.out.print("ha!");
}
}catch(Exception e){
e.printStackTrace();
}
}
}
Thanks in advance!
If you want to write a String you need to have the StringEncoder in the ChannelPipeline.
Otherwise you can only send ByteBuf instances.
I have set up a network and I've set up the reading and writing stream to a socket as so:
//Set up socket reads and writes
final BufferedReader in = new BufferedReader(
new InputStreamReader(client.getInputStream()));
final PrintWriter out = new PrintWriter(
client.getOutputStream(), true);
I wanted to pass the two variables, 'in' and 'out', as parameters of another class' constructor. This is how it looks in the other class
BufferedReader in;
PrintWriter out;
public ClientThread(BufferedReader in, PrintWriter out) {
this.in = in;
this.out = out;
}
I then wanted to use those class variables to write to the output stream of the same socket like this (the class implements Runnable):
public void run() {
while (true) {
try {
String userCommand = in.readLine();
} catch (IOException e) {
// Die if something goes wrong.
System.err.println(e.toString());
System.exit(1);
}
}
}
However, whenever the code gets to this point, I get a SocketException:
java.net.SocketException: Socket closed
How can I fix this? I want to separate the setting up of the server and the socket from the processing of any commands given by the client.
EDIT: Here's what the BufferedRead gets the input from
//create server socket
ServerSocket server = new ServerSocket(portNum);
// Accept a client if it appears
Socket client = server.accept();
EDIT 2: I used these three files:
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.PrintWriter;
import java.net.Socket;
import java.net.UnknownHostException;
public class Client {
public static void main(String[] args) throws UnknownHostException, IOException {
//Change the socket if it doesn't work
Socket sock = new Socket("localhost", 5920);
//keyboard
final BufferedReader stdin = new BufferedReader(
new InputStreamReader(System.in));
//input from socket
final BufferedReader in = new BufferedReader(
new InputStreamReader(sock.getInputStream()));
//writer to socket
final PrintWriter out = new PrintWriter(
sock.getOutputStream(), true);
//new thread for incoming messages
(new Thread(){
#Override
public void run() {
String serverMessage;
try {
while ((serverMessage = in.readLine()) != null) {
System.out.println(serverMessage);
}
} catch (IOException e) {
System.err.println("Something went wrong whilst trying "
+ "to retrieve a message from the server");
System.exit(-1);
}
}
}).start();
//new thread for outgoing messages
(new Thread(){
#Override
public void run() {
String clientMessage;
try {
while ((clientMessage = stdin.readLine()) != null) {
out.println(clientMessage);
}
} catch (IOException e) {
System.err.println("Something went wrong whilst trying "
+ "to send a message to the server.");
System.exit(-1);
}
}
}).start();
}
}
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.PrintWriter;
import java.net.ServerSocket;
import java.net.Socket;
public class Server {
public static void main(String[] args) throws IOException {
ServerSocket server = new ServerSocket(5920);
Socket client = server.accept();
//Set up socket reads and writes
final BufferedReader in = new BufferedReader(
new InputStreamReader(client.getInputStream()));
final PrintWriter out = new PrintWriter(
client.getOutputStream(), true);
new Thread(new ClassWithParam(in, out)).start();
}
}
import java.io.BufferedReader;
import java.io.IOException;
import java.io.PrintWriter;
public class ClassWithParam implements Runnable {
BufferedReader in;
PrintWriter out;
public ClassWithParam(BufferedReader in, PrintWriter out) {
this.in = in;
this.out = out;
}
#Override
public void run() {
while (true) {
try {
System.out.println("HERE");
String userCommand = in.readLine();
System.out.println("HERE2");
} catch (IOException e) {
// Die if something goes wrong.
System.err.println(e.toString());
System.exit(1);
}
}
}
}
And now it works. Don't know what happened. Will proceed to bang head against wall. Thanks.
For some reason there's no problem now. The code (I recreated) which I used, which now works, is in the description.
I am currently writing this code for my client and server,
and I want to test it out using my emulator, but I'm stuck.
is this the correct IP address that I should be using?
socket = new Socket("10.0.2.2", 6000);
If i want to use my phone to test this out, what ip address should i be using?
thanks.
if you want to send messages between server/client, here is a sample code that i have made before.
please refer to the code below and feel free to comment!
also, that is the correct ip address to use when using emulator for simulation.
in addition, don't forget to change your permission to "android.permission.INTERNET" in your manifesto.
=================================myClient==================================
package com.example.myclient;
import java.io.IOException;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.net.Socket;
import java.net.UnknownHostException;
import android.os.Bundle;
import android.app.Activity;
import android.util.Log;
import android.view.Menu;
import android.view.View;
import android.widget.Button;
public class MainActivity extends Activity {
/** Manifest --> uses permission --> "android.permission.INTERNET" */
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
Button btn = (Button) findViewById(R.id.btn);
btn.setOnClickListener(new View.OnClickListener() {
#Override
public void onClick(View v) {
class MyThread extends Thread {
#Override
public void run() {
super.run();
Log.d("client", "thread is running...");
String str = "Do you want to eat hamburger?";
Socket socket;
try {
socket = new Socket("10.0.2.2", 6000);
ObjectOutputStream out = new ObjectOutputStream(socket
.getOutputStream());
ObjectInputStream in = new ObjectInputStream(
socket.getInputStream());
out.writeObject(str);
String rcv = (String) in.readObject();
Log.d("client", "Server :" + rcv);
out.close();
in.close();
socket.close();
} catch (UnknownHostException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (ClassNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
MyThread t = new MyThread();
t.start();
}
});
}
}
============================MyServer========================================
import java.io.IOException;
import java.io.InputStream;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.io.OutputStream;
import java.net.ServerSocket;
import java.net.Socket;
public class MyServer {
public static void main(String[] args) throws IOException, ClassNotFoundException {
ServerSocket server = new ServerSocket(6000);
System.out.println("waiting.....");
while (true) {
Socket socket = server.accept();
System.out.println("a client has connected...");
InputStream in = socket.getInputStream();
OutputStream out = socket.getOutputStream();
ObjectInputStream objIn = new ObjectInputStream(in);
ObjectOutputStream objOut = new ObjectOutputStream(out);
String str = (String) objIn.readObject();
System.out.println("client : " + str);
objOut.writeObject("No, I'm on a diet!!!");
objIn.close();
objOut.close();
socket.close();
}
}
}
10.0.2.2 will be the correct IP you are using emulator. 127.0.0.1 will be the IP if you are developing on the machine(client and server on same machine). As you said you want to test it in your mobile run the following code and you will get your IP(it will also work if you are on computer):
public class net
{
net() throws UnknownHostException
{
InetAddress ia=InetAddress.getLocalHost();
System.out.println(ia);
ia=InetAddress.getByName("local host");
System.out.println(ia);
}
public static void main(String args[])throws UnknownHostException
{
net a=new net();
}
}