Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
I currently have a Scala echo server, how to do I implement Akka into the code so it would serve HTML files that I've implemented?
Don't even know how to begin to work this out, would I have to import something like an Akka thing? Or to change my dependencies? So instead of using imports Java and Scala would I change it to Akka instead?
My echo server
import java.net._
import java.io._
import scala.concurrent.{ExecutionContext, Future}
import scala.util.control.Breaks
object EchoServer {
def read_and_write(in: BufferedReader, out: BufferedWriter): Unit = {
out.write(in.readLine())
out.flush()
in.close()
out.close()
}
def serve(server: ServerSocket): Unit = {
val s = server.accept()
val in = new BufferedReader(new InputStreamReader(s.getInputStream))
val out = new BufferedWriter(new OutputStreamWriter(s.getOutputStream))
read_and_write(in, out)
s.close()
}
def getListOfFiles(dir: String): List[File] = {
val file = new File(dir)
if (file.exists && file.isDirectory) {
file.listFiles.filter(_.isFile).toList
} else {
List[File]()
}
}
def main(args: Array[String]) {
val server = new ServerSocket(9999)
var readString = "C:\\Users\\Desktop\\EchoServer\\src\\main\\htmlFiles"
println("What file are you looking for with extension?")
implicit val ec = ExecutionContext.global
while (true) {
Future {
implicit val ec = ExecutionContext.global
val userInput = scala.io.StdIn.readLine()
val result = getListOfFiles(readString)
val loop = new Breaks
try {
result.foreach { file =>
if (file.getName.endsWith(".html") == (userInput.endsWith(".html"))) {
if (file.getName.equals(userInput)) {
println(file.getName + " file found")
}
else {
println("file not found")
}
}
}
}
catch {
case e: FileNotFoundException => println("Couldn't find that file.")
}
serve(server)
}
}
}
}
I had implemented futures, not sure if it serves HTML files correctly even. How would I change this to akka concurrency?
Related
I am trying to process some files asynchronously, with the ability to choose the number of threads the program should use. But I want to wait till processFiles() is completed processing all the files. So, I am searching for ways to stop function from returning until all the Futures are done executing. And it would be very helpful if anyone gives any ideas to approach this problem. Here is my sample code.
object FileReader{
def processFiles(files: Array[File]) = {
val execService = Executors.newFixedThreadPool(5)
implicit val execContext = ExecutionContext.fromExecutorService(execService)
val processed = files.map { f =>
Future {
val name = f.getAbsolutePath()
val fp = Source.fromFile(name)
var data = ""
fp.getLines().foreach(x => {
data = data ++ s"$x\n"
})
fp.close()
// process the data.
println("Processing ....")
data
}
}
execContext.shutdown()
}
def main(args: Array[String]): Unit = {
println("Start")
val tmp = new File("/path/to/files")
val files = tmp.listFiles()
val result = processFiles(files)
println("done processing")
println("done work")
}
}
I am thinking if my usage of Future here is wrong, please correct me if I am wrong.
My expected output :
Start
Processing ....
Processing ....
Processing ....
Processing ....
Processing ....
Processing ....
Processing ....
Processing ....
Processing ....
Processing ....
done processing
done work
My current output:
Start
done processing
done work
Processing ....
Processing ....
Processing ....
Processing ....
Processing ....
Processing ....
Processing ....
Processing ....
Processing ....
Processing ....
You need to use Future.traverse to combine all the Future's for individual file processing and Await.result on them after:
import java.io.File
import java.util.concurrent.Executors
import scala.io.Source
import scala.concurrent._
import scala.concurrent.duration._
import scala.language.postfixOps
object FileReader {
def processFiles(files: Array[File]) = {
val execService = Executors.newFixedThreadPool(5)
implicit val execContext = ExecutionContext.fromExecutorService(execService)
//Turn `Array[Future[String]]` to `Future[Array[String]]`
val processed = Future.traverse(files.toList) { f =>
Future {
val name = f.getAbsolutePath()
val fp = Source.fromFile(name)
var data = ""
fp.getLines()
.foreach(x => {
data = data ++ s"$x\n"
})
fp.close()
// process the data.
println("Processing ....")
data
}
}
//TODO: Put proper timeout here
//Execution will be blocked until all futures completed
Await.result(processed, 30 minute)
execContext.shutdown()
}
def main(args: Array[String]): Unit = {
println("Start")
val tmp = new File(
"/path/to/file"
)
val files = tmp.listFiles()
val result = processFiles(files)
println("done processing")
println("done work")
}
}
Thanks to #Ivan Kurchenko. The solution worked. I am posting my final version of the code that worked.
object FileReader {
def processFiles(files: Seq[File]) = {
val execService = Executors.newFixedThreadPool(5)
implicit val execContext = ExecutionContext.fromExecutorService(execService)
//Turn `Array[Future[String]]` to `Future[Array[String]]`
val processed = Future.traverse(files) {
f =>
Future {
val name = f.getAbsolutePath()
val fp = Source.fromFile(name)
var data = ""
fp.getLines()
.foreach(x => {
data = augmentString(data) ++ s"$x\n"
})
fp.close()
// process the data.
println("Processing ....")
f
}
}
// TODO: Put proper timeout here
// Execution will be blocked until all futures completed
Await.result(processed, 30.minute)
execContext.shutdown()
}
def main(args: Array[String]): Unit = {
println("Start")
val tmp = new File(
"/path/to/file"
)
val files =tmp.listFiles.toSeq
val result = processFiles(files)
println("done processing")
println("done work")
}
}
I want to stream some files and zip them on the fly, so users can download multiple files into a single zipped file without writing anything to the local disk. However, my current implementation holds everything in the memory, and will no work for large files. Is there any way to fix it?
I was looking at this implementation: https://gist.github.com/kirked/03c7f111de0e9a1f74377bf95d3f0f60, but couldn't figure out how to use it.
import java.io.{BufferedOutputStream, ByteArrayInputStream, ByteArrayOutputStream}
import java.util.zip.{ZipEntry, ZipOutputStream}
import akka.stream.scaladsl.{StreamConverters}
import org.apache.commons.io.FileUtils
import play.api.mvc.{Action, Controller}
class HomeController extends Controller {
def single() = Action {
Ok.sendFile(
content = new java.io.File("C:\\Users\\a.csv"),
fileName = _ => "a.csv"
)
}
def zip() = Action {
Ok.chunked(StreamConverters.fromInputStream(fileByteData)).withHeaders(
CONTENT_TYPE -> "application/zip",
CONTENT_DISPOSITION -> s"attachment; filename = test.zip"
)
}
def fileByteData(): ByteArrayInputStream = {
val fileList = List(
new java.io.File("C:\\Users\\a.csv"),
new java.io.File("C:\\Users\\b.csv")
)
val baos = new ByteArrayOutputStream()
val zos = new ZipOutputStream(new BufferedOutputStream(baos))
try {
fileList.map(file => {
zos.putNextEntry(new ZipEntry(file.toPath.getFileName.toString))
zos.write(FileUtils.readFileToByteArray(file))
zos.closeEntry()
})
} finally {
zos.close()
}
new ByteArrayInputStream(baos.toByteArray)
}
}
Instead of using a ByteArrayOutputStream to buffer the contents in an array then putting them into a ByteArrayInputStream you could use Java's piping mechanism.
Here's a sketch solution:
def zip() = Action {
// Create Source that listens to an OutputStream
// and pass it to `fileByteData` method.
val zipSource: Source[ByteString, Unit] =
StreamConverters
.asOutputStream()
.mapMaterializedValue(fileByteData)
Ok.chunked(zipSource).withHeaders(
CONTENT_TYPE -> "application/zip",
CONTENT_DISPOSITION -> s"attachment; filename = test.zip")
}
// Send the file data, given an OutputStream to write to.
def fileByteData(os: OutputStream): Unit = {
val fileList = List(
new java.io.File("C:\\Users\\a.csv"),
new java.io.File("C:\\Users\\b.csv")
)
val zos = new ZipOutputStream(os)
val buffer: Array[Byte] = new Array[Byte](2048)
try {
for (file <- fileList) {
zos.putNextEntry(new ZipEntry(file.toPath.getFileName.toString))
val fis = new Files.newInputStream(file.toPath)
try {
#tailrec
def zipFile(): Unit = {
val bytesRead = fis.read(buffer)
if (bytesRead == -1) () else {
zos.write(buffer, 0, bytesRead)
zipFile()
}
}
zipFile()
} finally fis.close()
zos.closeEntry()
}
} finally {
zos.close()
}
}
This is just an outline of an approach. You'll also want to make sure:
- the threading is OK - the fileByteData will hopefully run on a different thread to the sending thread
- the error handling is OK - e.g. all streams are closed properly if there's an error on either the server (e.g. file not found) or client side (early disconnect)
The example mentioned in
http://spark.apache.org/docs/latest/streaming-programming-guide.html
Lets me receive data packets in a TCP stream and listening on port 9999
import org.apache.spark._
import org.apache.spark.streaming._
import org.apache.spark.streaming.StreamingContext._ // not necessary since Spark 1.3
// Create a local StreamingContext with two working thread and batch interval of 1 second.
// The master requires 2 cores to prevent from a starvation scenario.
val conf = new SparkConf().setMaster("local[2]").setAppName("NetworkWordCount")
val ssc = new StreamingContext(conf, Seconds(1))
// Create a DStream that will connect to hostname:port, like localhost:9999
val lines = ssc.socketTextStream("localhost", 9999)
// Split each line into words
val words = lines.flatMap(_.split(" "))
import org.apache.spark.streaming.StreamingContext._ // not necessary since Spark 1.3
// Count each word in each batch
val pairs = words.map(word => (word, 1))
val wordCounts = pairs.reduceByKey(_ + _)
// Print the first ten elements of each RDD generated in this DStream to the console
wordCounts.print()
ssc.start() // Start the computation
ssc.awaitTermination() // Wait for the computation to terminate
I am able to send data over TCP by creating a data server by using in my Linux system
$ nc -lk 9999
Question
I need to receive stream from an android phone streaming using UDP and the Scala/Spark
val lines = ssc.socketTextStream("localhost", 9999)
receives ONLY in TCP streams.
How can I receive UDP streams in a similar simple manner using Scala+Spark and create Spark DStream.
There isn't something built in, but it's not too much work to get it done youself. Here is a simple solution I made based on a custom UdpSocketInputDStream[T]:
import java.io._
import java.net.{ConnectException, DatagramPacket, DatagramSocket, InetAddress}
import org.apache.spark.storage.StorageLevel
import org.apache.spark.streaming.StreamingContext
import org.apache.spark.streaming.dstream.ReceiverInputDStream
import org.apache.spark.streaming.receiver.Receiver
import scala.reflect.ClassTag
import scala.util.control.NonFatal
class UdpSocketInputDStream[T: ClassTag](
_ssc: StreamingContext,
host: String,
port: Int,
bytesToObjects: InputStream => Iterator[T],
storageLevel: StorageLevel
) extends ReceiverInputDStream[T](_ssc) {
def getReceiver(): Receiver[T] = {
new UdpSocketReceiver(host, port, bytesToObjects, storageLevel)
}
}
class UdpSocketReceiver[T: ClassTag](host: String,
port: Int,
bytesToObjects: InputStream => Iterator[T],
storageLevel: StorageLevel) extends Receiver[T](storageLevel) {
var udpSocket: DatagramSocket = _
override def onStart(): Unit = {
try {
udpSocket = new DatagramSocket(port, InetAddress.getByName(host))
} catch {
case e: ConnectException =>
restart(s"Error connecting to $port", e)
return
}
// Start the thread that receives data over a connection
new Thread("Udp Socket Receiver") {
setDaemon(true)
override def run() {
receive()
}
}.start()
}
/** Create a socket connection and receive data until receiver is stopped */
def receive() {
try {
val buffer = new Array[Byte](2048)
// Create a packet to receive data into the buffer
val packet = new DatagramPacket(buffer, buffer.length)
udpSocket.receive(packet)
val iterator = bytesToObjects(new ByteArrayInputStream(packet.getData, packet.getOffset, packet.getLength))
// Now loop forever, waiting to receive packets and printing them.
while (!isStopped() && iterator.hasNext) {
store(iterator.next())
}
if (!isStopped()) {
restart("Udp socket data stream had no more data")
}
} catch {
case NonFatal(e) =>
restart("Error receiving data", e)
} finally {
onStop()
}
}
override def onStop(): Unit = {
synchronized {
if (udpSocket != null) {
udpSocket.close()
udpSocket = null
}
}
}
}
In order to get StreamingContext to add a method on itself, we enrich it with an implicit class:
object Implicits {
implicit class StreamingContextOps(val ssc: StreamingContext) extends AnyVal {
def udpSocketStream[T: ClassTag](host: String,
port: Int,
converter: InputStream => Iterator[T],
storageLevel: StorageLevel): InputDStream[T] = {
new UdpSocketInputDStream(ssc, host, port, converter, storageLevel)
}
}
}
And here is how you call it all:
import java.io.{BufferedReader, InputStream, InputStreamReader}
import java.nio.charset.StandardCharsets
import org.apache.spark.SparkContext
import org.apache.spark.storage.StorageLevel
import org.apache.spark.streaming.dstream.InputDStream
import org.apache.spark.streaming.{Seconds, StreamingContext}
import scala.reflect.ClassTag
object TestRunner {
import Implicits._
def main(args: Array[String]): Unit = {
val sparkContext = new SparkContext("local[*]", "udpTest")
val ssc = new StreamingContext(sparkContext, Seconds(4))
val stream = ssc.udpSocketStream("localhost",
3003,
bytesToLines,
StorageLevel.MEMORY_AND_DISK_SER_2)
stream.print()
ssc.start()
ssc.awaitTermination()
}
def bytesToLines(inputStream: InputStream): Iterator[String] = {
val dataInputStream = new BufferedReader(
new InputStreamReader(inputStream, StandardCharsets.UTF_8))
new NextIterator[String] {
protected override def getNext(): String = {
val nextValue = dataInputStream.readLine()
if (nextValue == null) {
finished = true
}
nextValue
}
protected override def close() {
dataInputStream.close()
}
}
}
abstract class NextIterator[U] extends Iterator[U] {
protected var finished = false
private var gotNext = false
private var nextValue: U = _
private var closed = false
override def next(): U = {
if (!hasNext) {
throw new NoSuchElementException("End of stream")
}
gotNext = false
nextValue
}
override def hasNext: Boolean = {
if (!finished) {
if (!gotNext) {
nextValue = getNext()
if (finished) {
closeIfNeeded()
}
gotNext = true
}
}
!finished
}
def closeIfNeeded() {
if (!closed) {
closed = true
close()
}
}
protected def getNext(): U
protected def close()
}
}
Most of this code is taken from the SocketInputDStream[T] provided by Spark, I simply re-used it. I also took the code for the NextIterator which is used by bytesToLines, all it does is consume the line from the packet and transform it to a String. If you have more complex logic, you can provide it by passing converter: InputStream => Iterator[T] your own implementation.
Testing it with simple UDP packet:
echo -n "hello hello hello!" >/dev/udp/localhost/3003
Yields:
-------------------------------------------
Time: 1482676728000 ms
-------------------------------------------
hello hello hello!
Of course, this has to be further tested. I also has a hidden assumption that each buffer created from the DatagramPacket is 2048 bytes, which is perhaps something you'll want to change.
The problem with the Yuval Itzchakov's solution is that the receiver receives one message and restarts itself. Just replace restart for receive as shown below.
def receive() {
try {
val buffer = new Array[Byte](200000)
// Create a packet to receive data into the buffer
val packet = new DatagramPacket(buffer, buffer.length)
udpSocket.receive(packet)
val iterator = bytesToLines(new ByteArrayInputStream(packet.getData, packet.getOffset, packet.getLength))
// Now loop forever, waiting to receive packets and printing them.
while (!isStopped() && iterator.hasNext) {
store(iterator)
}
if (!isStopped()) {
// restart("Udp socket data stream had no more data")
receive()
}
} catch {
case NonFatal(e) =>
restart("Error receiving data", e)
} finally {
onStop()
}
}
I try write some simple akka-http and akka-streams based application, that handle http requests, always with one precompiled stream, because I plan to use long time processing with back-pressure in my requestProcessor stream
My application code:
import akka.actor.{ActorSystem, Props}
import akka.http.scaladsl._
import akka.http.scaladsl.server.Directives._
import akka.http.scaladsl.server._
import akka.stream.ActorFlowMaterializer
import akka.stream.actor.ActorPublisher
import akka.stream.scaladsl.{Sink, Source}
import scala.annotation.tailrec
import scala.concurrent.Future
object UserRegisterSource {
def props: Props = Props[UserRegisterSource]
final case class RegisterUser(username: String)
}
class UserRegisterSource extends ActorPublisher[UserRegisterSource.RegisterUser] {
import UserRegisterSource._
import akka.stream.actor.ActorPublisherMessage._
val MaxBufferSize = 100
var buf = Vector.empty[RegisterUser]
override def receive: Receive = {
case request: RegisterUser =>
if (buf.isEmpty && totalDemand > 0)
onNext(request)
else {
buf :+= request
deliverBuf()
}
case Request(_) =>
deliverBuf()
case Cancel =>
context.stop(self)
}
#tailrec final def deliverBuf(): Unit =
if (totalDemand > 0) {
if (totalDemand <= Int.MaxValue) {
val (use, keep) = buf.splitAt(totalDemand.toInt)
buf = keep
use foreach onNext
} else {
val (use, keep) = buf.splitAt(Int.MaxValue)
buf = keep
use foreach onNext
deliverBuf()
}
}
}
object Main extends App {
val host = "127.0.0.1"
val port = 8094
implicit val system = ActorSystem("my-testing-system")
implicit val fm = ActorFlowMaterializer()
implicit val executionContext = system.dispatcher
val serverSource: Source[Http.IncomingConnection, Future[Http.ServerBinding]] = Http(system).bind(interface = host, port = port)
val mySource = Source.actorPublisher[UserRegisterSource.RegisterUser](UserRegisterSource.props)
val requestProcessor = mySource
.mapAsync(1)(fakeSaveUserAndReturnCreatedUserId)
.to(Sink.head[Int])
.run()
val route: Route =
get {
path("test") {
parameter('test) { case t: String =>
requestProcessor ! UserRegisterSource.RegisterUser(t)
???
}
}
}
def fakeSaveUserAndReturnCreatedUserId(param: UserRegisterSource.RegisterUser): Future[Int] =
Future.successful {
1
}
serverSource.to(Sink.foreach {
connection =>
connection handleWith Route.handlerFlow(route)
}).run()
}
I found solution about how create Source that can dynamically accept new items to process, but I can found any solution about how than obtain result of stream execution in my route
The direct answer to your question is to materialize a new Stream for each HttpRequest and use Sink.head to get the value you're looking for. Modifying your code:
val requestStream =
mySource.map(fakeSaveUserAndReturnCreatedUserId)
.to(Sink.head[Int])
//.run() - don't materialize here
val route: Route =
get {
path("test") {
parameter('test) { case t: String =>
//materialize a new Stream here
val userIdFut : Future[Int] = requestStream.run()
requestProcessor ! UserRegisterSource.RegisterUser(t)
//get the result of the Stream
userIdFut onSuccess { case userId : Int => ...}
}
}
}
However, I think your question is ill posed. In your code example the only thing you're using an akka Stream for is to create a new UserId. Futures readily solve this problem without the need for a materialized Stream (and all the accompanying overhead):
val route: Route =
get {
path("test") {
parameter('test) { case t: String =>
val user = RegisterUser(t)
fakeSaveUserAndReturnCreatedUserId(user) onSuccess { case userId : Int =>
...
}
}
}
}
If you want to limit the number of concurrent calls to fakeSaveUserAndReturnCreateUserId then you can create an ExecutionContext with a defined ThreadPool size, as explained in the answer to this question, and use that ExecutionContext to create the Futures:
val ThreadCount = 10 //concurrent queries
val limitedExecutionContext =
ExecutionContext.fromExecutor(Executors.newFixedThreadPool(ThreadCount))
def fakeSaveUserAndReturnCreatedUserId(param: UserRegisterSource.RegisterUser): Future[Int] =
Future { 1 }(limitedExecutionContext)
I have been struggling with this for a while, I can't find a way to instruct phantomjs about the viewport. I am using play 2.2 (code is based on: Set Accept-Language on PhantomJSDriver in a Play Framework Specification )
package selenium
import org.specs2.mutable.Around
import org.specs2.specification.Scope
import play.api.test.{TestServer, TestBrowser, FakeApplication}
import org.openqa.selenium.remote.DesiredCapabilities
import org.specs2.execute.{Result, AsResult}
import play.api.test.Helpers._
import scala.Some
import org.openqa.selenium.phantomjs.PhantomJSDriver
import org.openqa.selenium.NoSuchElementException
import java.io.File
import java.io.PrintWriter
import java.util.concurrent.TimeUnit
import collection.JavaConversions._
abstract class WithPhantomJS(val additionalOptions: Map[String, String] = Map()) extends Around with Scope {
implicit def app = FakeApplication()
implicit def port = play.api.test.Helpers.testServerPort
// for phantomjs
lazy val browser: TestBrowser = {
val defaultCapabilities = DesiredCapabilities.phantomjs
print(defaultCapabilities.toString)
val additionalCapabilities = new DesiredCapabilities(mapAsJavaMap(additionalOptions))
val capabilities = new DesiredCapabilities(defaultCapabilities, additionalCapabilities)
val driver = new PhantomJSDriver(capabilities)
val br = TestBrowser(driver, Some("http://localhost:" + port))
br.webDriver.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS)
br
}
override def around[T: AsResult](body: => T):Result = {
try {
running(TestServer(port, FakeApplication()))(AsResult.effectively(body))
} catch {
case e: Exception => {
val fn: String = "/tmp/test_" + e.getClass.getCanonicalName;
browser.takeScreenShot(fn + ".png")
val out = new PrintWriter(new File(fn + ".html"), "UTF-8");
try {
out.print(browser.pageSource())
} finally {
out.close
}
throw e;
}
} finally {
browser.quit()
}
}
}
and the actual test looks like this:
class ChangeName extends Specification {
"User" should {
"be able to change his name in account settings" in new WithPhantomJS() {
// ... this throws
I'd like the exception handler to render the site, which works, however, i can see the with matching a mobile device (narrow).
How can I change the width?
This seems to work:
import org.openqa.selenium.Dimension
...
browser.webDriver.manage().window().setSize(new Dimension(1280, 800))