I'm making a distributed sorting system, where each machine (client) has a specific range of values (partition) to store. I don't have any problem sending and receiving data, but the synchronization is my problem. Before retrieving the wanted partition from the server, it must first arrive to the server from one of the other clients.
I've tried with delay and various other RPC calls to try to try to synchronize the clients and server, but the clients run out of synchronization, and the program breaks. I think I have to incorporate future or promise, but I'm not sure how.
Every client has a unique ID (from 0 and up), and will store partitions according to that number. One line of data looks like this: AsfAGHM5om 00000000000000000000000000000000 0000222200002222000022220000222200002222000000001111
EDIT: updated code for example
Client
val numberOfClients = 3
var sentPartitionsCounter = 0
while(sentPartitionsCounter < numberOfClients) {
if(sentPartitionsCounter != id){
sendUnwantedPartition("src/main/scala/testFile."+sentPartitionsCounter.toString) //"partition."+client.sentPartitionsCounter
sentPartitionsCounter += 1
}
client.getWantedPartition()
}
def sendUnwantedPartition(filename: String): Unit = {
val dataList = Source.fromFile(filename).getLines.toList
val dataSeq = for {
dataLine <- dataList
dataValues = dataLine.split(" ", 2)
} yield (Data(key = dataValues(0), value = dataValues(1)))
val partitionID = filename takeRight 1
val request = Dataset(data = dataSeq, partitionID = partitionID.toInt)
val response = blockingStub.getUnwantedPartitions(request)
print("sent partitions")
}
def getWantedPartition(): Unit = {
val request = ID(id = id)
val response = blockingStub.sendWantedPartitions(request)
val wantedPartitions = new File("clientFiles/wantedPartitions"+id.toString+".txt")
val printWriter: PrintWriter = new PrintWriter(new FileWriter(wantedPartitions, true));
if(!wantedPartitions.exists()){
wantedPartitions.createNewFile()
}
for {
data <- response.data
} yield (printWriter.append(data.key + " " + data.value + "\n"))
printWriter.close();
}
Server
override def getUnwantedPartitions(req: Dataset) = {
val filename = "serverPartitions/partition"+req.partitionID+".txt"
val partition = new File(filename)
val printWriter: PrintWriter = new PrintWriter(new FileWriter(partition, true));
if(!partition.exists()){
partition.createNewFile()
}
for {
data <- req.data
} yield (printWriter.append(data.key + " " + data.value + "\n"))
printWriter.close();
Future.successful(Text("Got unwanted data"))
}
override def sendWantedPartitions(req: ID) = {
val filename = "serverPartitions/partition"+req.id+".txt"
try {
val dataList = Source.fromFile(filename).getLines.toList
val dataSeq = for {
dataLine <- dataList
dataValues = dataLine.split(" ", 2)
} yield (Data(key = dataValues(0), value = dataValues(1)))
val reply = Dataset(data = dataSeq, partitionID = req.id)
new File(filename).delete()
Future.successful(reply)
} catch {
// Partition hasn't been received yet
case e: FileNotFoundException => Future.successful(Dataset())
}
}
Proto
syntax = "proto3";
package protoPackage;
service DistrSorting {
rpc getUnwantedPartitions(Dataset) returns (Text) {}
rpc sendWantedPartitions(ID) returns (Dataset) {}
}
message ID {
int32 id = 1;
}
message Text {
string text = 1;
}
message Dataset {
repeated Data data = 1;
int32 partitionID = 2;
}
message Data {
string key = 1;
string value = 2;
}
Related
I am using this code to upload an image on server , that i get from this link play upload
def upload = Action(parse.multipartFormData) { request =>
request.body
.file("picture")
.map { picture =>
val dataParts = request.body.dataParts;
val filename = Paths.get(picture.filename).getFileName
val fileSize = picture.fileSize
val contentType = picture.contentType
val picturePaths =
picture.ref.copyTo(
Paths.get(
s"/opt/docker/images/$filename"
),
replace = true
)
if (dataParts.get("firstPoint") == None) {
val pointlocation = new Point_LocationModel(
dataParts.get("step").get(0),
dataParts.get("backgroundTargetName").get(0),
dataParts.get("x").get(0),
dataParts.get("y").get(0),
dataParts.get("z").get(0),
dataParts.get("rotation").get(0),
dataParts.get("note").get(0),
dataParts.get("tag").get(0),
dataParts.get("work_session_id").get(0),
(picturePaths).toString
)
point_LocationRepository.create(pointlocation).map { data =>
Created(Json.toJson(data._2))
}
} else {
val jValuefirstPoint =
Json.parse(dataParts.get("firstPoint").get(0)).as[PointModel]
val jValuesecondPoint =
Json.parse(dataParts.get("secondPoint").get(0)).as[PointModel]
val pointlocation = new Point_LocationModel(
dataParts.get("step").get(0),
dataParts.get("backgroundTargetName").get(0),
Some(jValuefirstPoint),
Some(jValuesecondPoint),
dataParts.get("rotation").get(0),
dataParts.get("note").get(0),
dataParts.get("tag").get(0),
dataParts.get("work_session_id").get(0),
(picturePaths).toString
)
point_LocationRepository.create(pointlocation).map { data =>
logger.info(s"repoResponse: ${data}");
Created(Json.toJson(data._2))
}
}
Ok(s"picturePaths ${picturePaths}")
}
.getOrElse(Ok("Invalid Format"))
}
This code works very well, but on the response I want to get the response from the repository. How can i await for the response of the repository to return this?
Can you give me any idea how can i do it?
Thanks in advance.
If we simplify your code to the essential bits, you have:
def upload = Action(parse.multipartFormData) { request =>
request.body
.file("picture")
.map { picture =>
if (someConditionA) {
someBusinessLogicA().map { data =>
Created(Json.toJson(data._2))
}
} else {
someBusinessLogicB().map { data =>
Created(Json.toJson(data._2))
}
}
Ok(s"picturePaths ${picturePaths}")
}
.getOrElse(Ok("Invalid Format"))
}
There are 2 problems:
the return of your if/else is swallowed by the Ok(s"picturePaths ${picturePaths}")
assuming your business logic returns Future[_] you need to "propagate" the Future everywhere
Something like this should work:
def upload = Action.async(parse.multipartFormData) { request =>
request.body
.file("picture")
.map { picture =>
if (someConditionA) {
someBusinessLogicA().map { data =>
Created(Json.toJson(data._2))
}
} else {
someBusinessLogicB().map { data =>
Created(Json.toJson(data._2))
}
}
}
.getOrElse(Future.successful(Ok("Invalid Format")))
}
Notice the Action.async and the Future in the getOrElse.
This is my origin code, I would like to merge them into one loop
//1
if (eventUuidList.nonEmpty) {
eventUuidTransactionTableCondition = for {
eventUuid <- eventUuidList
eventUuidTransactionTableConditionSet =
s"${SQLColumnHelper.EVENT_INFO_STRUCT_NAME}.${SQLColumnHelper.EVENT_UUID} = '".concat(eventUuid).concat("'")
} yield eventUuidTransactionTableConditionSet
eventUuidTransactionTableConditionSet = "and ".concat(eventUuidTransactionTableCondition.reduce(_.concat(" or ").concat(_)))
}
//2
if (eventUuidList.nonEmpty) {
eventUuidExceptionTableCondition = for {
eventUuid <- eventUuidList
eventUuidExceptionTableConditionSet =
s"${SQLColumnHelper.EVENT_INFO_STRUCT_NAME}.${SQLColumnHelper.EVENT_UUID} != '".concat(eventUuid).concat("'")
} yield eventUuidExceptionTableConditionSet
eventUuidExceptionTableConditionSet = "and ".concat(eventUuidExceptionTableCondition.reduce(_.concat(" and ").concat(_)))
}
The different for these two parts are those lines (the first has = and the second !=):
//1
s"${SQLColumnHelper.EVENT_INFO_STRUCT_NAME}.${SQLColumnHelper.EVENT_UUID} = '".concat(eventUuid).concat("'")
eventUuidTransactionTableConditionSet = "and ".concat(eventUuidTransactionTableCondition.reduce(_.concat(" or ").concat(_)))
//2
s"${SQLColumnHelper.EVENT_INFO_STRUCT_NAME}.${SQLColumnHelper.EVENT_UUID} != '".concat(eventUuid).concat("'")
eventUuidExceptionTableConditionSet = "and ".concat(eventUuidExceptionTableCondition.reduce(_.concat(" and ").concat(_)))
I moved both parts into the same block. Is what I have below the correct way to do that? Or there is a better solution?
var eventUuidTransactionTableConditionSet, eventUuidExceptionTableConditionSet = " "
if (eventUuidList.nonEmpty) {
val both = eventUuidList.map( eventUuid =>
s"${SQLColumnHelper.EVENT_INFO_STRUCT_NAME}.${SQLColumnHelper.EVENT_UUID} = '".concat(eventUuid).concat("'") ->
s"${SQLColumnHelper.EVENT_INFO_STRUCT_NAME}.${SQLColumnHelper.EVENT_UUID} != '".concat(eventUuid).concat("'"))
val (eventUuidTransactionTableCondition, eventUuidExceptionTableCondition) = both.unzip
eventUuidExceptionTableConditionSet = "and ".concat(eventUuidTransactionTableCondition.mkString(" or "))
eventUuidExceptionTableConditionSet = "and ".concat(eventUuidExceptionTableCondition.mkString(" and "))
}
There are so many things wrong with this code it's hard to know where to begin. But to your question...
Why not create a tuple of the 2 string interpolations and then unzip them?
val both = eventUuidList.map(id => s"blahblah = '${id}'" ->
s"blahblah != '${id}'")
val (eventUuidTransactionTableCondition
,eventUuidExceptionTableCondition) = both.unzip
I have to make a connect-four game using scala. I have attached the code but everytime the game runs and gets to row 3 it just continues to change the second rows entry instead of going to the next row. Any help would be appreciated. I found this code on another thread on here and couldn't figure out how to get it to work:
// makes the board
val table = Array.fill(9,8)('-')
var i = 0;
while(i < 8){
table(8)(i) = (i+'0').toChar
i = i+1;
}
// prints starting board
def printBoard(table: Array[Array[Char]]) {
table.foreach( x => println(x.mkString(" ")))
}
//player 1 moves
def playerMove1(){
val move = readInt
var currentRow1 = 7
while (currentRow1 >= 0)
if (table(currentRow1)(move) != ('-')) {
currentRow1 = (currentRow1-1)
table(currentRow1)(move) = ('X')
return (player2)}
} else {
table(currentRow1)(move) = ('X')
return (player2)
}
}
//player 2 moves
def playerMove2(){
val move = readInt
var currentRow2 = 7
while (currentRow2 >= 0)
if (table(currentRow2)(move) != ('-')) {
currentRow2 = (currentRow2-1)
table(currentRow2)(move) = ('O')
return (player1)}
} else {
table(currentRow2)(move) = ('O')
return (player1)
}
}
//player 1
def player1(){
printBoard(table)
println("Player 1 it is your turn. Choose a column 0-7")
playerMove1()
}
//player 2
def player2(){
printBoard(table)
println("Player 2 it is your turn. Choose a column 0-7")
playerMove2()
}
for (turn <- 1 to 32){
player1
player2
}
I've tried to make your code readable and compiling and also tried to fix some logic.
However, I've never worked with Scala so this is just a first sketch where you might want to continue ...
Some functions can be merged and the currentRow needed a fix. See here:
object ConnectFour{
val table = Array.fill(9,8)('-')
val currentRow = Array.fill(8)(8)
def main(args: Array[String]) {
var i = 0;
while(i < 8) {
table(8)(i) = (i+'0').toChar
i = i+1;
}
player(1)
}
def printBoard(table: Array[Array[Char]]) {
table.foreach( x => println(x.mkString(" ")))
}
def player(playerNr : Int){
printBoard(table)
println("Player " + playerNr + " it is your turn. Choose a column 0-7")
var column = readAndVerifyInt
var nextUser = 1 : Int
var symbol = 'O' : Char
if(playerNr == 1) {
symbol = 'X'
nextUser = 2
}
var curR = currentRow(column)
while (curR >= 0) {
if (table(curR)(column) != ('-')) {
curR = curR-1
currentRow(column) = curR
}
table(curR)(column) = symbol
player(nextUser)
}
}
def readAndVerifyInt() : Int = {
var column = readInt
if (column >= 0 && column <= 7) {
return column
} else {
println(" > Please try again")
return readAndVerifyInt
}
}
}
I'm trying a small program that would read a file, convert that to a array of bytes and zip the bytes and return the zipped bytes back as an array. However, I'm getting IndexOutOfBounds thrown. Here is what I have so far tried!
private def getBytePayload(): Array[Byte] = {
val inputJson: String = Source.fromURL(getClass.getResource("/small_json.txt")).mkString
val bin: ByteArrayInputStream = new ByteArrayInputStream(inputJson.getBytes())
val bos: ByteArrayOutputStream = new ByteArrayOutputStream()
println("The unCompressed stream size is = " + inputJson.size) // Prints 5330
val gzip: GZIPOutputStream = new GZIPOutputStream(bos)
val buff = new Array[Byte](1024)
var len = 0
while((len = bin.read(buff)) != -1) {
gzip.write(buff, 0, len) // java.lang.IndexOutOfBoundsException is thrown!
}
println("The compressed stream size is = " + bos.toByteArray.size)
bos.toByteArray
}
Any pointers as to what has gone wrong?
As far as what the problem is, I'm not exactly sure. However, the following code should be revealing. See what it it tells you:
val buff = new Array[Byte](1024)
var len = 0
while((len = bin.read(buff)) != -1) {
//Print out actual contents of buff:
for(int i = 0; i < buff.length; i++) {
System.out.print(i + "=" + buff[i] + ", ");
}
System.out.println();
gzip.write(buff, 0, len) // java.lang.IndexOutOfBoundsException is thrown!
}
This is just my style-opinion, but I personally find this confusing
while((len = bin.read(buff)) != -1) {
and would change it to
len = bin.read(buff);
while(len != -1) {
//do stuff
len = bin.read(buff);
}
I am using the twitter4j code and I get connect timed out for making about 5 queries a day.
I am using twitter4j with a servlet and I get the error a lot with timeout.
What happens is that I run the code and the system provides the error when searching and hence the system stops and no more details are provided.
TwitterFactory tf = new TwitterFactory(cb.build());
Twitter twitter = tf.getInstance();
try {
Query query = new Query(searchTerm);
twitter4j.QueryResult result1;
do {
result1 = twitter.search(query);
List<Status> tweets = result1.getTweets();
int i=0;int maxint = 20;
for (Status tweet : tweets) {
i++;
if (i<maxint)
{
out.println("procvess");
xmlStr=xmlStr+"<tweet>";
String tweetText = tweet.getText();
tweetText=cleanStringData(tweetText );
HashtagEntity[] hashtags = tweet.getHashtagEntities();
URLEntity[] urls = tweet.getURLEntities();
Date createdDate = tweet.getCreatedAt();
User twitteruser = tweet.getUser();
long tweetId = tweet.getId();
long id1 = tweet.getInReplyToStatusId();
long id2 = tweet.getInReplyToUserId();
String userImageURL = twitteruser.getProfileImageURL();
String userProfileURL = "http://twitter.com/"+twitteruser.getScreenName();
String realname = twitteruser.getName();
String authorname = twitteruser.getScreenName();
Calendar cal = Calendar.getInstance();
cal.setTime(createdDate);
int year = cal.get(Calendar.YEAR);
int month = cal.get(Calendar.MONTH);
int day = cal.get(Calendar.DAY_OF_MONTH);
String tweetDateStr = String.valueOf(year)+"/"+String.valueOf(month)+"/"+String.valueOf(day);
int l=0;
xmlStr=xmlStr+"</tweet>";
}
// System.out.println("#" + tweet.getUser().getScreenName() + " - " + tweet.getText());
}
} while ((query = result1.nextQuery()) != null);
} catch (TwitterException te) {
te.printStackTrace();
out.println("Failed to search tweets: " + te.getMessage());
// return "<error>" + te.getMessage()+"</error>";
}