I am reading zip file using ZipInputStream. Zip file has 4 csv files. Some files are written completely, some are written partially. Please help me find the issue with below code. Is there any limit on reading buffer from ZipInputStream.read method?
val zis = new ZipInputStream(inputStream)
Stream.continually(zis.getNextEntry).takeWhile(_ != null).foreach { file =>
if (!file.isDirectory && file.getName.endsWith(".csv")) {
val buffer = new Array[Byte](file.getSize.toInt)
zis.read(buffer)
val fo = new FileOutputStream("c:\\temp\\input\\" + file.getName)
fo.write(buffer)
}
You have not closed/flushed the files you attempted to write. It should be something like this (assuming Scala syntax, or is this Kotlin/Ceylon?):
val fo = new FileOutputStream("c:\\temp\\input\\" + file.getName)
try {
fo.write(buffer)
} finally {
fo.close
}
Also you should check the read count and read more if necessary, something like this:
var readBytes = 0
while (readBytes < buffer.length) {
val r = zis.read(buffer, readBytes, buffer.length - readBytes)
r match {
case -1 => throw new IllegalStateException("Read terminated before reading everything")
case _ => readBytes += r
}
}
PS: In your example it seems to be less than required closing }s.
I am subscript to a message feed for a number of fields, I need to set the values from the feed to the domain object and have code like below:
if (map.contains(quoteBidPriceAcronym)) {
quote.bid.price = Some(map.get(quoteBidPriceAcronym).get.asInstanceOf[Number].doubleValue());
quote.changed = true;
}
if (map.contains(quoteBidSizeAcronym)) {
quote.bid.size = Some(sizeMultipler() * map.get(quoteBidSizeAcronym).get.asInstanceOf[Number].intValue());
quote.changed = true;
}
if (map.contains(quoteBidNumAcronym)) {
quote.bid.num = Some(map.get(quoteBidNumAcronym).get.asInstanceOf[Number].shortValue());
quote.changed = true;
}
if (map.contains(quoteAskPriceAcronym)) {
quote.ask.price = Some(map.get(quoteAskPriceAcronym).get.asInstanceOf[Number].doubleValue());
quote.changed = true;
}
if (map.contains(quoteAskSizeAcronym)) {
quote.ask.size = Some(sizeMultipler() * map.get(quoteAskSizeAcronym).get.asInstanceOf[Number].intValue());
quote.changed = true;
}
if (map.contains(quoteAskNumAcronym)) {
quote.ask.num = Some(map.get(quoteAskNumAcronym).get.asInstanceOf[Number].shortValue());
quote.changed = true;
}
if (map.contains(quoteExchTimeAcronym)) {
quote.exchtime = getExchTime(String.valueOf(map.get(quoteExchTimeAcronym).get));
}
It look pretty redundant, any suggestion to improve it?
You can do something like:
map.get(quoteBidPriceAcronym).map { item =>
quote.bid.price = item.map(_.asInstanceOf[Number].doubleValue())
quote.changed = true
}
Other issues might be better to fix outside. E.g. why map[quoteBidPriceAcronym] is storing an Option, if your code assumes it's not going to be None?
Something like this perhaps?
val handlers = Map[String, Number => Unit] (
quoteBidPriceAcronym -> { n => quote.bid.price = Some(n.doubleValue) },
quoteBidSizeAcronym -> { n => quote.bid.size = Some(sizeMultipler() * n.intValue },
etc. ...
)
for {
(k,handler) <- handlers
values <- map.get(k).toSeq
quote.chanded = true
_ = handler(n.asInstanceof[Number])
}
Personally, I don't like code changing an object state (quote) but this is a question on Scala, not functional programming.
That said I would reverse the way you are using you map map keys. Instead of checking whether a value exists to perform some action, I'd have a map from your keys to actions and I'd iterate over your map elements.
e.g (assuming map is of the type Map[String, Any]):
val actions: Map[String, PartialFunction[Any, Unit]] = Map(
(quoteBidPriceAcronym, {case n: Number => quote.bid.price = Some(n.doubleValue())}),
(quoteBidSizeAcronym, {case n: Number => quote.bid.size = Some(sizeMultipler() * n.doubleValue())}),
...
...
)
for((k,v) <- map; action <- actions.get(k); _ <- action.lift(v))
quote.changed = true;
The for construct here iterates over map key-values, then (next level of iteration, over the possible action available for the key. If an action is found, which is a partial function, it gets lifted to make it a function from Any to Option[Unit]. That way, you can iterate in an additional inner level so quote.changed = true is only run when the action is defined for v.
I have a special class Model that needs to have its methods called in a very specific order.
I tried doing something like this:
val model = new Model
new MyWrappingClass {
val first = model.firstMethod()
val second = model.secondMethod()
val third = model.thirdMethod()
}
The methods should be called in the order listed, however I am seeing an apparently random order.
Is there any way to get the variable initialization methods to be called in a particular order?
I doubt your methods are called in the wrong order. But to be sure, you can try something like this:
val (first, second, third) = (
model.firstMethod(),
model.secondMethod(),
model.thirdMethod()
)
You likely have some other problem with your code.
I can run 100 million loops where it never gets the order wrong, as follows:
class Model {
var done = Array(false,false,false);
def firstMethod():Boolean = { done(0) = true; done(1) || done(2) };
def secondMethod():Boolean = { done(1) = true; !done(0) || done(2) };
def thirdMethod():Boolean = { done(2) = true; !done(0) || !done(1) };
};
Notice that these methods return a True if done out of order and false when called in order.
Here's your class:
class MyWrappingClass {
val model = new Model;
val first = model.firstMethod()
val second = model.secondMethod()
val third = model.thirdMethod()
};
Our function to check for bad behavior on each trial:
def isNaughty(w: MyWrappingClass):Boolean = { w.first || w.second || w.third };
A short program to test:
var i = 0
var b = false;
while( (i<100000000) && !b ){
b = isNaughty(new MyWrappingClass);
i += 1;
}
if (b){
println("out-of-order behavior occurred");
println(i);
} else {
println("looks good");
}
Scala 2.11.7 on OpenJDK8 / Ubuntu 15.04
Of course this doesn't prove it impossible to have wrong order, only that correct behavior seems highly repeatable in a fairly simple case.
I have a file like below:
0; best wrap ear market pair pair break make
1; time sennheiser product better earphone fit
1; recommend headphone pretty decent full sound earbud design
0; originally buy work gym work well robust sound quality good clip
1; terrific sound great fit toss mine profuse sweater headphone
0; negative experienced sit chair back touch chair earplug displace hurt
...
and i want to extract number and store it in a for each document, i've tried :
var grouped_with_wt = data.flatMap({ (line) =>
val words = line.split(";").split(" ")
words.map(w => {
val a =
(line.hashCode(),(vocab_lookup.value(w), a))
})
}).groupByKey()
expected output is :
(1453543,(best,0),(wrap,0),(ear,0),(market,0),(pair,0),(break,0),(make,0))
(3942334,(time,1),(sennheiser,1),(product,1),(better,1),(earphone,1),(fit,1))
...
after generating above results i used them in this code to generate final results:
val Beta = DenseMatrix.zeros[Int](V, S)
val Beta_c = grouped_with_wt.flatMap(kv => {
kv._2.map(wt => {
Beta(wt._1,wt._2) +=1
})
})
final results:
1 0
1 0
1 0
1 0
...
This code doesn't work well , Can anybody help me? I want a code like above.
val inputRDD = sc.textFile("input dir ")
val outRDD = inputRDD.map(r => {
val tuple = r.split(";")
val key = tuple(0)
val words = tuple(1).trim().split(" ")
val outArr = words.map(w => {
new Tuple2(w,key)
})
(r.hashCode, outArr.mkString(","))
})
outRDD.saveAsTextFile("output dir")
output
(-1704185638,(best,0),(wrap,0),(ear,0),(market,0),(pair,0),(pair,0),(break,0),(make,0))
(147969209,(time,5),(sennheiser,5),(product,5),(better,5),(earphone,5),(fit,5))
(1145947974,(recommend,1),(headphone,1),(pretty,1),(decent,1),(full,1),(sound,1),(earbud,1),(design,1))
(838871770,(originally,4),(buy,4),(work,4),(gym,4),(work,4),(well,4),(robust,4),(sound,4),(quality,4),(good,4),(clip,4))
(934228708,(terrific,5),(sound,5),(great,5),(fit,5),(toss,5),(mine,5),(profuse,5),(sweater,5),(headphone,5))
(659513416,(negative,-3),(experienced,-3),(sit,-3),(chair,-3),(back,-3),(touch,-3),(chair,-3),(earplug,-3),(displace,-3),(hurt,-3))
Looking to convert Java Swing DatePicker into Scala but facing difficulty in one area of the code. How should I probably translate the if (x > 6) part into scala?
Original Java taken from http://www.roseindia.net/tutorial/java/swing/datePicker.html
for (int x = 0; x < button.length; x++) {
final int selection = x;
button[x] = new JButton();
button[x].setFocusPainted(false);
button[x].setBackground(Color.white);
if (x > 6)
button[x].addActionListener(new ActionListener() {
public void actionPerformed(ActionEvent ae) {
day = button[selection].getActionCommand();
d.dispose();
}
});
if (x < 7) {
button[x].setText(header[x]);
button[x].setForeground(Color.red);
}
p1.add(button[x]);
}
Converted Scala
for (x <- 0 until buttons.length) {
val selection = x
buttons(x) = new Button {
focusPainted = false
background = Color.white
}
if (x > 6)
buttons(x).reactions += {
case ButtonClicked(_) => {
day = buttons(selection).action
d.dispose()
}
}
if (x < 7) {
buttons(x).text = header(x)
buttons(x).foreground = Color.red
}
contents += buttons(x)
}
What is wrong with your translation? Is it not working? The only thing I can see at a glance is that you do not listen to the button:
button(x) listenTo button(x)
But I'm not sure how wise a button listening to itself is, or whether there are any nasty consequences. You don't need to add the reactions to the button itself, you could probably add them to the date picker itself.
Also, something like zipWithIndex is possibly preferable to the way you have done it:
buttons.zipWithIndex foreach { case (button, x) =>
//no need to use buttons(x)
}
The if(x < 7) could be replaced by else, that would be clearer.
You could use a match statement as well:
x match {
case xx if xx > 6 => ...
case _ => ...
}