long time=System.currentTimeMillis();
MarketDataRequest request=new MarketDataRequest();
System.out.println("First: " +(System.currentTimeMillis()-time));
time=System.currentTimeMillis();
request=new MarketDataRequest();
System.out.println("Second: "+ (System.currentTimeMillis()-time));
Result:
First: 43
Second: 0
43ms seems very slow...any reason why the first creation of the object takes sooo long?
Two possible causes are Java class loading and JIT compiler overhead.
Related
I'm using Chronicle Map extensively in Scala and recently decided to experiment with Kryo serialisation. I added custom marshallers (code below) and while it reduced the size of my stores by 14G (roughly 62%), and everything still works, the speed was unacceptable.
I created a sample usecase and did a few runs on the same data
[Using kryo] took 6883, and then 7187, 6954, 7225, 13051
[Not using kryo] took 2326, and then 1352, 1493, 1500, 1187
So it's several times slower. Here's the marshaller for reading:
class KryoMarshallerReader[T] extends BytesReader[T] {
val kryo = // Reference to KryoPool from Twitter's Chill library
override def read(in: Bytes[_], using: T): T = {
val bytes = benchmark("array allocation") {
new Array[Byte](in.readRemaining().toInt)
}
benchmark("reading bytes") {
in.read(bytes)
}
benchmark("deserialising") {
kryo.fromBytes(bytes).asInstanceOf[T]
}
}
override def readMarshallable(wire: WireIn): Unit = {}
override def writeMarshallable(wire: WireOut): Unit = {}
}
I then averaged execution times (all in ms) over those three stages and realised that reading the bytes is the slowest:
stage Average time (ms)
(fctr) (dbl)
1 [array allocation] 0.9432907
2 [deserialising] 0.9944112
3 [reading bytes] 13.2367265
Now the question is - what am I doing wrong?
I looked through the interface of Bytes[_] and it looks like it reads bytes one by one - is there a way to use a buffer or something magically capable of bulk loading?
Update: Eventually I changed array allocation + reading bytes to in.toByteArray but it's still slow because under the hood it copies bytes one by one. Just running reads on the map shows that byte reading is the bottleneck:
in.readRemaining() of the Bytes, passed into BytesReader.read(), is not your object in serialized form, it's more than that. The serialized form of your object is guaranteed to start from in.readPosition(), but it typically end much earlier than in.readLimit() (as readRemaining() = readLimit() - readPosition()). Generally BytesReader/BytesWriter pair of implementations should care of determining the end of the object bytes itself (if it needs), e. g. see the implementation of CharSequenceArrayBytesMarshaller in BytesReader and BytesWriter section of the Chronicle Map tutorial:
public final class CharSequenceArrayBytesMarshaller
implements BytesWriter<CharSequence[]>, BytesReader<CharSequence[]> {
...
#Override
public void write(Bytes out, #NotNull CharSequence[] toWrite) {
out.writeInt(toWrite.length); // care about writing the size ourselves!
...
}
#NotNull
#Override
public CharSequence[] read(Bytes in, #Nullable CharSequence[] using) {
int len = in.readInt(); // care about reading the size ourselves!
...
}
}
But since your are implementing Kryo serialization which should be conceptually similar to Java standard serialization, you should probably take the source of SerializableReader and SerializableDataAccess and modify it to use Kryo instead of standard Java serialization (but note that those sources are LGPLv3 licensed). In particular those implementations use Bytes.inputStream() and Bytes.outputStream() to bridge the standard Java serialization, which doesn't know about Bytes, but knows about InputStream/OutputStream, without needlessly copying bytes. I'm pretty sure Kryo supports InputStream/OutputStream as well.
Be very careful with kryo as an instance field of any serializer interface (BytesReader in your case) without implementing StatefulCopyable. You may easily introduce concurrency bottlenecks, or concurrency bugs (data races). Check Understanding StatefulCopyable section in the Chronicle Map tutorial and Chronicle Map custom serialization checklist.
I'm completely new to Lift. My goal is to make a small application which will let me enter data about scientific articles (studies) into a database.
I haven't gotten around to making a database yet, still playing with getting an entry form to work. So I decided that I will hold a few studies in memory, in a list of a companion object to the Study class. I also created an accumulator variable to dispense unique IDs, as long as I don't have a DBMS managing the IDs.
As a smoke test, I visited the page showing the list of studies (seeded with two studies in code), then visited the form page, entered a new study, and navigated again to the list of studies. I was surprised to see that my new study has the ID of 1 instead of 3, so at some point, my accumulator variable must have been reset. But the ListBuffer collecting studies was not reset, because it showed all three studies. Adding more studies results in the counter incrementing by 1 every time.
The literature I have found on Lift (the two free books on Liftweb, as well as Gilberto Garcia's Lift Application Development Cookbook) are incomplete and are more like a collection of mini-tutorials, they don't explain how Lift works.
So what is the actual lifecycle of my Study object, and why did one mutable variable get reset after re-opening the page but not another?
package org.rumtscho.litsuche
package study
import scala.collection.mutable.ListBuffer
class Study private[study](
val id: Int,
val handmadeAuthorRef: String,
val humanReadableDescription: String )
{
}
object Study {
val seedStudies = List(
new Study(dispenseNextFreeId, "Brooks1975", "Tells us that throwing more programmers at a project makes it run late, not early"),
new Study(dispenseNextFreeId, "Dijkstra1968", "Recognizes that we need structured code")
)
private var studiesList = seedStudies.to[ListBuffer]
private var currentId = 0
private def dispenseNextFreeId: Int = {
currentId = currentId + 1
return currentId
}
def allStudies = studiesList.toList
def addStudy(reference: String, description: String): Unit = {
studiesList += new Study(dispenseNextFreeId, reference, description)
}
}
Here is the representation of the three studies:
update My understanding of what is happening (could be wrong, of course):
I open the page showing the list of studies. This calls allStudies. studiesList is initialized to contain Brooks1975 and Dijkstra1968. During the construction of the studies, currentId is initialized to 0 and then incremented to 2.
I open the entry form and add a third study. addStudy retrieves allStudies from memory, without initializing it again. It initializes currentId to 0. I end up with a third study with the ID 1.
I display all studies, then return to the entry form page. This time, addStudy retrieves both allStudies and currentId from memory. I get a fourth study with the ID of 2.
The comments have pointed out that this is probably Scala-specific and not related to Lift. But still, I don't understand why currentId is initialized two times (in steps 1 and 2), and not either once (as soon as the object itself is created) or every time it is read. I would have expected the first behavior, but even reinitializing every time seems more logical than randomly reinitializing one time only.
Go into the scala REPL, enter paste mode (:paste) command, and put in the following:
def increment {
currentId = currentId + 1
}
increment
increment
var currentId = 0
then try
var currentId = 0
def increment {
currentId = currentId + 1
}
increment
increment
In the first example, currentId ends up with value 0. In the second, it ends up with value 2. Why does this happen? I'm not an expert on Scala declaration, but it seems that this is the same problem you are running in to.
It seems that the solution is as #jcern suggests. In general, I'd say put all your declarations at the top of your classes or objects, and always declare before using a variable, and not the other way around.
I wrote a TimeFormatter to parse a String to an Option[LocalDateTime].
The API notes that either exception could be thrown.
private def convertToDateTime(date: String): Option[LocalDateTime] =
try {
Some( timeFormatter.parseLocalDateTime(date) )
}
catch { // DateTimeFormatter#parseLocalDateTime API throws these possible exceptions
case e # (_: IllegalArgumentException | _: UnsupportedOperationException) =>
log.error(s"Could not convert $date to LocalDateTime", e); None
}
Joshua Bloch notes:
Item 39: Use exceptions only for exceptional conditions.
I thought of using a regular expression to catch an error. But, I'm not sure if my reg-ex will always match jodatime's way of parsing the String to a LocalDateTime. I can trust that the API will throw those exceptions, but I'd rather not rely upon internals of the method call with a reg-ex.
From a functional point of view, I'd rather not use exceptions.
How can I write this function without exceptions?
There is nothing wrong with your code : you are not using exceptions to model non-exceptional conditions. You are handling conversion errors and turning them into a value to be returned which is perfectly acceptable.
Jodatime is using exceptions to signal invalid input (maybe not so exceptional) or the unavailability of a part of its API (completely exceptional), this is common practice in the Java world.
To actually parse dates without exceptions entirely you would need to find or reimplement a date handling library which is quite a huge endeavour.
A silent alternative to your method:
private def convertToDateTime(date: String): Option[LocalDateTime] =
Try(timeFormatter.parseLocalDateTime(date)).toOption
I have experimented with Joda-Time and found this half solution avoiding runtime exceptions even in case of wrong input:
String input = "2015-02-a29 15:24:33";
DateTimeFormatter dtf = DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss");
DateTimeParser parser = dtf.getParser();
Chronology chronology = ISOChronology.getInstanceUTC();
DateTimeParserBucket bucket =
new DateTimeParserBucket(0L, chronology, Locale.getDefault(), 2000, 2000);
int result = parser.parseInto(bucket, input, 0);
System.out.println(result);
if (result < 0) {
System.out.println("Parsing failed at position: " + ~result);
} else {
System.out.println(new LocalDateTime(bucket.computeMillis(), chronology));
}
The output is for the given input:
-9
Parsing failed at position: 8
This is in agreement with the javadoc. However, the bad news is that this solution is not perfect. If you use the input 2015-02-29 15:24:33 (an invalid date in non-leap-year) then you get an exception because Joda-Time does not recognize in the method parseInto(...) that the parsed day-of-month is out of range and unfortunately sees it too late, namely in the method computeMillis(). The output is here:
19 // should have been negative - probably a bug!!!
Exception in thread "main" org.joda.time.IllegalFieldValueException: Value 29 for dayOfMonth must be in the range [1,28]
at org.joda.time.field.FieldUtils.verifyValueBounds(FieldUtils.java:217)
at org.joda.time.field.PreciseDurationDateTimeField.set(PreciseDurationDateTimeField.java:78)
at org.joda.time.format.DateTimeParserBucket$SavedField.set(DateTimeParserBucket.java:483)
at org.joda.time.format.DateTimeParserBucket.computeMillis(DateTimeParserBucket.java:365)
at org.joda.time.format.DateTimeParserBucket.computeMillis(DateTimeParserBucket.java:318)
at JodaPeriod.main(JodaPeriod.java:31)
Of course, for the valid input 2015-02-28 15:24:33 all is fine, and the output is
19
2015-02-28T15:24:33.000
I agree with Joshua Bloch that catching exceptions should only be used in exceptional situations. In my company architects say: Don't use exceptions for programming logic. Sometimes it is good for performance to avoid exceptions when ever possible in use-cases like processing a lot of possibly wrong data. A good API should therefore support similar solutions.
Note that other date-and-time-libraries have this option, too. For example java.text.SimpleDateFormat using an instance of class ParsePosition (there the API is even much simpler and smoothly works - although not thread-safe).
You could use the Exception api in scala.control to transform exceptions:
Your function would look something like this:
import scala.util.control.Exception._
private def convertToDateTime(date: String): Option[LocalDateTime] =
catching(classOf[Exception]) opt timeFormatter.parseLocalDateTime(date)
I am trying to do some processing on a SynchronizedQueue using a tail recursive function. The function seems to work properly but the more I think about concurrency the more I believe I could have some race conditions when accessing this queue with different threads. Here is the function that I think I could use some help with:
val unsavedMessages = new SynchronizedQueue[CachedMessage]()
val MAX_BATCH = 256
val rowCount = new AtomicInteger()
private def validateCacheSize() = if (unsavedMessages.length > MAX_BATCH) {
implicit val batch = createBatch
val counter = rowCount.getAndIncrement
#tailrec
def processQueue(queue: SynchronizedQueue[CachedMessage]): Unit = if (queue.nonEmpty) {
val cm = queue.dequeue
addToBatch(cm.request, cm.timestamp, cm.brokerId, counter)
processQueue(queue)
}
processQueue(unsavedMessages)
executeBatch
resetQueue
}
def resetQueue = unsavedMessages.clear
Multiple threads call this function:
def add(request: WebserviceRuleMatch, timestamp: Long, brokerId: String) = {
validateCacheSize
//log.info("enquing request "+ unsavedMessages.length)
unsavedMessages.enqueue(CachedMessage(request, timestamp, brokerId))
}
Does anyone have any pointers on how to improve this so there would likely not be a race condition?
there could be a chance that the queue gets emptied between queue.nonempty and queue.dequeue
Avoid calling multiple queue operations that must be synchronized within your code. Use the power of SynchronizedQueue to do atomic thread-safe operations. E.g. avoid calling queue.nonempty altogether (alternative to tail-recursion):
for (cm <- unsavedMessages.dequeueAll(_ => true))
addToBatch(cm.request, cm.timestamp, cm.brokerId, counter)
executeBatch
//resetQueue -- Don't do this! Not thread-safe
I think messages could be added by a thread between processQueue and resetQueue
There will always be a point at which your code has taken a 'snapshot' of the queue and emptied it. My previous point ensured that the 'snapshot' and emptying are a single atomic operation. If new entries are enqueued at any point after that atomic 'snapshot & empty' operation - no problem. Your 'snapshot & empty' must occur somewhere and new items enqueued are a fact of life. Make the decision to allow new items to be enqueued at any point subsequent to the 'snapshot & empty'. They'll be processed on next cycle. i.e. nothing extra needed beyond above point.
Robin Green: (By the way, that method seems to have a very misleading name!)
Wot he said! :)
The add function gets gets called from a future so I feel as though there could be a chance that the queue gets emptied between queue.nonempty and queue.dequeue.
Yes, it could. You could use double-checked locking to make validateCacheSize single-threaded. (By the way, that method seems to have a very misleading name!)
Also I think messages could be added by a thread between processQueue and resetQueue.
Yes, they could. But why do you need to call unsavedMessages.clear at all? queue.dequeue already removes them from the queue. So the only unsavedMessages that should exist in the queue then are ones that still remain to be processed.
I'm writing some special purpose data structures in Java, intended for use in the browser, (compiled to JavaScript with GWT).
I'm trying to match the performance of some of the built-in JDK classes I'm noticing things run reasonably fast, but when I compare my code trace to some of the emulated JDK code, mine has lots of calls to dynamicCast and canCastUnsafe, while the JDK emulated classes do not. And it just about accounts for the difference in performance too...
Any GWT gurus out there know how to avoid this? It's amounting to a 20% overhead :-(
Details:
Here's the profile output (captured in Firebug) for 10,000 insertions of random integers, between 0 and 100,000 into two different data structures:
Google's TreeMap implementation for java.util.TreeMap (a red-black tree):
Profile (4058.602ms, 687545 calls)
Function Calls Percent Own Time
$insert_1 129809 41.87% 1699.367ms
$compare_0 120290 16% 649.209ms
$isRed 231166 13.33% 540.838ms
compareTo_0 120290 8.96% 363.531ms
$put_2 10000 6.02% 244.493ms
wrapArray 10000 3.46% 140.478ms
createFromSeed 10000 2.91% 118.038ms
$TreeMap$Node 10000 2.38% 96.706ms
initDim 10000 1.92% 77.735ms
initValues 10000 1.49% 60.319ms
$rotateSingle 5990 0.73% 29.55ms
TreeMap$Node 10000 0.47% 18.92ms
My Code (An AVL tree):
Profile (5397.686ms, 898603 calls)
Function Calls Percent Own Time
$insert 120899 25.06% 1352.827ms
$compare 120899 17.94% 968.17ms
dynamicCast 120899 14.12% 762.307ms <--------
$balanceTree 120418 13.64% 736.096ms
$setHeight 126764 8.93% 482.018ms
compareTo_0 120899 7.76% 418.716ms
canCastUnsafe 120899 6.99% 377.518ms <--------
$put 10000 2.59% 139.936ms
$AVLTreeMap$Node 9519 1.04% 56.403ms
$moveLeft 2367 0.36% 19.602ms
AVLTreeMap$State 9999 0.36% 19.429ms
$moveRight 2378 0.34% 18.295ms
AVLTreeMap$Node 9519 0.34% 18.252ms
$swingRight 1605 0.26% 14.261ms
$swingLeft 1539 0.26% 13.856ms
Additional observations:
Same problem for another data structure I made (SkipList).
dynamicCast is being applied in the compare function:
cmp = dynamicCast(right.key, 4).compareTo$(key);
dynamicCast goes away if the class does not implement Map (ie: just removing " implements Map" from the class. Doesn't matter if it's accessed through the interface or directly. This results in the same line compiling to:
cmp = right.key.compareTo$(key);
This is the relevant section of Java source from SkipList:
private int compare(Node a, Object o) {
if (comparator != null)
return comparator.compare((K) a.key, (K) o);
return ((Comparable<K>) a.key).compareTo((K) o);
}
public V get(Object k) {
K key = (K) k;
Node<K, V> current = head;
for (int i = head.height - 1; i >= 0; i--) {
Node<K, V> right;
while ((right = current.right[i]) != null) {
int cmp = compare(right, key);
...
}
}
}
Unfortunately I'm still not exactly clear on the cause, but from my experience, it seems from explicit casts, like:
((Comparable) obj).compareTo(other)
The Javascript generated looks like:
dynamicCast(obj, 1).compareTo(other);
Where 1 is a generated typeId representing the target of the cast. dynamicCast in turn calls canCastUnsafe and if false, it throws a ClassCastException. The value of this has been debated, since this would already be caught in hosted mode.
It can be sidestepped with JSNI:
public static native int compare(Object a, Object b) /*-{
return a.#java.lang.Comparable::compareTo(Ljava/lang/Object;)(b);
}-*/;
Dunno if you've seen this thread in the GWT Contributor's forum...
Basically, it starts with the same problem you've identified, proposes some new compiler flags, and goes on to show how to use some JSNI to get around the casts.
Edit In the GWT trunk there's a new compiler flag. See the wiki...
An updated answer for GWT version 2.1 and later:
Since GWT 2.1 (at least that's the first mention), the GWT compiler has a new compiler argument called -XdisableCastChecking that disables all runtime checking of casts.
Note, this option is marked as experimental (probably because this would make class cast exceptions very hard to debug).
In my app dynamicCast was called thousands of times in a short profile run, and were the the 3rd most time consuming method in the Firebug profiler. Using this compiler argument significantly reduced the number of "Long Duration Events" messages in the Chrome Speed Tracer.
See GWT Compiler Options for this and other Compiler arguments.
It's definitely a compiler problem: I have the problem on the following line:
final DefaultIconedSuggestBox<SuggestValueProxy, IconedValueHolderItem<SuggestValueProxy>> fieldValueWidget = getCategoryWidget().getFieldValueWidget();
I don't really know how I can workaround it: this line happens in a moment I'm changing from a module to another (it is maybe related to the code splitter issue: even though I'm not using code split: I'm just loading another page with another module)
Does the use of java 1.5 generics and wildcards could avoid this ?