I'm using the Codahale metrics in my Scala application, but don't seem to see a way to get the last value for a timed operation. I can get mean, median, and percentiles - all great stuff from the histogram - but I just want to know how long the last operation took!
Look at entry zero in the snapshot and scale it from the value in nanoseconds down to milliseconds? That doesn't seem to be right...
I must be missing the obvious!
t.time {
iService.availCheck(q)
} match {
case ReturnInfo(Some(true)) =>
// The HUB says TRUE, this name is available. That's gold.
logger.debug("Avail check for (" + q + ") HUB TRUE: " + (t.snapshot.getValues()(0) / 100000.0).toString + "ms")
Ok(Some(true))
}
Found the answer - the stop() function, embedded in the t.time call, returns the elapsed time. I just had to refactor my code to have access to it. If not consumed, it's lost. Poof.
Go figure ;)
Related
I am searching for GitHub files containing the string "torch." Since, the search API limits searches to the first 100 results, I am searching based on file sizes as suggested here. However, I keep hitting the secondary rate limit. Could someone suggest if I am doing something wrong or if there is a way to optimize my code to prevent these rate limits? I have already looked at best practices to deal with rate limits. Here is my code -
import os
import requests
import httplink
import time
# This for loop searches for code based on files sizes from 0 to 500000 containing the string "torch"
for i in range(0,500000,250):
print("i = ",i," i + 250 = ", i+250)
url = "https://api.github.com/search/code?q=torch +in:file + language:python+size:"+str(i)+".."+str(i+250)+"&page=1&per_page=10"
headers = {"Authorization": f'Token xxxxxxxxxxxxxxx'} ## Please put your token over here
# Backoff when secondary rate limit is reached
backoff = 256
total = 0
cond = True
# This while loop goes over all pages of results => Pagination
while cond==True:
try:
time.sleep(2)
res = requests.request("GET", url, headers=headers)
res.raise_for_status()
link = httplink.parse_link_header(res.headers["link"])
data = res.json()
for i, item in enumerate(data["items"], start=total):
print(f'[{i}] {item["html_url"]}')
if "next" not in link:
break
total += len(data["items"])
url = link["next"].target
# Except case to catch when secondary rate limit has been reached and prevent the computation from stopping
except requests.exceptions.HTTPError as err:
print("err = ", err)
print("err.response.text = ", err.response.text)
# backoff **= 2
print("backoff = ", backoff)
time.sleep(backoff)
# Except case to catch when the given file size provides no results
except KeyError as error:
print("err = ", error)
# Set cond to False to stop the while loop
cond = False
continue
Based on this answer, it seems like it is a common occurrence. However, I was hoping someone could suggest a workaround.
I have added the tag Octokit, although I am not using that, to increase visibility and since this seems like a common problem.
A big chunk of the above logic/code was obtained through SO answers, I highly appreciate all support from the community.
Note that search has its primary and secondary rate limiting that is lower than others. For JavaScript, we have a throttle plugin that implements all the recommended best practices. For search we limit requests to 1 per 2 seconds. Hope that helps!
I am trying to get EDS spectra on every scanned pixel using STEM.
I am using the EDSStartAcquisition( 2048, 10,fexposure*2, 1) command and
I have attached the following simple listener object into the shown 1D spectrum image:
string messagemap = "data_value_changed:MyImageAction"
Class MyListenerClass1
{
String event_desc;
MyListenerClass1(Object self); //Result("\n");
~MyListenerClass1(Object self);// Result("\n");
Void MyImageAction(Object self, Number e_fl, Image Img)
{
ImageGetEventMap().DeconstructEventFlags( e_fl, event_desc )
Result(GetTime(1)+": Image message : " + event_desc + " 0x" + Binary(e_fl) + "\n" )
}
}
ListenerID1 = EDSIm.ImageAddEventListener( Listener1, messagemap)
Since speed is the issue here, I figured to try the continuous mode of the EDS acquisition. But then I would need to listen which counts belong to each scanned pixel. The following topic (How to getting acquired frames at full speed ? - Image Event Listener does not seem to be executing after every event) shows how to listen to the last pixel change of an image. But what would be the fastest way to directly see which slice of the 1D spectrum has changed on every event? Without going through every slice...
thanks in advance!
An images' data_value_changed is fired whenever a) an image locker (the object that ensures there is single access to the memory) is released, or b) a specific update-call is made in the code.
As such, when a cumulative EDS spectrum is acquired, the whole array gets "locked", then modified (on one or more positions) and then 'updated'. There is no specific information carried on where the array was modified.
Therefore, the only way to find out where the spectrum changed is by comparing a copy of the "before" with the "now" - which is not super efficient.
I'm new to Kafka Streams.
I use the suppress method of KTable in order to handle only the final result of a window like this:
myStream
.windowedBy(TimeWindows.of(Duration.ofSeconds(10)).grace(Duration.ofMillis(500)))
.aggregate(new Aggregation(),
(k, v, a) -> a, // Disabled the actual aggregation in order to eliminate possiblities of latency
materialized.withLoggingDisabled())
.suppress(untilWindowCloses(Suppressed.BufferConfig.unbounded()))
.toStream().peek((k, v) -> log.info("delay " + (System.currentTimeMillis() - k.window().endTime().toEpochMilli())));
This way I get a log with the delay every 10 seconds with the difference between the window end and the actual time the peek was called.
I would exect a very small number here, since this code practically does nothing...
Nevertheless, I get delay of 4-20 sec for each key/window.
I use a thread per task (5 threads for this topic).
Can someone please point out if I'm doing anything wrong?
Thanks!
Edit:
Using VirtualVM shows that ~99% of the time consumed over sun.nio.ch.SelectorImpl.select(). This means AFAIU, that the process is "idle" most of the time.
Edit:
It seems that changing "commit.interval.ms" (which was by default 30000) reduced the delay drastically.
Still delay has peaks of event 15 seconds, so the problem isn't solved yet...
I'm trying to use delay and amb to execute a sequence of the same task separated by time.
All I want is for a download attempt to execute some time in the future only if the same task failed before in the past. Here's how I have things set up, but unlike what I'd expect, all three downloads seem to execute without delay.
Observable.amb([
Observable.catch(redditPageStream, Observable.empty()).delay(0 * 1000),
Observable.catch(redditPageStream, Observable.empty()).delay(30 * 1000),
Observable.catch(redditPageStream, Observable.empty()).delay(90 * 1000),
# Observable.throw(new Error('Failed to retrieve reddit page content')).delay(10000)
# Observable.create(
# (observer) ->
# throw new Error('Failed to retrieve reddit page content')
# )
]).defaultIfEmpty(Observable.throw(new Error('Failed to retrieve reddit page content')))
full code can be found here. src
I was hoping that the first successful observable would cancel out the ones still in delay.
Thanks for any help.
delay doesn't actually stop the execution of what ever you are doing it just delays when the events are propagated. If you want to delay execution you would need to do something like:
redditPageStream.delaySubscription(1000)
Since your source is producing immediately the above will delay the actual subscription to the underlying stream to effectively delay when it begins producing.
I would suggest though that you use one of the retry operators to handle your retry logic though rather than rolling your own through the amb operator.
redditPageStream.delaySubscription(1000).retry(3);
will give you a constant retry delay however if you want to implement the linear backoff approach you can use the retryWhen() operator instead which will let you apply whatever logic you want to the backoff.
redditPageStream.retryWhen(errors => {
return errors
//Only take 3 errors
.take(3)
//Use timer to implement a linear back off and flatten it
.flatMap((e, i) => Rx.Observable.timer(i * 30 * 1000));
});
Essentially retryWhen will create an Observable of errors, each event that makes it through is treated as a retry attempt. If you error or complete the stream then it will stop retrying.
I'm building an application to record when my cat has an asthma attack. I'm not interested in the exact time since glancing at the time in interval of 15 minutes is easier to review (e.g. rounding 9:38am should be recorded as 9:45am).
I looked for a UDF at cflib.org for this but couldn't find one. I tinkered with CF's round function but I'm not getting it to do what I want.
Any advice?
This could do with a bit more polish (like data type validation) but it will take a time value and return it rounded to the nearest 15-minute increment.
<cfscript>
function roundTo15(theTime) {
var roundedMinutes = round(minute(theTime) / 15 ) * 15;
var newHour=hour(theTime);
if (roundedMinutes EQ 60) {
newHour=newHour + 1;
roundedMinutes=0;
}
return timeFormat(createTime(newHour,roundedMinutes,0),"HH:mm");
}
</cfscript>
I'm not familiar with the format of the timestamp here, but generally when I want to round I do something like floor((val + increment / 2) / increment) * increment
I like Al Everett's answer, or alternatively store the actual time to preserve the most accurate time, then use query of query in the view and use between :00 and :15 to show the time in 15min period.
If you use Henry's suggestion to store the precise time in the database (principle: if there's no cost, prefer to preserve data), then you can simply use Al Everett's rounding function whenever you display the data to the user.