I have a large file I need to load into the cache (As a hash) which will be shared between perl processes. Loading the data into cache takes around 2 seconds, but we have over 10 calls per second.
Does using the compute method cause other processes to be locked out?
Otherwise, I would appreciate suggestions on how to manage the load process so that there's a guaranteed lock during load and only one load process happening!
Thanks!
Not sure about the guaranteed lock, but you could use memcached with a known key as a mutex substitute (as of this writing, you haven't said what your "cache" is). If the value for the key is false, set it to true, start loading, and then return the result.
For the requests happening during that time, you could either busy-wait, or try using a 503 "Service Unavailable" status with a few seconds in the Retry-After field. I'm not sure of the browser support for this.
As a bonus, using a time-to-live of, say, 5 minutes on the mutex key will cause the entire file to be reloaded every 5 minutes. Otherwise, if you have a different reason for reloading the file, you will need to manually delete the key.
Related
In order to not overload our database server we are trying to flush each server with a 60 second delay between them. I'm having a bit of issue determining when a server was actually flushed when a delay is given.
I'm using BeITMemcached and calling the FlushAll with a 60 second delay and staggered set to true.
I've tried using command line telnet host port followed by stats to see if the flush delay is working, however when I look at the cmd_flush the value goes up instantly on all of the host/port combinations being flushed without a delay. I've tried stats items and stats slabs but can't find information on what all the values represent and if there is anything that shows that it has been invalidated.
Is there another place I can look to determine when the server was actually flushed? Or does that value going up instantly mean that the delay isn't working as expected?
I found a round about way of testing this. Even though the cmd_flush gets updated right away the actual keys don't until after the delay.
So I connected with telnet to the server/port I wanted to monitor. Then used gets key to find a key with a value set. Once found I ran the flushall with a delay between the first servers and this one and continued to monitor that key value. After the delay was up the key started to return no value.
Doing a regression on a large dataset, I have a huge read-only matrix that I'd like to share among several threads. I've looked at various way of doing this and found sharedmatrix toolkit to be exactly what I need. Reading through the tutorial, I came up with the following setup:
Session 0 - just loads up the matrix and makes it available
Session 1..n - worker sessions
The problem is that Session 0 has to finish only after all the other n sessions have finished. Do you know how to make a session wait? The best solution would be to make it wait until I kill it as I'm running the scripts on a remote linux system and am not connected to it all the time.
UPDATE:
In the end I've changed my approach to the problem, after reading this part of the tutorial:
The “free” directive marks the shared memory segment for deletion.
Note: it is not actually deleted until every attached session
explicitly detaches or is terminated. As soon as the last session
detaches, the system will reclaim the allocated segment.
This means that I've created one "master" session that loads up the matrix, makes it available and then starts its own computation, and several "slave" sessions that use the shared matrix. Even if the master session finishes earlier, it causes no problem to the slave sessions as the shared matrix remains in the memory until the last process that uses it is terminated.
If you really want to have it wait indefinitely, use Eitan's solution based on pause. More generally, something like labBarrier is what you should use to perform this kind of synchronization.
I'm starting to integrate libmemcached into my application and reading the documentation, there is a non-blocking mode flag. After a quick google, there seems to be a performance advantage to non blocking mode, but are there any disadvantages to running libmemcached in non blocking mode?
Of course there are. The disadvantage would only arise if you needed to ENSURE that the written value actually was written to memcached and did not fail. For example - you're using memcached to store a counter variable which has a sentinel that checks to see if the counter has reached a certain value before performing an operation.
In blocking mode - the memcached client will wait to get a write success response from memcached before proceeding and produce an error if it fails. This way you know the counter was updated. If you tell it to write in non-blocking mode, the client sends the request to increment the counter, but never waits to ensure that it really occurred. Because it doesn't wait, you code execution after the call resumes more quickly, but with the uncertainty of not knowing for sure the counter was ever incremented.
However, since memcached values are destroyed on a service restart (think system crash) you can't ever really be sure a value will be there. Also, with low-memory pruning you also cannot ever be sure the value is 100% correct as it may get pruned by the LRU algorithm - you'd need persistent storage to alleviate this uncertainty.
Given this inherent uncertainty, many people use non-blocking mode to get the performance gain because they can't ever be totally certain the counter value in memcached isn't reset/innacurate anyway, so why not get some performance for the tradeoff.
Hope this clarifies the issue. As a side note - MongoDB has non-blocking writes in persistent storage - which while awesome in its flexibility, gives people using non-blocking mode more of a false sense of security that the write will always succeed...
R
I've been using Proc::Daemon in an attempt to make a start/stop daemon script, something allows me to do:
X start
X stop
X status
etc. However, in the source code it looks like that Proc::Daemon uses either a "pid" file, or a search of the process table. I'm concerned with both of these approaches, firstly as "pid"s are reused, which may give the impression a service is up when it's actually down, and secondly that process table entries are easily faked, and the checking doesn't look particularly robust.
Is there any robust way to make a start/stop daemon script/program like I've described, or has someone already made one? Note that I haven't got root access, and I'm also on Solaris if that's important.
Although pids are reused, I believe that they round-robin through a (large) fixed size set. e.g. on Solaris this used to be 30,000 (it may be different now). So 30,000 processes would have to start/finish before your pid was reused.
The approach used by Proc::Daemon doesn't look unreasonable and is a fairly common approach to this problem.
An approach I use is to have the daemon process obtain an exclusive (write) lock on a file.
You can test to see if anyone is holding the lock by trying to obtain the lock yourself, and there are various ways of obtaining the PID of a process holding a lock on a file - i.e. fcntl and probably something in /proc.
Some words of advice:
Use local files (ie. not NFS) for locks.
Make sure the lock file exist before the daemon is started.
Never delete the lock file.
The kernel associates locks with the inode number of the file, so you always want the lock file to have the same inode number throughout all time. Deleting and recreating the lock file will change the inode associated with the lock.
A simple keep alive mechanism can be implemented as a cron job - the cron job just tries to spawn the daemon process every N minutes, and then have the daemon quietly exit if it can't obtain the exclusive lock.
I'm working on a multiplayer game and it needs a message queue (i.e., messages in, messages out, no duplicates or deleted messages assuming there are no unexpected cache evictions). Here are the memcache-based queues I'm aware of:
MemcacheQ: http://memcachedb.org/memcacheq/
Starling: http://rubyforge.org/projects/starling/
Depcached: http://www.marcworrell.com/article-2287-en.html
Sparrow: http://code.google.com/p/sparrow/
I learned the concept of the memcache queue from this blog post:
All messages are saved with an integer as key. There is one key that has the next key and one that has the key of the oldest message in the queue. To access these the increment/decrement method is used as its atomic, so there are two keys that act as locks. They get incremented, and if the return value is 1 the process has the lock, otherwise it keeps incrementing. Once the process is finished it sets the value back to 0. Simple but effective. One caveat is that the integer will overflow, so there is some logic in place that sets the used keys to 1 once we are close to that limit. As the increment operation is atomic, the lock is only needed if two or more memcaches are used (for redundancy), to keep those in sync.
My question is, is there a memcache-based message queue service that can run on App Engine?
I would be very careful using the Google App Engine Memcache in this way. You are right to be worrying about "unexpected cache evictions".
Google expect you to use the memcache for caching data and not storing it. They don't guarantee to keep data in the cache. From the GAE Documentation:
By default, items never expire, though
items may be evicted due to memory
pressure.
Edit: There's always Amazon's Simple Queueing Service. However, this may not meet price/performance levels either as:
There would be the latency of calling from the Google to Amazon servers.
You'd end up paying twice for all the data traffic - paying for it to leave Google and then paying again for it to go in to Amazon.
I have started a Simple Python Memcached Queue, it might be useful:
http://bitbucket.org/epoz/python-memcache-queue/
If you're happy with the possibility of losing data, by all means go ahead. Bear in mind, though, that although memcache generally has lower latency than the datastore, like anything else, it will suffer if you have a high rate of atomic operations you want to execute on a single element. This isn't a datastore problem - it's simply a problem of having to serialize access.
Failing that, Amazon's SQS seems like a viable option.
Why not use Task Queue:
https://developers.google.com/appengine/docs/python/taskqueue/
https://developers.google.com/appengine/docs/java/taskqueue/
It seems to solve the issue without the likely loss of messages in Memcached-based queue.
Until Google impliment a proper job-queue, why not use the data-store? As others have said, memcache is just a cache and could lose queue items (which would be.. bad)
The data-store should be more than fast enough for what you need - you would just have a simple Job model, which would be more flexible than memcache as you're not limited to key/value pairs