Hi so I want to know what's the recommended cache or memory size for memcached? I'm running a drupal site on AwS Thanks!
This is a very subjective question as it depends on what you are caching and how big your data is. That you will have to figure out yourself.
After you do that you should know that on average you get a 60 to 70% utilization on memcache.
So if you are planning to cache around 6GB of data then its good to allocate 10GB to Memcache.
You can take a look here to see how memcache works. This will explain why it is not possible to fully utilize memcache
Related
I have a JSF application hosted on Wildfly 10. For some reason when web clients are 75+ I see blank page.
Probably there is a default threat pool limit in Wildfly? Can you give some general advice how I can solve the problem?
Have you tried inspecting the memory usage of the server on this load.
You can get very useful stats using VisualVM.
If you see the server is running out of memory, solution could be as simple as
increasing the heap memory, might as well want to tweek MetaspaceSize parameter.
One recommendation would be to use jmeter and gradually increase the load and check stats using profiling tool.
This will help you narrow down if you have some memory leaks in your application it self.
I have a document that holds a big data structure in certain fields inside an array, it is slowing down my application due to frequent hits to read such data. am thinking on few solutions to implement but I need advice before i proceed and possibly even a better solution, here are my thoughts/questions:
would it help to cache data?
should I use memcached or redis as a caching engine and why?
would it help to read single fields from this document instead of reading it all every time?
should I do something else?!
Caching will help because it would avoid your db to be hit too often
Memcache or redis it's up to you. I prefere redis but if you already have a memcache it's fine.
If you have a cluster of servers, think if you need a centralized cache or not
Caching a full document won't help for getting a single field because you cache the result of a query without knowing what it contains.
your question need more clarification. for example how big is the data that you are speaking of is it couple of megabytes or gigabytes. All these factors change the solution. But if we consider that you have couple of megabytes and you want to prevent to call database every time the best solution is cache. How to choose a cache is also completely depends on what is your situation. If your web application runs on one server you can use the in-memory cache like ASP.Net cache which is very quick and fast for in-memory cache. this cache is stored in your heap so you can put all your object in the cache without serialization.But consider that whenever your application is restarted like most of deployments. your heap will be deleted and all the cache is cleared inside the heap.
if you have more than one server then you can start to think about an out-of-memory cache because two servers are not sharing heap memory and using all in-memory cache are useless because it duplicate the data and invalidating is nightmare. However, this is more reliable cache while it is not in the heap and in term of persistence is more than in-memory cache. But whatever you want to put in this kind of cache should be serializable while you are transferring the object over network connection. So you cannot put all your object in cache. Both Redis and memcached can be used for this purpose. Redis is more complicated with more functionality than Memcached but for your purpose memcached is quite good.
Whatever caching system you choose, approach it in a wide perspective. Design a caching system in your application while over time you need to put more things in cache. so its better to prepare everything for that time from now.
another things which is very important in cache is that whenever you set something in cache you have to consider when you are going to invalidate it.
Whether or not caching will help depends on the accession of the document. If the document is being accessed multiple times then caching will not help due to how MongoDB to memory caching actually works.
First, you need to understand your data accession patterns.
I have been browsing through a lot of websites. I need experts advice on this one.
can anyone please explain me what exactly is memcache ?
From what I understand that it is a distributed memory caching system used for dynamic web apps but my main question is do we need a database when we say 'memcache' or the term 'memcache' doesnt need a database ?
please answer. Thank you
No, you don't require a traditional database when you say memcache, it's an in memory hash table(dictionary) with key,value storage and so it resides in RAM as a lookup table.For this fact, it is not persistent, so whenever you restart your server, memcache gets reset.
memcached is a specific program that runs a server that other programs can use to keep things in memory. It's something like an in-memory database, depending on your definition of database.
Caching something in memory can also be done generically, without memcached (pronounced memcache dee).
What is a good tool for applying a layer of caching between a webserver and an application server.
Basic Requirements:
The application server needs a way to remove items from the cache and put items in the cache with an expiration date.
The webserver needs a way to pull items out of the cache in a very light-weight, fast manner without requiring thread allocation on the application server.
It does not neccessarily need to be a distributed cache (accessible from multiple machines), but it wouldn't hurt.
Strategies I have considered:
Static file caching. Request comes in, gets hashed, if a file exists we serve it, if not we route the request to the app server. Is high I/O a problem or file locking problems due to concurrency? Is it accurate that the file system is actually very fast due to kernel level caching in memory.
Using a key-value DB like mongodb, or redis. This would store the finished HTML/JSON fragments in db. The webserver would be equipped to read from the DB and route to the app server if needed. The app server would be equipped to insert/remove from the DB.
A memory cache like memcached or Varnish (don't know much about Varnish). My only concern with memcached is that I'm going to want to cache 3 - 10 gigabytes of data at any given time, which is more than I can safely allocate in memory. Does memcached have a method to spill to the filesystem?
Any thoughts on some techniques and pitfalls when trying this type of caching layer?
You can also use GigaSpaces XAP in memory data grid for caching and even hosting your web application. You can choose just the caching option or combine the power of two and gain single management of your environment along other things.
Unlike the key value pair approach you suggested, using GigaSpaces XAP you'll be able to have complex queries such as SQL, object based temples and much more. In your caching scenario you should check out more specifically the local cache related features.
Local Cache
Web Container
Disclaimer, I am a developer in GigaSpaces.
Eitan
Just to answer this from the POV of using Coherence (http://coherence.oracle.com/):
1. The application server needs a way to remove items from the cache and put items in the cache with an expiration date.
// remove one item from cache
cache.remove(key);
// remove multiple items from cache
cache.keySet().removeAll(keylist);
2. The webserver needs a way to pull items out of the cache in a very light-weight, fast manner without requiring thread allocation on the application server.
// access one item from cache
Object value = cache.get(key);
// access multiple items from cache
Map mapKV = cache.getAll(keylist);
3. It does not neccessarily need to be a distributed cache (accessible from multiple machines), but it wouldn't hurt.
Elastic. Just add nodes. Auto-discovery. Auto-load-balancing. No data loss. No interruption. Every time you add a node, you get more data capacity and more throughput.
Automatic high availability (HA). Kill a process, no data loss. Kill a server, no data loss.
A memory cache like memcached or Varnish (don't know much about Varnish). My only concern with memcached is that I'm going to want to cache 3 - 10 gigabytes of data at any given time, which is more than I can safely allocate in memory. Does memcached have a method to spill to the filesystem?
Use both RAM and flash. Transparently. Easily handle 10s or even 100s of gigabytes per Coherence node (e.g. up to a TB or more per physical server).
For the sake of full disclosure, I work at Oracle. The opinions and views expressed in this post are my own, and do not necessarily reflect the opinions or views of my employer.
I was discussing the requirements of an app and I am a little lost with a few things.
1)
I will be getting data back from a web server via JSON. The user wants the data cached for a month so that there is no unnecessary data usage. My concern is that the data base on the webserver is about 300mb in size. If the user is very active this could mean there could be a lot of data cached on the phone. Is there a limit to the amount of storage you can use? Would 50mb of cached data be realistic to store?
2)
If i was to store all this data, what would be the best way to handle it? Since i would need to be able to search and manage the data from the device. Is this a realistic request from the client, or is the iphone really not suited to this kind of operation. Would it be best to not cache the data at all and just pull it down with JSON as required.
Many Thanks,
-Code
You can use CoreData framework with SQLite for storage.
According to the accepted answer to this question, there is no specific limit to how much data you can store other than the device's flash storage size.
Is there a limit to the amount of storage you can use? Would 50mb of cached data be realistic to store?
No, there is no limit except the capacity of the iPhone's flash memory. 50 MB is not that much IMO, considering there are apps that are hundreds of megabytes large.