There's no persistent memory in webtasks, so passport's default MemoryStore won't work. I do know webtask offers persistant storage so perhaps I could use that somehow?
Although it would be possible to use context.storage by hacking with a compiler (e.g. a fork of webtask-tools) it's probably not worth it as storage size limit is 500kb.
I would just get a free 500mb mongodb database from mlab and use connect-mongo.
Related
I have set up a free tier MongoDB-atlas database and have a script that is storing tweets on it. Using db.collection.stats() it says storage size is 32768 which will fill up quite fast. Firstly, what happens when you exceed this limit? are new entries rejected or something else? Secondly, is there a way to deal with this without upgrading? For example, is it possible to clear entries before exceeding capacity?
When you exceed the limit the atlas cluster node will have exceeded the limit will be unavailable. It may be possible that the entire cluster will go down and then you will need to contact the MongoDB support to make the cluster up.
Although the best option is this that you need to upgrade to next tier for having more storage capacity. But in case you don't want that in that case you may write a script to delete old data from your cluster and after deleting the data make sure to run the compact command to reclaim the data storage.
We plan to use MongoDB in production for our website which will be hosted (for the start) on a single ec2 instance.
What is the recommandation for a MongoDB, which will have around 25k documents for the start with low traffic? So far i am not used to AWS, therefore a have no comparison to other dedicated hosters.
The "storagesize" of the collection in question will be arond 400MB, "totalindexsize" maybe around 20MB
If it's not a production instance, you can start with t2.medium instance. If its a production instance, start with m5.large.
Attach a new ebs volume of size 10GB and configure mongodb to use this new volume as the data directory. This helps to scale up your storage easily at later point of time.
Make sure you format your ebs volume to have xfs file system before installing mongodb which is required for best performance by mongo.
Also, later if you feel like increasing the instace size wheb your traffic increases, just use the "instance modify" option to get it done.
I cannot give you a specific answer, but the cloud provides you with the option to test your setup very quickly. Just start with some instance (e.g. m3.medium) and you can create an Amazon Machine Image of your running MongoDB instance anytime and just start it on a larger or smaller instance type.
You can find the instance types here: https://aws.amazon.com/ec2/instance-types/
Deeper thought about the instance type choice can be found here: https://www.mongodb.com/blog/post/maximizing-mongodb-performance-on-aws
If you have any doubt about sizing, err on the side of going larger and then scaling down as needed. Undersizing your instances when deploying MongoDB can, and likely will, create performance problems. In order to achieve the best performance, your MongoDB working set should fit in memory.
The working set is the portion of data and related indexes that your clients access most frequently.
We are currently using the Wired Tiger storage engine with MongoDB 3.2
We have tweaked a server (196Go RAM, disable journal) in order to use mongodb as a cache server (no replication, write concern = 0 for fire & forget writes).
I'd like to know if it could be interesting for us to switch to the In Memory storage engine knowing that our data already fits in memory. Is there another benefits?
Thank you
As you have already disabled journaling and the data fits your memory, switching to In-memory storage engine makes more sense as it provides the performance benefits that comes with the engine.
Just make sure that writeConcernMajorityJournalDefault to false, not sure if write concern=0 servers the same purpose during write ACKs.
I have a web app on Heroku and I'm trying to understand the difference/ trade-offs between adding a Memcached instance with 1GB of RAM and adding 1GB of RAM to my Postgres server.
If I added a Memcached instance, I would probably use Johnny Cache (for Django - http://packages.python.org/johnny-cache/).
Should I expect similar performance improvement from the two options? In general, what is the advantage of using memcache vs. increasing the size of the Postgres cache. (I understand that people often run memcache on the DB server so there must be one).
I appreciate this is probably a very naive question but I haven't been able to find anything to clear up my confusion through Google.
Postgres for best performance needs enough cache to keep most frequently used object (indexes, tables). So there is a tipping point in setting of shared_buffers. After that point,
increasing shared buffers does not help much.
It is good to leave some part of RAM for filesystem-level caching.
For much more see http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server
As for memcache, it's totally different beast... It can be used directly from application to have ultra-fast non-persistent key-value storage.
All three traits make memcached differ from relational database (RDB).
ultra-fast (RDB are not)
non-persistent (RDB are)
key-value only (RDB much better)
I would like to know if Memcache is recommended when using a NoSQL database like mongoDB.
The concept of using memcache stems from the idea that you have "extra RAM" sitting around somewhere. Both MongoDB and MySQL (and most DBs) will take every meg of RAM that they can get.
In the case of the very common MySQL / Memcache, it is very well documented that using Memcache is more about reducing query load on the server than it is about speeding up queries. A good memcache implementation basically just tries to keep the most common data in memory so that the database server can churn away on bigger stuff.
In fact, it's been my experience that use of memcache generally becomes a reliance on memcache to maintain system performance.
So back to the original question, where do you have extra RAM?
If you have extra RAM on web servers, you may be able to use Memcache. Of course, you could also run Mongo locally on the web server. Just slave the data you need from the master.
If you have extra RAM on other computers, then there's not really a point in using memcache. Just add more nodes to your MongoDB replica set or shard. This is where MongoDB actually shines. Because of sharding / replication, you can add more RAM to Mongo Horizontally to increase performance. With SQL it's very difficult to "just add more servers" because joins don't scale very well. But with Mongo, it's quite possible to simply "add more nodes" to a problem.
MongoDB stores everything in memory anyway and works in a similar vein, being a key-value based system, however I believe MongoDB is more flexible, as it allows for storing BSON objects within themselves.
(Just for clarification, MongoDB uses BSON, a specialised form of JSON, for storing all its data, which includes objects within objects.)
At first no. If you run into performance problems later add a caching layer (memcache). But you won't gain anything if you're going to use Redis for example, as Redis already stores everything in memory.
The answer would depend on your use cases.
In general, accessing RAM is orders of magnitude faster than accessing disk.
Even the fastest SSD drives are about 100 times slower to access than RAM.
Now, I don't know if Mongo has a caching system in place (most likely it does), or what the eviction policy is, but as a programmer i would prefer a cache where i can store/retrieve and delete items at will. Therefore i would prefer using a caching solution even with Mongo.
In summary, it really depends what you are using these solutions for. There is no one answer to cover all possible uses.