Or rather how does remote RAM compare against local Disk access?
If the answer is "it depends", what are the conditions?
Data access patterns, ratio of read-to-writes, distance etc.
Finally, what if the local disks are NetApp filers?
Thanks.
It is as always a question of latency versus bandwidth. The access time to remote memory will be on the order of microseconds for gigabit networking, 100s of nanoseconds for InfiniBand and 10 GigE compared with milliseconds for local disks (example Seagate ST3250318AS - 4.1ms). As soon as the disk has hit a contiguous block it's transfer rate will exceed gigabit networking which is why local disk is predominantly considered faster.
In general the local HDD will be faster, as the remote connection will be limited by the network bandwidth + protocol overhead.
If "local" (they're not really local if they're on a remote netapp) are netapps, they will be generally slower than remote RAM. Since netapps will most likely have the same network bottleneck + protocol overhead, plus file system overhead (nfs/smb), and most likely some sort of RAID overhead.
Related
when I study in Computer Architecture and System Programming, some question rises up.
First of all, program is in SSD or Hard Disk but when it executed, this load to memory(RAM). Why program is not executed on HardDisk directly?? Why need to load on RAM?
Thanks
This is simply done because your RAM is way faster than your Hard Disk.
When your computer executes a program, the CPU reads all the instructions from memory one after another and executes them. The CPU itself cannot store the whole program while executing it, so it has to be read from somewhere else. If the CPU had to read the instructions from a hard disk, it would be crazy slow.
Now that we have SSDs this is becoming somewhat less relevant, but in the old days the difference between RAM ("Random Access Memory") and HDD ("Hard Disk Drive") was that the RAM could access any memory address at any point in time, thus "Random Access". The HDD would have to rotate the hard disk your data is stored on to read from a certain address. Accessing random memory addresses is very hard for an HDD.
When the CPU executes a program it has to jump around all the time. It also has to store the program's memory somewhere and access that as quickly as possible whenever needed. An HDD is very bad at those two things, a RAM is very good.
So why did we use HDDs at all? Because RAM
is way to expensive
does not persist data when turned off
And what about SSDs? They are a lot better at random access that HDDs, but they're still considerably slower than RAM.
Also, you have to take swap files into account. The computer can use some of your HDD or SSD storage as system memory if it needs to. This can be very useful if the data that's using up your RAM does not get accessed by the CPU very often.
I've got a task to do and some limited hardware resources, as always.
I need to setup postgres server with single database, with a table of largeobjects (3TB+) and a few small, heavily accessed tables (<10 GB).
I've got old physical server with ~5 TB of harddisk space, with limited CPU and RAM, I can also use much faster (in CPU and RAM) virtual server - but limited in storage.
I won't have much DELETE statements, most SELECT statements will be to recent data. There will be one simultanous connection doing all the job, client on one host only.
I see a few scenarios:
Postgres on virtual machine with remote storage (single instance)
Postgres on old hardware with local storage (single instance)
Postgres on both, with some kind of replication (high speed virtual machine for new data, low speed for older data on the old hardware)
Any other ideas?
Is it even possible to replicate just the most recent part of the postgres database?
90% of SELECT queries will be to the most recent ~5-10 gigabytes of data, but I need seamless access to the rest 2,990 TB.
What should I do? (except buying appropriate hardware;)
It doesn't really matter as long as you have enough RAM to buffer the 10GB of heavily accessed data.
You'll need some additional RAM to read large objects without pushing the 10GB out of the cache, but that shouldn't be a problem on today's machines.
If all your work is done on one connection, that sounds like there will be no high load on the database.
So I wouldn't really worry about scaling with requirements like that.
Your biggest worry should probably be how to backup 3TB of data in a reasonable time.
Edit: If you have much less memory, you should take the machine with the faster storage.
Finally I've checked several different scenarios and decided not to keep files/largeobjects in database.
Postgres with database location mounted over NFS (v4) had some lags - It was faster but it was choking for a few seconds periodically, i decided to store plain files over NFS which is significantly slower but more stable.
I'm sure there was a way to tune it, but this solution is fine too.
Postgres is used for file index and keeps their files on local harddisk.
I need to transfer huge amount of data between Java and C++ programs under Linux(CentOS). Performance is the first concern.
What will be the best choice? RAMDisk (/dev/shm/) or local socket?
A socket is fastest because the other end can start processing the data (on a separate cpu core) before you have finished sending data.
Say you're sending 100KB of data, the other end can begin processing as soon as it recieves a couple of kilobytes. And by the time all 100KB has been sent, it has probably finished processing 90KB or thereabouts, so it only has 10KB left.
While with a RAM disk, you have to write the entire 100KB before it can even start processing data. Making it about 10x faster to use a socket than a ram disk, assuming both ends need to do about the same amount of work.
Maybe it takes 1 millisecond to write 100KB to a RAM disk and then 1 millisecond to process it. With a socket it would take 1 millisecond to send the data but only 0.1 millisecond to finish processing after all the data has been sent.
The larger the amount of data being sent, the bigger the performance gain for sockets. 10 seconds to write all the data, and another 0.1 millisecond to fnish processing after all data has been sent.
However, a RAM disk is easier to work with. Sockets use streams of data, which is more time consuming in terms of writing the code and debugging/testing it.
Also, don't assume you need a ram disk. Depending on how the operating system has been configured writing 100MB to a spinning platter hard drive might simply write it to a RAM cache and then put it on the hard drive later on. You can read it from the temporary RAM cache immediately without waiting for the data to be written to the HDD. Always test before making performance assumptions. Do not assume a HDD is slower than RAM, because it might be optimised out for you silently.
The mac I'm typing this on, which is UNIX just like CentOS, currently has about 8GB of RAM dedicated to holding copies of files it guesses I'm going to read at some point in the near future. I didn't have to create a RAM disk manually, it just put them in RAM heuristically. CentOS does the same sort of thing, you have to test it to see how fast it actually is.
But sockets are definitely the fastest option, since you do not need to write all the data to start processing it.
Can anyone highlight ways by which inter-core communication can be reduced in a NUMA multicore architecture. Case study Intel NEHALEM micro architecture.
The Nehalem processor uses QuickPath Interconnect (QPI) for inter-processor/node/package communication. In a NUMA system each node has its own local memory, which is shared with other nodes in the system. When the working set of a program fits in the L1 cache and is read-only then it doesn't matter much which NUMA node owns the memory. Communication between NUMA nodes is necessary when a core gets a cache miss and the memory is owned by another node. However, this doesn't mean that it is slower to access memory owned by another node, it depends on whether the other node has it cached in the cache associated with its local memory, what Intel calls the Last Level Cache (LLC). Access by a core to a memory location that is local to that node is faster than access to memory owned by another node, but only if it misses in the LLC on both nodes. It is faster to access memory that hits in the LLC on another node than it is to go to memory on the local node, that is because memory is so much slower than the CPU and QPI is optimized for this sort of communication. Most systems don't bother trying to reduce inter-processor communication because, as you can imagine, it is not an easy problem - it requires setting affinity of threads to cores, setting affinity of the memory working set of that thread to the local memory of that node. You can read more about this in Drepper Ulrich's paper, search for NUMA. In this paper Ulrich refers to QPI as Common System Interface (CSI), which was the Intel name for QPI before announcement.
our Amazon EBS backed instance has slowed down considerably (maybe shifted physical host?).
I've checked the instance using top and the CPU use is very low when the process is activated (like 1%). Using iotop I have monitored the disk read speed of postgresql. When there is only one postgresql thread running it's reporting about a 5M/s read speed. Is this rather slow or is this in the parameters of usual disk read speeds?
Thanks
5MB/s its more or less the typical for one single hard drive. I mean sequential accesses of course. If you have only one hard disk then your CPU its fine since one hard disk its not enough to stress out your CPU probably, if you are not reaching any more speed than that even with constant queries then your hard disk its the bottleneck.