I am using Rx2AndroidNetworking in my project to get getObjectListObservable thousands of records from the Server JSON List of Objects about 150+ attributes of SQLite Database of 3 Tables which i have to resync(Insert) the data to SQLite tables using ActiveAndroid ORM.
While executing the Rx2AndroidNetworking.get method the following logs are showing in the logcat.
D/LoginActivity: onCompleted: only_login
D/LoginActivity: onComplete: before_executing_the_resyncing_code
D/LoginActivity: InsertData: before_Rx2AndroidNetworking_GetPatients
I/art: Do partial code cache collection, code=57KB, data=58KB
I/art: After code cache collection, code=56KB, data=57KB
I/art: Increasing code cache capacity to 256KB
I/art: Background sticky concurrent mark sweep GC freed 12747(2MB) AllocSpace objects, 0(0B) LOS objects, 11% free, 17MB/19MB, paused 12.598ms total 37.164ms
While executing the public void onNext method the following logs are showing in the logcat.
D/LoginActivity: onNext: resyncing_insert_data_in db
I/art: Do full code cache collection, code=125KB, data=101KB
I/art: Starting a blocking GC JitCodeCache
I/art: After code cache collection, code=69KB, data=44KB
I/art: Do partial code cache collection, code=81KB, data=80KB
I/art: After code cache collection, code=80KB, data=80KB
I/art: Increasing code cache capacity to 512KB
I/art: Compiler allocated 6MB to compile void .onNext(java.util.List)
I/art: Compiler allocated 8MB to compile void .onNext(java.util.List)
While executing network call and inserting the data in the SQLite the UI of my App is stucks. My app wants to get Memory from the OS but the device (Lenovo TB-X704L having 3 GB RAM) unable to allocate more than of 64 MB of memory to my app(screenshot is attached hereby).
Diagnose the issue and suggest the solution.
On which thread you are doing sql operation? Try to move sql operation on separate worker thread.
Related
I understand that the working set is cached every time a query is run in the mongoDB.
What happens if one working set exceeds the caching memory when I know that the data on the previous page is removed and cached?
Ex)
cacheSizeGB: 0.5g,
total document size about query: 1g
And is it helpful to reduce the size of the document being cached by using the project command?
The cache is managed by WiredTiger, the mongod will request documents it needs, and if WT doesn't find it in the cache, it will read it in from disk. If this makes the cache exceed the threshold (default 80% of maximum size), the background eviction workers will start removing the least recently used items from the cache.
The result set is constructed/sorted in heap memory, not cache. If your result set is too large, mongod may be able to use disk for temporary space while sorting.
Large results may also cause the operating system to use swap, and in extreme cases it might run out of memory and get killed by the OOM killer.
So I have database with 140 million entries. First Query takes around 500ms, repeating the query takes 8ms. So there is some caching.
I would like to learn more about how long this cache exists what are the conditions it's refreshed etc - time or number of new entries made to the table.
Adding ~1500 new entries to table still makes queries around 10ms. But adding 150k will reset the cache. So there could be both, or only transaction count.
Anyone has a good resource, so that I could get more information on it?
I made this gist to make the load tests.
PostgreSQL uses a simple clock sweep algorithm to manage the cache (“shared buffers” in PostgreSQL lingo).
Shared buffers are in shared memory, and all access to data is via this cache. An 8kB block that is cached in RAM is also called a buffer.
Whenever a buffer is used, its usage count is increased, up to a maximum of 5.
There is a free list of buffers with usage count 0. If the free list is empty, anybody who searches a "victim" buffer to replace goes through shared buffers in a circular fashion and decreases the usage count of each buffer they encounter. If the usage count is 0, the buffer gets evicted and reused.
If a buffer is "dirty" (it has been modified, but not written to disk yet), it has to be written out before it can be evicted. The majority of dirty buffers get written out during one of the regular checkpoints, and a few of them get written out by the background writer between checkpoints. Occasionally a normal worker process has to do that as well, but that should be an exception.
I'm using pyspark.
Previously i had a similar problem, I collected a lot of data in driver program and didn't realize that lineage of dataframe kept source of data in driver until action was completed. A solution of this problem was to checkpoint dataframe (so lineage became lazy sourced).
But recently I met the same problem, and I don't collect data in driver, but write result to db. Is there some other heavy objects that can cause this error?
I checked all logs, and didn't find any warnings or suspicious messages. Last started action was checkpoint. Of course I can increase driver memory limit, but reason of such consumption is still unknown and I'm afraid I have some bottleneck.
Error message:
Current usage: 4.5 GB of 4.5 GB physical memory used; 6.4 GB of 9.4 GB virtual memory used. Killing container.
And it sends SIGKILL to executors
UPDT:
After some investigations I suppose that problem can be in BroadcastHashJoin. We had a lot of small tables and broadcast join is used to join them, but to broadcast table spark need to materialize it in driver so I disable it by setting spark.sql.autoBroadcastJoinThreshold=-1. And looks like memory consumption on driver decreased, and now AM not raising previous error. But on writing result table in DB(postgres) executors stuck on few last task of last stage. In thread dump of executor i see this stuck task:
Executor task launch worker for task 635633 RUNNABLE Lock(java.util.concurrent.ThreadPoolExecutor$Worker#1287601873}), Monitor(java.io.BufferedOutputStream#1935023563}), Monitor(org.postgresql.core.v3.QueryExecutorImpl#625626056})
It not falling, just waiting, can this be lock from postgres or something ?
I am running into serious memory issues with my kdb process. Here is the architecture in brief.
The process runs in slave mode (4 slaves). It loads a ton of data from database into memory initially (total size of all variables loaded in memory calculated from -22! is approx 11G). Initially this matches .Q.w[] and close to unix process memory usage. This data set increases by very little in incremental operations. However, after a long operation, although the kdb internal memory stats (.Q.w[]) show expected memory usage (both used and heap) ~ 13 G, the process is consuming close to 25G on the system (unix /proc, top) eventually running out of physical memory.
Now, when I run garbage collection manually (.Q.gc[]), it frees up memory and brings unix process usage close to heap number displayed by .Q.w[].
I am running Q 2.7 version with -g 1 option to run garbage collection in immediate mode.
Why is unix process usage so significantly differently from kdb internal statistic -- where is the difference coming from? Why is "-g 1" option not working? When i run a simple example, it works fine. But in this case, it seems to leak a lot of memory.
I tried with 2.6 version which is supposed to have automated garbage collection. Suprisingly, there is still a huge difference between used and heap numbers from .Q.w when running with version 2.6 both in single threaded (each) and multi threaded modes (peach). Any ideas?
I am not sure of the concrete answer but this is my deduction based on following information (and some practical experiments) which is mentioned on wiki:
http://code.kx.com/q/ref/control/#peach
It says:
Memory Usage
Each slave thread has its own heap, a minimum of 64MB.
Since kdb 2.7 2011.09.21, .Q.gc[] in the main thread executes gc in the slave threads too.
Automatic garbage collection within each thread (triggered by a wsful, or hitting the artificial heap limit as specified with -w on the command line) is only executed for that particular thread, not across all threads.
Symbols are internalized from a single memory area common to all threads.
My observations:
Thread Specific Memory:
.Q.w[] only shows stats of main thread and not the summation of all the threads (total process memory). This could be tested by starting 'q' with 2 threads. Total memory in that case should be at least 128MB as per point 1 but .Q.w[] it still shows 64 MB.
That's why in your case at the start memory stats were close to unix stats as all the data was in main thread and nothing on other threads. After doing some operations some threads might have taken some memory (used/garbage) which is not shown by .Q.w[].
Garbage collector call
As mentioned on wiki, calling garbage collector on main thread calls GC on all threads. So that might have collected the garbage memory from threads and reduced the total memory usage which was reflected by reduced unix memory stats.
I created a database containing a total of 3 tables for a specific purpose. The total size of all tables is about 850 MB - very lean... out of which one single table contains about 800 MB (including index) of data and 5 million records (daily addition of about 6000 records).
The system is PG-Windows with 8 GB RAM Windows 7 laptop with SSD.
I allocated 2048MB as shared_buffers, 256MB as temp_buffers and 128MB as work_mem.
I execute a single query multiple times against the single table - hoping that the table stays in RAM (hence the above parameters).
But, although I see a spike in memory usage during execution (by about 200 MB), I do not see memory consumption remaining at at least 500 MB (for the data to stay in memory). All postgres exe running show 2-6 MB size in task manager. Hence, I suspect the LRU does not keep the data in memory.
Average query execution time is about 2 seconds (very simple single table query)... but I need to get it down to about 10-20 ms or even lesser if possible, purely because there are just too many times, the same is going to be executed and can be achieved only by keeping stuff in memory.
Any advice?
Regards,
Kapil
You should not expect postgres processes to show large memory use, even if the whole database is cached in RAM.
That is because PostgreSQL relies on buffered reads from the operating system buffer cache. In simplified terms, when PostgreSQL does a read(), the OS looks to see whether the requested blocks are cached in the "free" RAM that it uses for disk cache. If the block is in cache, the OS returns it almost instantly. If the block is not in cache the OS reads it from disk, adds it to the disk cache, and returns the block. Subsequent reads will fetch it from the cache unless it's displaced from the cache by other blocks.
That means that if you have enough free memory to fit the whole database in "free" operating system memory, you won't tend to hit the disk for reads.
Depending on the OS, behaviour for disk writes may differ. Linux will write-back cache "dirty" buffers, and will still return blocks from cache even if they've been written to. It'll write these back to the disk lazily unless forced to write them immediately by an fsync() as Pg uses at COMMIT time. When it does that it marks the cached blocks clean, but doesn't flush them. I don't know how Windows behaves here.
The point is that PostgreSQL can be running entirely out of RAM with a 1GB database, even though no PostgreSQL process seems to be using much RAM. Having shared_buffers too high just leads to double-caching and can reduce the amount of RAM available for the OS to cache blocks.
It isn't easy to see exactly what's cached in RAM because Pg relies on the OS cache. That's why I referred you to pg_fincore.
If you're on Windows and this won't work, you really just have to rely on observing disk activity. Does performance monitor show lots of uncached disk reads? Does operating system memory monitoring show lots of memory used for disk cache in the OS?
Make sure that effective_cache_size correctly reflects the RAM used for disk cache. It will help PostgreSQL choose appropriate query plans.
You are making the assumption, without apparent evidence, that the query performance you are experiencing is explained by disk read delays, and that it can be improved by in-memory caching. This may not be the case at all. You need to look at explain analyze output and system performance metrics to see what's going on.