is there any way to use volatility in order to extract the callstack of all threads of a single process given a PID? By extracting the callstack I intend to view all the system calls currently stacked.
Thanks.
Related
I have created a pipeline where I have used an until activity to iterate through the files and load the data into a SQL table. The logic used in the until activity expression is compared to the number of files available in the data lake with an incremental variable value. If the variable value is equal to or greater than the total number of files the loop will exit.
If we have less than 5 files in the data lake then the loop may exit within 1 minute but if we have more than 25 it takes nearly 8 minutes to exit the loop. AS the value of the count of files increases the time taken to exit the loop also increases.
For Eg: We have 35 Files, all the files got processed and even the last activity inside the loop also successfully got executed, but the "until" activity again runs for another 12-14 minutes without any further activity to get executed.
Any help to improve the performance of the until activity loop exit would be really helpful.
NB: Our pipeline requires sorting/failure exit/and other logical implementation methods that's why we did not use the for-each loop activity.
Regards,
Sandeep
According to the documentation, we can scale up the DIU to improve the performance.
We can increase the Computer type and Core count when creating the IR.
I'm counting the number of rows in a 1TB tplog using -11!(-2;`:tplog) internal function, which takes a long time as expected.
Is there a way to track progress (preferably as a percentage) of this operation?
No I don't believe this would be possible for the -2 option. It is possible with every other use of -11! by defining a custom .z.ps but it seems .z.ps isn't invoked when counting number of chunks.
Your best bet might be to check the raw file size on disk using a linux system command or using hcount and then based on some previous estimates tested on your machine you could create logic to come up with a ballpark estimate (maybe with a linear interpolation) of timing based only on the file size.
Of course - once you have the chunk count from the -2 option then you can easily create a progress estimate of an actual replay, just not for the chunk count part itself
Fundamentally, entries are variable length and there is no tracking built into the file, so there's no exact answer.
You might be able to do this:
Replay a short segment (say 1,000 entries) and a custom handler:
.z.ps:{estimate+:-22!x}
-11!(1000;`:tplog)
-1"estimate of entry size: ", string estimate%:1000
-1"estimate of number of entries: ", string floor n:hcount[`:tplog]%estimate
\x .z.ps
Then repeat the full file with a standard upd handler.
I have an expensive function
mainfunc:{[batch;savepath]
...compute large table depending on batch and save it to savepath...
}
which is copied to slave processes and each process handles a different batch input to the function.
Say I need to execute the function on input:til 1000 then process 1 executes mainfunc[input[til 50];savepath] and process 2 executes mainfunc[input[50+til 50]]. Since they are processes and not threads saving to file is possible.
The reason why I am trying to avoid threads is that the output from mainfunc is a large table and there would be a lot of serialization overhead if the table was sent to the main process and saved from there.
Now I am running into the limitation that I want to splay the large table in each process. However splaying (in the case of table with symbols) requires to modify the sym file kept for enumeration and I get an exception as soon as 2 processes are trying to access that sym file. I use something like
splaydirname set .Q.en[hsym `$savepath] largetable; / .Q.en locks the sym file enumeration is done
How can I work around that?
Thanks for the help
Im using Windows10, i need to check CPU usage and memory usage for a power shell script which is scheduled to run every 3mins.
I have created a data collector set with following details
I use perfmon, to monitor CPU Usage i have added:
\process(Powershell_12345)%ProcessorTime
to monitor memory usage i have added:
\Memory%Committed bytes in Use
\Memory%Committed bytes
But problem is every time powershell script gets triggered through scheduler a new PID is created and the process has names concatenated with PID like powershell_
If i add the powershell process only till that thread is used, it would get monitored and not for entire day
How do i use perfmon to monitor powershell.exe for a day ?
This should probably be on SF rather than SO but the easiest way to do this is going to just be to monitor All process instances and then go back and grab the powershell instance when you look through the data later. Other option would be to programmatically create the data collector set as the first step in your script, more info on that Here A lot more work but you'll end up with cleaner data so it just depends what's important to you.
A final option if you'd like to avoid using perfmon at all is to take a look at This Function I created for my blog about a year ago, in all of my testing it's results correlate to those seen in perfmon quite well, if your interested in going that way let me know and I can show you how to set it up as a job and have it keep running in a loop while your script runs to generate the data you are looking for.
I have to deliver an application as a standalone Matlab executable to a client. The code include a series of calls to a function that internally creates several cell arrays.
My problem is that an out-of-memory error happens when the number of calls to this function increases in response to the increase in the user load. I guess this is low-level memory fragmentation as the workspace variables are independent from the number of loops.
As mentioned here, quitting and restarting Matlab is the only solution for this type of out-of-memory errors at the moment.
My question is that how I can implement such a mechanism in a standalone application to save data, quit and restart itself in the case of out-of-memory error (or when high likelihood of such an error is predicted somehow).
Is there any best practice available?
Thanks.
This is a bit of a tough one. Instead of looking to restart to clear things out, could you change the code to break the work in to chunks to make it more efficient? Fragmentation is mostly proportional to the peak cell-related memory usage and how much the size of data items varies, and less to the total usage over time. If you can break a large piece of work in to smaller pieces done in sequence, this can lower the "high water mark" of your fragmented memory usage. You can also save on memory usage by using "flyweight" data structures that share their backing data values, or sometimes converting to cell-based structures to reference objects or numeric codes. Can you share an example of your code and data structure with us?
In theory, you could get a clean slate by saving your workspace and relevant state out to a mat file and having the executable launch another instance of itself with an option to reload that state and proceed, and then having the original executable exit. But that's going to be pretty ugly in terms of user experience and your ability to debug it.
Another option would be to offload the high-fragmentation code in to another worker process which could be killed and restarted, while the main executable process survives. If you have the Parallel Computation Toolbox, which can now be compiled in to standalone Matlab executables, this would be pretty straightforward: open a worker pool of one or two workers, and run the fraggy code inside them using synchronous calls, periodically killing the workers and bringing up new ones. The workers are independent processes which start out with non-fragmented memory spaces. If you don't have PCT, you could roll your own by compiling your application as two separate apps - the driver app and worker app - and have the main app spin up a worker and control it via IPC, passing your data back and forth as MAT files or bytestreams. That's not going to be a lot of fun to code, though.
Perhaps you could also push some of the fraggy code down in to the Java layer, which handles cell-like data structures more gracefully.
Changing the code to be less fraggy in the first place is probably the simpler and easier approach, and results in a less complicated application design. In my experience it's often possible. If you share some code and data structure details, maybe we can help.
Another option is to periodically check for memory fragmentation with a function like chkmem.
You could integrate this function to be called silently from you code each couple of iterations, or use a timer object to have it called every X minutes...
The idea is to use thse undocumented functions feature memstats and feature dumpmem to get the largest free memory blocks available in addition to the largest variables currently allocated. Using that you could make a guess if there is a sign of memory fragmentation.
When detected, you would warn the user and instruct them you how to save their current session (export to MAT-file), restart the app, and restore the session upon restart.