I am studying about operating systems(Silberscatz, Galvin et al). My programming experiences are limited to occasional coding of exercise problems given in a programing text or an algorithm text. In other words I do not have a proper application programming or system programming experience. I think my below question is a result of a lack of experience of the above and hence a lack of context.
I am specifically studying IPC mechanisms. While reading about shared memory(SM) I couldn't imagine a real life scenario where processes communicate using SM. An inspection of processes attached to the same SM segment on my linux(ubuntu) machine(using 'ipcs' in a small shell script) is uploaded here
Most of the sharing by applications seem to be with the X deamon. From what I know , X is the process responsible for giving me my GUI. I infered that these applications(mostly applets which stay on my taskbar) share data with X about what needs to change in their appearances and displayed values. Is this a reasonable inference??
If so,
my question is, what is the difference between my applications communicating with 'X' via shared memory segments versus my applications invoking certain API's provided by 'X' and communicate to 'X' about the need to refresh their appearances?? BY difference I mean, why isn't the later approach used?
Isn't that how user processes and the kernel communicate? Application invokes a system call when it wants to, say read a file, communicating the name of the file and other related info via arguments of the system call?
Also could you provide me with examples of routinely used applications which make use of shared memory and message-passing for communication?
EDIT
I have made the question more clearer. I have formatted the edited part to be bold
First, since the X server is just another user space process, it cannot use the operating system's system call mechanism. Even when the communication is done through an API, if it is between user space processes, there will be some inter-process-communication (IPC) mechanism behind that API. Which might be shared memory, sockets, or others.
Typically shared memory is used when a lot of data is involved. Maybe there is a lot of data that multiple processes need to access, and it would be a waste of memory for each process to have its own copy. Or a lot of data needs to be communicated between processes, which would be slower if it were to be streamed, a byte at a time, through another IPC mechanism.
For graphics, it is not uncommon for a program to keep a buffer containing a pixel map of an image, a window, or even the whole screen that then needs to be regularly copied to the screen. Sometimes at a very high rate...30 times a second or more. I suspect this is why X uses shared memory when possible.
The difference is that with an API you as a developer might not have access to what is happening inside these functions, so memory would not necessarily be shared.
Shared Memory is mostly a specific region of memory to which both apps can write and read from. This off course requires that access to that memory is synchronized so things don't get corrupted.
Using somebody's API does not mean you are sharing memory with them, that process will just do what you asked and perhaps return the result of that operation to you, however that doesn't necessarily go via shared memory. Although it could, it depends, as always.
The preference for one over another I'd say depends on the specifications of the particular application and what it is doing and what it needs to share. I can imagine that a big dataset of some kind or another would be shared by shared memory, but passing a file name to another app might only need an API call. However largely dependent on requirements I'd say.
Related
I've been researching about memory segmentation, and it's been giving me lots of ideas for potential ways in which user code could benefit from swapping out segment registers. Specifically, I am interested in the x86-64 architecture because that's what I have.
Is there any way in which a user-mode program can create a new segment, for internal use?
To what extent can a program configure its own address space?
I imagine the GDT is way outside of a process' reach, but can a process modify the LDT?
Sorry if I sound naive, this is new stuff for me.
I also imagine that, if there even is a way to do this, it will almost certainly pass through OS specific functions. How would one do this in, say, Win32? I have found GetThreadSelectorEntry (https://learn.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-getthreadselectorentry), but I haven't been able to find the equivalent SetThreadSelectorEntry.
I was given this exact question on a quiz.
Question
Answer
Does the question make any sense? My understanding is that the OS schedules a process and manages what instructions it needs the processor to execute next. This is because the OS is liable to pull all sorts of memory management tricks, especially in main memory where fragmentation is a way of life. I remember that there is supposed to be a special register on the processor called the program counter. In light of the scheduler and memory management done by the OS I have trouble figuring out the purpose of this register unless it is just for the OS. Is the concept of the Stored Program Computer really relevant to how a modern computer operates?
Hardware fetches machine code from main memory, at the address in the program counter (which increments on its own as instructions execute, or is modified by executing a jump or call instruction).
Software has to load the code into RAM (main memory) and start the process with its program counter pointing into that memory.
And yes, if the OS wants to page that memory out to disk (or lazily load it in the first place), hardware will trigger a page fault when the CPU tries to fetch code from an unmapped page.
But no, the OS does not feed instructions to the CPU one at a time.
(Unless you're debugging a program by putting the CPU into "single step" mode when returning to user-space for that process, so it traps after executing one instruction. Like x86's trap flag, for example. Some ISAs only have software breakpoints, not HW support for single stepping.)
But anyway, the OS itself is made up of machine code that runs on the CPU. CPU hardware knows how to fetch and execute instructions from memory. An OS is just a fancy program that can load and manage other programs. (Remember, in a von Neumann architecture, code is data.)
Even the OS has to depend on the processing architecture. Memory today often is virtualized. That means the memory location seen by the program is not the real physical location, but is indirected by one or more tables describing the actual location and some attributes (e.g. read/write/execute allowed or not) for memory accesses. If the accessed virtual memory has not been loaded into main memory (these tables say so), an exception is generated, and the address of an exception handler is loaded into the program counter. This exception handler is by the OS and resides in main memory. So the program counter is quite relevant with today's computers, but the next instruction can be changed by exceptions (exceptions are also called for thread or process switching in preemptive multitasking systems) on the fly.
Does the question make any sense?
Yes. It makes sense to me. It is a bit imprecise, but the meanings of each of the alternatives are sufficiently distinct to be able to say that D) is the best answer.
(In theory, you could create a von Neumann computer which was able to execute instructions out of secondary storage, registers or even the internet ... but it would be highly impractical for various reasons.)
My understanding is that the OS schedules a process and manages what instructions it needs the processor to execute next. This is because the OS is liable to pull all sorts of memory management tricks, especially in main memory where fragmentation is a way of life.
Fragmentation of main memory is not actually relevant. A modern machine uses special hardware (and page tables) to deal with that. From the perspective of executing code (application or kernel) this is all hidden. The code uses virtual addresses, and the hardware maps them to physical addresses. (This is even true when dealing with page faults, though special care will be taken to ensure that the code and page table entries for the page fault handler are in RAM pages that are never swapped out.)
I remember that there is supposed to be a special register on the processor called the program counter. In light of the scheduler and memory management done by the OS I have trouble figuring out the purpose of this register unless it is just for the OS.
The PC is fundamental. It contains the virtual memory address of the next instruction that the CPU is to execute. For application code AND for OS kernel code. When you switch between the application and kernel code, the value in the PC is updated as part of the context switch.
Is the concept of the Stored Program Computer really relevant to how a modern computer operates?
Yes. Unless you are working on a special custom machine where (say) the program has been transformed into custom silicon.
Can processes share a single stack?
I'm currently thinking yes and no. That they "share" stack but it need to copy and save the information already there elsewere before using it and return it when the first process is getting picked up by the CPU again. But I might be confusing this with registers in general.
Can someone help me shed some light on this?
Processes do not share CPU stacks.
While processes can potentially share memory using shared-memory facilities, processes do not share memory by default. Operating systems try to minimize the amount of sharing between processes as a way to ensure security.
Sharing CPU stack between processes A and B would be detrimental to security, because process A would be able to poke around "junk" left on the stack by process B, and vice versa. Hackers managed to exploit an indirect sharing on a much smaller scale to create a major security vulnerability (you can read more about Meltdown and Spectre here). Sharing CPU stacks would create a much bigger problem.
It goes without saying that an attempt to share stacks would require a degree of inter-process synchronization that would be prohibitive to overall performance. An ability to make independent operations on CPU stack is so critical to concurrency that threads inside the same process are allocated separate stacks, even though they already share all the memory allocated to the process, so security is not a concern. Sharing stacks would effectively kill concurrency, because maintaining a shared stack would require frequent exclusive access with lots of synchronization.
Some systems use an interrupt stack shared by all processes. Generally, there is one interrupt stack per processor.
User stacks (and there is usually one for each processor mode used by the system) are unique to each process (or thread).
The difference between the registers and the stack is that the latter can be anywhere in memory (it is indirectly referenced by appropriate registers) while the formers are fixed (there is only one set of architecturally visible registers).
The stack is part of the state of a program, just like it make no sense to mix program instruction, data and context together, mixing two stacks make no sense.
If program A pushes X it expects to pop X and not an eventual value pushed by program B meanwhile.
It's possible to make all program shares the same memory area for the stack but this is, in general, counterproductive.
As you correctly noted, the stack must be swapped in an out, thus, in the case of two program A and B, two additional memory areas are needed: one for saving the stack of A and one for the stack of B.
In the end, three memory areas are used instead of two.
There are cases where the swapping is necessary: when the shared is at a fixed location.
This is the case, in some degenerate form, of registers but other structure can have a fixed location.
One simple example are the page table entries, if a program A is used to generate two processes A1 and A2, most OSs will copy-on-write them.
Under this circumstances, the two processes end up sharing a lot of pages, even all but a few. For the OS may be easier to swap in and out the few different pages rather than make the page table (or part of it) point to two different locations.
In general, if we cannot afford to have multiple instances of a resource, we need to time-share it.
Since we can afford to have more than one instance of the stack, we prefer to not share it.
I'm reading this article on how to : correctly retain variable state in Android and I'm reminded that I've never gotten a good answer (and can't find one here) for why it's better to tussle with the Bundle (which isn't a HUGE hassle, but definitely has its limitations) rather than just always have an Application overridden in your App, and just store all your persistent data members there. Is there some leakage risk? Is there a way that the memory can be released unexpectedly? I'm just not clear on this... it SEEMS like it's a totally reliable "attic" to all the Activities, and is the perfect place to store anything that you're worried might be reset when the user turns the device or suspends the app.
Am I wrong on this? Would love to get some clarity on what the true life cycle of the memory is in the Application.
Based on the answers below, let me extend my question.
Suppose I have an app that behaves differently based on an XML file that it loads at startup.
Specifically, the app is a user-info gathering app, and depending on the XML settings it will follow an open ended variety of paths (collecting info A, but not J, and offering Survey P, followed by an optional PhotoTaking opportunity etc.)
Ideally I don't have to store the details of this behavior path in a Bundle (god forbid) or a database (also ugly, but less so). I would load the XML, process it, and have the Application hold onto that structure, so I can refer to it for what to do next and how. If the app is paused and the Application is released, it's not *THAT big a hassle to check for null in my CustomFlow object (that is generated as per the XML) and re-instantiate it. It doesn't sound like this would happen all that often, anyway. Would this be a good example of where Application is the *best tool?
The question as to which method is better largely depends upon what information you are storing and need access to and who (which components, packages, etc.) needs access to that information. Additionally, settings like launchMode and configChanges which alter the lifecycle can help you to determine which method is best for you.
First, let me note, that I am a huge advocate for extending the Application object and often extend the Application class, but take everything stated here in its context as it is important to understand that there are circumstances where it simply is not beneficial.
On the Lifecycle of an Application: Chubbard mostly correctly stated that the Application has the same life as a Singleton component. While they are very close, there are some minute differences. The Application itself is TREATED as a Singleton by the OS and is alive for as long as ANY component is alive, including an AppWidget (which may exist in another app) or ContentResolver.
All of your components ultimately access the same object even if they are in multiple Tasks or Processes. However, this is not guaranteed to remain this way forever (as the Application is not ACTUALLY a Singleton), and is only guaranteed in the Google Android, rather than the manufacturer overridden releases. This means that certain things should be handled with care within the Application Object.
Your Application object will not die unless all of your components are killed as well. However, Android has the option to kill any number of components. What this means is that you are never guaranteed to have an Application object, but if any of your components are alive, there IS an Application to associate it to.
Another nice thing about Application is that it is not extricably bound to the components that are running. Your components are bound to it, though, making it extremely useful.
Things to Avoid in Application Object:
As per ususal, avoid static Contexts. In fact, often, you shouldn't store a Context in here at all, because the Application is a Context itself.
Most methods in here should be static, because you are not guaranteed to get the same Application object, even though its extremely likely.
If you override Application, the type of you data and methods store here will help you further determine whether you need to make a Singleton component or not.
Drawables and its derivatives are the most likely to "leak" if not taken care of, so it is also recommended that you avoid references to Drawables here as well.
Runtime State of any single component. This is because, again, you are not guaranteed to get back the same Application object. Additionally, none of the lifecycle events that occur in an Activity are available here.
Things to store in the Application (over Bundle)
The Application is an awesome place to store data and methods that must be shared between components, especially if you have multiple entry points (multiple components that can be started and run aside from a launch activity). In all of my Applications, for instance, I place my DEBUG tags and Log code.
If you have a ContentProvider or BroadcastReceiver, this makes Application even more ideal because these have small lifecycles that are not "renewable" like the Activity or AppWidgetProvider and can now access those data or methods.
Preferences are used to determine, typically, run options over multiple runs, so this can be a great place to handle your SharedPreferences, for instance, with one access rather than one per component. In fact, anything that "persists" across multiple runs is great to access here.
Finally, one major overlooked advantage is that you can store and organize your Constants here without having to load another class or object, because your Application is always running if one of your components is. This is especially useful for Intent Actions and Exception Messages and other similar types of constants.
Things to store in Bundle rather than Application
Run-time state that is dependent upon the presence or state of a single component or single component run. Additionally, anything that is dependant upon the display state, orientation, or similar Android Services is not preferrable here. This is because Application is never notified of these changes. Finally, anything that depends upon notification from that Android System should not be placed here, such as reaction to Lifecycle events.
And.... Elsewhere
In regard to other data that needs to be persisted, you always have databases, network servers, and the File System. Use them as you always would have.
As useful and overlooked as the Application is, a good understanding is important as it is not ideal. Hopefully, these clarifications will give you a little understanding as to why gurus encourage one way over the other. Understand that many developers have similar needs and most instruction is based on what techniques and knowledge a majority of the community has. Nothing that Google says applies to all programmer's needs and there is a reason that the Application was not declared Final.
Remember, there is a reason Android needs to be able to kill your components. And the primary reason is memory, not processing. By utilizing the Application as described above and developing the appropriate methods to persist the appropriate information, you can build stronger apps that are considerate to the System, the User, its sibling components AND other developers. Utilizing the information that everyone here has provided should give you some great guidance as to how and when to extend your Application.
Hope this helps,
FuzzicalLogic
I prefer to subclass Application and point my manifest to that. I think that's the sane way of coding android although the Android architects from Google think you should use Singletons (eek) to do that. Singletons have the same lifetime as Application so everything that applies to them applies to Application except much less dependency mess Singletons create. Essentially they don't even use bundles. I think using subclass Application has dramatically made programming in Android much faster with far less hassle.
Now for the downside. Your application can be shutdown should the phone need more memory or your Application goes into the background. That could mean the user answered the phone or checked their email. So for example, say you have an Activity that forces the user to login to get a token that other Activities will use to make server calls. That's something you might store in your service object (not android service just a class that sends network calls to your server) that you store in your subclass of Application. Well if your Application gets shutdown you'll loose that token, and when the user clicks the back button your user might return to an Activity that assumes you are already authenticated and boom your service class fails to work.
So what can you do? Continue to use Bundle awfulness? Well no you could easily store security tokens into the bundle (although there might be some security issues with that depending on how this works for your app), or you have to code your Activities to not assume a specific state the Application is in. I had to check for a loss of the token and redirect the user back to the login screen when that happens. But, depending on how much state your Application object holds this could be tricky. But keep in mind your Application can know when it's being shutdown and persist it's internal state to a bundle. That at least allows you to keep your Objects in memory for 99% of the time your Application, and only save/restore when it gets shutdown rather than constantly serializing and deserializing with boiler plate code whenever you move between Activities. Using Application lets you centralize how your program can be brought up and shutdown, and since it normally lives longer than any one activity it can reduce the need for the program to reconstitute the guts of your App as the user moves between Activities. That makes your code cleaner by keeping out details of the app from every Activity, reduces overhead if your Application is already built, shares common instances/code, and allows Activities to be reclaimed without loosing your program all together. All good programs need a centralized hub that is the core, and subclassing Application gives you that while allowing you to participate in the Android lifecycle.
My personal favorite is to use http://flexjson.sourceforge.net/ to serialize my Java objects into bundles as JSON if I need to send objects around or save them. Far easier than writing to sqlite DB when all you need to do is persist data. And nice when sending data between two Activities using objects instead of broken apart primitives.
Remember by centralizing your model in the Application you create a place to share code between multiple Activities so you can always delegate an Activities persistence to an object in the Application by hooking the onPause() as well allowing persistence to be centrally located.
The short answer is: use bundles as it makes saving your state out when you're backgrounded easier. Also, it's complicated.
The long answer:
My understanding is, as soon as you Activity's onPause method is called (and onSaveInstanceState which gives you a bundle into which you should store your Activity's data) your process can be terminated without further warning. Later, when the user comes back to your application, your activity is given an onCreate call with that original bundle from which to restore its state. This will happen to all your activitys in what was your original stack.
Being able to restore your state from the bundle (which Android will save for you as your process goes away) is how Android maintain's the myth of multi-tasking. If you don't dump your activity's state out to a bundle each time onSaveInstanceState is called, your app will look like it's been restarted when the user may have just switched out for a second. This can be especially troubling when the system is resource constrained as the system would need to kill off processes more often in order to keep the device running quickly
Why the Application can be Bad
The Application does not actually get a chance to save any of its data if the process is shut down. It does have an onDestroy method but the docs will tell you that this actually never gets called by the system on an actual device. This means that, in the constrained case I mentioned above, any incidental information about what's going on within an Activity (if you've saved it in the Application) will be lost if the process is ended.
Developer's often miss this case (and it can be really annoying for users) because they're either running on a dev phone which never gets hit with using many applications at the same time. We're also never using the app for a while, then switching to another application and, after a while, switching back again.
I'm quite impressed by what Lift 2.0 brings to the table with Actors and StatefulSnippets, etc, but I'm a little worried about the memory overhead of these things. My question is twofold:
How does Lift determine when to garbage collect state objects?
What does the memory footprint of a page request look like?
If a web crawler dances across the footprint of the site, are they going to be opening up enough state objects to drown out a modest VPS (512M)? The question is very obviously application dependent, but I'm curious if anyone has any real world figures they can throw out at me.
Lift stores state information in a session, so once the session is destroyed the state associated with that session goes away.
Within the session, Lift tracks each page that state is allocated for (e.g., mapping between an ajax button in the browser and a function on the server) and have a heart-beat from the browser. Functions for pages that have not seen the heartbeat in 10 minutes are unreferenced so the JVM can garbage collection them. All of this is tunable, so you can change heart-beat frequency, function lifespan, etc., but in practice the defaults work quite well.
In terms of session explosion, yeah... that's a minor issue. Popular sites (including http://demo.liftweb.net/ ) experience it. The example code (see http://github.com/lift/lift/tree/master/examples/example/ ) detects sessions that were created by a single request and then abandoned and expires those early. I'm running demo.liftweb.net with 256MB of heap size (that'd fit in a 512MB VPS) and occasionally, the session count rises over 1,000, but that's quickly tamped down for search engine traffic.
I think the question about memory footprint was once answered somewhere on the mailing list, but I can’t find it at the moment.
Garbage collection is done after some idle time. There is, however, an example on the wiki which uses some better heuristics to kill off sessions spawned by web crawlers.
Of course, for your own project it makes sense to check memory consumption with something like VisualVM while spawning a couple of sessions yourself.