Recently, I have been reading about Operating Systems, and this bugs me a lot.
How is it really possible for one process to manage other process.
Basically a CPU simply executes instructions, after executing one instruction, then it executes the instruction at address pointed by IP and increments the IP.
Let me elaborate my doubt with an example. Lets say I have an User process (or simply a process) which is being executed by CPU. Lets say, it has 'n' instruction and currently executing 'i'th instruction. IP points to (i+1)th instruction.
So, at this point how can all other OS processes like Scheduler, dispatcher etc... comes into play, Since CPU is already executing another process.
One solution (Just a guess), I could think of is , the use of Interrupts and Interrupt Service Routines.
But its only a guess.
PS: I searched and couldn't find any satisfying answer.
With the help of the hardware, ticks causes the CPU to execute operating system code. This code checks the system state and the time that has elapsed since the beginning of this process execution. At this point, the operating system can decide to schedule a different process. All it has to do is save the current state of the running process with the process that is about to start running. (basically changing the content of the registers and saving the registers state before changing to the new process).
Eventually, the CPU is taken away even if the process doesn't want to yield it.
To address your concern, there are no operating system processes in the way you think... it isn't like there are OS processes in the queue waiting among other processes....
I've been using Proc::Daemon in an attempt to make a start/stop daemon script, something allows me to do:
X start
X stop
X status
etc. However, in the source code it looks like that Proc::Daemon uses either a "pid" file, or a search of the process table. I'm concerned with both of these approaches, firstly as "pid"s are reused, which may give the impression a service is up when it's actually down, and secondly that process table entries are easily faked, and the checking doesn't look particularly robust.
Is there any robust way to make a start/stop daemon script/program like I've described, or has someone already made one? Note that I haven't got root access, and I'm also on Solaris if that's important.
Although pids are reused, I believe that they round-robin through a (large) fixed size set. e.g. on Solaris this used to be 30,000 (it may be different now). So 30,000 processes would have to start/finish before your pid was reused.
The approach used by Proc::Daemon doesn't look unreasonable and is a fairly common approach to this problem.
An approach I use is to have the daemon process obtain an exclusive (write) lock on a file.
You can test to see if anyone is holding the lock by trying to obtain the lock yourself, and there are various ways of obtaining the PID of a process holding a lock on a file - i.e. fcntl and probably something in /proc.
Some words of advice:
Use local files (ie. not NFS) for locks.
Make sure the lock file exist before the daemon is started.
Never delete the lock file.
The kernel associates locks with the inode number of the file, so you always want the lock file to have the same inode number throughout all time. Deleting and recreating the lock file will change the inode associated with the lock.
A simple keep alive mechanism can be implemented as a cron job - the cron job just tries to spawn the daemon process every N minutes, and then have the daemon quietly exit if it can't obtain the exclusive lock.
I was wondering if it was possible to run threads in Perl across multiple machines. I'm working in a clustered environment and need to run some of my process in parallel, but am unable to use MPI.
If threading is not able to be used across machines, are there any other alternatives I should look at that will allow me to do something similar and not require special modules?
Threads (and forks) in Perl are tied to the same computer as the parent thread / process, so no cross-computer threading / forking. That said, you can use AnyEvent::MP / Coro::MP modules, message-passing extensions to the AnyEvent asynchronous event-loop framework and the Coro co-routine, cooperative threading framework respectively, which let you create a network of nodes doing different tasks on one or multiple machines. See AnyEvent::MP::Intro for details.
As for alternatives not requiring special modules (by which, I guess, you mean modules not in the perl distribution), you could conceivably write a daemon for your tasks and have them communicate over TCP or UDP. Anything beyond that would probably require at least a few modules not installed with Perl, but available from the CPAN.
Have a look at Gearman, a multi-machine job manager queue. It does require special modules; I answered here "just in case" you can in fact use additional modules/infrastructure.
There are Perl bindings, Gearman::XS, which I successfully use in projects where I want specific tasks to be done in an environment where either requesters or workers processes may reside on multiple machines. Works well also for multiple worker processes on one machine and one requester (example: a certain web scraper which requests all links from a page parsed by any worker, but wants to keep control of the results).
The way it works is that you create a "worker" Perl program which has a number of subroutines which perform the action you'd like to perform in a distributed fashion. You launch those worker programs on whichever machines you want and as many times as you want, and have them connect to one (or multiple) master gearman "managers".
You then create a requester (gearman client) Perl program which will perform the requests. This can also run on any machine, and will contact the master gearman manager to request a number of the workers' specific actions completed. Any worker does it, and your requester gets the result back.
If your requesters don't need a result back but "just" need a task to happen, have instead a look at TheSchwartz which has a similar architecture but does not provide a facility for getting messages from the workers back to the requesters, IIRC.
Check GRID::Machine.
I stumbled upon GNU parallel a week or two ago, while not across separate machines it helps reduce time by allowing regular programs to take advantage of multiple cores. May help speed up whatever you're doing.
what is the difference between these concepts?
“Process” is well-defined; “job” and “task” are ambiguous.
Fundamentally a job/task is what work is done, while a process is how it is done, usually anthropomorphised as who does it. A job is an overall unit of work, and is composed of tasks. In practice usage is very inconsistent, and often “task” == “process”, though formally a process performs a task.
Process is a well-defined operating systems concept, as is thread: a process is an instance of a program that is being executed, and is the basic unit of resources: a process consists of or “owns” its image, execution context, memory, files, etc.; etymologically a process is the steps done by a processor. A process consists of one or more threads, which are the unit of scheduling, and consist of some subset of a process (possibly shared with other threads): execution context and perhaps more. Traditionally a thread is the unit of execution on a processor (a thread is “what is executing”), but with multi-core processors and hardware threads, some scheduling is done even at the level of a single core. There are various kinds of processes and threads, and the exact definition varies between platforms.
Job and task are today vague, ambiguous terms, especially task. A “job” often means a set of processes, while a “task” may mean a process, a thread, a process or thread, or, distinctly, a unit of work done by a process or thread.
To give an idea how confused the naming is,
Windows Task Manager manages (running) processes, while
Windows Task Scheduler schedules programs to execute in future, what is traditionally known as a job scheduler, and uses the .job extension!
The term “job” traditionally means a “piece of work” (as opposed to “occupation”), and is used as such in manufacturing, in the phrase “job production”, meaning “custom production”, where it is contrasted with batch production (many items at once, one step at a time) and flow production (many items at once, all steps at the same time, by item). Note that these distinctions have become blurred in computing, notably in the oxymoronic term “batch job”.
In computing, “job” originates in non-interactive processing on mainframes, notably in IBM’s Job Control Language for the DOS/360 and OS/360 of the mid-1960s, and formally means a “unit of work for an operating system”, which consists of steps, each of which is a request to execute a specific program. Early computers primarily did batch processing (running the same program over many input data), like census or billing, and a standard type of one-off job was compiling a program from source, which could then process batches of data. Later batch came to be applied to all non-interactive computing, whether one-off or multiple items.
In Unix shells, a “job” is the shell’s representation for a process group – a set of processes that can all be sent a signal – concretely a pipeline and its descendent processes; note that running a script starts a job, exactly as in mainframes. The job is not done until the processes complete, and a job can be stopped, resumed, or terminated, which corresponds to suspending, resuming, or terminating the processes. Thus while formally a job is distinct from the process group, this is a subtle distinction and thus people often use “job” to mean “set of processes”.
Traditional jobs (and batches) have finite input data and should complete processing, successfully or not. By contrast, when running a server, such as a web server, the input, such as a stream of requests, is unlimited (formally codata). This is analogous to flow production, and the process (or “job”) never completes, though it can be terminated or “canceled”. In a quip, “a server’s job is never done” (formally, exit status will be CANCELED, not COMPLETED/SUCCESS).
The term “step” makes sense for sequential computing – one step follows another – but once you have concurrent computing, you have a set of tasks, which do not necessarily run in a particular order, rather than a sequence of steps. The term “task” was popularized by OS/360, which featured “Multiprogramming with a Fixed number of Tasks (MFT)” and “Multiprogramming with a Variable number of Tasks (MVT)”, though in this case “task” was used synonymously with “process” or “thread”, as the basic task is “execute this program” (so the resulting process/thread performs the task), which is probably the source of the ambiguity.
Formally “multitasking” means “working on multiple tasks concurrently”, but in practice means an operating system (or virtual machine, or runtime, or individual process) “running multiple processes/threads concurrently”.
A clear distinction between tasks as work and process/threads as how the work is done is given in a
task queue, as in this diagram of a thread pool: there is a (big, potentially unlimited) queue of incoming tasks (pending), which are performed by a (small, often fixed) set of threads, each task being performed by a single thread, and each thread performing a single task at a time: the active tasks correspond to the active threads. Concretely, consider a multithreaded web server, where the tasks are “service this web page request”, and each thread fetches (from disk or memory) or renders the web page (say by a template or PHP), then returns the result.
As you can see from this last example, it is often useful to distinguish tasks from threads or processes, and in particular contexts “job” and “task” have specific meanings, though in general they are ambiguous.
Clearest is thus to avoid using “job” or “task” and instead refer to a “set of processes”, “process”, or “thread”, and for servers to refer to requests (or queries) rather than tasks.
They can be all considered the same thing, really depends on the context. A process though is usually an isolated entity that's managed by the operating system. A job is often more of an application level term or just some script that's executed to do a specific set of task(s). A task is often a part of a job - sometimes the only part.
A job is a unit of work that has been submitted by user. It is usually associated with batch systems. A batch job might be a request to run multiple programs in succession [pg 144]. However, it can be assumed that a job is a request to run a single program. Hence, depending on the context, a job can be a program (we usually assume this), or a set of programs (e.g. batch systems) [pg 8].
A process is an active entity, which requires a set of resources, including a processor and special registers to perform its function. It is a single instance of an executable program. So from here, you can see the connection between a process and a program, hence, a job.
The Linux kernel internally represents processes as tasks [pg 742].
Source: Modern Operating Systems (3rd edition) by Tanenbaum, published by Pearson Education, Inc, 2009
A task represents the execution of a single process or multiple processes on a compute node. A collection of tasks that is used to perform a computation is known as a job. Jobs are used to reserve the resources required by tasks.
source: jobs and tasks http://msdn.microsoft.com/en-us/library/bb525214%28v=vs.85%29.aspx
Well...
This might not be as clear as described here. It may very well depend of the operating system some one is dealing with.
For example when compiling a DIGITAL Equipment OSF1 kernel (also known as TruUnix64) -- when that Unix was still existant, end of the nineties, beginning of the century -- the term TASK was dedicated to the number of parallel tasks the kernel was able to handle.
It was a fixed array of tasks the kernel could perform at a given moment.
Thus it was the sum of the processes it could spawn as well as internal tasks it has to do even if not seen as processes by ps. Then it was a very low level count of actions allowed to the kernel on each NUMA node, not something accessible outside the kernel.
On the other hand a previous operating system like DEC VMS was known for having its base OS unit as a job (you interactively logged under a job) executing possibly (depending on system and account parameters and privileges) many processes at a time. An image (an executable) then occupying a process and (most of the time) multiple threads (the OS took care of multithreading by itself) at a time.
So, then, the job was not application related but really OS related.
Somewhat similarly Windows, which does not natively support fork() as a lightweight process creator, tends to create processes (using a spawn - CreateProcess - primitive that looks very much alike the one that existed onto VMS / OpenVMS 40 years ago) that are heavier that the Unix ones. Here, we have the same word (process) to describe (in term of OS) two realities that are quite different: a Windows process tends to be closer to a VMS job than a true Unix process.
As I did not configure/build any Unix Kernel since TrueUnix64, I am not able to discuss the TASK kernel parameter of a Debian or Linux OS if any. It might be interesting that someone with inner knowledge of the tasks limit of those kind of OS could explain us further on this concept in these systems.
To conclude: task, process, job, spawn, fork, thread... the more you dig into different OS, the more varieties you get and possible contradictory definitions you face.
Gilles
[non native English speaker, pardon my English].
In the course of testing the code for the question How can I store per-thread state between calls in Perl? I noticed that the first I time execute the script the threads execution are fairly well interleaved with each other. But on all subsequent executions of the script all the threads run almost perfectly in the order of their creation with very little interleaving. This is Perl ithreads on Ubunutu 9.04.
Maybe someone could enlighten me about what's going on?
Your threads are running in creation order largely due to implementation details in Perl and the operating system (mainly because their individual execution time is below the operating system's shortest execution-time slice). To interleave the threads, you can use sleep rather than yield (or make the threads do some real work).
Keep in mind that yield in Perl threads is just a suggestion, which is unlike the way yield works in some other languages. Since Perl threads are concurrent and are largely managed by the operating system's scheduler, unless you use some sort of mutex to block execution, its not really possible to predict their execution order.
Thread scheduling is a complex implementation detail of the OS. Trying to figure out how it works by observing how threads are scheduled is next to impossible. And you shouldn't really make any assumptions based on these observations. On different hardware the OS may schedule the threads differently.