So I have this iterative program which needs to perform the same calculations for a large number of different possibilities. Obviously I would like to take advantage of multiple cores when available. Ive determined that because none of the possibilities need to communicate with each other, multiprocessing is likely the best way of doing this over multithreading.
I know fork and the exec family can create new processes but I am having difficulty finding which command will work best. Specifically what I need to do is launch from my main program, say four child processes that have a specific set of data to crunch, while leaving the main program running to catch child process terminations, log the results, and start the next one. I can't see any way to do this with fork or exec.
So my next thought was to just launch multiple programs and let the OS handle the core distributions. I would have them store their data to a temp file for later reintegration, but I cant find a way to launch the program with a copy of objects in my main program.
That said, the only solution I've come up with so far is to create data files prior to launching the new program and in its command line call pass it a path to the file which contains the data to crunch. It seems to be a simple solution, but I cant imagine it is the most efficient with all the file IO going on.
I'm open to suggestions....if it sounds like I've bit off more than I can chew, you're likely right, but I've found there is no better way to learn then when go head long into a problem.
Well In addition/revision to the above, I just realized using the system call to launch the programs wont work if i want to launch multiple instances of it so I guess my work around wont work anyway. Any suggestions?
[EDIT]: My apologies guys....I went back and re-read the man pages after taking a break from it all, and the solution seems rather clear:
Use fork to create the new processes, then call exec from within them. As far as getting the data in I will use the multiple temp files and pass the file name as an argument in the exec call. Still requires a lot of file IO though. Does anyone have any suggestions for improvements?
You don't need exec(). exec() is used to launch a different program.
The gist of what you want is
1 2 3 4 5 6 7 8 9 10
// Run the program using the values 0...9 as input
for( int i = 0; i < 10; i++ ) {
pid_t p = fork();
if( p == 0 ) { // child
Crunch( i );
} elseif( p > 0 ) { // parent
// Store p somewhere
} else { // fork failed
}
}
There is a synchronization problem you need to solve with the above code
(namely there is no guarantee that the child will get far enough to copy i before the parent modifies it.)
You do _not_ want the individual processes to write their output to a common file since the synchronization you need to ensure the file isn't corrupted a) is hard, and b) will probably kill the parallelism in your program (unless the time to write to file is dominated by processing time).
Thanks for the input jsmith. Your suggestion sounds good but there is still the matter of dealing with ensuring the child can save the data before the parent modifies it. Would a Boolean flag that the parent waits for the child to flip work?
Also, regarding the files, assuming I went the execv() route there would be no file sharing between processes...the parent would wait for the child to finish then read in the results....keeping it all to ram though is obviously better though so I'll attempt what you suggested first.
IMO. Using a thread-pooling system would be a better design approach. It'd be faster because your not going to have the file system I/O or having to continually spawn new processes.
My program spawns each new process only once. Each process is assigned a fraction of the total data and then it executed continously until it has finished it's share. It logs the data back into the father program every so often and the father program writes it to a file to keep the memory footprint low. I may have left some ambiguity in my previous post...there is only one file IO opperation going on now, and it is in a process that is doing nothing else other than monitoring the others.
In response to your other response, I am not confused. I am aware of the added overhead of creating new processes, hence the reason for leaving them running. Also, this program may one day soon be improved to run on a cluster, so I will apparently be one step closer to that acording to you. My intention was that once I got the multiprocessing working I would add multitudeading to each process. I don't really know for sure about other forms of unix or Linux, but OS X will handle moving multiple processes to multiple cores just as it would threads.