I've written a shell, and instead of adding useful features like a scripting language I've decided to give it its own built-in terminal emulator (which can be turned off with a command-line option), firstly because I can (the best part of writing your own software is that you can put an e-mail client in a virtual machine) and secondly because I can.
As I said I've already written the bulk of the code and it all uses standard streams. So, I'm brainstorming ideas for getting I/O to-and-from SFML. Currently I'm thinking of putting the GUI in a different thread and having it poll the standard streams. The problem is I can't think of how that will work when the shell opens something in another process.
I'm curious because this would effect how I would go about the issue you are stuck on. How complex of a shell are we talking? It's a pretty generic term and could mean anything from cmd.exe being a shell that handles the "command consule" stuff, to explorer.exe\X.exe that handle the whole user GUI experiance and has handles into system memory space.
It's a cmd/bash-esque (it's more like bash, I'm writing it to be customisable and scriptable) shell. It's not supposed to be "professional" (read: usable) like bash, though; it's more of an academic exercise for me.
Edit: it also has a few bugs that need to be ironed out ;]
So this is interprocess communication, in which case I know that for the Win32 API CreateProcess(...) and CreateThread(...) both return a handle to the program for reasons just like this.
The following feels like I'm trying to dig too deep into things but here it is anyway:
AFAIK in the Win32 API this is done in the memory space for the new thread, it is allocated so that the parent has read access to it's handle and can therefore pass it commands. When the new process is created a structure is filled out with the relavent data including the memory addresses of it's startup information, this structure is kept up to date in the parent process. Keep in mind though this is my interpretation of what I read in a book written by Greg Hougland, and if you've been following the news recently I'll let you make your own decisions about his abilities. Despite the recent troubles though I've personally seen enough proof to absolve him of that snafu, his word is still cannon in my mind, and I also may be misunderstanding everything he had written.
Meanwhile I am reading through the header files included in the Windows 7 DDK, if I come across a function for this I'll post what I find.
Thing is, I'm trying to keep my shell cross-platform. Cygwin and MinGW provide POSIX headers on Windows but only Wine allows Windows programs to run on non-Windows platforms and, well... Wine kinda sucks.
I have to disagree with your assesment of WINE. I've seen it do some impressive things with minimal help from other applications.
But I can understand wanting to keep this open, especially for a learning experiance, there is no need and no reason to cop out and just go for a Windows app because it's easier.