Reading Anonymous Pipes in Real-Time WinAPI

[Dennis M]

Hello everyone,

I'm working on an application which reads the output of another program (which I did not write) through anonymous pipes. Now, I have the reading working fine; the problem is doing this in real-time. I've been doing research for a little over a week and can't seem to figure out how to properly accomplish this. It normally doesn't read until the child process has returned. I will post the relevant code. Now that I have a commented line in the beginning of my code (ping 127.0.0.1 -t) which works exactly how I am looking, but this could be because the program is returning with every ping? Not entirely sure.

void Arc_Redirect::createProcesses()
{
	//TCHAR programName[]=TEXT("ping 127.0.0.1 -t");
	TCHAR programName[]=TEXT("Hola.exe");
	PROCESS_INFORMATION pi; 
	STARTUPINFO si;

	ZeroMemory(&pi, sizeof(PROCESS_INFORMATION));
	ZeroMemory(&si, sizeof(STARTUPINFO));
	si.cb = sizeof(STARTUPINFO); 
	si.hStdError = outStd[0].hOutWrite;
	si.hStdOutput = outStd[0].hOutWrite;
	si.hStdInput = outStd[0].hInRead;
	si.dwFlags = STARTF_USESHOWWINDOW|STARTF_USESTDHANDLES;

	if(!CreateProcess(NULL,programName,NULL,NULL,TRUE,0,NULL,NULL,&si,&pi))
		throw "Error: Could not start Program!";

	// Create thread
	outStd[0].hThreadRd = CreateThread(NULL,0,Arc_readPipe,(LPVOID)outStd[0].hOutRead,0,&outStd[0].threadIdRd);
	outStd[0].hThreadWr = CreateThread(NULL,0,Arc_writePipe,(LPVOID)outStd[0].hInWrite,0,&outStd[0].threadIdWr);
	if(outStd[0].hThreadRd == NULL || outStd[0].hThreadWr == NULL) throw "Error creating thread!";

	// Cleanup the useless handles
	if(!CloseHandle(pi.hThread) || !CloseHandle(pi.hProcess))
		throw "Error: Could not CloseHandle();";
}

This is the class member which creates the processes. The callbacks are defined as follows. Please note I am using FindWindow() because the HWND is not defined in the global scope of this file. Also, the other callback is defined, but at the moment I am not having problems with it and would simply like to focus on this aspect right now. bRunThread is defined as a BOOL FALSE until told otherwise to terminate.

DWORD WINAPI Arc_readPipe(LPVOID threadParam)
{
	Arc_Redirect ar;
	CHAR chBuf[BUFSIZE];
	DWORD dwRead;
	HANDLE hPipe = (HANDLE)threadParam;
	HWND g1  = FindWindow("ArcGUI",NULL);
	HWND dlg = GetDlgItem(g1,IDO_WORLDOUT);
	
	while(bRunThread)
	{
		while(TRUE)
		{
			if(!ReadFile(hPipe,chBuf,BUFSIZE,&dwRead,NULL) || !dwRead)
				if(GetLastError() == ERROR_BROKEN_PIPE)
				{
					ar.appendText(dlg,chBuf);
					memset(chBuf,0,sizeof(chBuf));
					break; // Normal exit
				} else {
					MessageBox(g1,(LPCSTR)GetLastError(),"Cannot Read!",MB_ICONWARNING);
				}

			chBuf[dwRead] = '\0';
			//MessageBox(g1,"Read data","Read",MB_OK);
			if(chBuf != NULL && dwRead != 0)
			{
				ar.appendText(dlg,chBuf);
				FlushFileBuffers(hPipe);
			}
		}
	}

	return 1;
}

EDIT: outStd is defined as PIPE_HANDLES (the struct below)

	typedef struct
	{
		HANDLE hOutRead;
		HANDLE hOutWrite;
		HANDLE hInRead;
		HANDLE hInWrite;
		HANDLE hProc;
		HANDLE hThreadRd;
		HANDLE hThreadWr;
		OVERLAPPED o1;
		DWORD threadIdRd;
		DWORD threadIdWr;
	} PIPE_HANDLES, *LPSTDPIPE;

Not that the is entirely relevant, just an extra bit for you guys to help with. The pipes ARE created correctly even though I do not show the creation of the pipes.

Regards,
Dennis M.

[kbw]

the problem is doing this in real-time

What do you consider real-time and what are you dong differently?

CreateThread
You can't create a thread using this and safely use the C runtime library.

// Cleanup the useless handles
The handles aren't useless. You can wait on the process handle if you want check it it's still running.

[Dennis M]

I would like to read the input within 5 seconds of having new information available. Right now it doesn't send anything until the program exits.

[kbw]

Within 5 seconds is hardly realtime.

If the source has buffered output, you'll have to wait until it flushes its buffer. It's possible that the data simply isn't written yet.

[Dennis M]

Is there a way to force this from my end? I wrote a sample application and it writes immediately upon starting, then it sleeps for 5 seconds and writes some more information and then exits after another sleep for 3 seconds. I don't receive information until the exit. I understand 5 secs is hardly real-time, but at this point beggars can't be choosers. I just can't wait 2mins for the output to finish before it's read on the real application.

Regards,
Dennis M.

[kbw]

Is there a way to force this from my end?

No.

Does the app write to stdout when you've not redirected its output? Or is there no output there either?

[Dennis M]

Which app? The child does properly write to stdout. Since it's a GUI program I haven't looked at a console stdOut for my program.

[kbw]

What is outStd? How is it declared and initialised?

[Dennis M]

The OP has the struct which it is declared as. It is initialized using ZeroMemory();

[kbw]

Don't you give it a valid handle?

[Dennis M]

Yes. outStd[0] is an array of variables. outStd[0].hReadOut is defined as HANDLE

[kbw]

Yes, but you don't actually assign a valid handle that you read. You just assign zero.

[Dennis M]

I am not entirely understanding what you are saying, I am sorry. Do you mean when I do this?

	if(!CreatePipe(&outStd[0].hOutRead,&outStd[0].hOutWrite,&sa,0) || !CreatePipe(&outStd[0].hInRead,&outStd[0].hInWrite,&sa,0))
		throw "Error: Could not CreatePipe();!";

	if(!SetHandleInformation(outStd[0].hOutRead,HANDLE_FLAG_INHERIT,0) || !SetHandleInformation(outStd[0].hInWrite,HANDLE_FLAG_INHERIT,0))
		throw "Error: Could not SetHandleInformation();";

I didn't include that in OP.

[kbw]

Take a look at this:
http://msdn.microsoft.com/en-us/library/ms682499(VS.85).aspx

Output is not captured until child process ends (resolved)
With some experiments, I found the output is buffered at child side's STDOUT with every 4096 bytes. So, I add a "fflush(stdout)" after printf(), the output redirect to parent without problem then.

[Dennis M]

I have read that from MSDN before. The problem with that is I cannot edit the child process so it can flush its stdout.

[kbw]

Unitl the flush occurs, the data isn't written to the stream, so there's nothing to read.

You get the same issue with tail -f. It's not aways obvious, but if you start filtering tail into a couple of greps, the output takes for ever to arrive.