Stop python interpreter in C++

[joaopedro]

I'm embedding python in a C++ plug-in. The plug-in calls a python algorithm dozens of times during each session, each time sending the algorithm different data. So far so good.

But now I have a problem: The algorithm takes sometimes minutes to solve and to return a solution, and during that time often the conditios change making that solution irrelevant. So, what I want is to stop the running of the algorithm at any moment, and run it immediately after with other set of data.

Here's the C++ code for embedding python that I have so far:

    void py_embed (void*data){

	counter_thread=false;

	//To inform the interpreter about paths to Python run-time libraries
	Py_SetProgramName(arg->argv[0]);

	if(!gil_init){
		gil_init=1;
		PyEval_InitThreads();
		PyEval_SaveThread();
	}
	PyGILState_STATE gstate = PyGILState_Ensure();

    // Build the name object
    pName = PyString_FromString(arg->argv[1]);

    // Load the module object
    pModule = PyImport_Import(pName);
	
    // pDict is a borrowed reference 
    pDict = PyModule_GetDict(pModule);
	
    // pFunc is also a borrowed reference 
    pFunc = PyDict_GetItemString(pDict, arg->argv[2]);
	
	PyObject *pValue2,*pArgs2;

	pArgs2=Py_BuildValue("siii",arg->msg,arg->TypeOfCall,arg->InstructionsMax,arg->DeviationMax);
	pValue2 = PyObject_CallObject(pFunc, pArgs2); //line where it takes it time...

    /* Treat the data that comes from python
    ...
    */

 	
    // Clean up
	Py_XDECREF(pArgs2);
	Py_XDECREF(pValue2);
    Py_DECREF(pModule);
    Py_DECREF(pName);

	PyGILState_Release(gstate);

	delete data;
	counter_thread=true;
	_endthread(); 
		
};

and the way I call the py_embed from the main thread:

handle=(HANDLE) _beginthread(py_embed,0,(void*)cmd2);
//cmd2 é a informação que envio do C++ para o python 

So basically what I need is a way to stop `PyObject_CallObject` from the main or another thread. How can I do this?

I tried TerminateThread already. The thread terminated but I was unable to create another thread and I think that it's because this solutions creates GIL lock problems.

Note: The python algorithm is not mine so the solution where the python code has to be changed in order to test for a condition on each cycle is the very last solution.

[Computergeek01]

What you want is called an "Asynchronous Procedure Callback". This has to be setup inside of the function that you are applying it to so if that function is just calling the Python interpreter then it gets tricky.

- http://msdn.microsoft.com/en-us/library/windows/desktop/ms681951(v=vs.85).aspx

[helios]

An APC won't solve anything if it's a problem with locks, since he still won't be able to release them from within the callback.

The most reliable solution is to move the call to Python to a different process and do IPC. If the process finds a solution on time, you leave it idling for the next problem. If it doesn't, you kill it and start a new process. Unlike with threads, this is perfectly safe.

[Computergeek01]

@ helios: To be clear here, your not talking about calling the Python stuff in another process and calling TerminateProcess when you want to cancel it, you're saying to make a clean exit right?

The only reason that OP is seeing a lock of any kind to begin with is because he's using TerminateThread. Point taken about the APC not being able to release the locks. But there is no reason that this can't all be in the same process. The OP just needs a flag of some sort that the Python portion of the application checks on periodically so that when that flag gets set the thread knows to exit cleanly. They could do this with a global Boolean if they wanted to. Normally I'd say tack a pointer to a localized bool on to the threads argument list, but since the Python interpreter is locked anyway he won't have more then one thread using it so it won't make a difference in this case.

[helios]

To be clear here, your not talking about calling the Python stuff in another process and calling TerminateProcess when you want to cancel it, you're saying to make a clean exit right?

I'm saying to call TerminateProcess(). Making the process kill itself is also viable, but unnecessary.

The OP just needs a flag of some sort that the Python portion of the application checks on periodically

That's the preferred solution, yes. However,

the solution where the python code has to be changed in order to test for a condition on each cycle is the very last solution.

[chrisname]

Computergeek01 wrote:

you're saying to make a clean exit right?

TerminateProcess is a clean exit. The OS (assuming it's a modern, production-quality one) always frees any resources a process is using when that process ends. The only time it would be a problem is if the Python program was doing output, because then it might not get an opportunity to finish its output.

I want to point out, though, that usually a managed program calls native code when it wants to do CPU-intensive processing more quickly. Why is the OP doing it the other way around?