I don't know anything about OpenMP, neither Boost nor pthread. So, before begin learning one of these, I'd like to explain to you how I want my application to work.
It deals with running two parallel loops, one in LabView, and the other in C++. It doesn't matter if you don't know anything about LabView, this explanation is easy enough to understand. I want to know whether my application would be possible to achieve using OpenMP, or whether it'd be easier with another API.
First, the execution begins in LabView. Here, I call a dll function written in C++. This is the "initialization" function. Its aim is to create another thread that will execute the main loop in C++. Then, this "initialization" function ends, because it has to return the control execution to LabView, but the new thread which has just been created must continue executing in its own loop.
Now we have two loops in parallel: LabView and C++. In LabView's loop, we call periodically another dll function that has to send and retrieve data from the independent thread in C++. This function, like the other, has to end executing to pass control to LabView, after it has done its purpose.
This will be the behaviour until we call from LabView another dll function to stop the independent thread. Then, the LabView program ends.
Which multithreading library would you recommend me for this application?
So, in other words, you have one thread that's relevant, not two.
OpenMP is designed to easily parallelize algorithms, not to do things like, say, write a non-blocking function. Boost and pthreads would work, but I think the simplest solution would be to use the Windows API to create your threads. If you're not afraid to add dependencies to your project, Boost is easier to use, but its interface is slightly more complicated. pthreads only really makes sense if you're going to compile for both Windows and UNIX.
EDIT: Come to think of it, use Boost. The Windows API isn't very friendly when it comes to synchronization. Boost's mutexes are much simpler to use.