boost::thread, pthread (valgrind/helgrind/drd)

[kfmfe04]

Recently, I've been reading up on Butenhof's "Programming with POSIX threads" when I discovered the boost::thread library. I thought I would ask some advice form veteran thread developers on the best approach for this simple first program.

What I'd like to do is, have the main execution thread and a clock thread.

The main execution thread may do calculations and other work, and periodically write to cout/cerr. The clock thread will display the current time via cout/cerr, every second, regardless of what the main thread is doing.

Would the correct approach be to wrap all I/O to cout/cerr in a class with a Mutex? Any suggestions to unit-testing this kind of program or ensuring robustness would be greatly appreciated! TIA.

[helios]

Not exactly the best threading example, is it? One of the threads is doing almost nothing at all. You could accomplish the same by, say, checking the elapsed time every 1000th iteration and print a progress message if it's been a second or more since the last update.

[kfmfe04]

Hmm... ...I am not trying to show the efficiency of threading through example - rather, I am trying to build a simple test case to ensure that I know how to use boost::thread properly, for use with more complex problems later.

I'm simplifying things a bit (always room to make things harder, later) - of course, I could make both threads do a lot of work, but I was trying to come up with an example where each thread is doing different things.

Thread1 - not time-critical and can be calculation-intensive, but not cout intensive
Thread2 - want to cout the clock at regular 1 sec intervals (as close as possible) - I could be blocked by Thread1, but want to minimize this

Maybe a simple Mutex wrapper around cout will do. I will probably try that first - having problems finding a good unit-testing framework for C++ multithreading though. I am currently using googletest - there must be something better.

Maybe something specifically made to test multithreaded apps?

[helios]

If you're using Linux, Valgrind can find concurrency problems in compiled programs at the cost of a huge performance overhead.

[kfmfe04]

If you're using Linux, Valgrind can find concurrency problems in compiled programs at the cost of a huge performance overhead.

That's good to know. Thx!

(I've used Valgrind for its memory checking - didn't know it can also handle concurrency checks. I will try it out.)

[Galik]

Unit testing with threads can be tricky. In your case you have two threads writing to the same stream. This can make it difficult to test the stream output for conformance to an expected standard.

I never like to code to std::cin/std::cout directly. I always make the input/output as switchable as possible. That way you can supply pre-determined input and compare against pre-recorded output easily. Concurrency makes that harder because the order of the output data will likely change from one run to another.

So I would be tempted to make a wrapper stream that distinguished all output with the thread_id of the calling thread. The wrapper stream could send the output from different thread_ids to different places to be compared separately or it could wrap each output with the thread_id like this:

{1:<output-line>}{2:<output-line>}{2:<output-line>}...

You will likely need to write your code in such a way as to facilitate regression testing. I always output to std::ostream, rather than std::ofstream so the outputs can be switched for testing purposes.

[kfmfe04]

Thanks, Galik - looks like a good idea.

Making code thread-safe forces us to write really clean code - no Singletons or other static data members, try to eliminate state as much as possible, encapsulate calls to 3rd party libraries and other code that we need to synchronize so we can make it reentrant, etc...

The merits of having built-in language keywords like Java's synchronized is becoming more apparent. Without such a feature in C++, I think I will have to either comment more often or come up with a consistent naming convention so it's obvious what classes or methods are thread-safe.

[kfmfe04]

OK, I've tried out valgrind/drd on a simple boost::thread program:

#include <iostream>

#include <boost/thread/thread.hpp>
#include <boost/thread/mutex.hpp>

#include <gtest/gtest.h>

static boost::mutex io_mutex;

struct MyCount {
  MyCount(int id) : id(id) { }
  void operator()() {
    for (int i = 0; i < 1000; ++i) {
      boost::mutex::scoped_lock lock( io_mutex );
      const char* c = (id==1) ? "." : "+"; 
      std::cerr << c;
    }
  }
  int id;
};

TEST( Test_thread, EnsureJoin )
{
  MyCount mc1(1),mc2(2);
  boost::thread thrd1(mc1);
  boost::thread thrd2(mc2);
  thrd1.join();
  thrd2.join();
  std::cerr << std::endl;
}

valgrind --tool=drd ./tst/bin/myapp_ut

and I get:

==16036== Conflicting load by thread 1 at 0x006086b0 size 8
==16036== Conflicting store by thread 1 at 0x006086b0 size 8
...
==16036== ERROR SUMMARY: 4042 errors from 46 contexts (suppressed: 9 from 9)

Googling around, it seems that valgrind's drd support for boost::thread appears to be on-going.

http://old.nabble.com/-drd--Race-condition-in-boost-thread-library--td28786561.html

Perhaps I should try to call pthread directly in hopes of getting better current support from valgrind/drd.
I am wary of attempting any thread programming without some reliable unit tests.

Helios, under what lib/environment have you used valgrind's drd successfully? TIA.

[helios]

I've never used it. Pthreads should work fine.

[kfmfe04]

ic. From what I can tell, valgrind/drd seems to err on the side of safety (reporting too many false positives, where a positive==some_bad_multithreading_condition).

So far, I have not been able to get any simple boost::thread to run cleanly through valgrind/drd.

Surprisingly, many pthread examples I have found also fail to pass valgrind/drd. I have managed to get one to work, so I will continue to play with that for now.

If anyone has any examples of mutexes, barriers, semaphores, etc... in either boost::thread or in Pthreads which does clear valgrind/drd, please post.

Thx - I will post mine later, once I get something reasonable working, in case anyone is interested.

[kfmfe04]

The valgrind/drd/helgrind problems seem to go deeper than just boost::thread or my assessment of false positives in the previous thread.

http://www.mail-archive.com/[e-mail]/msg00357.html

In an attempt to be POSIX-compliant with respect to making many common I/O routines threadsafe, the mechanisms used tend to be unorthodox and hard to detect, even for valgrind.

[kfmfe04]

ok - getting valgrind/drd/helgrind to work was much more difficult than I expected.

Here are notes to self and to others who may be interested:

1. compile flags to gcc may result in significantly different output. I found out the hard way that drd/helgrind doesn't seem to like -fprofile-arcs. In order the find out what sets of flags are ok, I advise downloading the helgrind source and running stuff in the tests directory (eg ~/src/valgrind/helgrind/tests). On a pass, you can pay attention to what flags were passed to the compiler and try to sync-up your build to these flags, as you see fit.

2. when you find an error there are a few possibilities: something is wrong with valgrind/drd/helgrind, something is wrong with your code/system libraries (out of date? using unorthodox memory or pthread code?), or both. If you can convince yourself that valgrind is producing a spurious error, you can run it with --gen-suppressions=yes . Valgrind will then prompt you to see if you want to generate suppression code. You can copy and paste this code into /usr/local/lib/valgrind/default.supp or some other non-default location to suppress these warnings/errors. In this case, it's useful to first google around and/or look at the source to determine if, indeed, the errors are spurious.

3. one small note on whether or not printf() is thread-safe. Apparently, there is a great deal of distrust and/or confusion in the community regarding the claims that printf() is thread-safe, as required by POSIX. I believe some of the confusion may be driven by the problem that printf() wasn't always thread-safe, bugs seem to pop up periodically which make it unsafe, or library maintainers use unorthodox methods (eg non-pthread) to ensure its thread-safe, so that valgrind throws false positives.

Ref 1: http://www.kernel.org/doc/man-pages/online/pages/man7/pthreads.7.html
Ref 2: http://manpages.ubuntu.com/manpages/intrepid/man7/pthreads.7.html