Another Performance check

[vivmen]

Hi all
after my 1'st performance check
http://cplusplus.com/forum/general/56290/

i try another one this time i am using

AMD Phenom 2 X6 1090T 3.2 GHZ, Windows 7 , boost 1.48 , MINGW GCC (NetBeans 7.1)

This time 2D array to test boost multidimensional array

seems boost 1.48 did good job with array it gives even better result then normal array, however boost multidimensional did not do well.(i may be wrong as usual)
i try below code please check and suggest.

#include <cstdlib>
#include <windows.h>
#include <iostream>
#include <vector>
#include <boost/multi_array.hpp>
#include <boost/array.hpp>
using namespace std;
using namespace boost;
 /*
  * 
 */
void arrr()
{
    int arr[10][10];
    LARGE_INTEGER start,end;
    QueryPerformanceCounter(&start);
    
    for(int i=0 ; i <10 ; ++i)
    {
        for(int j=0 ; j <10 ; ++j)
        {
            arr[i][j] = 10;
        }
    }
    
    for(int i=0 ; i <10 ; ++i)
    {
        for(int j=0 ; j <10 ; ++j)
        {
            cout<<arr[i][j] <<" ";
        }
    }
    QueryPerformanceCounter(&end);

    cout<<end.LowPart - start.LowPart<<endl;
}

void barr()
{
    array<array<int,10>,10 > arr;
    LARGE_INTEGER start,end;
    QueryPerformanceCounter(&start);
    
    for(int i=0 ; i <10 ; ++i)
    {
        for(int j=0 ; j <10 ; ++j)
        {
            arr[i][j] = 10;
        }
    }
    
    for(int i=0 ; i <10 ; ++i)
    {
        for(int j=0 ; j <10 ; ++j)
        {
            cout<<arr[i][j] <<" ";
        }
    }
    QueryPerformanceCounter(&end);

    cout<<end.LowPart - start.LowPart<<endl;
}

void vec()
{
    vector<vector<int> > vec(10);
    for(int i = 0 ; i <10 ; ++i)
    {
        vector<int> v(10);
        vec.push_back(v);
    }
    LARGE_INTEGER start,end;
    QueryPerformanceCounter(&start);
    
    for(int i=0 ; i <10 ; ++i)
    {
        for(int j=0 ; j <10 ; ++j)
        {
            vec[i].push_back(10);
        }
    }
    for(int i=0 ; i <10 ; ++i)
    {
        for(int j=0 ; j <10 ; ++j)
        {
            cout<<vec[i][j] <<" ";
        }
    }
    QueryPerformanceCounter(&end);

    cout<<end.LowPart - start.LowPart<<endl;
}

void bmarr()
{
    boost::multi_array<double, 2> A(boost::extents[10][10]);
    
    LARGE_INTEGER start,end;
    QueryPerformanceCounter(&start);
    
    for(int i = 0; i < 10; ++i)
    {
        for(int j = 0; j < 10; ++j)
        {
            A[i][j] = 10;
        }
    }

    for(int i = 0; i < 10; ++i)
    {
        for(int j = 0; j < 10; ++j)
        {       
            cout<<A[i][j]<<" ";
        }
    }
    
    QueryPerformanceCounter(&end);

    cout<<end.LowPart - start.LowPart<<endl;
}

int main(int argc, char** argv)
{
    arrr();
    barr();
    vec();
    bmarr();
    return 0;
}

result:
153
107
190
940

[Athar]

There are several problems with this test:

1. Don't use such small samples for measuring, it's extremely unreliable.
Setting 100 values is such an insignificant amount of work that calling QueryPerformanceCounter might take significantly more time than the task itself. Repeat each test at least one million times or use larger arrays with less repetitions.

2. Never include any output in measuring. It takes a lot of time and is extremely unreliable (for example, the first function you call might take longer printing the values than the following ones), so what you're measuring here is mainly output performance.

Instead of printing, do something with the values like summing them up and returning the result. You have to print the result in main() to prevent the entire function from being optimized away. You should also initialize each member with different values, otherwise the compiler might figure out that the result is 10*10*10=1000 and optimize the function away anyway.

3. You use double in the last test. Printing doubles takes longer than printing ints, so it's not surprising that it takes longer.

4. The vector test is incorrect. You start with 10 elements for both the outer and inner vectors and then add ten each.
Replace the entire initialization with vector<vector<int> > vec(10,vector<int>(10)); and change vec[i].push_back(10); to vec[i][j] = 10;

[vivmen]

of course i don't have any standard test environment , i am doing this just as a fun activity with c++.
in real practice some one have to test it in stable & standard environment to get actual & proper result. (i wish if i could do that at my home).
any way don't rely or draw any conclusion from above test just try it on your machine for little fun.

[Athar]

It doesn't have anything to do with standard environments or anything, the test is just broken for the reasons I mentioned. It could give the impression that one method is faster than the other, when this might not be true at all.

After fixing some of the problems, the results only indicate for certain that multi_array is slower than the rest (at least with gcc 4.6 -O3 -m64). Note that this is NOT visible from your test, it just shows that printing doubles takes longer than printing integers.

Also, artificial tests like this are only rarely useful. Not only are you testing the performance of something that won't ever be used in the same way in a real program, but there's often too much compiler optimization potential that can mess things up (e.g. using a constant for each array element, possibly allowing the compiler to compute parts of the problem at compile-time in some situations).

So whenever possible, you should test performance of different approaches in a program that solves a real problem.

[therockon7throw]

Hi

Athar said

2. Never include any output in measuring. It takes a lot of time and is extremely unreliable (for example, the first function you call might take longer printing the values than the following ones), so what you're measuring here is mainly output performance.

I do agree, remove all output, they have nothing to do with the performance of the library you use. One library may compute an inverse of a matrix of size 120X120 just in a few seconds(1-3), but the output process for the same matrix may required more than some minutes :-)

[vivmen]

thanks for your suggestions i will try other possible ways as well to make it much closer to real world practice so people will get clear idea of performance. (i think my current test might put wrong picture)