std::ifstream::getline empty, in rather "big" project

[hansaaa]

Hi,

I am doing some matrix operations, and, in the process of it, need to read in matrices, or, rather, matrix lines (vectors, if you so will), into c++.

I have a system of equations with 2079 lines and 12 variables.

It is stored in a comma-separated text file E:\foot\mat.txt

a sample line looks like this:

3.727272749,0.071428575,0.071428575,0.214285716,0,0.071428575,3.636363745,0.071428575,0.071428575,0.142857149,0,0,-1000

From these 2079 lines, I want to choose 12, so that I can solve this "resulting SubMatrix" for the 12 variables.

I want to do this a fair amount, choosing different 12 lines each time. I have chosen, for now, to do 20 Choose 12 (20C12) combinations of lines, resulting in 20C12, or, 125970 sub-matrices.

I have code for solving them and all, and that is not the problem (although It'll take a while ;)). BUT, when reading in the lines, there is a problem at some point.

Let me try and break this down even further, into parts:

Part (a): Generate all possible 20C12 combinations, and store them in a manner that they can be used for any purpose (in this case, for "indexing" the lines I want to read from the "mat.txt" file.

I have an algorithm for this, which looks like:

void do_next(std::vector<set> *newSets, int t[], int i, int n, int N )
{
	for( int j = t[i-1] + 1; j <= N - n + i; ++j)
    {
        t[i] = j;
        if( i == n)
        {
			set newSet;
            for(int k = 1; k <= n; ++k)
            {
              
				newSet.intSet.push_back(t[k]);
            }
			newSets->push_back(newSet);
        }                                
        else
        {
            do_next(newSets, t, i + 1, n, N);
        }
    }
}

and works fine. It a struct set, in order to hold an individual combination in the 20C12 set of combinations, i.e., just a set of integers. it is this:

struct set
{
	std::vector<int> intSet;
};

,

and pushes_back all these individual sets into a std::vector<set>, and, then, returns this std::vector.


So, Part (a), generating the combinations of possible lines, is done.

Then part (b):

Reading in the lines into the code, and, then, storing them in my customly defined avector struct:

struct avector{
	std::vector<double> doubles;
};

.

This struct stored the individual (in this case double) "numbers"/"entries", or what you will, entries in an individual vector.

The resulting avector object instances are then stored in a matrix struct:

struct matrix{
	std::vector<avector> vectors;
};

So far, so good.

My code, for reading in the vectors from the text file into an individual avector, and, subsequently, matrix instance, is:

matrix getMatrixFromSubFile(set rows, std::string fn, int factor)
{
	matrix ma;
	std::ifstream *f = new std::ifstream(fn);
	char a[10000];

	for (int i = 0; i < rows.intSet.size(); i++)
	{
		for (int j = 0; j <rows.intSet.at(i)*factor; j++)
		{
			if (j == rows.intSet.at(i)*factor -1)
			{
				f->getline(a, 10000);
				std::string szVec = std::string(a);
				avector curvec = szVectorToRealVec(szVec);
				ma.vectors.push_back(curvec);
				f = new std::ifstream(fn);
			}
			else
			{
				f->getline(a,10000);
			}

		}
	}
	f->close();
	delete f;
	return ma;
}

which should be pretty straight forward. It "loops" through the file, using std::ifstream::getline, and, once it reaches a certain line, converts the char buffer *a, or a[10000] to a std::string.

It then uses szVectorToRealVec, defined here:

vector szVectorToRealVec(std::string szVec)
{
	avector vec;
	bool bEOL = false;	

	int posS = 0;
	int posE = 100000000;
	int l = 0;
		
	while(bEOL != true)
	{
		std::string af;
		if(szVec.find(",") != -1)
		{
			posE = szVec.find(",");
			l = posE - posS;
			af = szVec.substr(posS, l);
			szVec = szVec.substr(af.length()+1);
		}
		else
		{
			bEOL = true;
			af = szVec.substr(posS);
		}
		_CRT_DOUBLE *curDStr = new _CRT_DOUBLE;
		_atodbl(curDStr,const_cast<char*>(af.c_str()));
		double d = double(curDStr->x);
		vec.doubles.push_back(d);
		delete curDStr;
	}

	return vec;
}

to create an avector with the current vector/line stored in the text file.

This avector is then stored in the matrix.

The condition
if (j == rows.intSet.at(i)*factor -1), i.e., the factor-1 part, in getMatrixFromSubFile is used so that I can read in lines in the range of the entire system of equations, i.e. lines in range [1,2079]. I chose 77 for the value of factor, because, I will probably later use 27C12 combinations, rather than 20C12, and, 27*77 is 2079, resulting in the max line I have in the file.

So, again, so far, so good. This works, I can generate the 20C12 combinations, and then read in the matrices. However, my part(c), subsequent solving of the matrices, later crashes, and it ALWAYS (at least so far), crashes at matrix number 43. This happens, because, for some reason, when READING IN matrix number 43, from the "mat.txt" file, using the code posted above, the line

f->getline(a, 10000);, to place the line into the char buffer a, "fails", leaving the char buffer empty.

Tracking this down further, I could see that, for this "matrix number 43", it was still able to read in the first FIVE lines of the matrix, from the text file. Using the combination number 43, generated from the previuos algorithm, and using the factor of 77, which I used, I can see that it, thus, reads in lines
77*1 (so 77), 77*2 (so 154), 77*3 (so 231), 77*4(so 308), and line 77*5 (so, 385), successfully, but, then, on the line number six, 77*6, which is 462, the line:
f->getline(a, 10000);,

fails to place anything in the char buffer, just leaving it absolutely empty (or, rather, I THINK, as an empty string "").

Verifying line 462 in my "mat.txt" file, I can see that it is:

3.363207579,0.091954023,0.068965517,0.130268201,0,0.003831418,3.6340909,0.122137405,0.110687025,0.106870227,0,0,1000

, and, that, at least from my perspective, nothing seems to be "weird" or "funny" about neither the previous, nor the next lines:

461:

3.795348883,0.114503816,0.087786257,0.156488553,0.003816794,0.003816794,3.799086809,0.07662835,0.07662835,0.107279696,0,0.007662835,-1000

or 463:

3.751162767,0.08429119,0.057471264,0.183908045,0.003831418,0,3.775701046,0.080769233,0.065384619,0.115384616,0.007692308,0.003846154,0

.

So, I have really no idea why it is failing, and, always on matrix number 43, line 6.

I am only guessing, that it might have something to do with memory allocation, somehow?

The task is at about 420,000 KB of private memory allocated, reported by task mon, at the time it crashes.


Finally, I will post my calling code, so you can see how I call the stuff defined above:

int _tmain(int argc, _TCHAR* argv[])
{
	std::vector<matrix> matrices;
	int toI;
	int combosize;
	while (true)
	{
		std::vector<set> *combos = new std::vector<set>;	
		std::cout << "from 1 to ";
		std::cin >> toI;
		std::cout << std::endl << "pick ";
		std::cin >> combosize;
		int * t = new int[combosize + 1];
		t[0] = 0;
		
		time_t s = time(NULL);
		do_next(combos, t, 1, combosize, toI);
		delete t;
		time_t a = time(NULL);
		time_t diff = difftime(a,s);
		std::cout << std::endl;
			
		std::cout << combos->size() << " combinations" << std::endl;
		std::cout << "t: " << diff << std::endl;

		std::string fn = std::string("E:\\foot\\combos\\combos.txt");
		std::cout << combos->size() << " printing to " << fn.c_str();
		std::cout << std::endl;

		std::cout << "size of combos is " << sizeof(combos) << std::endl;

		std::ofstream *f = new std::ofstream(fn);
		setsToFile(*combos, f); 
		f->close();
		delete f;

		std::cout << "printed" << std::endl;


		for (int i = 0; i < combos->size(); i++)
		{
			std::cout << "getting matr " << i+1 << std::endl;
			matrices.push_back(getMatrixFromSubFile(combos->at(i), "E:\\foot\\mat.txt", 77));
		}

		std::cout << "size of matrices is " << sizeof(matrices) << std::endl;
		
		for (int i = 0; i < matrices.size(); i++)
		{
			std::cout << "solving mat " << i+1 << std::endl;
			matrix ma = solveSystem(matrices.at(i));
		}


		std::cout << std::endl;

	
		delete combos;
		matrices.erase(matrices.begin(), matrices.end());
	}	
	return 0;
}