problem reading big flat file

[kinchan]

Dear Friends,

I have a flat file that has 219789 lines(each line has 5 records separated by Pipe).
I have a C Program that reads that flat file and inserts data into the database. When I run the C program to
process the whole flat file, it fails to read few records from few rows. For e.g. Below is the portion of the
original flat file that could not read completely.
1646090|1645522|100LAY|16|1|
1646090|1645529|100LAY|16|1|
1646090|1645537|100LAY|16|1|
1646090|1645544|100LAY|16|1|
1646090|1645551|100LAY|16|1|

Rather than reading the whole records, program is reading only the following
1646090|1645522|100
1646090|1645529|100
1646090|1645537|100
1646090|1645544|100
1646090|1645551|100

I am having problem with just above rows from 219789 lines from the flat file. Next time, I processed part of
the flat file, just 20 rows, including the above rows; at that time, program is reading all the records completely.
Why the heck C program is not reading when we process the big file(219789 lines). I am suspecting that there might be
memory issue but I am not sure. could somebody help me in this regard.

Thanks.
Kinchan

[jsmith]

Try a test where the file contains only the lines that are failing to be read and see what happens.

(And how are you reading the file? Obviously the third column cannot be read into an integer
data type because it will read only the 100, and leave the LAY.)

[Fenor]

Hi,

try this read-in loop, it should read all lines without problems.

int first,second,third,fourth,fifth;
		char dummy[4];
		while (!source.eof()){
			source >> first;
			source.ignore(1); //remove |
			source >> second;
			source.ignore(1);
			source >> third;
			if (source.peek() == 'L'){ 
				source.get(dummy,4,'|'); //remove LAY
				source.ignore(1);
				source >> fourth;
				source.ignore(1);
				source >> fifth;
				source.ignore(1);
				//if (!source.eof())	//remove duplicate of the last element in file
				//list.push_back(xx(first,second,third,fourth,fifth); //5 elements
			}
//			else if (!source.eof())
//				list.push_back(xx(first,second,third); // 3 elements
		}

[turbozedd]

How about reading a string instead of stripping? Splitting it into something else should be delegated to a function, that way you don't have to key on the peek value.

[kinchan]

Thanks guys.... The real problem was different. There was a hiden weired character between 100 and LAY in the third column. I came to know about this when I imported the flat file from Unix box to windows text pad. Thanks a lot for your help...