Read .csv file with 365 columns and 5000 rows

[jjardin61]

Hi,

I am trying to write a function that reads information from a .csv file. I find this very difficult in c++ especially if the format is dynamic. Please let me know the best way to handle this.

The file has 5 columns with text data and then 360 columns with numerical information. The 360 columns are 30 years of data split by month. In my program I want to read the 360 data columns into a vector variable. The rows repeat new information for different scenarios.

Normally I read in information using the "fscan_s" function. The problem here is that since there are 360 columns of information I would need to have 360 of the "%lf" and this seems like there must be another way with loops. Below is a sample code for reading a file with only 11 columns.

How do I code this efficiently for a file with 360 + 5 columns?

    int ReadLokum(void)
    {
    	int iVal_1, iVal_2;
    	int ret = 0;
    	int i = 1;
    	int j = 1;
    	FILE * infile;
    	char buf[12];
    	char temp_name[1000];
    	strcpy_s(temp_name, cRunLocation);
    	_itoa_s(CurrentSet, buf, 10);
    	strcat_s(temp_name, "Lokum_");
    	strcat_s(temp_name, buf);
    	strcat_s(temp_name, ".csv");
    	errno_t err;
    	err = fopen_s(&infile, temp_name, "r");
    
    	if (infile == NULL)
    	{	LogFile << "Error reading file Lokum_.csv" << endl;
    		exit(2);	}
    	else
    		LogFile << "Successfully opened file Lokum_.csv" << endl;
    
    	char chrX[1000];
    	fgets(chrX, sizeof(chrX) - 1, infile); //Skip past header
    	while (1)
    	{
    		for (i = 0; i <= 360; i++)
    		{
    			ret = fscanf_s(infile, "%d,%d,%lf,%lf,%lf,%lf,%lf,%lf,%lf,%lf,%lf",
    				&iVal_1,
    				&iVal_2,
    				&LokumFIXED[j][i],
    				&LokumRate5[j][i],
    				&LokumUST7[j][i],
    				&LokumBeta[j][i],
    				&LokumUS[j][i],
    				&LokumAGGR[j][i],
    				&LokumSMALL[j][i],
    				&LokumINTL[j][i],
    				&LokumMONEY[j][i]);
    			if (ret == EOF) break;
    		}
    		if (ret == EOF) break;
    		j++;
    		if (j > LastScenario) return 0; // Exit to save time
    	}
    
    	fclose(infile);
    	return 0;
    }

[salem c]

An example of using %n to step your way through a char buffer.

#include <stdio.h>
#include <stdlib.h>

int main()
{
    char buff[] = "12,3456,7890123\n";
    char *p = buff;
    double d;
    int n;
    // The format does the following
    // Extracts a float
    // Skips a comma or a newline, but doesn't store either (doesn't count as a conversion)
    // Store the number of characters consumed by this format (doesn't count as a conversion).
    while( sscanf(p,"%lf%*[,\n]%n",&d,&n) == 1 ) {
        printf("Extracted %f, using %d chars\n", d, n );
        p += n;
    }
    return 0;
}


$ ./a.out 
Extracted 12.000000, using 3 chars
Extracted 3456.000000, using 5 chars
Extracted 7890123.000000, using 8 chars

Then read your file one line at a time using a large buffer (make sure you always get a \n in it), then use the above to step your way through it.

You could also use strtok() to do this.

[thmm]

Why this mix of C and C++ ?
Why don't you use a string and stringstream ?
Can you upload a small file with some sample data so we can see the excact format

[jjardin61]

Yeah, my compiler is 2017 and it only uses the _s functions.

I put together the below, but it is not iterating to the next number assigned to "x1" inside the while loop.

#include <stdio.h>
#include <stdlib.h>

int main(){

        char string1[] = "1,2,3,4,5,6,7,8,9,10,11";
	char seps[] = " ,\t\n";
	char *next_token1 = NULL;
	printf("Tokens:\n");
	double x1;

	while (1) {
		char *token1 = strtok_s(string1, seps, &next_token1);
		sscanf_s(token1, "%lf", &x1);
	}
}