How to use copy_n and istream_iterator to copy integers from binary file to vector?

[dfchu]

Hello,

I'm trying to read from a binary file of unsigned integers and copy them into a vector using the copy_n function. However, when it gets to the copy_n command, my debugger gives me an assertion error: "The stored stream pointer in_stream must be non-null". Looking at the code, it seems as though std::istream_iterator<unsigned int>(index_is) maybe has null values? I'm not sure why. Am I misunderstanding how the istream_iterator works?

#include <vector>
#include <iostream>
#include <fstream>

int main(int argc, char* argv[])
{
    std::ofstream os(argv[1], std::ios::out | std::ios::binary);
    std::vector<unsigned int> vec1 = { 1,2,3,4,5,6,7,8,9,10 };
    if (os)
    {
        for (auto i : vec1)
        {
            os.write(reinterpret_cast<const char*>(&i), sizeof(i));
        }
        os.close();
    }
    else
    {
        std::cout << "unable to open output file";
        return 1;
    }

    std::ifstream index_is(argv[1], std::ios::in | std::ios::binary);  
    if (index_is)
    {
        std::vector<unsigned int> vec2;
        std::copy_n(std::istream_iterator<unsigned int>(index_is), 
                          10, 
                          std::back_inserter(vec2));
    }
    else
    {
        std::cout << "unable to open input file";
        return 1;
    }
}

Thank you for your time.

[dutch]

istream_iterator uses operator>> for input.

BTW, you're missing both the <algorithm> and <iterator> headers.
Also, you don't need to say std::ios::in for an ifstream (that's what the i means)
or std::ios::out for an ofstream (that's what the o means).

It's also best to deal with file opening problems right away and to not put your regular processing in an if statement.

#include <algorithm>
#include <fstream>
#include <iostream>
#include <iterator>
#include <vector>

int main(int argc, char** argv)
{
    if (argc != 2)
    {
        std::cout << "Usage: " << argv[0] << " FILENAME\n";
        return 1;
    }

    std::ofstream out(argv[1], out.binary);
    if (!out)
    {
        std::cerr << "unable to open output file\n";
        return 1;
    }

    for (unsigned i = 1; i <= 10; ++i)
        out.write(reinterpret_cast<const char*>(&i), sizeof(i));
    out.close();

    std::ifstream in(argv[1], in.binary);
    if (!in)
    {
        std::cerr << "unable to open input file\n";
        return 1;
    }

    // This part still doesn't work, of course!
    std::vector<unsigned> v;
    std::copy_n(std::istream_iterator<unsigned>(in),
                10,
                std::back_inserter(v));

    for (unsigned n: v)
        std::cout << n << '\n';
}

[Handy Andy]

Hello dfchu,

If yo are going to use "copy_n", from the "<algorithm>" header then you first need to include the "algorithm" at the top of your program.

Then you could take a look at this: http://www.cplusplus.com/reference/algorithm/copy_n/

This does not read from a file, you should be able to adjust it to.

The rest I will have to work on to see what I come op with.

Andy

[dfchu]

Thanks. So does that mean I can't use an istream_iterator for this? I'm not sure how >> works with binary files.

So would the best way be to just have a for loop like:

unsigned int num;
for (int i=0; i<10; i++)
{
    index_is.read(reinterpret_cast<char*>(&num), sizeof(num));
    vec2.push_back(num);
}

I'm not really familiar with how iterators work, so I thought maybe there is a more c++ way of doing it.

[Handy Andy]

Hello dfchu,

Like you I am not that familiar with iterators.

Just for comparison and something to think about I came up with this:

#include <algorithm>
#include <iostream>
#include <vector>

#include <fstream>

int main(int argc, char* argv[])
{
    if (argc < 2)  // <--- Or !=2.
    {
        std::cerr <<
            "\n     Invalid command line arguement list!\n"
            "     Usage is (Prog. Name) (File Name) [anything else]\n";

        return 1;
    }

    std::vector<unsigned int> vec1 = { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 };

    std::ofstream outFile(argv[1], std::ios::out | std::ios::binary);

    if (!outFile)
    {
        //std::cout << "\n File " << std::quoted(outFileName) << " did not open" << std::endl;
        std::cout << "\n File \"" << argv[1] << "\" did not open." << std::endl;

        return 2;
    }

    for (auto i : vec1)
    {
        outFile.write(reinterpret_cast<const char*>(&i), sizeof(i));
    }

    outFile.close();

    std::ifstream inFile(argv[1], std::ios::in | std::ios::binary);

    if (!inFile)
    {
        //std::cout << "\n File " << std::quoted(inFileName) << " did not open." << std::endl;
        std::cout << "\n File \"" << argv[1] << "\" did not open." << std::endl;

        return 3;
    }

    int num{};
    std::vector<unsigned int> vec2;

    while (inFile.read(reinterpret_cast<char*>(&num), sizeof(num)))
    {
        vec2.push_back(num);
    }

    return 0;  // <--- Not required, but makes a good break point.
}

Something to consider.

Andy

[mbozzi]

As @dutch points out, std::istream_iterator performs formatted input and is therefore not suitable for unformatted binary files. Formatted input operations may skip whitespace, for example.

You could use std::istreambuf_iterator instead, but that only gets you half-way there.

I thought maybe there is a more c++ way of doing it

What would replacing this simple for-loop mean for the quality of your final product?

[JLBorges]

> So does that mean I can't use an istream_iterator for this?

You can't use std::istream_iterator for a standard type for this.
With a user defined type, stream extraction and insertion operators behave as we instruct them to behave.

For example, with a user defined type to wrap stream extraction and insertion for the items in the vector:

#include <iostream>
#include <type_traits>
#include <vector>
#include <algorithm>
#include <iterator>
#include <string>
#include <fstream>

template < typename T > struct as_bits // a wrapper that reads and writes object of type T as bits
{
    static_assert( std::is_trivially_copyable<T>::value ) ;

    T value{} ;

    as_bits( const T& v = T{} ) : value(v) {}

    operator T& () noexcept { return value ; }
    operator const T& () const noexcept { return value ; }

    friend std::ostream& operator<< ( std::ostream& stm, const as_bits<T>& v )
    { return stm.write( reinterpret_cast< const char* >( std::addressof(v.value) ), sizeof(T) ) ; }

    friend std::istream& operator>> ( std::istream& stm, as_bits<T>& v )
    { return stm.read( reinterpret_cast<char*>( std::addressof(v.value) ), sizeof(T) ) ; }
};

int main()
{
    const std::string file_name = "test_file.dat" ;

    const std::vector< unsigned int > vec { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9  };

    // write contents (as bits) into a file
    {
        std::ofstream file { file_name, std::ios::binary } ;

        file << as_bits<std::size_t>( vec.size() ) ; // write the size of the vector

        // write out the integer values
        std::copy_n( vec.begin(), vec.size(), std::ostream_iterator< as_bits<unsigned int> >(file) ) ;
    }

    // read it back into another vector
    {
         std::vector< unsigned int > vec2 { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9  };

         std::ifstream file { file_name, std::ios::binary } ;

         // read the size of the vector
         as_bits<std::size_t> sz ;
         file >> sz ;

         vec2.resize(sz) ; // provide enough space for sz values

         // read the sz integer values into the vector
         using istream_iterator = std::istream_iterator< as_bits<unsigned int> > ;
         std::copy_n( istream_iterator(file), vec2.size(), vec2.begin() ) ;

         // check if what was read is the data that was written out earlier
         if( vec2 == vec ) std::cout << "vec2 == vec: ok. this is what is expected\n" ;
         else std::cerr << "something is wrong\n" ;
    }
}

http://coliru.stacked-crooked.com/a/49bd0561c32fd85a

[seeplus]

to read from a binary file of unsigned integers

Before even thinking about writing some code, you need to know a) the number of bits stored for each integer b) the endianess used (little or big)

Is this a file you're written (as seems from the first post or is that just a test?) or a file you're given? If it's always a file you're written, then you know a) and b) but if it's not then you need to know.

copy them into a vector using the copy_n function

Why copy_n? Is this a given requirement - because usually you wouldn't.

Consider:

#include <vector>
#include <iostream>
#include <fstream>

int main(int argc, char* argv[])
{
	const std::vector<unsigned int> vec1 {1, 2, 3, 4, 5, 6, 7, 8, 9, 10};

	if (argc < 2)
		return (std::cout << "File name not specified\n"), 1;

	std::ofstream os(argv[1], std::ios::binary);

	if (!os)
		return (std::cout << "Unable to open output file\n"), 2;

	for (const auto i : vec1)
		os.write(reinterpret_cast<const char*>(&i), sizeof(i));

	os.close();

	std::ifstream index_is(argv[1], std::ios::binary);

	if (!index_is)
		return (std::cout << "Unable to open input file\n"), 3;

	std::vector<unsigned int> vec2;

	for (unsigned int i {}; index_is.read(reinterpret_cast<char*>(&i), sizeof(i)); vec2.push_back(i));

	for (const auto n : vec2)
		std::cout << n << ' ';

	std::cout << '\n';
}

[JLBorges]

The elements in a vector are stored contiguously; if there is no requirement that copy_n must be used,
we do not have to read and write each item one by one in a loop.

#include <iostream>
#include <string>
#include <vector>
#include <fstream>

int main()
{
    const std::string file_name = "test_file.dat" ;

    const std::vector< unsigned int > vec { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9  };

    // write contents (as bits) into a file
    {
        std::ofstream file { file_name, std::ios::binary } ;
        std::size_t vec_sz = vec.size() ;
        file.write( reinterpret_cast< const char* >( std::addressof(vec_sz) ), sizeof(vec_sz) ) ; // write the size of the vector

        // write out the vec_sz integer values in one shot (vector storage is contiguous)
        file.write( reinterpret_cast< const char* >( vec.data() ), vec_sz * sizeof( vec.front() ) ) ;
    }

    // read it back into another vector
    {
         std::ifstream file { file_name, std::ios::binary } ;

         // read the size of the vector
         std::size_t vec_sz ;
         file.read( reinterpret_cast<char*>( std::addressof(vec_sz) ), sizeof(vec_sz) ) ;

         std::vector< unsigned int > vec2(vec_sz) ; // a vector that holds vec_sz values

        // read the vec_sz integer values in one shot (vector storage is contiguous)
        file.read( reinterpret_cast<char*>( vec2.data() ), vec_sz * sizeof( vec.front() ) ) ;

         // check if what was read is the data that was written out earlier
         if( vec2 == vec ) std::cout << "vec2 == vec: ok. this is what is expected\n" ;
         else std::cerr << "something is wrong\n" ;
    }
}

http://coliru.stacked-crooked.com/a/d761e1b9c91d2988

[oggin]

Then there is the old fashioned way from C to read/write in a block.

#include<iostream>
#include <fstream>
#include<vector>
#include <string>
using namespace std;


int range(int f,int l)
   { return (rand() % ((l-f)+1)) + f;}

std::ifstream::pos_type filesize(const char* filename)
{
    std::ifstream in(filename, std::ifstream::ate | std::ifstream::binary);
    return in.tellg(); 
}

template <class S>
void save(string filename,S array)
{
    FILE * f = fopen(filename.c_str(), "wb");
    int size= array.size()*sizeof(array[0]);
   auto ap = array.data();
    fwrite(ap,size,1,f) ;
    fclose(f);
}

template <class L>
void load(string filename,L  &array)
{
	FILE * f = fopen(filename.c_str(), "rb");
	auto ap = array.data();
	fread(ap,array.size()*sizeof(array[0]),1,f); 
   fclose(f);
}

template <class P>
void print(P u)
{
	for (int i=0;i<u.size();i++) cout<<u[i]<<",";
	cout<<endl;
}


int main(int argc, char* argv[])
{
	vector<unsigned int> u;
	char * filename;
	if(argv[1])
	 filename=argv[1];
	 else
	 filename = (char *)"integers.txt";
	
	u.resize(200);
	for(int i=0;i<u.size();i++) u[i]=range(0,1000); //fill the array
	

    cout<<"Sent to file "<<filename<<endl;
	print(u);
	
	save<vector<unsigned int>>(filename,u);
	
	int s=filesize(filename)/sizeof(unsigned int); // get the number of elements
	cout<<endl;
	cout<<"number of elements to reload "<<s<<endl;
	
	cout<<endl;
	vector<unsigned int> out;
	out.resize(s);
	load<vector<unsigned int>>(filename,out);
	cout<<"Back from file and loaded into a vector<...> "<<endl;
	print<vector<unsigned int>>(out);
	cout <<"  "<<endl;
    cout <<"Press return to delete the binary file "<<filename<<". . ."<<endl; 
    cin.get();	
	//get rid
	 if( remove( filename ) != 0 )
     cout<<"Error deleting file, please complete the task manually"<<endl;
     else
     cout<<endl;
     cout<< "File successfully deleted"<<endl;
  
  
cout <<"  "<<endl;
cout <<"Press return to end . . ."<<endl; 
cin.get();	  	
}


 

Either from the command line or a straight run.
A question, I can use print(vector) or save(vector) or load(vector) directly, but for a template I though that :
print<vector<unsigned int>>(out);
should be used.
i.e.
<vector<unsigned int>>(vector) can be left out.

[lastchance]

#include <iostream>
#include <fstream>
#include <iterator>
#include <vector>
#include <algorithm>
using namespace std;

int main()
{
    ofstream out( "test.dat", ios::binary );
    for ( unsigned i = 1; i <= 10; ++i ) out.write( reinterpret_cast<char*>( &i ), sizeof i );
    out.close();

    ifstream in( "test.dat", ios::binary );
    union { char c; unsigned e; } u;
    vector<unsigned> V;
    for ( unsigned i = 1; i <= 10; ++i )
    {
       copy_n( istream_iterator<char>( in >> noskipws ), sizeof u.e, &u.c );
       V.push_back( u.e );
    }
    for ( auto x : V ) cout << x << '\n';
}

For cpp.sh:

#include <iostream>
#include <fstream>
#include <sstream>
#include <iterator>
#include <vector>
#include <algorithm>
using namespace std;

int main()
{
    ostringstream out( ios::binary );
    for ( unsigned i = 1; i <= 10; ++i ) out.write( reinterpret_cast<char*>( &i ), sizeof i );

    istringstream in( out.str() );
    union { char c; unsigned e; } u;
    vector<unsigned> V;
    for ( unsigned i = 1; i <= 10; ++i )
    {
       copy_n( istream_iterator<char>( in >> noskipws ), sizeof u.e, &u.c );
       V.push_back( u.e );
    }
    for ( auto x : V ) cout << x << '\n';
}