Create a variable with type based on a value from a binary file

alexqt (5)
I need to get the data and set the type, with data type based on a specific value from a binary file.

I don't have a problem with retrieving data from the binary file. My problem is how can I specify the type of the variable that will hold the data, where type will by dynamically set based on a value from the binary file?

I have researched auto, template specialization, and read articles from this site and other sites like stackoverflow, but can't seem to figure out how to correctly do it for almost a week now.

I need a function that reads the data type (numeric, occupying 1 byte in the file) and the value (8 bytes, can be any numeric data type like short, int, long, long, signed and unsigned). This function must send the value with the correct type back to the calling function.

This is sort of like a mini database, where anyone can set a value with a specific type, store it in the db, and then retrieve it.

I would greatly appreciate if someone can share the code that can do it.

BTW, I would like only to use standard c++ libraries and not boost.

template<typename T>
void ReadValue(const int& recordID, T& value)
{
// read type and value from binary file <- no problem with this
// seek the location of the recordID and read the typeID and value
// typeID is any number between 0-20.

// the following is where I'm having a difficult time
// set a variable with type based on a numeric value from the binary file
// set value of this variable to a value from the binary file.
}

Thanks in advance!
Last edited on
JLBorges (13770)
Use a union-like class, perhaps. For instance:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
 
#include <iostream>
#include <algorithm>
#include <fstream>
#include <cstring>

struct any // discriminated union (union-like-class) to hold int, double or long long
{
    enum type_t : char { INT = 1, DOUBLE = 2, LONG_LONG = 3 /* ... */ };
    static constexpr std::size_t MAX_VALUE_SIZE = std::max( sizeof(double), sizeof(long long) ) ;

    constexpr any( int v = 0 ) : type(INT), ival(v) {}
    constexpr any( double v ) : type(DOUBLE), dval(v) {}
    constexpr any( long long v ) : type(LONG_LONG), lval(v) {}
    // ...

    constexpr operator int() const { return operator long long() ; }
    constexpr operator double() const { return type == DOUBLE ? dval : operator long long() ; }
    constexpr operator long long() const { return type == INT ? ival : type == DOUBLE ? dval : lval ; }
    // ...

    operator int&() { if( type != INT ) throw type_error() ; return ival ; }
    operator double&() { if( type != DOUBLE ) throw type_error() ; return dval ; }
    operator long long&() { if( type != LONG_LONG ) throw type_error() ; return lval ; }
    // ...

    type_t type ;

    union
    {
        int ival ;
        double dval ;
        long long lval ;
        // ...
    };

    struct type_error : virtual std::domain_error { type_error() : std::domain_error("invalid type" ) {} } ;
};

std::ostream& write_any( std::ostream& binary_stm, any a )
{
    char type = a.type ;
    char value[any::MAX_VALUE_SIZE]{} ;

    switch(type)
    {
        case any::INT: std::memcpy( value, &a.ival, sizeof(a.ival) ) ; break ;
        case any::DOUBLE: std::memcpy( value, &a.dval, sizeof(a.dval) ) ; break ;
        case any::LONG_LONG: std::memcpy( value, &a.lval, sizeof(a.lval) ) ; break ;
        default: throw any::type_error() ;
    }

    binary_stm.write( &type, sizeof(type) ) ;
    return binary_stm.write( value, sizeof(value) ) ;
}

// function that reads the data type (numeric, occupying 1 byte in the file)
// and the value (8 bytes, can be any numeric data type like short, int, long, long, signed and unsigned).
std::istream& read_any( std::istream& binary_stm, any& a )
{
    char type ;
    char value[any::MAX_VALUE_SIZE] ;

    if( binary_stm.read( &type, sizeof(type) ) && binary_stm.read( value, sizeof(value) ) )
    {
        switch(type)
        {
            case any::INT: a = *reinterpret_cast< const int* >(value) ; break ;
            case any::DOUBLE: a = *reinterpret_cast< const double* >(value) ; break ;
            case any::LONG_LONG: a = *reinterpret_cast< const long long* >(value) ; break ;
            default: binary_stm.clear( std::ios::failbit ) ;
        }
    }

    return binary_stm ;
}

int main() // minimal test driver
{
    const char* const test_file_name = "test.bin" ;

    {
        std::ofstream test_file( test_file_name, std::ios_base::binary ) ;
        const any many[] { 123456, -67.89, 1234567890123456LL } ;
        for( any a : many ) write_any( test_file, a ) ;
    }

    {
        std::ifstream test_file( test_file_name, std::ios_base::binary ) ;
        any a ;
        while( read_any( test_file, a ) )
        {
            switch(a.type)
            {
                case any::INT: std::cout << int(a) << " (int)\n" ; break ;
                case any::DOUBLE: std::cout << double(a) << " (double)\n" ; ; break ;
                case any::LONG_LONG: std::cout << a.lval << " (long long)\n" ; break ;
                default: std::cout << "this is insane\n" ;
            }
        }
    }
}

http://coliru.stacked-crooked.com/a/18d6d6dacfa51a24
alexqt (5)
@JLBorges thank you for this! wow! I'll add additional data types like a "short" and a "string" to the code you gave and see how it does. Will a "string" also work on this?
Last edited on
JLBorges (13770)
> Will a "string" also work on this?

No, it won't. For more than one reason.

1. std::string has non-trivial foundation operations.
If a union contains a non-static data member with a non-trivial special member function (default constructor, copy/move constructor, copy/move assignment, or destructor), that function is deleted by default in the union and needs to be defined explicitly by the programmer.
http://en.cppreference.com/w/cpp/language/union

We would have to write a lot of boiler-plate code to work around this.

2. std::string is not TriviallyCopyable; we can't use binary i/o (read/write as in the code above) with types that are not TriviallyCopyable


I would strongly suggest:

a. Use text mode instead of binary mode.
When you feel the urge to design a complex binary file format, or a complex binary application protocol, it is generally wise to lie down until the feeling passes.
ESR in 'The Art of Unix Programming' http://catb.org/~esr/writings/taoup/html/ch05s01.html


b. Use boost::variant. It has already done all the hard work. http://www.boost.org/doc/libs/1_60_0/doc/html/variant.html

For instance:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
 
#include <iostream>
#include <boost/variant.hpp>
#include <string>
#include <iomanip>
#include <fstream>

using any = boost::variant< int, double, long long, std::string > ;

std::ostream& operator<< ( std::ostream& stm, const any& a )
{
    const auto which = a.which() ;
    stm << which << ' ' ;
    switch( which )
    {
        case 0 : return stm << boost::get<int>(a) << '\n' ;
        case 1 : return stm << boost::get<double>(a) << '\n' ;
        case 2 : return stm << boost::get<long long>(a) << '\n' ;
        case 3 : return stm << boost::get<std::string>(a) << '\n' ;
    }

    stm.clear( std::ios_base::failbit ) ;
    return stm ;
}

std::ostream& pretty_print( const any& a, std::ostream& stm = std::clog )
{
    switch( a.which() )
    {
        case 0 : return stm << boost::get<int>(a) << " (int)\n" ;
        case 1 : return stm << boost::get<double>(a) << " (double)\n" ;
        case 2 : return stm << boost::get<long long>(a) << " (long long)\n" ;
        case 3 : return stm << std::quoted( boost::get<std::string>(a) ) << " (std::string)\n" ;
    }
    return stm << "insane\n" ;
}

std::istream& operator>> ( std::istream& stm, any& a )
{
    int which ;
    if( stm >> which )
    {
        switch( which )
        {
            case 0 : { int i ; stm >> i ; a = i ; return stm ; }
            case 1 : { double d ; stm >> d ; a = d ; return stm ; }
            case 2 : { long long ll ; stm >> ll ; a = ll ; return stm ; }
                     // note: assumes that the string did not contain an embedded new line
            case 3 : { std::string s ; std::getline( stm >> std::ws, s ) ;  a = s ; return stm ; }
        }
    }
    stm.clear( std::ios_base::failbit ) ;
    return stm ;
}

int main()
{
    const char* const test_file_name = "test.bin" ;

    {
        std::ofstream test_file(test_file_name) ;
        const any many[] { 123456, "hello world!", -67.89, 1234567890123456LL, -729, "bye!" } ;
        for( any a : many ) test_file << a << '\n' ;
    }

    {
        std::ifstream test_file(test_file_name) ;
        any a ;
        while( test_file >> a ) pretty_print(a) << '\n' ;
    }
}

http://coliru.stacked-crooked.com/a/d4176de184b0a96b
alexqt (5)
@JLBorges, thank you for the advice. I guess I really need to use boost.

I added more data types to the variant variable, which includes a bool. After running the program, the string is not read correctly. It is now displaying as a bool. I think it is not reading the entire string. I need to delete bool in the variant variable definition and in the switches to make the string display correctly.

see code here http://coliru.stacked-crooked.com/a/1b22af35fe4bcafa

If I comment out the section that writes to a file, and edit the file directly instead, the string was read correctly. I guess I need to use literal for a string and bool like what you did for long long which is LL. I just don't know if there exist a literal for bool and string. Another solution is to just create a bool and string variable, assign them values, then use these variables for the contents of the many[] array.

Editing the file directly, I tried adding a signed char type, but I also can't read correctly a signed char value (1 byte for numeric –128 to 127). If it has a negative sign, it just displays the negative and ignores the rest of the number. I think it is only reading it as a single char value. same goes for unsigned char type (0-255). Issue still the same if I use signed/unsigned char variables to contain the values and use them in many[] array.

How can I store to a file and retrieve/display correctly signed/unsigned char types?

Many thanks!
JLBorges (13770)
> I added more data types to the variant variable, which includes a bool.
> After running the program, the string is not read correctly.

const char* to bool is a standard conversion;
it is favoured over the user-defined conversion const char* to std::string.


> I just don't know if there exist a literal for bool and string.

bool literals: true, false, bool(45)
std::string literals: "hello world!"s "abcd"s http://en.cppreference.com/w/cpp/string/basic_string/operator%22%22s


> How can I store to a file and retrieve/display correctly signed/unsigned char types?

signed char literal: (signed char)'S'
unsigned char literal: (unsigned char)'U'
char literal: 'C'

char should be added as an additional type in the variant; or else, 'C' would get promoted to int.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
 
#include <iostream>
#include <boost/variant.hpp>
#include <string>
#include <iomanip>
#include <fstream>

// added type 'char' (which is distinct from both 'signed char' and 'unsigned char'
using any = boost::variant< bool, unsigned char, signed char, unsigned short, unsigned int, unsigned long long, 
                            signed short, signed int, signed long long, float, double, std::string, char >;

std::ostream& operator<< (std::ostream& stm, const any& a)
{
        const auto which = a.which();
        stm << which << ' ';
	switch (which)
	{
	case 0: return stm << boost::get<bool>(a) << '\n';
	case 1: return stm << boost::get<unsigned char>(a) << '\n';
	case 2: return stm << boost::get<signed char>(a) << '\n';
	case 3: return stm << boost::get<unsigned short>(a) << '\n';
	case 4: return stm << boost::get<unsigned int>(a) << '\n';
	case 5: return stm << boost::get<unsigned long long>(a) << '\n';
	case 6: return stm << boost::get<signed short>(a) << '\n';
	case 7: return stm << boost::get<signed int>(a) << '\n';
	case 8: return stm << boost::get<signed long long>(a) << '\n';
	case 9: return stm << boost::get<float>(a) << '\n';
	case 10: return stm << boost::get<double>(a) << '\n';
	case 11: return stm << boost::get<std::string>(a) << '\n';
	case 12: return stm << boost::get<char>(a) << '\n'; // *** added
	}

	stm.clear(std::ios_base::failbit);
	return stm;
}

std::ostream& pretty_print(const any& a, std::ostream& stm = std::clog)
{
	switch (a.which())
	{
	case 0: return stm << std::boolalpha << boost::get<bool>(a) << " (bool)\n"; // *** boolalpha added
	case 1: return stm << boost::get<unsigned char>(a) << " (unsigned char)\n";
	case 2: return stm << boost::get<signed char>(a) << " (signed char)\n";
	case 3: return stm << boost::get<unsigned short>(a) << " (unsigned short)\n";
	case 4: return stm << boost::get<unsigned int>(a) << " (unsigned int)\n";
	case 5: return stm << boost::get<unsigned long long>(a) << " (unsigned long long)\n";
	case 6: return stm << boost::get<signed short>(a) << " (signed short)\n";
	case 7: return stm << boost::get<signed int>(a) << " (signed int)\n";
	case 8: return stm << boost::get<signed long long>(a) << " (signed long long)\n";
	case 9: return stm << boost::get<float>(a) << " (float)\n";
	case 10: return stm << boost::get<double>(a) << " (double)\n";
	case 11: return stm << std::quoted(boost::get<std::string>(a)) << " (std::string)\n";
	case 12: return stm << boost::get<char>(a) << " (default char)\n"; // *** added
	}
	return stm << "insane\n";
}

std::istream& operator>> (std::istream& stm, any& a)
{
	int which;
	if (stm >> which)
	{
		switch (which)
		{
		case 0: { bool b; stm >> b; a = b; return stm; }
		case 1: { unsigned char uc; stm >> uc; a = uc; return stm; }
		case 2: { signed char sc; stm >> sc; a = sc; return stm; }
		case 3: { unsigned short us; stm >> us; a = us; return stm; }
		case 4: { unsigned int ui; stm >> ui; a = ui; return stm; }
		case 5: { unsigned long long ull; stm >> ull; a = ull; return stm; }
		case 6: { signed short sc; stm >> sc; a = sc; return stm; }
		case 7: { signed int si; stm >> si; a = si; return stm; }
		case 8: { signed long long sll; stm >> sll; a = sll; return stm; }
		case 9: { float f; stm >> f; a = f; return stm; }
		case 10: { double d; stm >> d; a = d; return stm; }
				// note: assumes that the string did not contain an embedded new line
		case 11: { std::string s; std::getline(stm >> std::ws, s);  a = s; return stm; }
	        case 12: { char c ; stm >> c ; a = c ; return stm ; } // *** added
		}
	}
	stm.clear(std::ios_base::failbit);
	return stm;
}

int main()
{
    const char* const test_file_name = "test.bin" ;

    {
        std::ofstream test_file(test_file_name) ;
        using namespace std::literals ; // C++14
        
        // note: bool(4), (signed char)'A', (unsigned char)'B'
        // std::string literals  "hello world!"s "bye!"s (namespace std::literals)
        const any many[] { 123456, bool(4), (signed char)'A', (unsigned char)'B', 'C', "hello world!"s,
                           -67.89, 1234567890123456LL, false, -729, "bye!"s } ; 

        for( any a : many ) test_file << a << '\n' ;
    }

    {
        std::ifstream test_file(test_file_name) ;
        any a ;
        while( test_file >> a ) pretty_print(a) << '\n' ;
    }
}

http://coliru.stacked-crooked.com/a/b13cafdf0c2361db
Last edited on
alexqt (5)
will it also work for numbers for the signed and unsigned char instead of letters? say for signed chars it can store and display numbers between –128 to 127, for unsigned char between 0-255?
Cubbi (4774)
Those are some scary switches, btw. With boost variants, that operator<< is just
1
2
3
4
5
6
 
std::ostream& operator<< (std::ostream& stm, const any& a)
{
    stm << a.which()  << ' ';
    apply_visitor([&](auto&& x){ stm << x << '\n'; }, a);
    return stm;
}


...or you could even skip it entirely and use boost.variant's own operator<<
for( any a : many ) test_file << a.which() << ' ' << a << '\n' ;

(the pretty-print would require slightly less trivial visitor, of course)
Last edited on
JLBorges (13770)
> for signed chars it can store and display numbers between –128 to 127, for unsigned char between 0-255?

For that, we would have to promote and write them as integers; and then read the value as an integer, (verify that it is not out of range), and narrow it to back to the appropriate character type.

Something like:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
 
#include <iostream>
#include <boost/variant.hpp>
#include <string>
#include <iomanip>
#include <fstream>
#include <cctype>

using any = boost::variant< char, unsigned char, signed char >;

std::ostream& operator<< (std::ostream& stm, const any& a)
{
    const auto which = a.which();
    stm << which << ' ';
    switch (which)
    {
        case 0: return stm << +boost::get<char>(a) << '\n';
        case 1: return stm << +boost::get<unsigned char>(a) << '\n'; // note: integral promotion with unary +
        case 2: return stm << +boost::get<signed char>(a) << '\n';
    }

    stm.clear(std::ios_base::failbit);
    return stm;
}

std::ostream& pretty_print_char_as_int( int v, const char* type, std::ostream& stm )
{
    stm << v ;
    if( std::isprint(v) ) stm << " '" << char(v) << "' " ;
    return stm << " (" << type << ")\n" ;
}

std::ostream& pretty_print(const any& a, std::ostream& stm = std::clog)
{
    switch (a.which())
    {
        case 0: return pretty_print_char_as_int( boost::get<char>(a), "char", stm ) ;
        case 1: return pretty_print_char_as_int( boost::get<unsigned char>(a), "unsigned char", stm ) ;
        case 2: return pretty_print_char_as_int( boost::get<signed char>(a), "signed char", stm ) ;
    }

    return stm << "insane\n";
}

std::istream& operator>> (std::istream& stm, any& a)
{
    int which;
    if (stm >> which)
    {
        switch (which)
        {
            // note: range check elided for brevity
            case 0: { int c ; stm >> c ; a = char(c) ; return stm; }
            case 1: { int uc ; stm >> uc ; a = (unsigned char)uc; return stm; }
            case 2: { int sc ; stm >> sc ; a = (signed char)sc; return stm; }
        }
    }
    stm.clear(std::ios_base::failbit);
    return stm;
}

int main()
{
    const char* const test_file_name = "test.txt" ;

    {
        std::ofstream test_file(test_file_name) ;
        using namespace std::literals ; // C++14

        using schar = signed char ;
        using uchar = unsigned char ;
        const any many[] { 'A', schar(-78), uchar(246), schar('B'), 'C', char(12), uchar('D') } ;
        for( any a : many ) test_file << a << '\n' ;
    }

    {
        std::ifstream test_file(test_file_name) ;
        any a ;
        while( test_file >> a ) pretty_print(a) << '\n' ;
    }

    { std::cout << "\n\nfile " << test_file_name << " contains:\n" << std::ifstream(test_file_name).rdbuf() ; }
}

http://coliru.stacked-crooked.com/a/6d3a0731a849a9e6
alexqt (5)
Thank you again JLBorge. Thanks to you too Cubbi. :-) Happy New yr!
Topic archived. No new replies allowed.