how to serialize binary data?

[density]

i have an 144-bit array of ints that are 1s and 0s. i need to split up the array into 24 6-bit symbols and store each symbol in a single variable. i am unsure of what kind of variable is best for this kind of thing. this is what i have but will it work? it does compile but i am unsure.

generate_gf(  ) ;
  gen_poly(  ) ;

  int i;
  uint8_t u0, u1, u2, u3, u4, u5, u6, u7, u8, u9, u10, u11, u12, u13, u14, u15, u16, u17, u18, u19, u20, u21, u22, u23 ;
  uint8_t cw[nn] = {u0, u1, u2, u3, u4, u5, u6, u7, u8, u9, u10, u11, u12, u13, u14, u15, u16, u17, u18, u19, u20, u21, u22, u23 };  

  for (i=0;   i<6; i++)   u0  = rs_codeword[i] ; 
  for (i=6;   i<12; i++)  u1  = rs_codeword[i] ;  
  for (i=12;  i<18; i++)  u2  = rs_codeword[i] ; 
  for (i=18;  i<24; i++)  u3  = rs_codeword[i] ;  
  for (i=24;  i<30; i++)  u4  = rs_codeword[i] ; 
  for (i=30;  i<36; i++)  u5  = rs_codeword[i] ;  
  for (i=36;  i<42; i++)  u6  = rs_codeword[i] ; 
  for (i=42;  i<48; i++)  u7  = rs_codeword[i] ;  
  for (i=48;  i<54; i++)  u8  = rs_codeword[i] ; 
  for (i=54;  i<60; i++)  u9  = rs_codeword[i] ;  
  for (i=60;  i<66; i++)  u10 = rs_codeword[i] ; 
  for (i=66;  i<72; i++)  u11 = rs_codeword[i] ;  
  for (i=72;  i<78; i++)  u12 = rs_codeword[i] ;  
  for (i=78;  i<84; i++)  u13 = rs_codeword[i] ; 
  for (i=84;  i<90; i++)  u14 = rs_codeword[i] ;  
  for (i=90;  i<96; i++)  u15 = rs_codeword[i] ; 
  for (i=96;  i<102; i++) u16 = rs_codeword[i] ;  
  for (i=102; i<108; i++) u17 = rs_codeword[i] ; 
  for (i=108; i<114; i++) u18 = rs_codeword[i] ;  
  for (i=114; i<120; i++) u19 = rs_codeword[i] ; 
  for (i=120; i<126; i++) u20 = rs_codeword[i] ;  
  for (i=126; i<132; i++) u21 = rs_codeword[i] ; 
  for (i=132; i<138; i++) u21 = rs_codeword[i] ;  
  for (i=138; i<144; i++) u23 = rs_codeword[i] ; 
  
  
  for (i=0; i<12; i++)  data [i] = cw[i] ;          //data hex-bits
  for (i=12; i<24; i++)  bb [i] = cw[i] ;          //parity hex-bits

  for (i=0; i<nn-kk; i++)  recd[i] = bb[ i ] ;         // the 12 parity hex-bits first
  for (i=0; i<kk; i++)     recd[i+nn-kk] = data[ i ] ;   // the 12 data hex-bits second 
  for (i=0; i<nn; i++)     recd[i] = index_of[recd[i]] ;   //  put 24 hex-bit codeword into index form;  

any help will be much appreciated. Thanks!

[Chervil]

for (i=0; i<6; i++) u0 = rs_codeword[i] ;
This is just repeatedly overwriting the previous value of u0
You might just as well put this it gives the same result:
u0 = rs_codeword[5] ;

Please, how is rs_codeword declared? I see it is an array, but it isn't clear of what type.

At a guess you may need some bit-shifting (or just plain ordinary multiplication by 2) to get the correct value when combining 6 values into a single result.

[density]

its a pointer declared the function that calls the function containing the code i posted above
uint16_t * rs_codeword = new uint16_t[144];

How could i get binary data grouped into 6-bit symbols?? Thanks!

[coder777]

Like so:

for (i=0; i<144; i++)
{
  cw[i / 6] |= rs_codeword[i] << (i % 6);
}

Make sure that cw is filled with 0

EDIT: not tested!

[JLBorges]

#include <iostream>
#include <cstdint>
#include <bitset>
#include <string>
#include <vector>

std::string to_string( const std::uint8_t* a, std::size_t n )
{
    std::string result ;

    using bits = std::bitset<8> ;
    for( std::size_t i = 0 ; i < n ; ++i ) result += bits( a[i] ).to_string() ;

    return result ;
}

template< std::size_t NBITS >
std::vector<std::uint8_t> to_n_bit_values( const std::string& str )
{
    std::vector<std::uint8_t> result ;

    using bits = std::bitset<NBITS> ;
    for( std::size_t i = 0 ; i < str.size() ; i += NBITS )
        result.push_back( bits( str.substr( i, NBITS ) ).to_ulong() ) ;

    return result ;
}

template< std::size_t NBITS = 6 >
std::vector<std::uint8_t> to_n_bit_values( const std::uint8_t* a, std::size_t n )
{ return to_n_bit_values<NBITS>( to_string(a,n) ) ; }

int main() // minimal test driver
{
    constexpr std::size_t N = 6 ;
    const std::uint8_t a[N] = { 0, 1, 25, 3, 255, 49 } ;
    for( int i : a ) std::cout << i << ' ' ;
    std::cout << '\n' ;

    std::string str = to_string(a,N) ;
    for( std::size_t i = 0 ; i < str.size() ; i += 8 )
        std::cout << str.substr( i, 8 ) << ' ' ;
    std::cout << '\n' ;

    for( std::size_t i = 0 ; i < str.size() ; i += 6 )
        std::cout << str.substr( i, 6 ) << ' ' ;
    std::cout << '\n' ;

    for( auto i : to_n_bit_values<>(a,N) ) std::cout << int(i) << ' ' ;
    std::cout << '\n' ;
}

http://coliru.stacked-crooked.com/a/03405dc14340565a

[density]

Like so:

for (i=0; i<144; i++)
{
cw[i / 6] |= rs_codeword[i] << (i % 6);
}

Make sure that cw is filled with 0

EDIT: not tested!

i am sorry if i am not understanding what you have posted or if i have not explained myself here. so cw should be named like
cw[24]={0};
?? the cw[] array needs to have 24 positions each filled with one 'hex-bit' or 6-bit symbol.

will this small amount of code accomplish that?

for (i=0; i<144; i++)
{
  cw[i / 6] |= rs_codeword[i] << (i % 6);
}

[coder777]

will this small amount of code accomplish that?

It takes 6 consecutive values from rs_codeword and converts it to 6 bits in cw.
That 24 times (accidentally: 144 = 6 * 24)

I don't know what a hex-bit or a 6-bit symbol is?

[Chervil]

I briefly looked at this, but found I wasn't sure of the requirements.
Let's say we have 144 bits like this:

000011100001100011111010101101111001110100001001110100010011... etc.

So we take the first 6 bits, thus:

000011

Now there are two ways to convert that into a 6-bit number, depending upon whether the lowest-order bit comes first, or the highest.

Hence the result for those 6 bits could have a decimal value of either 3 or 48. Either value could be valid, but it needs to be interpreted consistently both here and in whatever destination process it is that will receive the data.

[density]

I don't know what a hex-bit or a 6-bit symbol is?

it is just 6 bits like 110010 so that is a 6-bit symbol, or some say a hex-bit. so in octal it is 062 . 'symbols' are used a lot in digital communication.

Now there are two ways to convert that into a 6-bit number, depending upon whether the lowest-order bit comes first, or the highest.

this is an excellent question. what i am dealing with is error corrected data transmission. The data payload of interest consists of 72 bits, and this is serialized into 12 hexbits, or 6-bit symbols. These hex-bits are then encoded with the (24,12,13) RS code to yield 24 hex bits, or a 24 hex-bit codeword. the irreducible polynomial used for encoding is x^6 + x + 1 = 100011 or 43 in octal. so this leads me to believe that, how do they say, 'the most significant bit is first'. that being said, this is what i know in practice: if i just extract the data bits from the transmission, the received bit vector has 1728 bits and i just specify the ordinals to extract in order from lowest to highest, without the parity bits and ignore the error correction, those bits will correctly display the data when converted to hexadecimal like this:

   uint8_t mi[9];
   extract(frame_body(), MI_BITS, MI_BITS_SZ, mi);
   ostringstream os;
   os << "0x";
   for(size_t i = 0; i < (sizeof(mi) / sizeof(mi[0])); ++i) {
      uint16_t octet = mi[i];
      os << hex << setfill('0') << setw(2) << octet;
   }
   return os.str();
}

Thanks!

[density]

can you give me code you put in the command line? cuz i am getting a crap load of errors trying to compile. Thanks!

#include <iostream>
#include <cstdint>
#include <bitset>
#include <string>
#include <vector>

std::string to_string( const std::uint8_t* a, std::size_t n )
{
std::string result ;

using bits = std::bitset<8> ;
for( std::size_t i = 0 ; i < n ; ++i ) result += bits( a[i] ).to_string() ;

return result ;
}

template< std::size_t NBITS >
std::vector<std::uint8_t> to_n_bit_values( const std::string& str )
{
std::vector<std::uint8_t> result ;

using bits = std::bitset<NBITS> ;
for( std::size_t i = 0 ; i < str.size() ; i += NBITS )
result.push_back( bits( str.substr( i, NBITS ) ).to_ulong() ) ;

return result ;
}

template< std::size_t NBITS = 6 >
std::vector<std::uint8_t> to_n_bit_values( const std::uint8_t* a, std::size_t n )
{ return to_n_bit_values<NBITS>( to_string(a,n) ) ; }

int main() // minimal test driver
{
constexpr std::size_t N = 6 ;
const std::uint8_t a[N] = { 0, 1, 25, 3, 255, 49 } ;
for( int i : a ) std::cout << i << ' ' ;
std::cout << '\n' ;

std::string str = to_string(a,N) ;
for( std::size_t i = 0 ; i < str.size() ; i += 8 )
std::cout << str.substr( i, 8 ) << ' ' ;
std::cout << '\n' ;

for( std::size_t i = 0 ; i < str.size() ; i += 6 )
std::cout << str.substr( i, 6 ) << ' ' ;
std::cout << '\n' ;

for( auto i : to_n_bit_values<>(a,N) ) std::cout << int(i) << ' ' ;
std::cout << '\n' ;
}


http://coliru.stacked-crooked.com/a/03405dc14340565a

[JLBorges]

The command line (C++11) is there in the link http://coliru.stacked-crooked.com/a/03405dc14340565a
g++-4.8 -std=c++11 -O2 -Wall -pedantic-errors -pthread main.cpp && ./a.out

This is the C++98 version:

#include <iostream>
#include <bitset>
#include <string>
#include <vector>

typedef unsigned char byte ;

std::string to_string( const byte* a, std::size_t n )
{
    std::string result ;

    typedef std::bitset<8> bits ;
    for( std::size_t i = 0 ; i < n ; ++i ) result += bits( a[i] ).to_string() ;

    return result ;
}

template< std::size_t NBITS >
std::vector<byte> to_n_bit_values( const std::string& str )
{
    std::vector<byte> result ;

    typedef std::bitset<8> bits ;
    for( std::size_t i = 0 ; i < str.size() ; i += NBITS )
        result.push_back( bits( str.substr( i, NBITS ) ).to_ulong() ) ;

    return result ;
}

template< std::size_t NBITS >
std::vector<byte> to_n_bit_values( const byte* a, std::size_t n )
{ return to_n_bit_values<NBITS>( to_string(a,n) ) ; }

int main() // minimal test driver
{
    const std::size_t N = 6 ;
    const byte a[N] = { 0, 1, 25, 3, 255, 49 } ;
    for( std::size_t i = 0 ; i < N ; ++i ) std::cout << int( a[i] ) << ' ' ;
    std::cout << '\n' ;

    std::string str = to_string(a,N) ;
    for( std::size_t i = 0 ; i < str.size() ; i += 8 )
        std::cout << str.substr( i, 8 ) << ' ' ;
    std::cout << '\n' ;

    for( std::size_t i = 0 ; i < str.size() ; i += 6 )
        std::cout << str.substr( i, 6 ) << ' ' ;
    std::cout << '\n' ;

    const std::vector<byte> six_bit_values = to_n_bit_values<6>(a,N) ;
    for( std::size_t i = 0 ; i < six_bit_values.size() ; ++i )
        std::cout << int( six_bit_values[i] ) << ' ' ;
    std::cout << '\n' ;
}

http://coliru.stacked-crooked.com/a/87d5b611970e0db8

[density]

The command line (C++11) is there in the link http://coliru.stacked-crooked.com/a/03405dc14340565a
g++-4.8 -std=c++11 -O2 -Wall -pedantic-errors -pthread main.cpp && ./a.out

This is the C++98 version:

Thanks a bunch. but i cant get the tarball to extract on gcc4.8.2 ?? i am on ubuntu12.10. Is this an issue with the new gcc ??

[JLBorges]

The version of GCC that you are currently using would be adequate to compile the code. Even if it is quite old, it should have no problems with the C++98 version.

[density]

omg i should have just put 4.7 in the command line. its running now. i will have to play with this for a while. Thanks a bunch! sweet piece of code btw!!

[coder777]

I guess the order of the bits are a problem. Try this:

for (i=0; i<144; i++)
{
  cw[i / 6] |= rs_codeword[i] << ((5 - i) % 6); // Now the order is maintained
}

Or simplified:

for (i=0; i<144; i++)
{
  const int idx = i / 6;
  cw[idx] <<= 1;
  cw[idx] |= rs_codeword[i];
}

[density]

Hi!

ok so i have been trying the code from above, the C++98 version. i am having trouble trying to come up with a way to make the input a 144-bit array (pointer really), split that into the 24 6-bit symbols, and then convert each one of those to octal, (like 111111 = 77) and then turn the first 12 of the 24 'double-octals' back into a bit vector.

i will be crazy appreciative for any help. Thanks!

[JLBorges]

It would be just an extension of the same technique, wouldn't it?

a. convert each byte into a bit string of size eight (zeroes or ones)
b. concatenate the bit strings to get a big bit string
c. extract six bits at a time and create a byte array where each value is the range 000 to 077

d. convert each byte in the byte array into a bit string of size six (zeroes or ones)
e. concatenate the bit strings to get a big bit string
g. extract eight bits at a time and create new a byte array where each value is the range 0 to 0xff

#include <iostream>
#include <bitset>
#include <string>
#include <vector>

typedef unsigned char byte ;

template< std::size_t NBITS >
std::string to_string( const byte* a, std::size_t n )
{
    std::string result ;

    typedef std::bitset<NBITS> bits ;
    for( std::size_t i = 0 ; i < n ; ++i ) result += bits( a[i] ).to_string() ;

    return result ;
}

template< std::size_t NBITS >
std::vector<byte> to_n_bit_values( const std::string& str )
{
    std::vector<byte> result ;

    typedef std::bitset<NBITS> bits ;
    for( std::size_t i = 0 ; i < str.size() ; i += NBITS )
        result.push_back( bits( str.substr( i, NBITS ) ).to_ulong() ) ;

    return result ;
}

int main()
{
    const std::size_t NBITS = 144 ;
    const std::size_t N8 = NBITS / 8 ;
    const std::size_t N6 = NBITS / 6 ;

    const byte a[N8] = { 0, 1, 25, 3, 255, 49 /* , ... */ } ;

    std::vector<byte> six_bit_vals = to_n_bit_values<6>( to_string<8>( a, N8 ) ) ;
    std::vector<byte> bytes = to_n_bit_values<8>( to_string<6>( &six_bit_vals.front(), N8 ) ) ;

    byte bit_vector[ N6/2 ] ;
    std::copy( bytes.begin(),  bytes.begin() + N6/2, bit_vector ) ;
}

[density]

i guess i am still unclear about why we are using bytes, or 8-bit values, at all? because if we start with the 144 bit array (assume that we have this in our possession), and not bytes; is it possible to go from the 144 bits straight to 24 6-bit values, convert those 24 6-bit values to 24 2-digit octal values, and convert the 24 2-digit octal values back to 144 bits? Thanks!!

[density]

Hi coder777! if use your code:

Or simplified:

for (i=0; i<144; i++)
{
  const int idx = i / 6;
  cw[idx] <<= 1;
  cw[idx] |= rs_codeword[i];
}

like this:

main()
{  
   int cw[48] = {0};
   int i;
   int rs_codeword[144] = //array of bits to convert to  octals 
   {
       0,0,0,0,0,0,    // each line is two octals
       0,0,0,0,0,1,
       0,0,0,0,1,0,
       0,0,0,0,1,1,
       0,0,0,1,0,0,
       0,0,0,1,0,1,
       0,0,0,1,1,0,
       0,0,0,1,1,1,
       0,0,1,0,0,0,
       0,0,1,0,0,1,
       0,0,1,0,1,0,
       0,0,1,0,1,1,
       0,0,1,1,0,0,
       0,0,1,1,0,1,
       0,0,1,1,1,0,
       0,0,1,1,1,1,
       0,1,0,0,0,0,
       0,1,0,0,0,1,
       0,1,0,0,1,0,
       0,1,0,0,1,1,
       0,1,0,1,0,0,
       0,1,0,1,0,1,
       0,1,0,1,1,0,
       0,1,0,1,1,1,
   };

   for (i=0; i<144; i++)
   {
      const int idx = i / 3;
      cw[idx] <<= 1;
      cw[idx] |= rs_codeword[i];
   }
   for (i=0; i<48; i++) printf( "%d", cw[i]);  //print each element of the cw[]
}

the output is:
000102030405060710111213141516172021222324252627
there are 48 octals here. so this is correct, sort of. how can we get these values into a 24 element array filled like this:

cw2[0] = {00};
cw2[1] = {01};
cw2[2] = {02};  
. . . . . .
cw2[22] = {26};
cw2[23] = {27};

so that each element is filled with two digits?? Thanks!

[density]

i believe this works. converts binary to octals, and converts octals to decimal

#include <math.h>
#include <stdio.h>

main()
{  
   int octal_array[48] = {0};
   int decimal_array[24] = {0};
   int i;
   int binary_array[144] = //array to convert to 48 octals and 24 dec 
   {
       0,0,0,0,0,0,    // each line has two octals
       0,0,0,0,0,1,
       0,0,0,0,1,0,
       0,0,0,0,1,1,
       0,0,0,1,0,0,
       0,0,0,1,0,1,
       0,0,0,1,1,0,
       0,0,0,1,1,1,
       0,0,1,0,0,0,
       0,0,1,0,0,1,
       0,0,1,0,1,0,
       0,0,1,0,1,1,
       0,0,1,1,0,0,
       0,0,1,1,0,1,
       0,0,1,1,1,0,
       0,0,1,1,1,1,
       0,1,0,0,0,0,
       0,1,0,0,0,1,
       0,1,0,0,1,0,
       0,1,0,0,1,1,
       0,1,0,1,0,0,
       0,1,0,1,0,1,
       0,1,0,1,1,0,
       0,1,0,1,1,1,
   };

   for (i=0; i<144; i++)
   {
      const int idx = i / 3;
      octal_array[idx] <<= 1;
      octal_array[idx] |= binary_array[i];
   }
   for (i=0; i<48; i++) printf( "%d", octal_array[i]);  //print each element of the array
                        printf("\n");
   for (i=0; i<48; i++)
   {
      const int idx = i / 2;
      decimal_array[idx] <<= 3;
      decimal_array[idx] |= octal_array[i];
   }
   for (i=0; i<24; i++) printf( "%d", decimal_array[i]);  //print each element of the array
                        printf("\n");
}

output is:

000102030405060710111213141516172021222324252627
01234567891011121314151617181920212223

if anyone cares to test please share results. Thanks everybody!!