C++

Forum

 
openSSL RSA Enc/Dec

Aug 20, 2010 at 2:36am
Phantom139 (41)
Hello, I'm really close to having all the pieces I need for my program. The last piece that is causing me issues is RSA Encrypt/Decrypt.

This is what I have so far (I put the class info this time):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
 
std::string PGDCrypto::doRSADecryption(std::string message) {
   std::string out = "";
   std::string correctedMessage = pgd.hexToStr(message);
   //int length = correctedMessage.length();
   unsigned char * decrypted = (unsigned char *)malloc(1024);
   unsigned char * in = (unsigned char *)(correctedMessage.c_str());
   pgd.clientKey = pgd.StoreDataStruct();
   RSA_private_decrypt(RSA_size(pgd.clientKey), in, decrypted, pgd.clientKey, RSA_PKCS1_OAEP_PADDING);
   //
   ofstream outstrm("RSATest.txt", ios::app);
   outstrm << "cnv: " << in << "\n";
   outstrm << "dec: " << decrypted << "\n\n\n";
   outstrm.close();
   //
   out.append(string((const char *)decrypted));
   free(in);
   free(decrypted);
   return out;
}

std::string PGDCrypto::doRSAEncryption(std::string message) {
   std::string out = "";
   int length = message.length();
   unsigned char *encrypted = (unsigned char*)malloc(1024);
   unsigned char * in = (unsigned char *)(message.c_str());
   pgd.clientKey = pgd.StoreDataStruct();
   RSA_public_encrypt(length, in, encrypted, pgd.clientKey, RSA_PKCS1_OAEP_PADDING);
   //
   ofstream outstrm("RSATest.txt", ios::app);
   outstrm << "org: " << encrypted << "\n";
   outstrm.close();
   //
   out.append(pgd.strToHex(string((const char *)encrypted)));
   //
   free(in);
   free(encrypted);
   //
   return out;
}

std::string PGDCrypto::hexToStr(std::string &input)
{
	std::ostringstream output;
	int len = input.length();
	char tmp[5], *ptr;
	tmp[0] = '0';
	tmp[1] = 'x';
	tmp[4] = '\0';
	for (int i = 0; i < len; i += 2)
	{
		if (i + 1 >= len)
			return output.str();
		tmp[2] = input[i];
		tmp[3] = input[i+1];
		//std::cout << tmp << "\n";
		output << (char)strtol(tmp, &ptr, 16);
	}
	return output.str();
}

std::string PGDCrypto::strToHex(std::string &input)
{
	std::ostringstream output;
	int len = input.length();
	int val;
	//std::ofstream outfile("tohex.txt", ios::out);
	for (int i = 0; i < len; i++)
	{	
		val = (int)input[i];
		//std::cout << val << "\n" << input[i] << "\n\n";
		if (val < 0)
			val = 256 + val;
		if (val < 16)
			output << "0"; // >_>
		output << std::hex << val;
	}

	return output.str();
}


I have already checked the clientKey variable, everything is in it's proper place there (from the StoreDataStruct function).

The function that I'm generating the data to encrypt/decrypt is as follows:
1
2
3
4
5
6
7
8
9
10
11
12
 
std::string PGDCrypto::generateChallenge(int length) {
	std::string challenge = "";
	std::string hexclg = "0123456789abcdef";
	std::string storage = "";
	for(int i = 0; i < length; i++) {
	   int randNum = (int)((16 * rand()) / RAND_MAX);
	   //cout << "RND: " << randNum << endl;
	   storage = hexclg.substr(randNum, 1);
	   challenge.append(storage);
	}
	return challenge;
}


The main issues with this currently is with the decoded data, it decodes a portion of it, but the rest remains encrypted (ASCII), like so: (test dump):

dec: b8960b19d7b76c85fa252bf9050a2dd9bd02b93129555acc89ce2a642d169483а8i#µºÉÉ9p¢//z¾©ý]ÏÿåÄ8—†
D<,‹6`λÒåTÿx–bµ”û[cíêd6/Åï¥|m]hzÿ]hzÿ]hzÿ>EQÿ

The other thing is with the function that calls it, after I call the function about 3 - 4 times, the output is all ASCII:

dec: ÿÿíæ²4zÜ/cW;E],JF`ÌÿÌ\B^„]«Qäð",½PóźÝÃA+RX"ÈM&.áÇt[
~¾³à”±-z“6w´§ÁÈ+ ?¼³0¡:sŒ¼äP‹³9˜ß¿wO:Ö¼dùj¡-

How can I go about getting this fixed up?
Aug 20, 2010 at 8:38am
kbw (9488)
I'll take a look at the RSA stuff later when I can reference some code I wrote. In the mean time, I'm surprised you're not getting a crash in lines 16, 35 with:
free(in);
Aug 21, 2010 at 2:51am
Phantom139 (41)
alright thanks.

And I should probably remove those two then...
Aug 27, 2010 at 2:36am
Phantom139 (41)
I'm getting a little closer.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
 
std::string PGDCrypto::doRSADecryption(std::string message) {
   std::string out = "";
   //std::string correctedMessage = pgd.hexToStr(message);
   //int length = correctedMessage.length();
   unsigned char * decrypted = (unsigned char *)malloc(message.length());
   unsigned char * in = (unsigned char *)(message.c_str());
   ofstream outstrm("RSATest.txt", ios::app);
   outstrm << "cnv: " << message << " -> " << in << "\n";
   RSA * workkey = pgd.StoreDataStruct();
   RSA_private_decrypt(RSA_size(workkey), in, decrypted, workkey, RSA_PKCS1_OAEP_PADDING); //
   //
   outstrm << "dec: " << decrypted << "\n\n\n";
   outstrm.close();
   //
   //out.append(string((const char *)decrypted));
   out.assign(string((const char *)decrypted));
   return out;
}

std::string PGDCrypto::doRSAEncryption(std::string message) {
   std::string out = "";
   int length = message.length();
   unsigned char * encrypted = (unsigned char *)malloc(1024);
   unsigned char * in = (unsigned char *)(message.c_str());
   RSA * workkey = pgd.StoreDataStruct();
   RSA_public_encrypt(length, in, encrypted, workkey, RSA_PKCS1_OAEP_PADDING);
   //
   ofstream outstrm("RSATest.txt", ios::app);
   outstrm << "msg: " << message << "\n";
   outstrm << "org: " << encrypted << "\n";
   outstrm.close();
   //
   //out.append(pgd.strToHex(string((const char *)encrypted)));
   out.assign(string((const char *)encrypted));
   return out;
}


there is still some noticeable errors in some decryption dumps, see below, but there are now some of them that actually are coming out to be correct.

Still not entirely sure exactly where the issue is being caused at.

Latest dump set:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
 
msg: 093c975edb2db8401522f7100689992a7509cc84ebfe8273d3cdff9644d061a0
org: …̳E{«Ûn2ßæò[ö©n}¦A)­’+&Åð…ý›H?Q1Ÿ‰úÿÂùl—…øº­¬6Kßmuڏ`¼ÄÕ]ùñ'‹©ð¯ê÷2f/·7ÚO ußBy÷dÓÈ+š¢ÛªÙ~V~³ÍW“e>8ZäøéRY~ʁ…Fú}1
cnv: …̳E{«Ûn2ßæò[ö©n}¦A)­’+&Åð…ý›H?Q1Ÿ‰úÿÂùl—…øº­¬6Kßmuڏ`¼ÄÕ]ùñ'‹©ð¯ê÷2f/·7ÚO ußBy÷dÓÈ+š¢ÛªÙ~V~³ÍW“e>8ZäøéRY~ʁ…Fú}1 -> …̳E{«Ûn2ßæò[ö©n}¦A)­’+&Åð…ý›H?Q1Ÿ‰úÿÂùl—…øº­¬6Kßmuڏ`¼ÄÕ]ùñ'‹©ð¯ê÷2f/·7ÚO ußBy÷dÓÈ+š¢ÛªÙ~V~³ÍW“e>8ZäøéRY~ʁ…Fú}1
dec: 093c975edb2db8401522f7100689992a7509cc84ebfe8273d3cdff9644d061a0s


msg: 0e449bdb73b77fb196b995236c8fc52a71b82f2eb461fa2ed932d728b649acb7
org: •ÓÅÄ|‹êñ'öÛÎõ	fw1M$®`8•Å{°Š«’áu\‹;W‚0së–& Ðô^nÂÖ¡xýöÝ
ÆÓÖ´j%Ù²m—¥¸;c>ð¯ºh—õCG„¶Xìd!Ý&Zçó´6;Ä#9ˆ£!BÀ‰TÓÂÙ¡ÒlŽ3ÉK¢)ó‘wT •
cnv: •ÓÅÄ|‹êñ'öÛÎõ	fw1M$®`8•Å{°Š«’áu\‹;W‚0së–& Ðô^nÂÖ¡xýöÝ
ÆÓÖ´j%Ù²m—¥¸;c>ð¯ºh—õCG„¶Xìd!Ý&Zçó´6;Ä#9ˆ£!BÀ‰TÓÂÙ¡ÒlŽ3ÉK¢)ó‘wT • -> •ÓÅÄ|‹êñ'öÛÎõ	fw1M$®`8•Å{°Š«’áu\‹;W‚0së–& Ðô^nÂÖ¡xýöÝ
ÆÓÖ´j%Ù²m—¥¸;c>ð¯ºh—õCG„¶Xìd!Ý&Zçó´6;Ä#9ˆ£!BÀ‰TÓÂÙ¡ÒlŽ3ÉK¢)ó‘wT •
dec: 0e449bdb73b77fb196b995236c8fc52a71b82f2eb461fa2ed932d728b649acb7ene


msg: 15d08a61fe8575d574f4b93cd68e0f9083daa3d11b5f452bdb9b4200cd318017
org: dª,1éX|~˜—¯¢zaÛ©¨ö{<Kô¤ßË9L
cnv: dª,1éX|~˜—¯¢zaÛ©¨ö{<Kô¤ßË9L -> dª,1éX|~˜—¯¢zaÛ©¨ö{<Kô¤ßË9L
dec: 3,1éX|~˜—¯¢zaÛ©¨ö{<Kô¤ßË9L


msg: ca8163b46398a28fbe352f7f1a17e0e43c63a9779add9b962b8e339b9571b99b
org: ·¿'‰*ƒ%ŸðWd¡M)Ê¡ŽD€ºD¡iü}‰ŒÛ«h”ù7Q ©¤®Ÿí=œ&[ëöäí揋ü\LƒrccHÑ©zvXrjž‰ö0L×æzy*~uhs¡;ü9˜l!”¤
›ñ8þnŸùß.FF‹šQ‚>‹Øóä
cnv: ·¿'‰*ƒ%ŸðWd¡M)Ê¡ŽD€ºD¡iü}‰ŒÛ«h”ù7Q ©¤®Ÿí=œ&[ëöäí揋ü\LƒrccHÑ©zvXrjž‰ö0L×æzy*~uhs¡;ü9˜l!”¤
›ñ8þnŸùß.FF‹šQ‚>‹Øóä -> ·¿'‰*ƒ%ŸðWd¡M)Ê¡ŽD€ºD¡iü}‰ŒÛ«h”ù7Q ©¤®Ÿí=œ&[ëöäí揋ü\LƒrccHÑ©zvXrjž‰ö0L×æzy*~uhs¡;ü9˜l!”¤
›ñ8þnŸùß.FF‹šQ‚>‹Øóä
dec: ca8163b46398a28fbe352f7f1a17e0e43c63a9779add9b962b8e339b9571b99bÿÿ


msg: c29de9b0af59421ddb2511ad87647825b5dfdb4f21c89f1ea51834e8a1767c86
org: &ÿ",…™Iî× ]¸0Ýv¹9¥ÓÚr;¢!bäºê¥ȃ½]z^PijKA?5qš>F»³‘º²²èp˜¼Ï¹RØ‘q6ö­æc&`â´zD+b⌠h‘‹åIk-Ø<¸o‰ø°òo— mn빓:e˜Ž¢uÛ½V!
cnv: &ÿ",…™Iî× ]¸0Ýv¹9¥ÓÚr;¢!bäºê¥ȃ½]z^PijKA?5qš>F»³‘º²²èp˜¼Ï¹RØ‘q6ö­æc&`â´zD+b⌠h‘‹åIk-Ø<¸o‰ø°òo— mn빓:e˜Ž¢uÛ½V! -> &ÿ",…™Iî× ]¸0Ýv¹9¥ÓÚr;¢!bäºê¥ȃ½]z^PijKA?5qš>F»³‘º²²èp˜¼Ï¹RØ‘q6ö­æc&`â´zD+b⌠h‘‹åIk-Ø<¸o‰ø°òo— mn빓:e˜Ž¢uÛ½V!
dec: c29de9b0af59421ddb2511ad87647825b5dfdb4f21c89f1ea51834e8a1767c86


msg: b8960b19d7b76c85fa252bf9050a2dd9bd02b93129555acc89ce2a642d169483
org: 	5©èŸ¹à’…‰‰i&QÄø~êI®Ù??ÛÔñð÷ò‚"é*,àˆ}7fÌ…0€õT§•	® ø5ý<³âdílÕÏ£§û'ܨžtåáô7Ï3ßÊU»Ï4ÔÛwb(h²©Ãp¸þ±E
3&K&›‰¤Ä¡ 08笿¶k,™uC
cnv: 	5©èŸ¹à’…‰‰i&QÄø~êI®Ù??ÛÔñð÷ò‚"é*,àˆ}7fÌ…0€õT§•	® ø5ý<³âdílÕÏ£§û'ܨžtåáô7Ï3ßÊU»Ï4ÔÛwb(h²©Ãp¸þ±E
3&K&›‰¤Ä¡ 08笿¶k,™uC -> 	5©èŸ¹à’…‰‰i&QÄø~êI®Ù??ÛÔñð÷ò‚"é*,àˆ}7fÌ…0€õT§•	® ø5ý<³âdílÕÏ£§û'ܨžtåáô7Ï3ßÊU»Ï4ÔÛwb(h²©Ãp¸þ±E
3&K&›‰¤Ä¡ 08笿¶k,™uC
dec: b8960b19d7b76c85fa252bf9050a2dd9bd02b93129555acc89ce2a642d169483


msg: 41c24cecb95d3d338f018614e322e1005e66fd8db6408f6394af6e0f9c923606
org: ^A |SE®HþÛ'ÊÛsÆ|ÎiªfÑB²qÕÃ(«slÙ3~ßcªXÇãJ¢çœ	Í)ŽÉ»â?ÍXù®C¨]
‡r<Ýaù?¼
cnv: ^A |SE®HþÛ'ÊÛsÆ|ÎiªfÑB²qÕÃ(«slÙ3~ßcªXÇãJ¢çœ	Í)ŽÉ»â?ÍXù®C¨]
‡r<Ýaù?¼ -> ^A |SE®HþÛ'ÊÛsÆ|ÎiªfÑB²qÕÃ(«slÙ3~ßcªXÇãJ¢çœ	Í)ŽÉ»â?ÍXù®C¨]
‡r<Ýaù?¼
dec: 

msg: e7ea2a96f8984650c3169132a79b7a6d93fe6f2e97ab0446ebbe66e4695bbbb2
org: N^*£T:¾­×8xP¦³4g2k7gÈ\¤èPÅ
‹Wò<‘ùf
cnv: N^*£T:¾­×8xP¦³4g2k7gÈ\¤èPÅ
‹Wò<‘ùf -> N^*£T:¾­×8xP¦³4g2k7gÈ\¤èPÅ
‹Wò<‘ùf
dec: t

msg: d644096007de6b9ce991c7999da50453c5d1bae28ecf3ecfdea2944bc2b71cd1
org: <`؛5N=`
?»ƒ™æÎÙ»?CÙƒ-»[¹ŒBgŒê
ÅM©Hó,æ¼t…9AÜ||.î é>Üsýûî_#%]5õ	wPý:‹k%jAa£Ëj6ánÄ-3dO)£;ã)‹1ƒŠ”`ß6;ò]*“ϹRóÑg2ÕôÆÏ^xÿ
cnv: <`؛5N=`
?»ƒ™æÎÙ»?CÙƒ-»[¹ŒBgŒê
ÅM©Hó,æ¼t…9AÜ||.î é>Üsýûî_#%]5õ	wPý:‹k%jAa£Ëj6ánÄ-3dO)£;ã)‹1ƒŠ”`ß6;ò]*“ϹRóÑg2ÕôÆÏ^xÿ -> <`Ø›5N=`
?»ƒ™æÎÙ»?CÙƒ-»[¹ŒBgŒê
ÅM©Hó,æ¼t…9AÜ||.î é>Üsýûî_#%]5õ	wPý:‹k%jAa£Ëj6ánÄ-3dO)£;ã)‹1ƒŠ”`ß6;ò]*“ϹRóÑg2ÕôÆÏ^xÿ
dec: d644096007de6b9ce991c7999da50453c5d1bae28ecf3ecfdea2944bc2b71cd1Thread
Aug 27, 2010 at 10:28pm
Phantom139 (41)
More Progress, this bit of code will encode/decode properly most of the time, however, faults are still occurring, I'm not entirely sure if this is an issue with the encoding function, or just an issue with openSSL.

int ret comes out as -1 on all of the improper decodes.

any input on how to fix it?

*EDIT*

removed above code, I modified my function to use openSSL's err checking lib, and this is what came out of that:

error: 04065072:lib(4):func(101):reason(114)

new function:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
 
char * PGDCrypto::doRSADecryption(char * message) {
	std::string encoded = message;
	encoded = hexToStr(encoded);
	unsigned char * decodeHold = (unsigned char *)encoded.c_str();
	unsigned char * returnedDecMessage = (unsigned char *)malloc(1024);
	std::string finalDecode;
	//
	//RSA_public_decrypt(RSA_size(pgd.key), decodeHold, returnedDecMessage, pgd.key, RSA_PKCS1_OAEP_PADDING);
	int ret = RSA_private_decrypt(RSA_size(pgd.key), decodeHold, returnedDecMessage, pgd.key, RSA_PKCS1_OAEP_PADDING); //RSA_PKCS1_OAEP_PADDING
	//
	if(ret == -1) {
	   char errBuf[1024];
       ERR_error_string(ERR_peek_last_error(), errBuf);
       printf("Error: %s\n", errBuf);
	}

	//
	//cout << "RET: " << ret << endl;
	finalDecode = string((const char *)returnedDecMessage);
	char * out_d = (char *)malloc(1024);
	strcpy(out_d, finalDecode.c_str());
	return out_d;
}


the error only shows on the decodes that don't work, so that should be a point in the correct direction.
Last edited on Aug 28, 2010 at 1:11am
Topic archived. No new replies allowed.