Download file from https server using sockets

Apr 20, 2010 at 2:50am
Hi,
I have written the code for downloading a file from a https server using sockets in C++(Unix). However the file is only partially downloaded and the problem is in setting the receive buffer for the data received from the server.
Code snippet for the same is
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
char buf[100];     			 /* buffer for data from the server     */
SSL_write(ssl, downloadString, strlen(downloadString));			/* encrypt & send message */
		memset(buf, 0, sizeof(buf));
       		int bytes = SSL_read(ssl, buf, sizeof(buf)-1);	/* get reply & decrypt */
        	buf[bytes] = 0;
           
	while( bytes > 0)
		{
	       
		buf[bytes] = 0;
		memset(buf, 0, sizeof(buf));
		bytes = SSL_read(ssl, buf, sizeof(buf)-1);
		outfile << buf;
		}

The file to be downloaded is about 12 megs and only about 10 megs gets downloaded.
If the buf size is increased the downloaded file size decreases.Appreciate if any one can advice me the right way to do this, as i am fairly new to C++ socket programming.

Thanks,
Apr 20, 2010 at 4:17am
Personally, I prefer cURL or cURLpp for this sort of thing: http://curlpp.org/
Apr 20, 2010 at 10:01am
I think you have a mistake in your while loop.

You are not saving the first block of data you read. Line 11 overwrites the data read in line 4 (outside the loop) without writing it into a file first. Even worse, you don't set the null-terminator in your loop accordingly after reading the bytes in line 11, which could lead to buffer overflows.

I guess you want to move the line 13 ("outfile<<buf") up two lines, so it comes before the memset and after the null-termination thingie. (and you can remove line 5 - it's not necessary).


But with the 10vs12 MB problem, it's somewhere else: Your routine only works for text-data. But I guess you are downloading binary data, right? Binary data can contain 0-bytes, so writing them using operator<< and a char-buf is not possible (as its null-terminated. Imagine that accidently, there is a byte with value 0 at position 50 in your byte buffer. "outfile<<buf" will just stop after writing 50 bytes and the remaining data will silently be ignored!

You have to use other output functions, e.g. if outfile is an ostream, use ostream::write which takes a number of bytes to write. Additional advantage: You don't need any null-termination thingies anymore (all those "buf[bytes]=0" can be erased).

Ciao, Imi.
Apr 22, 2010 at 4:45pm
Hi imi,
Thanks for you input. However I am still facing the problem of more bytes being written to the downloaded file.
The reade buffer is buf[1024] and the final downloaded file has exactly 1024 bytes more than the actual file.
The code snippet is as follows - appreciate if you can point out the error for the same, i am opening the file to write in binary mode.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
SSL_write(ssl, downloadString, strlen(downloadString));			/* encrypt & send message */
		int ret;
       		int bytes = SSL_read(ssl, buf, sizeof(buf));	/* get reply & decrypt */
        int err;// = SSL_get_error(ssl, bytes);
	//buf[bytes] = '\0';
	while(bytes > 0)
	{
		//buf[bytes] = '\0';
		err = SSL_get_error(ssl, bytes);
		if(SSL_ERROR_WANT_READ || SSL_ERROR_WANT_WRITE)
		outfile.write(buf, sizeof(buf));
		else
		break;
		//memset(buf, '\0', sizeof(buf));
		bytes = SSL_read(ssl, buf, sizeof(buf));
			
		}


Thanks,
Apr 23, 2010 at 12:57pm
Try to find your problem in line 11.
Topic archived. No new replies allowed.