Trying to download a binary file from HTTP server

[Jason2gs]

Hi everyone :)

I'm trying to download a binary file from an HTTP server using sockets and fstreams.

However, when I get the response back from the server (in this case, imageshack), it only gives me a few characters; not the whole image.

When I send the same exact request with telnet, it works fine. I'm a bit confused about this :S

I'll post the code below:

#include <netdb.h>
#include <string.h>
#include <fstream.h>
#include <arpa/inet.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>

int main()
{
    int sockfd, c;
    struct sockaddr_in addr;
    struct hostent *host = gethostbyname("img260.imageshack.us");

    sockfd = socket(PF_INET, SOCK_STREAM, 0);

    addr.sin_family = AF_INET;
    addr.sin_port = htons(80);
    addr.sin_addr = *((struct in_addr *)host->h_addr);
    memset(addr.sin_zero, '\0', sizeof addr.sin_zero);

    c = connect(sockfd, (struct sockaddr *)&addr, sizeof addr);

    printf("Kk\r\n");

    while(c == -1)
    {
        sleep(5);
        c = connect(sockfd, (struct sockaddr *)&addr, sizeof addr);
    }

    printf("Kk\r\n");

    char *packet = "GET /img260/4994/haruhichristmascs7.png HTTP/1.0\r\n\r\n";

    send(sockfd, packet, strlen(packet), 0);

    printf("Kk\r\n");

    int bytes = 33054;

    char buffer[bytes];

    int r = recv(sockfd, buffer, bytes, 0);

    printf("%d\r\n", r);

    char *pch = strstr(buffer, "\r\n\r\n");

    char *mesg = (char *)malloc(bytes);

    strncpy(mesg, pch + 4, bytes);

    printf(mesg);

    fstream fp("/home/mike/Desktop/Haruhi.png", std::ios::out|std::ios::binary);
    fp.write(mesg, strlen(mesg));

    return 0;

}

Hope you can help me,

-Mike

[Zaita]

line 45 might be your problem.

You should try

while (recv() > 0) {

}

[bnbertha]

recv() is a blocking call so the only way it returns is if all the data has been read or the call is interrupted, for example by a signal or an error. It sounds like it's being interrupted so Zaitas idea of a loop should fix the problem. You will need to test how many bytes have been read and move a pointer along your buffer accordingly for the next call to recv()

[Jason2gs]

Well that was quick. Thank you :)

I'll try that, and post back how it went.

[Jason2gs]

Alrighty, here's the updated bit...

#include <netdb.h>
#include <string.h>
#include <fstream.h>
#include <arpa/inet.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>

int main()
{
    int sockfd, c;
    struct sockaddr_in addr;
    struct hostent *host = gethostbyname("localhost"); // I decided to use a picture on my server.

    sockfd = socket(PF_INET, SOCK_STREAM, 0);

    addr.sin_family = AF_INET;
    addr.sin_port = htons(80);
    addr.sin_addr = *((struct in_addr *)host->h_addr);
    memset(addr.sin_zero, '\0', sizeof addr.sin_zero);

    c = connect(sockfd, (struct sockaddr *)&addr, sizeof addr);

    while(c == -1)
    {
        sleep(5);
        c = connect(sockfd, (struct sockaddr *)&addr, sizeof addr);
    }

    char *packet = "GET /16997.jpg HTTP/1.0\r\n\r\n"; // Image size is 3,259 bytes.

    send(sockfd, packet, strlen(packet), 0);

    int r;

    char *buffer = (char*)malloc(128); // I tried setting it to 1024, but it wasn't working very well. What should I set it to?

    char *response = (char*)malloc(4000);

    printf("Kk\r\n");

    while(r = recv(sockfd, buffer, 128, 0) > 0) // I'll work on dynamically setting the size later on.
    {
        strcat(response, buffer);

        printf("%d, %d\r\n", strlen(response), strlen(buffer));
    }

    char *pch = strstr(response, "\r\n\r\n"); // To chop off the data the server sends before it sends the file. (Content type, etc.)

    char *mesg = (char *)malloc(4000);

    strncpy(mesg, pch + 4, 4000);

    fstream fp("/home/mike/Desktop/Haruhi.jpg", std::ios::out|std::ios::binary);
    fp.write(mesg, strlen(mesg));

    return 0;

}

It still doesn't write out the image, though...

[Zaita]

How big is your image? You've only allocated 4KBs for it in memory.

[Jason2gs]

Line 31 :p

3,259 bytes. Even with the headers the server sends back, the full response is still less than 4KB.

Also,

I tried setting the response chunks lower, and it started giving me more and more data. However, when I set it as low as one to four, it'd get a lot bigger than the full response should be. So big that the app would segfault.

[Jason2gs]

Bit of an update:

I've been doing some testing with this thing, and learned that if the file size is really really small (a few bytes), then the output repeats itself somehow. ("|Hello|" would become *Header data\r\n\r\n* "|Hell". But I cut off the Header data and two line breaks, so I only see |Hell|)

Also, it doesn't matter how big the file is, if it's over a certain size, it'll chop it off. (The 3000 byte files turns out to be 2000-some, a 330 byte file is 72 bytes... :S)

But again, how big of packets does the server send back at one time?

[Zaita]

That doesn't seem right to me at all. TCP has dynamic packet sizes.

It appears that something is wrong in your receiving code.

while(r = recv(sockfd, buffer, 128, 0) > 0)

Try
while(recv(sockfd, buffer, 128, 0) > 0)

[Jason2gs]

Mm, seems to be no different.

But what would that do? Just curious :)

And another update:

#include <netdb.h>
#include <string.h>
#include <fstream.h>
#include <arpa/inet.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>

int main()
{
    int sockfd, c;
    struct sockaddr_in addr;
    struct hostent *host = gethostbyname("localhost");

    sockfd = socket(PF_INET, SOCK_STREAM, 0);

    addr.sin_family = AF_INET;
    addr.sin_port = htons(80);
    addr.sin_addr = *((struct in_addr *)host->h_addr);
    memset(addr.sin_zero, '\0', sizeof addr.sin_zero);

    c = connect(sockfd, (struct sockaddr *)&addr, sizeof addr);

    while(c == -1)
    {
        sleep(5);
        c = connect(sockfd, (struct sockaddr *)&addr, sizeof addr);
    }

    char *packet = "GET /File HTTP/1.0\r\n\r\n";

    send(sockfd, packet, strlen(packet), 0);

    char *buffer = (char*)malloc(56);

    char *response = (char*)malloc(4000);

    while(recv(sockfd, buffer, 56, 0) > 0)
    {
        strcat(response, buffer);
    }

    printf("%s\r\n", response);

    fstream fp("/home/mike/Desktop/Haruhi", std::ios::out);
    fp.write(response, strlen(response));

    return 0;

}

Returns this:

mike@Juankubariz:~$ '/home/mike/Desktop/C++/Test'
HTTP/1.1 200 OK
Date: Thu, 12 Jun 2008 01:52:16 GMT
Server: Apache/2.2.4 (Ubuntu) PHP/5.2.3-1ubuntu6.3
Last-Modified: Wed, 11 Jun 2008 21:36:03 GMT
ETag: "5c0113-d-d4d996c0"
Accept-Ranges: bytes
Content-Length: 13
Connection: close
Content-Type: text/plain

Hello, thar!
ection: close
Content-Type: text/plain

Hello, thar!

Just to give you a better understanding of what's going on.

[Zaita]

That looks perfect. What is the problem?

[Jason2gs]

Well, down near the bottom it tries to repeat itself ^_^

Normally, I could just cut the headers out at the top by searching for \r\n\r\n and copying everything after that. Then copy exactly how many bytes I need, save that in a char*, and write it to the file. However, I tried that, and it didn't work :S

[Zaita]

That looks more like a problem with your memory allocation. You should memset() the memory you allocate to nulls. This will make sure you don't have any issues.

Remember strcat doesn't null-terminate your buffer.

edit: you can also use the content-length to work out how many bytes you need to read. This would be a good idea.

edit: I'll explain. Malloc will return you memory, but won't clear it. So you should clear the memory with nulls (0) before you use it.

[Jason2gs]

You mean after each pass of the while loop, memset() buffer to nothing?

Well, it works :)

Thank you! :D

I'll be back in a little bit. Then I'm gonna try and DL a picture/application again.

[Zaita]

right after your malloc thats where you put the memset.

[Jason2gs]

So this:

    char *buffer = (char*)malloc(56);

    memset(buffer, NULL, 56);

    char *response = (char*)malloc(512);

    while(recv(sockfd, buffer, 56, 0) > 0)
    {
        strcat(response, buffer);
    }

NOT this:

    char *buffer = (char*)malloc(56);

    char *response = (char*)malloc(512);

    while(recv(sockfd, buffer, 56, 0) > 0)
    {
        strcat(response, buffer);
        memset(buffer, NULL, 56);
    }

?

Mm, the second one works for me, the first doesn't :S

Although, this brings up another question:

Why, when I set the buffer really really small, doesn't it work?

[Jason2gs]

Ah, crap -_-

I guess that didn't fix it :(

It still seems to only grab part of it... *Scratches head*

[iL0pX9L8]

Thanks for your suggestion
But It would be more fruitful if you help me of what i want

[Jason2gs]

Heh, methinks you've got the wrong thread, deaduser :p

Edit: Reading through your other posts, me-also-thinks that you're a bit of an idiot.

You spam your thread into multiple sections, and when someone tells you not to, you tell them off and expect them to help you further?

I'd be very surprised if you get any help by anybody at all...

[bnbertha]

Are you trying to grab a binary file, or the filename? If you trying to get the actual file I wouldn't use strcat(), try something like memcpy()

ssize_t bytes_read = 1;
char *end_of_data = response;
while(bytes_read > 0)
{
    bytes_read = recv(sockfd, buffer, 56, 0) ;

    if (bytes_read > 0)
    {
        memcpy(end_of_data, buffer, bytes_read);
        end_of_data += bytes_read;
        //strcat(response, buffer);
        memset(buffer, NULL, 56);
    }
}