cUrl, how to retreive the contents of a url

[roelf01]

Hello,
I need to know how to use libcurl to retreive the contents of a url.
Just a simple example.
Because i can't find any docs about how to do this.

Help needed!!!

[Athar]

Google, first hit for "libcurl":
http://curl.haxx.se/libcurl/

[roelf01]

I already have it...
I just need to know how to use it.
There don't seem to be any docs.
So i figured, ask it to someone who might know.

[ModShop]

Here is a simple example to get the content of an URL into a string object.

#include <iostream>
#include <string>
#include <curl/curl.h> //your directory may be different
using namespace std;

string data; //will hold the url's contents

size_t writeCallback(char* buf, size_t size, size_t nmemb, void* up)
{ //callback must have this declaration
    //buf is a pointer to the data that curl has for us
    //size*nmemb is the size of the buffer

    for (int c = 0; c<size*nmemb; c++)
    {
        data.push_back(buf[c]);
    }
    return size*nmemb; //tell curl how many bytes we handled
}

int main()
{
    CURL* curl; //our curl object

    curl_global_init(CURL_GLOBAL_ALL); //pretty obvious
    curl = curl_easy_init();

    curl_easy_setopt(curl, CURLOPT_URL, "http://www.example.com/path");
    curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, &writeCallback);
    curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L); //tell curl to output its progress

    curl_easy_perform(curl);

    cout << endl << data << endl;
    cin.get();

    curl_easy_cleanup(curl);
    curl_global_cleanup();

    return 0;
}

Pretty easy. Check out libCurl's documentation on curl_easy_setopt() to get a comprehensive list of all the available options.

[roelf01]

Thx,
It worked (after i fixed an issue with my parser).

[Galik]

I generally use something rather like this:

#include <curl/curl.h>
#include <fstream>
#include <sstream>
#include <iostream>

// callback function writes data to a std::ostream
static size_t data_write(void* buf, size_t size, size_t nmemb, void* userp)
{
	if(userp)
	{
		std::ostream& os = *static_cast<std::ostream*>(userp);
		std::streamsize len = size * nmemb;
		if(os.write(static_cast<char*>(buf), len))
			return len;
	}

	return 0;
}

/**
 * timeout is in seconds
 **/
CURLcode curl_read(const std::string& url, std::ostream& os, long timeout = 30)
{
	CURLcode code(CURLE_FAILED_INIT);
	CURL* curl = curl_easy_init();

	if(curl)
	{
		if(CURLE_OK == (code = curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, &data_write))
		&& CURLE_OK == (code = curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 1L))
		&& CURLE_OK == (code = curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L))
		&& CURLE_OK == (code = curl_easy_setopt(curl, CURLOPT_FILE, &os))
		&& CURLE_OK == (code = curl_easy_setopt(curl, CURLOPT_TIMEOUT, timeout))
		&& CURLE_OK == (code = curl_easy_setopt(curl, CURLOPT_URL, url.c_str())))
		{
			code = curl_easy_perform(curl);
		}
		curl_easy_cleanup(curl);
	}
	return code;
}

int main()
{
	curl_global_init(CURL_GLOBAL_ALL);

	std::ofstream ofs("output.html");
	if(CURLE_OK == curl_read("http://google.com", ofs))
	{
		// Web page successfully written to file
	}

	std::ostringstream oss;
	if(CURLE_OK == curl_read("http://google.com", oss))
	{
		// Web page successfully written to string
		std::string html = oss.str();
	}

	if(CURLE_OK == curl_read("http://google.com", std::cout))
	{
		// Web page successfully written to standard output (console?)
	}

	curl_global_cleanup();
}

It uses std::ostream for flexibility. It allows you to easily send the retrieved url data to a file or a string or the standard output or any other std::ostream derived object.

[Athar]

I just need to know how to use it.
There don't seem to be any docs.

So doesn't the site load correctly for you or how did you manage to miss "API", "Example sources", "Using libcurl" or "Tutorial" to the left. Seriously, what the hell.

[roelf01]

I saw them but they either didn't work, or didn't tell me how to download a url.
The "simple.c" example only told me how to init and shutdown libcurl.
The others are for diffrent protocols.
The api docs don't tell me how to get the content either.
The tut's code won't work (i found out that it is because i need to return size*nmemb, they didn't say i had to do that in the tut).
And the "using libcurl" only told me how to compile.

[roelf01]

Oh,
When you take ModShop's code,
Put it in a seperate namespace.
Rename int main to string get_page.
Add a char* url as param.
Remove the return 0; and replace it with string tmp = data; data = ""; return tmp;
Change the CURL_URL param to url.
Add a int main and call get_page, you download a url. And it's repeatable.

[ashish verma]

have you guys noticed a thing ...

The code by ModShop do not download URL like www.google.com (when in India its redirected to www.google.co.in) and doesn't even download www.facebook.com..

but the code which Galik used can do it...

wow!

Anyone knows the reason behind this..??

Is it CURL_GLOBAL instead of CURL_EASY.

[modoran]

You need to use CURLOPT_FOLLOWLOCATION option to follow http redirects.

[ashish verma]

yeah I saw that thanks modoran