Copying Data Effeciently

[muswei]

I have an existing class underneath a http server that reads data from a stream into a std::auto_ptr<unsigned char> in chunks to get the total size of the data and then does another pull for all the data so we don't have to keep a reference to the steram around. The class also has another method for reading strips of the data which are written back out as the http server response. The code seems horribly inefficient with all the copying.

Does anyone have any suggestions on how to improve this code? Any tutorials about doing this would be appreciated. I have heard that taking ownership of the stream data would be the best route but I am not familiar with doing that.

void SomeClass::SetupTheData()
{
	size_t bufferSize = 1024;
	std::auto_ptr<unsigned char> buffer(new unsigned char[bufferSize]);
	unsigned bytesRead = 0;
	//read through the data once to grab the total size of the data
	do
	{
		bytesRead = myStream->readSomeBytes(buffer.get(), bufferSize);
		responseSize_ += bytesRead;
	} while (bytesRead > 0);

	closeTheStream(myStream);

	//copy the stream to the response
	theresponse_.reset(new unsigned char[responseSize_]);
	myStream = openTheStream();

	myStream->readSomeBytes(response_.get(), responseSize_);

	closeTheStream(myStream);
}

//buff_offset_ is a static at the class level
unsigned int SomeClass::ReadSomeResponse( char* buf, unsigned int size)
{
                unsigned int i;
                int bytesWritten = 0;
                for(i = 0; i < size && (i+buff_offset_ < responseSize_); i++)
                {
                        buf[i] = response_.get()[i+buff_offset_];
                        bytesWritten++;
                }
                buff_offset_ += i;
                return bytesWritten;
        }
        catch(...)
        {
		//some error happend
		//uh oh
        }
}

Thanks

[Abramus]

Well, I have doubts if copying the data multiple times in RAM can be a bottleneck of you program - net transfer is MUCH slower. Have you measured this?

Having said that, there are some problems with the code you posted:
1) Using std::auto_ptr to hold a pointer to a dynamically allocated array is wrong as std::auto_ptr destructs the referenced object using 'delete', not 'delete[]'.
2) Reading the data from a stream only to determine size of this data is not very smart.
3) You are also allocating memory each time SomeClass::SetupTheData is called.

What can be done about it? I don’t know the design of your program sufficiently to give you a specific answer, but there are some things you could try:
1) Maybe make a big buffer for the data at the start of the program (or one such buffer per connection or something) so that you don’t need to do allocations all the time. Read data directly into this buffer. If the stream still contains some data after reading then double the size of the buffer and read the remaining part.
2) Alternatively, create a fast allocator for chunks of the data (all chunks are of the same size). Lets say the allocator creates 100 such chunks using 'new', and then you can get these chunks one by one when you need them. If there are no more available chunks, the allocator again creates 100 in one operation. You will also need some kind of higher layer for this as the read data will be divided into some number of separate chunks.