Allocating large amounts of data on the heap

[keanedawg]

If something I understand is fundamentally wrong (which is quite likely), please correct me.


I'm creating a program which impersonates that of the heap. In it, I essentially create my own version of "malloc" and "free" functions.

So here's the thing, the way I understand it is that the heap is a growing amount of memory which depends on the size of the request. So if I only need an array of size 50 bytes, the operating system will give me a page of 16384 bytes - and then give me 50 bytes from that page. Next, if I allocate a second set of 50 bytes, I will get another 50 bytes from that same page.

So what the hell happens if I request a size of a billion integers? I can't put it all on a single page. A professor once told me the compiler can be tricky and actually put the data on different pages of memory (Even though it looks like a single long array). Though I'm not sure if that's true. If you look at the memory addresses, they never jump to a new location.

This has created a problem in my program. Essentially, I have a working version of malloc, but if the user requests anyting more than 16374 bytes at once, I can't do anything. I can't make a bigger page and there's no guarantee that two pages reside next to each other in memory.

I hope this has made sense. The problem I'm having is essentially that I can't create a "fake" malloc that allocates more than 16374 bytes because I don't know how to handle memory larger than the page. Any responses appreciated. Thanks.

[helios]

So if I only need an array of size 50 bytes, the operating system will give me a page of 16384 bytes - and then give me 50 bytes from that page. Next, if I allocate a second set of 50 bytes, I will get another 50 bytes from that same page.

Sort of. It's usually a bit more complicated than that. Keep in mind that the OS has its own allocator and malloc() implements an allocator on top of that. Additionally, a malloc() implementation may use any of several allocation strategies to decide where to get your memory from.

A professor once told me the compiler can be tricky and actually put the data on different pages of memory (Even though it looks like a single long array). Though I'm not sure if that's true. If you look at the memory addresses, they never jump to a new location.

malloc() is perfectly capable of returning a pointer to a buffer that crosses the boundaries of two or more consecutive pages, provided that memory is not so fragmented that it is unable to find an unallocated range of bytes large enough to fit the request.

if the user requests anyting more than 16374 bytes at once, I can't do anything. I can't make a bigger page and there's no guarantee that two pages reside next to each other in memory.

Sure there is. You just have to ask the OS to give you specific pages (the OS may of course return an error on some such requests). If you're using an OS where malloc() can allocate buffers larger than a single page, then the OS does provide this functionality.