Memory Leak
Hey all. I am having trouble with, what I believe is a massive memory leak. My goal is to display a 3d sinusoidal surface using OpenGL. To do that I make an XY grid and calculate the "Z" coordinate at each location. I store the vertex locations in a std vector vecv_s and the normal vectors at each vertex in a std vector vecn_s. The objects within are Vector3F objects which simply store the x,y,z components of the mathematical vectors.
I then have to index all of the vertices because I have to draw them in OpenGL in a specific orders. The precise indices are stores in a 2-D std vector in type "int".
I am able to run and display, in real time, with fair performance, when N<100. When N gets too large the rendering slows significantly, and at a certain point the program crashes with "bad_alloc" if N is too large.
Can anyone help me understand what's might be going wrong, or how I can do this more efficientl? I'd like to be able to generate a really fine grid so I can get a great approximation for the fuction. Is 25M quadrilaterals too many? I don't know how many polygons you can expect to render with today's hardware.
I then have to index all of the vertices because I have to draw them in OpenGL in a specific orders. The precise indices are stores in a 2-D std vector in type "int".
I am able to run and display, in real time, with fair performance, when N<100. When N gets too large the rendering slows significantly, and at a certain point the program crashes with "bad_alloc" if N is too large.
Can anyone help me understand what's might be going wrong, or how I can do this more efficientl? I'd like to be able to generate a really fine grid so I can get a great approximation for the fuction. Is 25M quadrilaterals too many? I don't know how many polygons you can expect to render with today's hardware.
|
|
First I don't see any memory leaks. You're not using any dynamic memory, so memory leaks should not be the problems.
Probably. 25M * 25M * (sizeof V1 * 4) seems like a very big number to me.
The "bad alloc" is probably because you don't have enough memory for the vector. How big is N when you get this message?
Have you run the program with your debugger to see where the problem is occurring? The compiler should be able to show you where it detected the problems.
By the way why the global variables (vecv_s, vecn_s, vecf_s)? By the way you really should use more meaningful variable names, these three names are so close to being the same that confusion seems inevitable, IMO.
| Is 25M quadrilaterals too many? |
Probably. 25M * 25M * (sizeof V1 * 4) seems like a very big number to me.
The "bad alloc" is probably because you don't have enough memory for the vector. How big is N when you get this message?
Have you run the program with your debugger to see where the problem is occurring? The compiler should be able to show you where it detected the problems.
By the way why the global variables (vecv_s, vecn_s, vecf_s)? By the way you really should use more meaningful variable names, these three names are so close to being the same that confusion seems inevitable, IMO.
Last edited on
The size of the 2-D grid is N by N. So 5,000*5,000 = 25M. Which means I have 25 million vertices to keep track of. Each vertex has 6 floats associated with it stored in two separate vectors (VECtor of Normals for Surface) and (VECtor of Vertices for Surface).
The 2-D grid will have (N-1)*(N-1) quadrilateral elements, which each have 4 numbers associated with it stored in vecf_s (VECtor of Faces for Surface).
These are all global in scope currently because this function simply "builds" the surface. The surface is drawn (over and over) in another function "DrawScene()" later. It made sense to me to have these vectors global so they never leave scope and have to be rebuilt.
I'm started to learn how to make a game by learning OpenGL (and c++ simultaneously). I'm trying to make the "ground floor", and a simple sinusoidal surface that appeared like hills made sense. I'm having trouble expanding the ground surface without running out of memory.
The debugger is still running, but it's doing something kind of funny in the last loop with V1 in it. It's running particularly slow. Like a few minutes to get through 300 iterations.
Does V1.clear() actually clear that vector from memory? Is this implementation good?
EDIT:
Debugger Failed:
The 2-D grid will have (N-1)*(N-1) quadrilateral elements, which each have 4 numbers associated with it stored in vecf_s (VECtor of Faces for Surface).
These are all global in scope currently because this function simply "builds" the surface. The surface is drawn (over and over) in another function "DrawScene()" later. It made sense to me to have these vectors global so they never leave scope and have to be rebuilt.
I'm started to learn how to make a game by learning OpenGL (and c++ simultaneously). I'm trying to make the "ground floor", and a simple sinusoidal surface that appeared like hills made sense. I'm having trouble expanding the ground surface without running out of memory.
The debugger is still running, but it's doing something kind of funny in the last loop with V1 in it. It's running particularly slow. Like a few minutes to get through 300 iterations.
Does V1.clear() actually clear that vector from memory? Is this implementation good?
EDIT:
Debugger Failed:
|
|
Last edited on
closed account (48T7M4Gy)
As a start adjust the value of N until the program runs. oops I see you've already done that.
Last edited on
| These are all global in scope currently because this function simply "builds" the surface. The surface is drawn (over and over) in another function "DrawScene()" later. It made sense to me to have these vectors global so they never leave scope and have to be rebuilt. |
So I take it you've never heard of passing by reference into your functions?
Have you compiled your code with debugging information enabled (-g)? By the way you need to back trace until you get into your code. The locations shown in the above code is in the stl code, which is probably not the root cause of the crash.
| Does V1.clear() actually clear that vector from memory? |
Yes, it clears the vector, however it may or may not change the capacity.
| Is this implementation good? |
I question the need for the two sets of loops and all the push_back() calls. You seem to know the size of the vectors so you would probably be better off sizing the vectors when you create them.
If a float is 4 bytes that means vecv_s and vecn_s will use more than 572 MB.
5001*5001*6*4 bytes ≈ 572 MB
If an int is 4 bytes that means vecf_s will use more than 381 MB.
5000*5000*4*4 ≈ 381 MB
It will actually be much more than that because a vector has some overhead. On a 64 bit compiler I expect it will be at least 24 bytes extra for storing a data pointer, the size and the capacity (other things like alignment might result in additional overhead). That gives at least an additional 572 MB.
5000*5000*24 ≈ 572 MB
572+381+572 MB ≈ 1.5 GB
If you increase N this number will grow quickly.
5001*5001*6*4 bytes ≈ 572 MB
If an int is 4 bytes that means vecf_s will use more than 381 MB.
5000*5000*4*4 ≈ 381 MB
It will actually be much more than that because a vector has some overhead. On a 64 bit compiler I expect it will be at least 24 bytes extra for storing a data pointer, the size and the capacity (other things like alignment might result in additional overhead). That gives at least an additional 572 MB.
5000*5000*24 ≈ 572 MB
572+381+572 MB ≈ 1.5 GB
If you increase N this number will grow quickly.
Last edited on
So I take it you've never heard of passing by reference into your functions? |
A callback to the drawscene function is called multiple times a second. I don't think there is any better way to store the information other than a global variable, because every function goes in and out of scope multiple times. I'm open to better suggestions though.
Have you compiled your code with debugging information enabled (-g)? By the way you need to back trace until you get into your code. The locations shown in the above code is in the stl code, which is probably not the root cause of the crash. |
I'm not sure how to do the traceback. I ran the debugger in code blocks from within the IDE which goes straight to the STL code. I'm sure there is an easy way to connect the dots back to my code, any guidance?
Not sure if I'm posting this in the right forum. Perhaps I need to find one with an emphasis on OpenGL and 3-D graphics. I'm likely not doing this in a wise manner. I have to imagine there is a better way to create a ton of polygons for the "ground". Any idea on who games like minecraft manage their memory with an "infinite" world? I can't even store a relatively small surface without taking up 1.5GB!!
If anyone has similar interests and could recommend resources/communities, I would appreciate that as well.
Umm... passing by reference is the way to go. If you're worried about speed, on my computer passing by reference is faster than calling globals.
For example: http://coliru.stacked-crooked.com/a/5c23dae8da91a2e0
But, anyway, the difference in speed is so little as to be negligible in all but the most performance-sensitive applications.
If you look at the stack trace (there should be a window in CodeBlocks showing the call stack) you should be able to find the bit of your code that called the STL code that did a bad_alloc.
Also, game like Minecraft 'chunk' the world; they don't actually render the whole world in one go. The idea is not to render anything you can't see; and, if that's still to far, add in a fog or something so that you can't see as far. That way you only need to render the surroundings, which could be in the order of 5,000 vertices, rather than the entire map.
For example: http://coliru.stacked-crooked.com/a/5c23dae8da91a2e0
But, anyway, the difference in speed is so little as to be negligible in all but the most performance-sensitive applications.
If you look at the stack trace (there should be a window in CodeBlocks showing the call stack) you should be able to find the bit of your code that called the STL code that did a bad_alloc.
Also, game like Minecraft 'chunk' the world; they don't actually render the whole world in one go. The idea is not to render anything you can't see; and, if that's still to far, add in a fog or something so that you can't see as far. That way you only need to render the surroundings, which could be in the order of 5,000 vertices, rather than the entire map.
Did you modify your project to add the debug information? If so then the problem line should have the cursor pointing to it in the editor window.
You should've also modified the project to increase your compiler warning levels.
You should've also modified the project to increase your compiler warning levels.
Hey all. Thanks for the replies. The stack trace is kind of cryptic to me:
http://i.imgur.com/7TmXMJK.png
I did isolate the code that causes the error though. It's really simple code, but it takes forever to run and eventually errors out. It's the double loop in the code below:
I took out any custom classes, so anyone should be able to run this code fairly easily.
http://i.imgur.com/7TmXMJK.png
I did isolate the code that causes the error though. It's really simple code, but it takes forever to run and eventually errors out. It's the double loop in the code below:
|
|
I took out any custom classes, so anyone should be able to run this code fairly easily.
It works on my VS 2013 CE.
Output:
Output:
|
|
Touche... It does work. It only breaks when I add the additional two vectors (which are unfortunately custom classes) The full code is shown below.
The vector classes can be found at the following location under "Assignment 0 starter code" if you wish to run it:
http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-837-computer-graphics-fall-2012/assignments/
The vectors are only about 25mb each. Any advice is appreciated. I'm still not clear on the stack trace debugging, you can see from my previous post what comes out of that.
|
|
The vector classes can be found at the following location under "Assignment 0 starter code" if you wish to run it:
http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-837-computer-graphics-fall-2012/assignments/
The vectors are only about 25mb each. Any advice is appreciated. I'm still not clear on the stack trace debugging, you can see from my previous post what comes out of that.
What operating system are you using? 32 or 64 bit? What compiler/version are you using?
How much memory does your system have?
What is the size of an int on your system?
Remember you are trying to build, in this "simple" function a vector of 5,000 * 5,000 vectors of 4 integer values. So you have at minimum 25,000,000 * (4 * 4) bytes, or 400,000,000 bytes at a minimum. This doesn't even consider any overhead being used by the vector class.
Your calculations seem a little strange as well. You seem to be forgetting that integer math has no fractions. Look at your second calculation:
Until i is larger than N + 1 that first calculation i / (N + 1) yields zero because both of the operands are integers. I also don't understand why you're casting an integer calculation to an integer.
The trace you've shown is in the standard library routines, but the actual problem is being caused by your code so you need to backtrace farther until you enter your code to see where the problem is actually located. I would guess that you're running out of memory trying to reallocate memory for that vector.
It's taking forever to run because of all the memory reallocation required during the loop. You may want to think about either reserving memory for the vector before the loop, or size the vector properly when you declare the vector to avoid the relocations.
How much memory does your system have?
What is the size of an int on your system?
Remember you are trying to build, in this "simple" function a vector of 5,000 * 5,000 vectors of 4 integer values. So you have at minimum 25,000,000 * (4 * 4) bytes, or 400,000,000 bytes at a minimum. This doesn't even consider any overhead being used by the vector class.
Your calculations seem a little strange as well. You seem to be forgetting that integer math has no fractions. Look at your second calculation:
V1.push_back((int(i/(N+1))+1)*(N+1)+j + i*(N+1));Until i is larger than N + 1 that first calculation i / (N + 1) yields zero because both of the operands are integers. I also don't understand why you're casting an integer calculation to an integer.
| The stack trace is kind of cryptic to me: |
The trace you've shown is in the standard library routines, but the actual problem is being caused by your code so you need to backtrace farther until you enter your code to see where the problem is actually located. I would guess that you're running out of memory trying to reallocate memory for that vector.
| but it takes forever to run and eventually errors out. |
It's taking forever to run because of all the memory reallocation required during the loop. You may want to think about either reserving memory for the vector before the loop, or size the vector properly when you declare the vector to avoid the relocations.
Does V1 really need to be a vector?
I changed V1 from a vector to a simple struct since it only has four values.
Using N=1000, your unmodified code ran 17.215 seconds on my machine.
Using a struct for V1, it ran in 0.374 seconds.
I changed V1 from a vector to a simple struct since it only has four values.
Using N=1000, your unmodified code ran 17.215 seconds on my machine.
Using a struct for V1, it ran in 0.374 seconds.
|
|
What operating system are you using? 32 or 64 bit? What compiler/version are you using? |
Win10 64 bit
How much memory does your system have? |
16 gb
What is the size of an int on your system? |
4 bytes
Your calculations seem a little strange as well. You seem to be forgetting that integer math has no fractions. Look at your second calculation: V1.push_back((int(i/(N+1))+1)*(N+1)+j + i*(N+1)); Until i is larger than N + 1 that first calculation i / (N + 1) yields zero because both of the operands are integers. I also don't understand why you're casting an integer calculation to an integer. |
I'm positive there is a better way to do it. The point is that at the end of the outer loop, the indexing needs to bump up by a factor of (N+1). I used "int()" as a clever (not so clever..??) way of rounding. Maybe floor() works just as well? I don't know I was NOT in the zone the day I wrote that code. I was just trying to get the damn surface to render.... It worked (for small N)!
The trace you've shown is in the standard library routines, but the actual problem is being caused by your code so you need to backtrace farther until you enter your code to see where the problem is actually located. I would guess that you're running out of memory trying to reallocate memory for that vector. |
Yes I can see that, I'm sure there is a way to make the IDE tell you which line of code is calling that particular library function, but I'm just not sure how. The higher level codes I'm used to give you the trace from failure point all the way up to the written script. I'm sure codeblocks will do the same thing, I'm just not sure how to get it to do that.
It's taking forever to run because of all the memory reallocation required during the loop. You may want to think about either reserving memory for the vector before the loop, or size the vector properly when you declare the vector to avoid the relocations. |
Previously I used a similar architecture to read an object file. It made more sense to dynamically allocate memory using vector.push() rather than declaring the size upfront, because I didn't know how many vertices the object file had without calculating it apriori. Calculating it apriori seemed to be wasteful, because it required a loop just to count the number of elements and nothing else.
In this application, however, you're correct.. I can do a simple calculation to determine the size necessary. This would increase performance, but not memory allocation issues I believe, no?
I'm also interested in Abstraction's recommendation of using structs. Is this method so much faster because V1 no longer is dyanmically allocating memory (which I understand is a lot of overhead). This code needs to be severely optimized, but I'm just beginning to understand the lay-of-the land when it comes to 3d graphics. This optimization will get runtime down to a fraction of what it currently is, but I don't think it is going to fix any out-of-memory errors.
I have to imagine games like Crysis and other high-detail games are rendering many more polygons than I am trying to, no? Plus they are doing a lot more complex physics, lighting, texturing etc... I'm going to have to do some more research it seems to figure out how to do this more efficiently!!
Last edited on
It still works for me, but only on a 64bit build. On a 32bit build I got a bad_alloc exception.
Thanks for running it Thomas. That is interesting. Anyone know why it might work for one and not the other?
On a 32bit build only 4GB are usable and this might not be enough.
http://stackoverflow.com/questions/8869563/how-much-memory-can-be-accessed-by-a-32-bit-machine
http://stackoverflow.com/questions/8869563/how-much-memory-can-be-accessed-by-a-32-bit-machine
Topic archived. No new replies allowed.