- Forum
- UNIX/Linux Programming
- Virtual memory assigned to my program
Virtual memory assigned to my program
Hi everyone!
I´ve got an application which periodically request for some GB of memory, and after some work is done, they are released.
When I check the virtual memory assigned to my proccess it seems to be bigger than it should be after some iterations of the memory request.
For example, if I am working with data that takes about 9GB the virtual memory assigned at the peaks looks something like this:
Iteration 1: 9.0GB
.
.
Iteration 50: 9.2GB
.
.
Iteration 100: 9.6GB
.
.
(and so on)
My concern is that if it keeps growing, eventually it will reach the limit and request for swap memory, which I must avoid.
I am pretty sure that there are no memory leaks.
I´ve looked for information about how virtual memory works in Linux. I think it could be that after some time executing the program, some parts of the memory are not dealed in an efficient way. For example, if the memory management uses 10MB to store an element that just needs 8MB.
I have no idea if this could be happening or if something else could be the reason.
I would appreciate if anyone knows some source of information about this topic or any kind of help.
Thanks!!
I´ve got an application which periodically request for some GB of memory, and after some work is done, they are released.
When I check the virtual memory assigned to my proccess it seems to be bigger than it should be after some iterations of the memory request.
For example, if I am working with data that takes about 9GB the virtual memory assigned at the peaks looks something like this:
Iteration 1: 9.0GB
.
.
Iteration 50: 9.2GB
.
.
Iteration 100: 9.6GB
.
.
(and so on)
My concern is that if it keeps growing, eventually it will reach the limit and request for swap memory, which I must avoid.
I am pretty sure that there are no memory leaks.
I´ve looked for information about how virtual memory works in Linux. I think it could be that after some time executing the program, some parts of the memory are not dealed in an efficient way. For example, if the memory management uses 10MB to store an element that just needs 8MB.
I have no idea if this could be happening or if something else could be the reason.
I would appreciate if anyone knows some source of information about this topic or any kind of help.
Thanks!!
- Is the memory rising linearly with each iteration, or does it appear to be leveling off?
- Can you provide excerpts of the code that requests and releases memory?
- Without more specific information, if this is an application that you yourself can compile, I would suggest using valgrind.
See: https://stackoverflow.com/a/13684022
- Can you provide excerpts of the code that requests and releases memory?
- Without more specific information, if this is an application that you yourself can compile, I would suggest using valgrind.
See: https://stackoverflow.com/a/13684022
Last edited on
- Is the memory rising linearly with each iteration, or does it appear to be leveling off?
It seems to be random. With some iterations it raises more than others. Sometimes you cannot see difference.
- Can you provide excerpts of the code that requests and releases memory?
Sorry, my company does not allow me to provide much information about the code.
- Without more specific information, if this is an application that you yourself can compile, I would suggest using valgrind.
I tried using valgrind but it was too slow with the project. However I programmed a counter everytime memory was requested/released, and everything seems to be released (the counter goes back to 0).
Thanks for your time!
It seems to be random. With some iterations it raises more than others. Sometimes you cannot see difference.
- Can you provide excerpts of the code that requests and releases memory?
Sorry, my company does not allow me to provide much information about the code.
- Without more specific information, if this is an application that you yourself can compile, I would suggest using valgrind.
I tried using valgrind but it was too slow with the project. However I programmed a counter everytime memory was requested/released, and everything seems to be released (the counter goes back to 0).
Thanks for your time!
1. Are you requesting the the 9GB in one piece, or are you allocating and deallocating smaller pieces totaling 9GB?
2. Are you allocating all the heap memory at one time, then deallocating all the memory at the same time?
2. Are you allocating all the heap memory at one time, then deallocating all the memory at the same time?
I don't think the "virtual memory" column is much to worry about.
The total virtual memory usage for all the processes that are currently running on my computer adds up to about 48 GB. I only have 8 GB RAM + 8 GB Swap and I don't experience any problems.
The total virtual memory usage for all the processes that are currently running on my computer adds up to about 48 GB. I only have 8 GB RAM + 8 GB Swap and I don't experience any problems.
Last edited on
If we are talking about gnome system monitor, then you can right click on the process and hit "memory maps".
That brings up a kind of explanation for what they mean by "Virtual Memory" as they report it. The best I understand is that it reports "Virtual Memory" as all the different points in memory that your program touches including the size as the program loaded from the harddrive and swap memory utilized. It also includes all the dependencies and kernel drivers that were utilized and/or depended upon and so on. It reports them as used even if they have been deleted/freed after their initial use, so it is not a representation of what you might call living memory.
"Virtual Memory" is not much use because of the previous paragraph. Still, I do think it's interesting to see where your program has been.
"Resident Memory" is a bit more useful as it is the memory that your program and its included shared libraries are currently using.
"Shared Memory" looks to be strictly what is being used by the included shared libraries; not allocated by your program. These shared libraries may have been activated by other programs, you are just a client of those libs. If this is continuously increasing then a memory leak may be occurring in an included shared library (this may not be your code's fault)
Looking further to the right, past CPU and ID, you will see the column labelled "Memory".
"Memory" is how much is allocated by your program. If this is continually and unexpectedly increasing then a memory leak is indicated somewhere in your own code.
"Memory" + "Shared Memory" approximately equals "Resident Memory".
Grain of salt: this is all best-guess-understanding. I am currently reading the documentation on Gnome System Monitor, and I'm finding it a bit lacking.
That brings up a kind of explanation for what they mean by "Virtual Memory" as they report it. The best I understand is that it reports "Virtual Memory" as all the different points in memory that your program touches including the size as the program loaded from the harddrive and swap memory utilized. It also includes all the dependencies and kernel drivers that were utilized and/or depended upon and so on. It reports them as used even if they have been deleted/freed after their initial use, so it is not a representation of what you might call living memory.
"Virtual Memory" is not much use because of the previous paragraph. Still, I do think it's interesting to see where your program has been.
"Resident Memory" is a bit more useful as it is the memory that your program and its included shared libraries are currently using.
"Shared Memory" looks to be strictly what is being used by the included shared libraries; not allocated by your program. These shared libraries may have been activated by other programs, you are just a client of those libs. If this is continuously increasing then a memory leak may be occurring in an included shared library (this may not be your code's fault)
Looking further to the right, past CPU and ID, you will see the column labelled "Memory".
"Memory" is how much is allocated by your program. If this is continually and unexpectedly increasing then a memory leak is indicated somewhere in your own code.
"Memory" + "Shared Memory" approximately equals "Resident Memory".
Grain of salt: this is all best-guess-understanding. I am currently reading the documentation on Gnome System Monitor, and I'm finding it a bit lacking.
Last edited on
That Virtual Memory metric is the size of the processes address space.
If you allocate pages and release them back to the OS repeatedly, and you get different pages, the Virtual Memory will grow. But that doesn't mean the application is using more memory.
Remember those memory limits of 16bit and 32bit systems, all solved be 64bit giving a huge (2^15-ish) bytes. Well, not all of that address space is mapped to virtual memory. It's only mapped on demand. That Virtual Memory metric is showing you how much has been mapped.
If you allocate pages and release them back to the OS repeatedly, and you get different pages, the Virtual Memory will grow. But that doesn't mean the application is using more memory.
Remember those memory limits of 16bit and 32bit systems, all solved be 64bit giving a huge (2^15-ish) bytes. Well, not all of that address space is mapped to virtual memory. It's only mapped on demand. That Virtual Memory metric is showing you how much has been mapped.
The application has been working for several days and the physical memory has stopped growing at certain point.
According to what Peter87 said, I think that the virtual memory is not a good metric to know how much memory a program is using with precision.
Thank you all for your time!
According to what Peter87 said, I think that the virtual memory is not a good metric to know how much memory a program is using with precision.
Thank you all for your time!
Topic archived. No new replies allowed.