- Forum
- General C++ Programming
- HardDisk Bottleneck
HardDisk Bottleneck
Hello,
Me and a couple of colleagues just finished creating a Client\Server Application.
I was responsible for the Server creation.
The server takes images from the client and preforms some Image Process methods with the images. (JPG's)
the tool i used to do this is : ImageMagick (www.imagemagick.org)
this tool is operating via Command line prompts, for example: convert.exe -greyScale Input.jpg Output.jpg.
My problem is :
these execution work on my Harddisk and takes along time to preform.
My quesiton is:
is it possible somehow to load all the Imagmagick files into my RAM and use it as a Harddisk alternative? this way all computation preforms on my RAM and is superfast in comparison to my Harddisk.
is it possible?
thanks!
Me and a couple of colleagues just finished creating a Client\Server Application.
I was responsible for the Server creation.
The server takes images from the client and preforms some Image Process methods with the images. (JPG's)
the tool i used to do this is : ImageMagick (www.imagemagick.org)
this tool is operating via Command line prompts, for example: convert.exe -greyScale Input.jpg Output.jpg.
My problem is :
these execution work on my Harddisk and takes along time to preform.
My quesiton is:
is it possible somehow to load all the Imagmagick files into my RAM and use it as a Harddisk alternative? this way all computation preforms on my RAM and is superfast in comparison to my Harddisk.
is it possible?
thanks!
That sounds rather strange.
How did you determine that the hard disk is the bottleneck?
The image loading/transformation/saving process is relatively expensive, so you would need a system with many, many CPUs processing images in parallel for the disk to become the bottleneck (if it is otherwise idle).
Please confirm that the disk accesses are not caused by swapping, which will occur if your system is low on RAM and you're trying to process a very large image file (as in dimensions, not the file size).
How are the images processed anyway? Are they received by the server, then saved and processed immediately? If so, the input file will still be in the file cache when ImageMagick reads it - no disk access will occur. Likewise, when you read back the output file immediately and then delete it, it's likely that no disk accesses will ever occur.
How did you determine that the hard disk is the bottleneck?
The image loading/transformation/saving process is relatively expensive, so you would need a system with many, many CPUs processing images in parallel for the disk to become the bottleneck (if it is otherwise idle).
Please confirm that the disk accesses are not caused by swapping, which will occur if your system is low on RAM and you're trying to process a very large image file (as in dimensions, not the file size).
How are the images processed anyway? Are they received by the server, then saved and processed immediately? If so, the input file will still be in the file cache when ImageMagick reads it - no disk access will occur. Likewise, when you read back the output file immediately and then delete it, it's likely that no disk accesses will ever occur.
OP said:
IMHO this is your underlying issue right here, weither or not you speed up your read\write times. A server is a central point of contact, not a work horse. So unless your clients are terminals or some other form of thin client then there is no reason that you should be offloading all of the processing to the server.
Which RAID are you using (if any)? Any form of stripeing is going to kill your disk write times.
Unless you're somehow writting directly to the disk with ASM, then your images are already loaded into memory.
| The server takes images from the client and preforms some Image Process methods with the images. |
IMHO this is your underlying issue right here, weither or not you speed up your read\write times. A server is a central point of contact, not a work horse. So unless your clients are terminals or some other form of thin client then there is no reason that you should be offloading all of the processing to the server.
Which RAID are you using (if any)? Any form of stripeing is going to kill your disk write times.
Unless you're somehow writting directly to the disk with ASM, then your images are already loaded into memory.
*mind-blown*
Thats amazing.
can you please refer me to a guide that shows a ram disk in c++?(with code)
| So unless your clients are terminals or some other form of thin client then there is no reason that you should be offloading all of the processing to the server. |
yes , the client is a smartphone with no huge computation capability
| How are the images processed anyway? Are they received by the server, then saved and processed immediately? If so, the input file will still be in the file cache when ImageMagick reads it - no disk access will occur. Likewise, when you read back the output file immediately and then delete it, it's likely that no disk accesses will ever occur. |
i dont understand you question - the input for the image process util is a file name (freshly recieved from client) and the output is another filename - to be further analyzed..
| i dont understand you question - the input for the image process util is a file name (freshly recieved from client) and the output is another filename |
When you write data to a file, it normally isn't written to disk immediately, but remains in memory for a while.
So when ImageMagick reads that file, it takes the data from memory, not the disk. And when you delete the file before the OS decides to finally write the data to disk, your chances are good that no disk access will occur at all.
The same applies to the output file. When the server application reads and deletes it right after ImageMagick completes, it will likely never make it to the disk.
athar - how can i verify if the file being used is cache\HD file?
is there any way to verify it?
is there any way to verify it?
I apologize, I assumed this was a school project and not a professional application. You have made exactly the right choice in your layout.
Not that easy I'm afraid. This would be heavily OS dependent and require a custom device driver. You'd be better off downloading one that was already made just to get up and running for now. Have you considered solid state drives? The R\W times are great for this type of thing.
| can you please refer me to a guide that shows a ram disk in c++?(with code) |
Not that easy I'm afraid. This would be heavily OS dependent and require a custom device driver. You'd be better off downloading one that was already made just to get up and running for now. Have you considered solid state drives? The R\W times are great for this type of thing.
SSD - yeah iv'e considered it .
now that you mention RAM DISK is complex- i might just pick this solution.
thanks
now that you mention RAM DISK is complex- i might just pick this solution.
thanks
| athar - how can i verify if the file being used is cache\HD file? is there any way to verify it? |
You can use Windows' perfmon tool to monitor disk accesses. Choose a partition for the ImageMagick input/ouput files that isn't being read or written from by other programs, then monitor "bytes read/s" on that partition and verify that little or no data is read.
Likewise, before you go off and buy an SSD for nothing, monitor the read/write queues and the percentage the disk is busy and verify that it is near 100%. As already mentioned, it's unlikely that the hard disk is the problem.
Besides, setting up a RAM disk takes less than a minute, including downloading and installing.
Last edited on
| Besides, setting up a RAM disk takes less than a minute, including downloading and installing. |
will i have to install the IMAGEMAGICK on the RAM disk?
i mean does it behave like a regular HDisk?
does it have persistency?
| will i have to install the IMAGEMAGICK on the RAM disk? |
That's not necessary, you just need to save the input and output files on it.
| i mean does it behave like a regular HDisk? |
More or less, yes.
| does it have persistency? |
Normally, no. However, some tools provide that functionality by saving the contents of the RAM disk to the hard disk when the computer is shut down.
Can't you just do the processing programmatically with a library like Magick++? It seems kind of silly to save to a file that ends up in memory and then do the processing when you (presumably) had the thing in memory to begin with.
OMG HELIOS
thats the perfect Solution!
thanks thanks thanks!
didnt know about its existence
thats the perfect Solution!
thanks thanks thanks!
didnt know about its existence
Last edited on
Topic archived. No new replies allowed.