Transposing files I can not load into memory
Hi,
I am trying to transpose a file to output into a new file. This would be easy if I could load it into memory and use a couple of for loops. However, the file is so large that this is not possible (~90Gb).
Could anyone give me some guidance here. I though that perhaps I could write to a file 'column' at a time, rather than the standard row/line at a time so that I only need 1 line of data in memory (which would fit) as opposed to the whole thing.
I am relatively new to c++ so any pointers (no pun intended) would be gratefully receieved.
Thanks,
Sam
I am trying to transpose a file to output into a new file. This would be easy if I could load it into memory and use a couple of for loops. However, the file is so large that this is not possible (~90Gb).
Could anyone give me some guidance here. I though that perhaps I could write to a file 'column' at a time, rather than the standard row/line at a time so that I only need 1 line of data in memory (which would fit) as opposed to the whole thing.
I am relatively new to c++ so any pointers (no pun intended) would be gratefully receieved.
Thanks,
Sam
What the hell is in that file? 90Gb ?
I want to know the resolution for this problem as well.
I want to know the resolution for this problem as well.
You may need OS-specific APIs for very large file support. In Windows, CreateFile can open a file up to 16EB in size because the file pointer is 64bit.
Last edited on
Topic archived. No new replies allowed.