please wait
fseek cracking under pressure
Mar 28, 2011 at 6:51pm
A year or so ago, I created a program in C/C++ (a mix) which took as input huge "output files" from another program and returns certain values of interest found within those files. These files are huge, easily 1GB large (the one making me write this is 3GB, 25 million lines long). Thus, since I only want a handful of values out of this mass of data, which is well-structured, I fseek constantly, jumping hundreds of thousands of characters at a time. This program has worked just fine until now.
With this file, fseek seems to be giving up after a while. The file is split into 130 steps (its from a numeric solution program). fseek() works fine (only returning 0) until step 88, after which one of its iterations starts returning -1 consistently.
The faulty line of code is the following:
where elem2=5469 and elem1=2619, making the seek=892388 characters. That seems like a lot, but the previous time (its in a loop), elem1=1, making seek=1716952 and that one worked. In fact, given the structure of the file (which is guaranteed to be correct), if the loop which contains this line works once, it should work always. And it worked 87 times.
I'd show you more of the code, but I can't think of a (non-)functional mini-example right now. I'm just curious (and desperately trying to find out!) if this has something to do with some sort of size limitation on fseek (this is one of the biggest files my program's had to deal with) or if you guys have any other ideas what this might be.
With this file, fseek seems to be giving up after a while. The file is split into 130 steps (its from a numeric solution program). fseek() works fine (only returning 0) until step 88, after which one of its iterations starts returning -1 consistently.
The faulty line of code is the following:
fseek(pos,157*(elem2-elem1)*2,SEEK_CUR); //ignore the magic numbers, I promise they're there for a reason. where elem2=5469 and elem1=2619, making the seek=892388 characters. That seems like a lot, but the previous time (its in a loop), elem1=1, making seek=1716952 and that one worked. In fact, given the structure of the file (which is guaranteed to be correct), if the loop which contains this line works once, it should work always. And it worked 87 times.
I'd show you more of the code, but I can't think of a (non-)functional mini-example right now. I'm just curious (and desperately trying to find out!) if this has something to do with some sort of size limitation on fseek (this is one of the biggest files my program's had to deal with) or if you guys have any other ideas what this might be.
Mar 29, 2011 at 6:39pm
errno returns "invalid argument". Which makes absolutely no sense, given how many other times it's worked.
And I can promise you that the file still has much, much more than 892388 characters to go. As I said, this file has a total of 25 million lines and we're only about 2/3 of the way through it when the failed call happens.
And I can promise you that the file still has much, much more than 892388 characters to go. As I said, this file has a total of 25 million lines and we're only about 2/3 of the way through it when the failed call happens.
Last edited on Mar 29, 2011 at 6:46pm
Mar 29, 2011 at 7:59pm
I have no idea what that is. But from what I can see on Google, its a Linux flag of some sort, but I'm using Windows...
Mar 29, 2011 at 8:20pm
YES! Got it!
You were on the right track. I found an article on the web of a guy trying to port a Linux program to Windows and stumbling on precisely this problem:
Thanks for the help. Though I do think this is pretty valuable information that perhaps should be added to the documentation here.
You were on the right track. I found an article on the web of a guy trying to port a Linux program to Windows and stumbling on precisely this problem:
fseek can't deal with offsets larger than 2GB. However, he found the "replacement" function _fseeki64(), which uses __int64 offset. I replaced all the fseek's in my program and it worked. w00t!Thanks for the help. Though I do think this is pretty valuable information that perhaps should be added to the documentation here.
Topic archived. No new replies allowed.