File I/O

[Disch]

I'm looking for File I/O routines which are a bit more specialized than cstdio or iostream. Specifically, I want to be able to resize files, and get the length of a file without requiring a seek. Although the file length thing isn't that important -- file resizing is definately a must.

To my knowledge, there is no way to explicitly set the size of a file with cstdio/iostream methods.

WinAPI offers a series of functions that work on HANDLEs that spawn from the CreateFile function which offer this functionality (and a lot of other stuff that you can't do with the generic standard lib stuff). However it's obviously not portable.

Is there something like this for Linux? Can anyone link me to a reference page?

Or better yet is there a platoform independent way to do this? I find this doubtful, but figured I should ask anyway.

Thanks in advance.

[elvis]

The size of a file is simply determined by its contents,
you can only influence it by writing stuff to the file.
File size is normally not an independent variable that
you can set at arbitrary values.

Can you indicate for what kind of application you would
"set" the size of a file? As it is now, your problem does
not make any sense.

[writetonsharma]

what you can do is, seek to the byte and then write something there. this will create a hole in the file and you have that much size of file.
like if you need 1000 bytes of file. then you do:

fseek(fp, 1000, SEEK_SET);
fwrite('\n', 1, 1, fp);

you seek to 1000th byte and wrote a '\n'. that will do i think.
your file is resized to 1000bytes.
im a little confused if you would use fseek or lseek.. try it, it will solve the problem.

[Disch]

The size of a file is simply determined by its contents, you can only influence it by writing stuff to the file. File size is normally not an independent variable that you can set at arbitrary values.

Err... wouldn't it have to be an independent variable? How else would the file system know how large a file is? There's no way to determine that from contents alone. There's no special "EOF" byte or string that marks the end of the file.

It's not an independent variable in cstdio (at least not one that's user accessable), which is my point. cstdio isn't good enough for what I want to do. That's why I used the WinAPI method of File I/O on Windows -- and why I'm looking for alternatives for Linux.

Can you indicate for what kind of application you would "set" the size of a file? As it is now, your problem does not make any sense.

Okay -- let's say I'm making a hex editor. The user goes in and deletes the last 5 bytes off a 2 GB file. How could this be accomplished without reading the entire file into memory, then writing it back (a process that could possibly take several minutes, and would use tons of RAM)

And to pre-empt the file mapping suggestion: that's no good because it doesn't allow you to buffer changes made to the file.

what you can do is, seek to the byte and then write something there. this will create a hole in the file and you have that much size of file.

This works for making a file larger, but this method can't be employed to make a file smaller, which is typically what I'd want to do this for (not to mention this seems a little hackish to me -- but I guess if it's standard behavior it's fine).

So far the only way I'm aware of to make a file smaller with cstdio is to read the entire file into memory (minus what I want to remove), then re-open the file with "w" to wipe it clean, then write everything back. That is such a huge waste of time and resources when the same thing should be able to be accomplished with a very minor function to change the filesize.

[writetonsharma]

hmmm...
making a file small..that's tricky..!!
because the normal method will take quite a long time, you read the file till you want and then create a file with these contents..just waste of time..

let us think about it..

[Disch]

Someone I know on IRC suggested <unistd.h> and the ftruncate() function:

http://opengroup.org/onlinepubs/007908799/xsh/ftruncate.html

Which appears to work with "open()" style file descriptors:

http://opengroup.org/onlinepubs/007908799/xsh/open.html

This looks like it would be a promising solution. I'll try it out and report back just in case anyone else here is interested.

[writetonsharma]

did you find anything??

[Disch]

Well I don't think I'll have time to fully test that before I have to leave on this trip I'm going on (will be gone for about a week).

Although I have it seeking and reading files (but haven't tried writing or changing filesize yet). I also want to check to see if it takes UTF-8 encoded filenames (another reason I want to avoid cstdio -- that's another thing it lacks). I'm a little nervous because that reference page doesn't make any mention of it.

[writetonsharma]

hmmmm...

[Disch]

After tweaking some stuff to get read/write permissions set properly when you create a new file with open()... I finally tested it out and it seems to be working great. Code snippits of interest from my wrapper class:

    ////////////////////////////////////////
    //  file opening flags
    enum
    {
        FL_READ =   0x01,       // want read permissions
        FL_WRITE =  0x02,       // want write permissions
        FL_CREATE = 0x04,       // create file if it doesn't exist
        FL_NEW =    0x08,       // create file always (destroying an existing file)

        // common combinations
        FL_R =      FL_READ,
        FL_RW =     FL_READ | FL_WRITE,
        FL_W =      FL_READ | FL_WRITE | FL_NEW
    };

bool BasicFile::Open(utfstr filename,int flags)
{
    Close();

    if(flags & FL_NEW)
        flags |= FL_WRITE;

    int oflgs;
    if(flags & FL_WRITE)        oflgs = O_RDWR;
    else                        oflgs = O_RDONLY;

    if(flags & FL_NEW)          oflgs |= O_TRUNC | O_CREAT;
    else if(flags & FL_CREATE)  oflgs |= O_CREAT;

    refarray<utf8> buf = filename.GetUTF8();
    char* chbuf = (char*)(utf8*)buf;

    int ret;
    if(oflgs & O_CREAT)
        ret = open(chbuf,oflgs,S_IRWXU | S_IRGRP | S_IROTH);
    else
        ret = open(chbuf,oflgs);

    if(ret < 0)
        return false;

    nFileDesc = ret;
    bOpened = true;
    if(flags & FL_WRITE)    bWritable = true;

    return true;
}

bool BasicFile::Resize(offset_t size)
{
    if(!IsWritable())       return false;
    if(size < 0)            return false;

    offset_t pos = Tell();
    if(pos < 0)             return false;

    offset_t cursiz = SeekEnd();
    if(cursiz < 0)          return false;

    // see if current size is less than the desired size
    if(cursiz < size)
    {
        Seek(size-1);
        if(!PutInt<u8>(0))  return false;
    }
    else if(cursiz > size)
    {
        // truncate the file
        if(ftruncate(nFileDesc,clip_cast<off_t>(size)) != 0)        return false;
    }

    Seek(pos);
    return true;
}

UTF-8 encoded filenames appear to work, but that might just be because that's how the filesystem I'm on works? It doesn't appear to be mentioned on the above page that the given filenames are to be UTF-8... but that appears to be the case for my implementation. I wish it was more standardized.

I never actually tried UTF-8 strings with cstdio under *nix, but I did under Windows (and it failed -- which is one reason I resorted to WinAPI calls).