Inheriting from std::iostream

[Disch]

So I'm reconsidering the idea of making an abstracted File + Filesystem lib primarily geared towards adding functionality standard libs lack and adding support for the possibility of different kinds of file systems (like archives).

Basically it will be a lib for using .zip files that doesn't suck -- but it'll have other perks as well (such as being easy to derive from).

To do this I'll need to make my own abstracted 'File' class from which I'll derive classes that read files from .zip and other kinds of archives.

I got it in my head that I want this lib to be as compatible as possible so that people won't have to rewrite their programs to make use of it. One way to do this would be to have my File class derive from std::iostream so that existing code which uses standard streams would still work.

The problem is, iostream is a beast and I have no idea how I'd go about doing this. I googled for a solution (briefly... I didn't put much effort into it) and I found 2 pages on point:

- One said "don't do it, users aren't supposed to derive from it":
http://stackoverflow.com/questions/4482116/inherit-stdostream

- The other was like a 200 page article and seemed overly wordy and complicated (but I only skimmed, so maybe it's not so bad?):
http://spec.winprog.org/streams/


From looking at the ref pages on this site, I get the impression that all I might have to do is derive another class from std::streambuf and just give a pointer to that to std::iostream on construction.

But even if that's right, deriving from streambuf doesn't seem any simpler.


Has anyone done this before? Is there any simple guide that explains how to do this? All I really need is a list of functions I need to implement and an explanation of what they need to do. And any other conditions/requirements I need to adhere to.

I vaguely recall someone (Duoas? helios?) posted a class that derived from iostream and used cstdio internally a while back.

Thanks!

[PanGalactic]

I use the Boost IOStreams library quite extensively. Using Boost IOStreams and maybe parts of the Boost Filesystem library (cribbing parts of the interface if the code won't do) may get you close to what you are after.

The Boost IOStreams library has support for gzip and bzip2 compressors.

[Disch]

Really! Maybe it's moot then.

I'll check it out after work.

Note to self: always check boost first


Although I seem to recall that boost's file system lib wasn't abstracted... which makes me wonder how it handles archives.

[jsmith]

NB - deriving from std::ostream would not suffice your requirements anyway -- namely that user code would be
able to use your object transparently. Why? If the user wrote this:

struct Foo {
    friend std::ostream& operator<<( std::ostream& os, const Foo& )
      { return os; }
};

int main() {
    YourDerivedStreamType stream;
    stream << Foo() << 42 << std::endl;
}

You'd find that Foo() gets written to your stream object fine, but 42 and std::endl wouldn't go through your
operator<< overloads because Foo's operator<< returns a reference to the base type, not YourDerivedStreamType,
hence would call ostream::operator<< instead of YourDerivedStreamType::operator<<. To get that to work properly,
every operator<< would have to be virtual in the base class.

[Disch]

@ jsmith

That doesn't really make sense. Consider this code:

int main() {
    std::ofstream stream("myfile");
    stream << Foo() << 42 << std::endl;
}

It's like you're saying that Foo would go to the file OK, but then 42 and endl wouldn't. Where else could they go?


It shouldn't matter that << returns an ostream reference. I figure all stream I/O at some point must go through some kind of virtual function call. All I'd have to do is implement those. I wasn't planning on implementing any << operators (unless they were virtual -- but that seems unlikely).

[jsmith]

What I'm saying, in my example, is that the type of the expression stream is YourDerivedType& and thus Foo's operator<<, if it output anything, would use YourDerivedType's methods; however the type of the expression stream << Foo() is std::ostream&, not YourDerivedType&, and thus ... << 42 would call std::ostream::operator<<(), not some overload in YourDerivedType.

Yes -- you can create your own streambuf -- that internal piece you are referring to when you say virtual function call. That would work fine. However that would not require deriving from std::ostream to implement.

It may just be a misunderstanding on my part.

[moorecm]

Doesn't that just mean you'd have to write a ton of compatible stream insertion operators?

[PanGalactic]

I'm not sure why you would need to ever do that. You would just overload the underlying stream buffer (zipfilebuf) to read/write a file within a ZIP archive. You may want a convenience zipifstream type that would have an open() function, using the rest of the ostream interface just as ifstream does.

Now, that zipfilebuf stream buffer would have to cooperate with some other ZIP filesystem manager that would know how to access and decompress the data from a ZIP archive or write data to a zip archive. There are almost certainly limits that don't exist with normal filesystems (how do you extend a file in a zip archive for example) that you'd have to handle.

[Disch]

jsmith wrote:

Yes -- you can create your own streambuf -- that internal piece you are referring to when you say virtual function call. That would work fine. However that would not require deriving from std::ostream to implement.

It may just be a misunderstanding on my part.

Maybe we are miscommunicating.

I want the class to be derived from std::iostream so that existing >> and << overloads (and any other I/O method) works without any modifications. Now however iostream handles actual I/O, I figure at some point it must all go through a series of virtual functions, or it uses a custom streambuf-derived object.... or something

My general quesiton is... what do I need to do to get this to work?

PanGalactic wrote:

Now, that zipfilebuf stream buffer would have to cooperate with some other ZIP filesystem manager that would know how to access and decompress the data from a ZIP archive or write data to a zip archive. There are almost certainly limits that don't exist with normal filesystems (how do you extend a file in a zip archive for example) that you'd have to handle.

I have all of that worked out already. I've done something similar to this before. The only thing new here is getting my File class to be derived from iostream.

All I have to do is map iostream's i/o system to my File class's own i/o system. If I can accomplish that, everything falls into place.

I find it a little hard to believe that mapping one I/O system to another is really so complicated. In theory it should be as easy as this:

void MyClass::iostream_method(int somedata)
{
  my_method(somedata);
}

Just implementing one method and passing the data to another method.

Why does iostream make it so complicated!

[PanGalactic]

Disch wrote:

I want the class to be derived from std::iostream so that existing >> and << overloads (and any other I/O method) works without any modifications. Now however iostream handles actual I/O, I figure at some point it must all go through a series of virtual functions, or it uses a custom streambuf-derived object.... or something

Exactly this. You create your own streambuf:

class ZipFileBuf : public std::streambuf { /* ... */ };

overriding the protected virtual member functions as necessary to read/write to a zip archive.

You then can create zipistream:

class zipistream : public std::istream
{
    zipistream();
    open();
    close();
};

All this does is allows the zipistream to associate a ZipFileBuf to your input stream, and open() and close() the buffer.

Do the same for zipostream and zipiostream (if desired).

[simeonz]

I have a dim spot in my head when it comes to streams. I mean, I think I have learned by now how to implement every container in STL. Sometimes in several ways. But I don't know anything about the streams hierarchy.

I say this because recently a guy asked why he can't see the changes he makes with ofstream, when he reads the file with ifstream. The two streams are opened concurrently. I played around, tested a bit, and it turned out that he needed to flush the ofstream before every read operation, or alternatively to tie the streams.

I wondered then, and I wonder still, how is this accomplished. The fact that the ifstream can not see the last changes made with the ofstream means that they do not share buffers. But since the objects are created without awareness of each other, then how does the ofstream signal the ifstream to reload it's buffer when I flush the ofstream?

Either way, it's black magic to me. And actually, this probably transfers to the C-style I/O.

Regards

[Disch]

Ok so I'm trying to do a simple test case where I have output going to a std::string, then I print it later.

From what I can tell on the docs on this site, the only protected virtual output function is streambuf::xsputn. However implementing this is apparently not enough, as this only seems to work for outputting strings.

Here's what I have:

#include <iomanip>
#include <iostream>
#include <string>

using namespace std;

class MyStream : public std::iostream, private std::streambuf
{
private:
    std::string     str;
    std::streampos  pos;

    std::streamsize xsputn( const char * s, std::streamsize n )
    {
        std::streamsize needed = n + pos;
        if(needed > str.size())
            str.resize(needed);

        std::copy(s,s+n,str.begin() + pos);
        pos += n;
        return n;
    }

public:
    MyStream() : std::iostream(this), pos(0) { }

    void print()
    {
        std::cout << str;
    }
};



int main()
{
    MyStream str;
    str << "This is a test:n" << hex << setw(5) << setfill('0') << 51 << "n" << setprecision(3) << 3.14 << "   ";

    str.print();

    std::cin.get();
}

All that's printing is "This is a testn". The rest of it does not appear to be getting through to my buffer.

What else do I need to implement? Does anyone know?


EDIT: backslashes seem to not be posting....

[simeonz]

Nope. It is printing "This is a test:n00033n3.14 " on my PC.

[Disch]

well those 'n's are supposed to be new lines but the forum is removing backslashes for some reason.

Anyway my iostream implementation must be different from yours because I'm only getting the first line. What compiler are you using?

[simeonz]

mingw gcc 4.5.0 and 4.4.0 both print everything.

[Disch]

Well VS 2010e doesn't want to play nice.

trying a smaller example:

    str << 5;

    if(!str.good())
        cout << "bad"; // this is printing 

Reveals that the stream is entering a bad state. I tried to step through it in the debugger, but it's way to convoluted to make any sense of.


Whatever. I'm going to give up on this for now. iostream compatibility can always be added in later.

[PanGalactic]

Try this instead:

class MyStream : public std::iostream, private std::streambuf
{
private:
    std::string     str;
    
    virtual std::streamsize xsputn(const char* s, std::streamsize n)
    {
        str.append(s, n);
        setp(&(*str.begin()), &(*str.end()));
        return n;
    }

    virtual int overflow(char c)
    {
        str.push_back(c);
        setp(&(*str.begin()), &(*str.end()));        
        return c;
    }


public:
    MyStream() : std::iostream(this), str() { }

    void print()
    {
        std::cout << str;
    }
};

This missing piece was the call to setp(). I believe overflow() is needed as well.

[Disch]

setp() doesn't seem to be mentioned on this site. What exactly does it do?

It looks like it sets begin/end pointers, but that's a little stupid if it's true. I don't want anything accessing the data directly, that's the whole point of having my own implementation... right?

EDIT nevermind. I was looking at stringbuf not streambuf. I see it now.

*looks*

EDIT2: Yeah it's what I thought...

So I have to buffer this? That's the dumbest thing I've ever heard. Who designed this mess!

Perhaps I'm just missing the bigger picture here.

Is there any guide that does a better job of explaining this?

[PanGalactic]

http://www.amazon.com/Standard-IOStreams-Locales-Programmers-Reference/dp/0201183951

I borrowed this from a friend long ago, but just skimmed through it. Pretty much everyone cites this book. I got most of my experience through lots of trial and error. I am at best an amateur with this stuff.