how to read thousand files

Jun 1, 2012 at 2:53pm
Dear All

Please share your thoughts and ideas!!!
I have thousands of files (.txt) which I have to parse and get bits out of them
(e.g. 5th bit of each file is 0|1?).Anyway of dealing such lot files to analysis ???

1
2
3
4
5
6
7
8
int main(int argc, const char* argv[]) {

    for (int i = 0; i < argc; ++i) {
        std::cout << argv[i] << std::endl;
    }
    std::cin.get();
    return 0;
}


I used above to push bits of each file into a vector, but it requires thousands of vectors to be defined.

providing any hint of reading and dealing with such a big number of files will be very helpful. Please shed some lights on it!!
Thanks in advance for your attention.
Last edited on Jun 1, 2012 at 3:02pm
Jun 1, 2012 at 3:08pm
Just open each file, read the first byte and extract the bit.

but it requires thousands of vectors to be defined.

How so? You just need one.
Jun 1, 2012 at 3:25pm
Thanks Athar.
Sorry i did not make it clear, I need all the bits out of all the files to deal with(e.g. from 1st bit to 9999th bit). Reading from folder seems to be one solution, but I could not do it even after following some descriptions. I will very appreciate of your further attentions.

Thanks again.
Jun 1, 2012 at 8:36pm
I need all the bits out of all the files to deal with(e.g. from 1st bit to 9999th bit).

Then just read everything into a vector, process the data and continue with the next file.

Reading from folder seems to be one solution

...?
Last edited on Jun 1, 2012 at 8:37pm
Jun 1, 2012 at 10:09pm
your OS, and compiler ?
Jun 2, 2012 at 1:41am
Only question is (1st) how to read big amount of text files??? (2nd) what are the ways of getting all bits of these thousand-file for the analysis??

Thanks for reading this, and please share your ideas..

P.S. Sorry, Reading from director seems to be one solution, but could manage such big amount of files ;(
Last edited on Jun 2, 2012 at 1:43am
Jun 2, 2012 at 4:24am
closed account (3hM2Nwbp)
If you're looking to implementing a cross-platform, lightning fast file analysis program and have need of a little extra muscle than the standard library provides, I would strongly recommend looking into boost::filesystem.

http://www.boost.org/doc/libs/1_49_0/libs/filesystem/v3/doc/index.htm

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

// Going from memory here, I haven't tried to compile it.
// Keep in mind, error-checking is excluded

#include <boost/filesystem.hpp>

int main(int argc, const char* argv[])
{

    boost::filesystem::path p = boost::filesystem::system_complete(boost::filesystem::path(argv[1]));

    boost::filesystem::directory_iterator end;
    for(boost::filesystem::directory_iterator iter(p); iter != end; ++iter)
    {
        // Process File Here
    }
}
Last edited on Jun 2, 2012 at 4:44am
Topic archived. No new replies allowed.