Time library to store millisecond data

[erixliechtenstein]

Hello,

I am reading a csv file which has a lot of time series data in it.

I am wondering what is the best way to store this - the data is millisecond-level and so I understand the ctime tm struct cannot hold this

What is the best way to deal with it?

[AbstractionAnon]

Not enough information.

Millisecond data relative to what? Relative to each other? Relative to a clock? Relative to something else?

How big are the values? If these are not clock values, simply store them as longs or as int64s depending on the scale of the data.

[gunnerfunner]

using the std::chrono library and suitable std::duration::casts you could read your data into variables of type std::chrono::milliseconds:

# include <iostream>
# include <chrono>

int main()
{
    using namespace std::chrono;

    std::chrono::milliseconds ms {123456789};

    auto hrs = std::chrono::duration_cast<hours>(ms);
    auto mins = std::chrono::duration_cast<minutes>(ms % std::chrono::hours(1));
    auto ss = std::chrono::duration_cast<seconds>(ms % std::chrono::minutes(1));
  //  auto msec = std::chrono::duration_cast<milliseconds>(ms % std::chrono::seconds(1));
    std::chrono::milliseconds msec = std::chrono::duration_cast<milliseconds>(ms % std::chrono::seconds(1));

    std::cout << hrs.count() << " " << mins.count() << " " << ss.count() << " " << msec.count() << " ";
}

but as mentioned above, seeing some sample lines of your data file would be more illuminating

[erixliechtenstein]

Thank you gunnerfunner - I will try that approach to reading the data - looks it should work with some slight modification


to clarify the original question:

millisecond data in absolute terms - for example:

20140501 00:00:00.298 - i.e. YYYYMMDD HH:MM:SS.LLL

The file has several thousand row of such numbers

I need to read and parse them into manipulable format - i.e. so I can add / subtract do other manipulation

Whole file looks something like this:
20140501 00:00:00.298, 34.56, 34.58 - i.e. date time | value 1 | value 2

[gunnerfunner]

boost/date_time/posix_time might be more suitable if you want to manipulate the date further :

so I can add / subtract do other manipulation

– it goes all the way down to microseconds, well beyond the milliseconds that you're looking for and has a wide range of operator overloads:
http://www.boost.org/doc/libs/1_43_0/doc/html/date_time/posix_time.html

for example, to read && display data from your file we could do:

# include <iostream>
# include <fstream>
# include <sstream>
# include <string>
# include <vector>
# include <algorithm>
# include <boost/date_time/posix_time/posix_time.hpp>

using namespace boost::posix_time;

struct MilliData
{
    ptime m_t;
    double m_val_1;
    double m_val_2;

    MilliData (const ptime& t, const double val_1, const double val_2)
    : m_t(t), m_val_1(val_1), m_val_2(val_2){}
};

int main()
{
    std::ifstream inFile {"C:\\test.txt"};
    std::vector<MilliData> data{};

    if(inFile)
    {
        std::string line{};
        while (getline(inFile, line))
        {
            std::istringstream stream{line};
            std::string date, time, str_val_1, str_val_2;
            if(stream)stream >> date >> time >> str_val_1 >> str_val_2;
            time.pop_back(); str_val_1.pop_back();
            time.erase(std::remove(time.begin(), time.end(), ':'), time.end());
            std::istringstream streamNum_1{str_val_1};
            double val_1;
            streamNum_1 >> val_1;
            std::istringstream streamNum_2{str_val_2};
            double val_2;
            streamNum_2 >> val_2;
            if(inFile) data.emplace_back(MilliData(from_iso_string(date + 'T' + time), val_1, val_2));
        }
    }
	for (const auto& elem : data)std::cout << elem.m_t << " " << elem.m_val_1 << " " << elem.m_val_2 << "\n";
}

OUTPUT

2014-May-01 00:00:00.298000 34.56 34.58

[erixliechtenstein]

thank you!! - worked perfectly. Ended up adapting it to read via csv.h parser by Ben Strasser to reduce csv reading time by about 60%