Display last 1000 lines

[TacoLover]

Hi,
I am writing a piece of code that requires me to display the last 1000 lines from a multiple text files (log files). FYI, I am running on Linux and using g++.

I have a log file from which - if it contains more than 1000 lines, I need to display the last 1000 lines. However, the log file could get rotated. So, in case where the current log file contains less than 1000 lines, I have to go to older log file and display the remaining. For e.g., if log got rotated and new log file contains 20 lines, I have to display the 980 lines from old log file + 20 from current log files.

What is the best way to do this? Even an outline algorithm will help.

thanks in advance
regards
TL

[Dy7SLyTq]

here is a basic class i threw together that should help you get started:

#include <string>
#include <deque>

using std::deque;
using std::basic_string;

template<typename T = basic_string<char>>
class LogStack : public deque<T>
{
    private:
		deque<T> Stack;

	public:
		LogStack(int size = 1)
			: Stack(size) {}

		void push_back(T object)
		{
			if(Stack.size() == 1000) Stack.pop_front();
			Stack.push_back(object);
		}
        
        T operator[](int i) { return Stack.at(i); }
};

so then you could just declare it like this: LogStack<> TheStack;
then just read in each line of the file(s) and if you push back a line and its the 1001st, then it pushes the first one off and you have a 1000 again

heres a really messy implementation to test it: http://coliru.stacked-crooked.com/a/b7148013f8e1485f

edit: fixed a lot of mistakes i didnt realize i made and linked to an example

[Smac89]

If you are on linux, why not write a shell script that does this. The unix system has a command called tail which can take an arguement -n where n specifies the number of lines of text (beginning at the last line) to display. You can even go further and pipe the output of this through to wc (another linux command) which you can call with wc -l which will tell you the number of lines that the last command was able to produce....

Example of what I just said

tail -1000 someFileName | wc -l

[TacoLover]

Thanks DTSCode. I was thinking along the same lines - some sort of a stack implementation of sorts.

Smac89. If it was just one file, it would be easy, but I have to check older files as well if the current log file isnt >1000 lines.

thanks for the input.

[Smac89]

TacoLover wrote:

Smac89. If it was just one file, it would be easy, but I have to check older files as well if the current log file isnt >1000 lines.

#!/bin/sh

# Half finished
# This is supposed to read files from stdin
# Then output the last 1000 lines of each file
# if the file contains less than 1000 lines, the program
# should try to output the rest of the lines using the
# previous versions of the file

for file in "$@" # $@ represents all files in current directory
	do
		if [ -r $file ] #File is readable
		then
			numLines=`wc -l $file | tr -d [:alpha:]`
			echo "==========$file=========="
			
			if [ "$numLines" -ge 1000 ]
			then
				tail -1000 $file
			
			else
				continue
				# Do something here			
			fi
		fi
	done

If you have used a standard method of naming each log file as well as older versions of it, the line that says "Do something here" can be replaced with a for-loop that will loop through all the files with similar version numbers until the line count has reached 1000.

[Duthomhas]

Why reinvent the wheel? Use tail:

tail -q -n 1000 file1 file2 file3 ...

It really is a pretty slick piece of software that comes preinstalled on every *nix desktop/workstation

[TacoLover]

Thanks for the input everyone. I ended up implementing the following.

const unsigned int maxRecords = 1000;

unsigned int get_line_count(std::string filename)
{
	std::string command = "wc -l " + filename;
	FILE* pipe = popen(command.c_str(), "r");
	if (!pipe) return 0;
	char buffer[128];
	std::string result = "";
	while(!feof(pipe)) {
		if(fgets(buffer, 128, pipe) != NULL)
			result += buffer;
	}
	pclose(pipe);

	std::string parsed = result.substr(0, result.find(' '));
	return atoi(parsed.c_str());
}


void get_available_files(deque<std::string> *avail)
{
        /*
           Log files are stored as /var/log/mydir/mylog.log
           Rotated files are timestamped - mylog.log.ts1, mylog.log.ts2 etc.
        */
	std::string command = "ls -t mylog.log*"; //sort by time
	FILE* pipe = popen(command.c_str(), "r");
	if (!pipe) return;
	char buffer[512];
	std::string result = "";
	while(!feof(pipe)) {
		if(fgets(buffer, 512, pipe) != NULL)
			result += buffer;
	}
	pclose(pipe);

	stringstream ss;
	std::string onefile;
	ss << result;
	while (ss >> onefile)
		avail->push_back(onefile);
}

void print_contents(deque<std::string> *ptr)
{
	for(int i=0; i<ptr->size(); ++i)
	{
		std::string f = ptr->at(i);
		int lineCount = get_line_count(f);
		cout<<"Cur file = "<< f <<" (No of lines = " << lineCount << ")" << endl;
	}
}

unsigned int determine_files_to_read(deque<std::string> *avail, deque<std::string> *toRead)
{
	unsigned int cumLineCount = 0;
	unsigned int startLineNo = 0; //on first file in 'toread' list

	while(!avail->empty())
	{
		std::string f = avail->front();
		int lineCount = get_line_count(f);
		avail->pop_front();
		toRead->push_front(f); //add to front of list
		//cout<<"adding " << f << " toread"<<endl;

		cumLineCount += lineCount;

		if(cumLineCount >= maxRecords) {
			//if cumLineCount remains < maxRecords, startLine on first file will be 0.
			unsigned int linesInThisFile = maxRecords - (cumLineCount - lineCount);
			startLineNo = lineCount - linesInThisFile;
			break;
		}
	}
	return startLineNo;
}

main()
{
  ...
  unsigned int startLineNo = 0;
  deque<std::string> availableFiles, filesToRead;
  string sLine = "";
  std::ifstream infile;

  get_available_files(&availableFiles);
  startLineNo = determine_files_to_read(&availableFiles, &filesToRead);

  bool firstFile = true;

    while(!filesToRead.empty())
	{
    	std::string file = filesToRead.front();

    	infile.open(file.c_str());
    	if(!infile.is_open()) {
    		cout<<"Unable to open file " << file << endl;
    		filesToRead.pop_front(); // pop front and continue with next file
    		continue;
    	}

    	if(firstFile)
    	{
    		firstFile = false;
			unsigned int tmpcounter = 0;
                        //skip the lines from first file.
			while(!infile.eof())
			{
				getline(infile, sLine);
				++tmpcounter;
				if(tmpcounter >= startLineNo)
					break;
			}
    	}

		//start printing from this line.
    	while(!infile.eof())
		{
			getline(infile, sLine);
			cout<< sLine << endl;
		}

		infile.close();
		filesToRead.pop_front();//continue with next file
	}
}

I realize that I could have also used tail, but in the future, I will be adding more functionality as my program evolves (some of the details are still not clear/unspecified) and needed a bit more control.

Thanks again.