QTableView memory consumption

[LordQt]

Hello friends,

I try a little routine to fill up a tableview from a textfile.

//**************************************************************************
void Window::openFileInTable(const QModelIndex & index )
//**************************************************************************
{
 
    int digits[6];
    int iSegCount(0);
    int iCopyFLines(0);
    QString selectedFile;
	if ( index.isValid())
	{
		 selectedFile = model->filePath( index );
		 selectedFile.replace(QString("/"),QString("\\"));
        bodyEdit->clear();
        bodyEdit->append(selectedFile);
 
        QFile file(selectedFile);
        if (!file.open(QIODevice::ReadOnly | QIODevice::Text))
            return;
        int fileLines(1);
        QTextStream InputDataFile(&file);
        while (!InputDataFile.atEnd()) {
            InputDataFile.readLine();
            fileLines++;
         }
        file.close();
        QTextStream InputDataFile(&file);
        if (!file.open(QIODevice::ReadOnly | QIODevice::Text))
            return;
        QStringList buffer_list;
        QString Delimiter(";");
        tableModel = new QStandardItemModel;
        tableModel->setRowCount(0) ;
        int Max_Number_of_Columns(36);
        int Max_Number_of_Lines(0);
        tableModel->setColumnCount(Max_Number_of_Columns + 3) ;
        progressDialog->setMaximum(fileLines);
        for(iSegCount=0;iSegCount<6;iSegCount++){digits[iSegCount]=0;}
        QString buf;
        while (!InputDataFile.atEnd())
        {
            buf = InputDataFile.readLine();
            buffer_list = buf.split(Delimiter);
 
            for (int column = 0; column <= Max_Number_of_Columns; column ++)
            {
                QStandardItem * item = new QStandardItem(buffer_list[column]);
                tableModel->setItem(Max_Number_of_Lines, column, item) ;
            }
            Max_Number_of_Lines++;
            progressDialog->setLabelText(tr("Transfer %1...").arg(selectedFile));
            progressDialog->setWindowModality(Qt::WindowModal);
            progressDialog->setValue(Max_Number_of_Lines);
 
        }
        iCopyFLines=Max_Number_of_Lines;
        iSegCount = 0;
                    do {
                        digits[iSegCount++] = iCopyFLines % 10;
                    } while ((iCopyFLines /= 10) > 0);
            progressCountEiner->setNumber(digits[0]);
            progressCountZehner->setNumber(digits[1]);
            progressCountHunderter->setNumber(digits[2]);
            progressCountTausender->setNumber(digits[3]);
            progressCountZehnTausender->setNumber(digits[4]);
            progressCountHundertTausender->setNumber(digits[5]);
 
        progressDialog->setValue(fileLines);
        trackView->setAutoFillBackground(true);
        trackView->setAlternatingRowColors(true);
        trackView->resizeColumnsToContents();
        trackView->setModel(tableModel);
        trackView->show() ;
    }
 
}

//**************************************************************************
void Window::openFileInTable(const QModelIndex & index )
//**************************************************************************
{

    int digits[6];
    int iSegCount(0);
    int iCopyFLines(0);
    QString selectedFile;
	if ( index.isValid())
	{
		 selectedFile = model->filePath( index );
		 selectedFile.replace(QString("/"),QString("\\"));
        bodyEdit->clear();
        bodyEdit->append(selectedFile);

        QFile file(selectedFile);
        if (!file.open(QIODevice::ReadOnly | QIODevice::Text))
            return;
        int fileLines(1);
        QTextStream InputDataFile(&file);
        while (!InputDataFile.atEnd()) {
            InputDataFile.readLine();
            fileLines++;
         }
        file.close();
        QTextStream InputDataFile(&file);
        if (!file.open(QIODevice::ReadOnly | QIODevice::Text))
            return;
        QStringList buffer_list;
        QString Delimiter(";");
        tableModel = new QStandardItemModel;
        tableModel->setRowCount(0) ;
        int Max_Number_of_Columns(36);
        int Max_Number_of_Lines(0);
        tableModel->setColumnCount(Max_Number_of_Columns + 3) ;
        progressDialog->setMaximum(fileLines);
        for(iSegCount=0;iSegCount<6;iSegCount++){digits[iSegCount]=0;}
        QString buf;
        while (!InputDataFile.atEnd())
        {
            buf = InputDataFile.readLine();
            buffer_list = buf.split(Delimiter);

            for (int column = 0; column <= Max_Number_of_Columns; column ++)
            {
                QStandardItem * item = new QStandardItem(buffer_list[column]);
                tableModel->setItem(Max_Number_of_Lines, column, item) ;
            }
            Max_Number_of_Lines++;
            progressDialog->setLabelText(tr("Transfer %1...").arg(selectedFile));
            progressDialog->setWindowModality(Qt::WindowModal);
            progressDialog->setValue(Max_Number_of_Lines);

        }
        iCopyFLines=Max_Number_of_Lines;
        iSegCount = 0;
                    do {
                        digits[iSegCount++] = iCopyFLines % 10;
                    } while ((iCopyFLines /= 10) > 0);
            progressCountEiner->setNumber(digits[0]);
            progressCountZehner->setNumber(digits[1]);
            progressCountHunderter->setNumber(digits[2]);
            progressCountTausender->setNumber(digits[3]);
            progressCountZehnTausender->setNumber(digits[4]);
            progressCountHundertTausender->setNumber(digits[5]);

        progressDialog->setValue(fileLines);
        trackView->setAutoFillBackground(true);
        trackView->setAlternatingRowColors(true);
        trackView->resizeColumnsToContents();
        trackView->setModel(tableModel);
        trackView->show() ;
    }

}

To copy to clipboard, switch view to plain text mode 

What I detect is that the bigger the textfile is the greater the memory consumption is.

The problem I think is the for loop:

for (int column = 0; column <= Max_Number_of_Columns; column ++)
            {
                QStandardItem * item = new QStandardItem(buffer_list[column]);
                tableModel->setItem(Max_Number_of_Lines, column, item) ;
            }

for (int column = 0; column <= Max_Number_of_Columns; column ++)
            {
                QStandardItem * item = new QStandardItem(buffer_list[column]);
                tableModel->setItem(Max_Number_of_Lines, column, item) ;
            }

To copy to clipboard, switch view to plain text mode 

Have somebody a tip to filling up my tableview with better performance or lower memory consumption ..???

[calmspeaker]

What I detect is that the bigger the textfile is the greater the memory consumption is.

I think what you detect is reasonable,doesn't if?

[LordQt]

What do you mean???

When I parse a textfile with for example 150.000 lines the Applications memory usage grows up to 500-600 mb. When the tableview is filled it goes to 400 mb and still point there.

There must be a better solution for handling big textfiles, or not???

[muellerp]

What do you mean???

When I parse a textfile with for example 150.000 lines the Applications memory usage grows up to 500-600 mb. When the tableview is filled it goes to 400 mb and still point there.

There must be a better solution for handling big textfiles, or not???

QString is Unicode, so 2 bytes per character. It's not the lines, but the number of characters you have.

As you store the content of the file in the StandardItem you have it temporarily twice in memory.
After closing the file "only" once, all in QStandardItems.

I'm not surprised by your numbers.

The only way is to handle the file like a "database" - read only chunks until you find the needed part - and your model is then looking temporarily for each line into the file. But with a text file this is really a timing issue.

You may first read the file junk by junk and store it in a SQL database on disk, here the access times are faster. Still I don't think we will get wonders in performance.

[LordQt]

Thank you for yours suggestions ;O)

but I need to check content of the File before inserting the file into Database.

When I understand you right
I must insert the hole file in DB then visualize the content in QTreeView then reinserts the hole content to my real DB ... hmm

Thats inefficient isn´t it?

[muellerp]

As you don't want to start with a database, then you should do it similar like a database.

Databases don't keep the data itself in memory, just the index to the data on a harddrive.

In your case, you don't store the text itself, but an index to the position in the file. The index is then fast enough and not that memory consuming.

So I would first parse the file and at each position of a new line, you store the position in an index, so later you can do:

pos = index (row)
data = readData(file, pos, length)

Maybe you also store the length for faster processing.

Then your model takes for each line through this index the data when displaying from the harddisk.

I think this is then reasonable fast and the index takes much less memory.

[LordQt]

Is there any example out there .....

[LordQt]

QFile has no member that returns an index of a row.......????