QFileInfo is very slow

[jho]

Hi,
i have a problem where QFileInfo is very slow.

I have a QStringList with over 15000 paths. I want to get the lastModified date, the fileSize and the fileName. It is very slow. It took about 60-80 seconds.

// collect files and exclude empty mask files
    for(int i=0;i<files3D.count();i++)
    {
        QFileInfo fileInfo = files3D[i];
 
        QDateTime date = fileInfo.lastModified();
        QString str;
        str = date.toString("dd/MM/yyyy hh:mm:ss");
 
            if(fileInfo.size() != 657660)
            {
                fileNames.append(fileInfo.fileName());
                fileDates << str;
                fileSizes << QString::number(fileInfo.size());
            }
 
            qApp->processEvents();
    }

// collect files and exclude empty mask files
    for(int i=0;i<files3D.count();i++)
    {
        QFileInfo fileInfo = files3D[i];

        QDateTime date = fileInfo.lastModified();
        QString str;
        str = date.toString("dd/MM/yyyy hh:mm:ss");

            if(fileInfo.size() != 657660)
            {
                fileNames.append(fileInfo.fileName());
                fileDates << str;
                fileSizes << QString::number(fileInfo.size());
            }

            qApp->processEvents();
    }

To copy to clipboard, switch view to plain text mode 

I think may code is quite simply or have i done something wrong here?
Is there a faster way to get the infos from a file?

[sulliwk06]

For that many paths 60 to 80 seconds doesn't sound entirely unreasonable to me for file actions. Do you have to get the info for all of the files at the same time?

[jho]

Yes I need all of the info at once.

If I compare it with the simple dir command from windows it takes really long.

qt: 80 sec
dir command: 3 sec

My fallback is to use the dir command. But this would be only a workaround.

[wysota]

Your dir command does not process events after reading each entry

[jho]

Hi wysota: I think you mean this line.

qApp->processEvents();

I put this in to keep my application responsive and not freezing. But I already disabled it. So it has nothing to do with the slow reading.

Or do you mean my dir command approach. This would be more of a hack and is not really a good solution.

[wysota]

This code:

#include <QtCore>
 
int main(int argc, char **argv) {
  QDir dir(argv[1]);
  QFileInfoList list = dir.entryInfoList(QDir::NoDotAndDotDot|QDir::Files);
  qDebug() << "File count:" << list.size();
  for(int i=0;i<list.size();++i) {
    const QFileInfo finfo = list.at(i);
    QDateTime mtime = finfo.lastModified();
  }
  return 0;
}

#include <QtCore>

int main(int argc, char **argv) {
  QDir dir(argv[1]);
  QFileInfoList list = dir.entryInfoList(QDir::NoDotAndDotDot|QDir::Files);
  qDebug() << "File count:" << list.size();
  for(int i=0;i<list.size();++i) {
    const QFileInfo finfo = list.at(i);
    QDateTime mtime = finfo.lastModified();
  }
  return 0;
}

To copy to clipboard, switch view to plain text mode 

takes this much time on a directory containing 15000 files:
$ time ./dirtest dir
File count: 15000

real 0m0.095s
user 0m0.074s
sys 0m0.020s

Bottom line: QFileInfo is not slow.

[jho]

I have to look into it again. Maybe with your code although it looks similar to mine. I have to say that the files are stored on a network. But the windows dir command reads it also from the network.

[jho]

I have tested it. It seems to me that using an entry from a QFileInfoList list is way faster than using a QString from a QStringlist with QFileInfo. Very strange...

[jefftee]

QDir::entryInfoList will create all of the QFileInfo objects in one pass of the directory. If you are getting file names into a QStringList and then creating a QFileInfo for each file, then you are accessing the directory multiple times.

Is that what you are referring to? If so, makes sense to me it's way slower, if not, post some code to show what you're referring to.

[jho]

Yes, that is what i meant. And your explanation sounds valid.

Thanks guys for your help.

[wysota]

QDir::entryInfoList will create all of the QFileInfo objects in one pass of the directory. If you are getting file names into a QStringList and then creating a QFileInfo for each file, then you are accessing the directory multiple times.

Is that what you are referring to? If so, makes sense to me it's way slower,

This code:

#include <QtCore>
 
int main(int argc, char **argv) {
  QDir dir(argv[1]);
  QStringList list = dir.entryList(QDir::NoDotAndDotDot|QDir::Files);                
  qDebug() << "File count:" << list.size();    
  for(int i=0;i<list.size();++i) {      
    const QFileInfo finfo = QFileInfo(list.at(i));
    QDateTime mtime = finfo.lastModified();
  }
  return 0;
}

#include <QtCore>

int main(int argc, char **argv) {
  QDir dir(argv[1]);
  QStringList list = dir.entryList(QDir::NoDotAndDotDot|QDir::Files);                
  qDebug() << "File count:" << list.size();    
  for(int i=0;i<list.size();++i) {      
    const QFileInfo finfo = QFileInfo(list.at(i));
    QDateTime mtime = finfo.lastModified();
  }
  return 0;
}

To copy to clipboard, switch view to plain text mode 

takes this much (different computer, don't compare with previous results):
$ time ./dirtest dir
File count: 15000

real 0m0.068s
user 0m0.055s
sys 0m0.012s

I don't see any significant differences.

[jefftee]

I don't see any significant differences.

I stand corrected. Looks like the 2nd test was actually a little faster than the first, go figure!

[jho]

I can post my solution tomorrow.

Maybe I have such great performance impact because i am reading the files from a network?

[wysota]

I stand corrected. Looks like the 2nd test was actually a little faster than the first, go figure!

No, as I said it was done on a different machine so you shouldn't compare them.

[jefftee]

Sorry, missed the part about run on different computer.

[yeye_olive]

Your code copies a QFileInfo instance at each iteration of the loop. I have no idea how efficient the copy is (the doc does not mention implicitly sharing, and I have not had a look at its implementation), but you could try

const QFileInfo &fileInfo = files3D.at(i);

const QFileInfo &fileInfo = files3D.at(i);

To copy to clipboard, switch view to plain text mode 

instead.

For very populated directories, QDirIterator is a better option that QDir.entry*List() because it lets you extract and store exactly the information you need.

In any case, the measurements provided by the other posters suggest that other, more significant, sources of inefficiency may exist in your code. You should fix those first.

[jho]

I already tried adding "const" and it definetly makes a difference here. Keep in mind that I run the code on a network drive.

Here is my code so far. It takes 1 sec for 25000 files and this is enough for me.

QFileInfoList list = dir.entryInfoList(QDir::NoDotAndDotDot|QDir::Files);
 
    for(int i=0;i<list.size();++i) 
{
 
 
                const QFileInfo fileInfo = list[i];
 
                const QDateTime date = fileInfo.lastModified();
                const QString str = date.toString("dd/MM/yyyy hh:mm:ss");
 
                if(fileInfo.size() != 657660)
                {
                    fileNames << fileInfo.fileName();
                    fileDates << str;
                    fileSizes << QString::number(fileInfo.size());
                }
 
        qApp->processEvents();
    }

QFileInfoList list = dir.entryInfoList(QDir::NoDotAndDotDot|QDir::Files);

    for(int i=0;i<list.size();++i) 
{


                const QFileInfo fileInfo = list[i];

                const QDateTime date = fileInfo.lastModified();
                const QString str = date.toString("dd/MM/yyyy hh:mm:ss");

                if(fileInfo.size() != 657660)
                {
                    fileNames << fileInfo.fileName();
                    fileDates << str;
                    fileSizes << QString::number(fileInfo.size());
                }

        qApp->processEvents();
    }

To copy to clipboard, switch view to plain text mode 

[yeye_olive]

I already tried adding "const" and it definetly makes a difference here.

That is not what I meant. Your code still copies a QFileInfo. The statement I suggested in my previous post defines fileInfo as a const C++ reference to the ith element in the list (the "&" in the declaration is what makes it a reference).

[wysota]

There is one thing which bothers me -- all your code does is that it has a list with the data you need (line #1) and you copy data from that list to three more lists (lines #14-16). What exactly is the point of doing that? It's a pure waste of time.

[jho]

i store them in a struct and return this struct at the end. I have three listWidgets where I need to add these three arrays.