PDA

View Full Version : QFileInfo is very slow



jho
20th January 2015, 15:14
Hi,
i have a problem where QFileInfo is very slow.

I have a QStringList with over 15000 paths. I want to get the lastModified date, the fileSize and the fileName. It is very slow. It took about 60-80 seconds.



// collect files and exclude empty mask files
for(int i=0;i<files3D.count();i++)
{
QFileInfo fileInfo = files3D[i];

QDateTime date = fileInfo.lastModified();
QString str;
str = date.toString("dd/MM/yyyy hh:mm:ss");

if(fileInfo.size() != 657660)
{
fileNames.append(fileInfo.fileName());
fileDates << str;
fileSizes << QString::number(fileInfo.size());
}

qApp->processEvents();
}


I think may code is quite simply or have i done something wrong here?
Is there a faster way to get the infos from a file?

sulliwk06
20th January 2015, 15:55
For that many paths 60 to 80 seconds doesn't sound entirely unreasonable to me for file actions. Do you have to get the info for all of the files at the same time?

jho
20th January 2015, 16:08
Yes I need all of the info at once.

If I compare it with the simple dir command from windows it takes really long.

qt: 80 sec
dir command: 3 sec

My fallback is to use the dir command. But this would be only a workaround.

wysota
20th January 2015, 16:31
Your dir command does not process events after reading each entry ;)

jho
20th January 2015, 16:36
Hi wysota: I think you mean this line. :)

qApp->processEvents();

I put this in to keep my application responsive and not freezing. But I already disabled it. So it has nothing to do with the slow reading.

Or do you mean my dir command approach. This would be more of a hack and is not really a good solution.

wysota
20th January 2015, 16:45
This code:

#include <QtCore>

int main(int argc, char **argv) {
QDir dir(argv[1]);
QFileInfoList list = dir.entryInfoList(QDir::NoDotAndDotDot|QDir::Files );
qDebug() << "File count:" << list.size();
for(int i=0;i<list.size();++i) {
const QFileInfo finfo = list.at(i);
QDateTime mtime = finfo.lastModified();
}
return 0;
}

takes this much time on a directory containing 15000 files:
$ time ./dirtest dir
File count: 15000

real 0m0.095s
user 0m0.074s
sys 0m0.020s

Bottom line: QFileInfo is not slow.

jho
20th January 2015, 17:13
I have to look into it again. Maybe with your code although it looks similar to mine. I have to say that the files are stored on a network. But the windows dir command reads it also from the network.

jho
20th January 2015, 20:29
I have tested it. It seems to me that using an entry from a QFileInfoList list is way faster than using a QString from a QStringlist with QFileInfo. Very strange...

jefftee
20th January 2015, 21:52
QDir::entryInfoList will create all of the QFileInfo objects in one pass of the directory. If you are getting file names into a QStringList and then creating a QFileInfo for each file, then you are accessing the directory multiple times.

Is that what you are referring to? If so, makes sense to me it's way slower, if not, post some code to show what you're referring to.

jho
20th January 2015, 22:07
Yes, that is what i meant. And your explanation sounds valid.

Thanks guys for your help. :)

wysota
20th January 2015, 22:09
QDir::entryInfoList will create all of the QFileInfo objects in one pass of the directory. If you are getting file names into a QStringList and then creating a QFileInfo for each file, then you are accessing the directory multiple times.

Is that what you are referring to? If so, makes sense to me it's way slower,

This code:


#include <QtCore>

int main(int argc, char **argv) {
QDir dir(argv[1]);
QStringList list = dir.entryList(QDir::NoDotAndDotDot|QDir::Files);
qDebug() << "File count:" << list.size();
for(int i=0;i<list.size();++i) {
const QFileInfo finfo = QFileInfo(list.at(i));
QDateTime mtime = finfo.lastModified();
}
return 0;
}

takes this much (different computer, don't compare with previous results):
$ time ./dirtest dir
File count: 15000

real 0m0.068s
user 0m0.055s
sys 0m0.012s

I don't see any significant differences.

jefftee
20th January 2015, 22:13
I don't see any significant differences.

I stand corrected. Looks like the 2nd test was actually a little faster than the first, go figure!

jho
20th January 2015, 22:18
I can post my solution tomorrow.

Maybe I have such great performance impact because i am reading the files from a network?

wysota
20th January 2015, 23:05
I stand corrected. Looks like the 2nd test was actually a little faster than the first, go figure!

No, as I said it was done on a different machine so you shouldn't compare them.

jefftee
20th January 2015, 23:43
Sorry, missed the part about run on different computer.

yeye_olive
21st January 2015, 11:23
Your code copies a QFileInfo instance at each iteration of the loop. I have no idea how efficient the copy is (the doc does not mention implicitly sharing, and I have not had a look at its implementation), but you could try

const QFileInfo &fileInfo = files3D.at(i);
instead.

For very populated directories, QDirIterator is a better option that QDir.entry*List() because it lets you extract and store exactly the information you need.

In any case, the measurements provided by the other posters suggest that other, more significant, sources of inefficiency may exist in your code. You should fix those first.

jho
21st January 2015, 12:04
I already tried adding "const" and it definetly makes a difference here. Keep in mind that I run the code on a network drive.

Here is my code so far. It takes 1 sec for 25000 files and this is enough for me. :)



QFileInfoList list = dir.entryInfoList(QDir::NoDotAndDotDot|QDir::Files );

for(int i=0;i<list.size();++i)
{


const QFileInfo fileInfo = list[i];

const QDateTime date = fileInfo.lastModified();
const QString str = date.toString("dd/MM/yyyy hh:mm:ss");

if(fileInfo.size() != 657660)
{
fileNames << fileInfo.fileName();
fileDates << str;
fileSizes << QString::number(fileInfo.size());
}

qApp->processEvents();
}

yeye_olive
21st January 2015, 12:20
I already tried adding "const" and it definetly makes a difference here.
That is not what I meant. Your code still copies a QFileInfo. The statement I suggested in my previous post defines fileInfo as a const C++ reference to the ith element in the list (the "&" in the declaration is what makes it a reference).

wysota
21st January 2015, 12:52
There is one thing which bothers me -- all your code does is that it has a list with the data you need (line #1) and you copy data from that list to three more lists (lines #14-16). What exactly is the point of doing that? It's a pure waste of time.

jho
21st January 2015, 13:30
i store them in a struct and return this struct at the end. I have three listWidgets where I need to add these three arrays.

wysota
21st January 2015, 13:43
I have three listWidgets where I need to add these three arrays.
It would be much better to have three listViews each serving a separate column from a single model based on QFileInfoList.

Something along the lines of:

#include <QtWidgets>

class InfoListModel : public QAbstractTableModel {
public:
InfoListModel(const QFileInfoList &list, QObject *parent = 0) : QAbstractTableModel(parent), m_data(list) {}
int rowCount(const QModelIndex &parent = QModelIndex()) const { return parent.isValid() ? 0 : m_data.size(); }
int columnCount(const QModelIndex &parent = QModelIndex()) const { return 3; }
QVariant data(const QModelIndex &index, int role = Qt::DisplayRole) const {
if(!hasIndex(index.row(), index.column(), index.parent())) return QVariant();
if(role != Qt::DisplayRole) return QVariant();
const QFileInfo &finfo = m_data.at(index.row());
switch(index.column()) {
case 0: return finfo.fileName();
case 1: return finfo.lastModified().toString("dd/MM/yyyy hh:mm:ss");
case 2: return finfo.size();
default: return QVariant();
}
}
private:
QFileInfoList m_data;
};

int main(int argc, char **argv) {
QApplication app(argc, argv);
QFileInfoList list = QDir(argv[1]).entryInfoList(QDir::Files|QDir::NoDotAndDotDot);
InfoListModel model(list);
QListView l1;
QListView l2;
QListView l3;
l1.setModel(&model);
l2.setModel(&model);
l3.setModel(&model);
l1.setModelColumn(0);
l2.setModelColumn(1);
l3.setModelColumn(2);
l1.show();
l2.show();
l3.show();
return app.exec();
}

And there is also QFileSystemModel.